A little rusty

Last week I was shopping at Walmart and I couldn't find the sugar cubes (because they don't carry them). I saw a man who worked there so I asked him where the sugar cubes were. He pointed to his ear and made signs like writing. Of course I didn't have a pen handy, but I thought, I can do this. I thought I remembered the sign for sugar, but it was awfully similar to the sign for cute, so I decided to finger spell it. My fingers were really slow and awkward, and I had to do it twice for him, but I made it. He answered me and I went on my way feeling proud I was able to use my ASL training, more to understand him than anything.
Well, this week I was back at Walmart, trying to find a hole punch without success. Out of the corner of my eye I could see a worker walking down the isle. I started to ask him where it was, when I recognized him as the deaf man from before. I couldn't think of the sign for hole or punch, so I just finger spelled it again. He helped me, I signed thank you, and then he quickly signed a question. Oh no. I could start to feel the pressure of a conversation in sign when I haven't really used it for 7 years. All I caught was "how you know..." Luckily I remembered the sign for again. He asked me how I knew he was deaf. I knew I couldn't finger spell that whole answer, so I struggled to remember enough signs to answer and plowed through a little conversation with him. It was fun to sign again, but it also made me flustered. Even though deaf people are always patient with my signing, I always feel more pressure signing with a deaf person. I would like to practice it more, but how do I find a hearing person who can sign?