Yeah I wouldn't admit that I'm a germophobe but honestly I know some people are negligent about handwashing and if there are cats (cats often jump on counters) and/or children in their house I'm extra wary. I'd politely decline without giving a reason. I became very aware of pathogens while attending massage therapy school 30+ years ago and I've been an avid handwasher ever since. I see people in public restrooms who don't adequately wash their hands, if they don't do it properly in public, how well are they washing up at home? The first thing I do before handling food is to thoroughly wash my hands and I often use food prep gloves.
I also have a weird phobia where I won't eat chips/crisps if someone else had their hand in the bag. When my kids were little I had my own loaf of bread and my own box of cereal, my own bag of chips, etc. I won't eat salads at a restaurant usually (hot foods only in general) and prefer restaurants with open kitchens where I can see how food is being handled. Also, I keep sanitizer in my purse and after looking at the menu or touching condiment bottles, salt shakes, etc. I sanitize.
Sorry, that was long-winded!