Well yes. If I started thinking too much about it I'd get OCD! Its funny really. There's this thing about using different chopping boards (covered in previous threads) for meat, fish, veg. That is mainly accepted as good practice in domestic kitchens as well as professional. Yet thirty years ago, you'd be hard pushed to find a domestic kitchen that used this system, perhaps with the exception of Kosher families. My parents would have thought it completely bonkers! If you start thinking about it too much, then using tea towels at all, more than once, might be hazardous. As you say 'where do you stop?' I think that's why I don't worry.
Here's a case in point: If you drop a clean tea towel on a clean (but walked on) floor, do you then have to put it in the laundry?
And another: Would you allow cats to walk on kitchen surfaces? I wouldn't, but I know people who do. I actually do find this horrible. The cat has been scratching in a cat tray or roaming outside. After all, we wouldn't walk in our shoes across a kitchen surface!