I've always hated fruit since I was a child. My dad, who loved all kinds of fruit, could never understand this so he force fed me fruit, he'd open my mouth and put pieces of orange, watermelon, grapes, etc, and forces me to stay at the table until I ate said fruit. I'd spend hours yelling and crying till I finally swallowed the fruit.
I've always been a picky eater but when I grew up I started trying foods I didn't like before and I found lots of new foods I actually enjoy. But I never wanted to try fruit again. One day it dawned on me why, it's because my dad force fed me fruit. Just the thought of the texture of fruit feels disgusting to me. So yes, bad memories do ruin food.