Yeah, they're a fact of life...
Some heads of broccoli/cauliflower are worse than others, but I look on the bright side. It does mean less pesticides in your veg.
In my home grown stuff, we'll pick half a dozen or so out, so I invert the broccoli on water and they climb out to be picked off, then I'll shake the broccoli well. Home grown heads of broccoli are not as compact as supermarket bought stuff, so they shake out quite easily.
I also have a session a couple of times a week picking caterpillars off my brassicas and my chooks love them. They'll fight each other over them if they can. I'll throw the caterpillars out of the veg plot, so they chooks have to hunt for them. Less fighting that way.