I sometimes include olive oil in my bread, but often don't - I've not tasted them side by side, but I don't think I've noticed any difference to my basic loaf....either in taste or in how long it keeps for.
I agree - unless its the type of bread with extra olive oil added before cooking, such as focaccia.