Apart from inventory weight, I see few downsides to carrying enchanted weapons. I rarely pick up any better weapons, but if I do, and my enchanted weapon has run out of charges, I can quickly swap to that one. That means that I'll end up carrying 2 melee weapons and 2 bows, thus taking up more inventory weight (as well as your example of soul gems, which can be a pain to obtain). I think Azura still gives Azura's star, which - if I'm correct - is a never ending soul gem, so that'll cut down on the gems you need.
My favourite enchanted weapons are the Daedric ones but sadly haven't upgraded all that much from previous games. Molag Bal's fatiguing mace has to be the most disappointing quest reward for one of the best quests in the game, for example.