Firstly, likes to wear what?
If it is what I think - lingerie or similar, giving it up could be detrimental to your well-being.
If your wife wants you to quit, it's probably because she feels that it's just a fetish; something kinky you like to do, but if the want to do it goes further, then I should talk to someone about it.
The fact is, you don't have to be gay to be a TV - for that's what you are if the above is what you do. In fact, according to some statistics, the vast majority of TV's are straight males.
For some it's a need, not just desire out of feeling horny, so you should be honest with yourself and decide whether your need to dress that way is in fact just something that's a turn-on or whether it runs deeper and take the appropriate action.