I'll answer this question from the perspective of my ex, who used to constantly complain about this issue, since I have no personal, objective experience in the matter ;)
1. Intellectual interactions. She would often complain that people don't take her seriously. For example: she would make an incredibly interesting comment (she was pretty clever), or try to engage in an interesting conversation with a man, and he just wouldn't stop trying to take the conversation elsewhere.
2. Social interactions. She would complain about always being stared at, it drove her nuts. Examples: awkward interactions at work when she would catch men checking her out; losing respect for intelligent men because she caught them checking her out; or simple things like being in a crowd, turning around and having every single man in the crowd try to make eye contact with her.
3. Obviously the cat-calls and aggressive behaviour; often offensive, frightening and even abusive.
Other than that........... lol. You've pretty much got it made. I'm positive that the benefits far outweigh the setbacks.