There are definite miserables souls out there. However, the response to people who drive our life insane; may not be to show them the same treatment. We should have some restraint. If its a coworker that drove you insane and you know that many of your coworkers would attend. There is no reason in hating someone that you did not love in the first place.
You are a lot bigger than you think. Its common courtesy to show some respect if you know them well. However, if you were the one who caused them much grief in their life. I would suggest not attending since you may irritate those whom know of your relationship with the person who is now dead. The last thing that you would want is to have a confrontation at the funeral.
However, personally, I have no qualms about being around people who hate me, unless of course, I have made a deliberate attempt to ruin their life. If I have not done anything purposefully and people are just annoyed because I am having a good turn of fortune. Then, I would not worry about that. Its more of their problem than yours. You should not take ownership for the ill feelings of others. Hope this advice helps. Great question by the way. Thanks.