It's been said, "Anger is the mask that (hurt) wears".
Apparently you must feel others are taking advantage of you, not respecting your wishes, needs, or wants. It's important to realize no one can read your mind therefore every "perceived slight" is not intentional. If indeed you do feel like you are spending time with people who are "toxic" or don't want you to be happy then you always have the option to not be around them!
Anger and fighting is usually about feeling the need to set boundaries or trying to (force) someone to behave the way you want them to. The healthy thing to do is (think) before you blurt out any words. Ask yourself if this person's goal was to piss you off? Next think about the outcome YOU want. Do you want to fight? or Do you want to get cooperation? When you start yelling they stop listening!