Each of us gets to choose our own friends, lovers, and spouse. We also get to choose who we spend our free time with.
One of the reasons people have a difficult time cutting toxic people out of their life is because they (feel) obligated to maintain a semblance of a relationship with them. They worry about what others will think or say if they pull away. They feel guilty.
The bottom line is each of us is responsible for our own happiness. If being around someone upsets us and yet we (choose) to continue to be around them then (we) are (choosing) to be upset.