Boy, I know what you mean!
Being a good person does not mean being in healthy relationship with everyone. Part of being good in a realistic way means ending unhealthy relationships, but doing it without hatred or blame. Ultimately, goodness has two aspects. Towards all people, we live in peace and non-harm (also called non-violence, spiritual love, unconditional love, or ahimsa). We agree to share the planet with this person. We wish them well. *and we decide we will have nothing to do with them.*
Why? Because, on Earth, no healthy relationship is possible with this person. To try is to drain our energy. Then our own bringing the life of our own soul into the world, fulfilling our creativity, our purpose, and our service to those we really can help, gets lost.
Let go of those with whom you cannot have a healthy relationship. Wish them well and farewell! And focus on bringing your gifts to the world with those who will benefit from them.