"Life is a personal journey"
If there is someone in your life that is making things difficult for you and you don't do anything about it then you are not "loving yourself".
Words like "bad", "right", "wrong" or "selfish" are often used to manipulate others. You have to dance to your own music in life.
Whatever YOU decide is best for you is what you ought to do.
(Each of us is responsible for our own happiness.)
We select our own friends, lovers, and spouse. "Choice" is your God given right. Only YOU can determine if being "wronged" justifies removing a person from your life.