"It's your life! Take the wheel!" Each of us is responsible for our own happiness.
You have the right to choose who you want to spend time with. Guilt is just a form of manipulation. Don't let someone talk you into being some place you don't want to be. Your life comes down to the choices and decisions that YOU make.
Once you get to a point where you stop worrrying about what others think you are truly free to be your "authentic self". That is the only way we find true happiness and peace of mind.