Yes! Sometimes it's in your best interest to avoid "toxic people".
Life is short! Why would anyone want to spend time with people they don't like or don't get along with? Ideally you want to make sure a situation has risen to the "deal breaker" stage before cutting your losses. We normally only regret when we believe we are in the wrong. If that's the case then make an apology. The ball is then placed in their court to decide if they want to stay in contact.