I've wondered about this as well. If they were following you but you weren't following them, I have yet to find a way to remove them which doesn't seem fitting. If there truly is no way to address that, perhaps your question and answers here may spur a change on HP. I really hope I'm wrong about this so I can do some pruning too. If you had been following them, the only things I've found you could do is unfollow them by going into your account and then followers and unfollowing them there or by going to your profile and clicking on followers.