I am not sure there is a way. I have asked one person to quit following me, but he did not, I have not found any buttons or anything that would let you remove that person from your list of followers. It is odd that you have to approve fan male, but you do not have to approve who follows you.
This is probably something you need to send to the HubPages team. I would like to know the answer as well. You can ignore the person comments by denying them so they do not show and not responding and he may get the message.