Yes it has definitely happened, but I wouldn't know how many times... I enjoy getting followers but I don't get hung up on the exact number all the time. The only reason I even noticed it the one time is that I got an accolade for getting 100 followers, then my followers dropped to 99, so I knew someone must have unfollowed.
If you are producing good content, the number of followers will increase overall in time, even if people unfollow occasionally. So I wouldn't get too worried about it. And you shouldn't feel obliged to follow someone just because they follow you... if they want to unfollow, so be it.