I don't think 1 or 2 people dropping out is such a big issue unless you only have 1 or 2 followers, when you have 10-20 dropping out or whatever then you have to start wondering what is going on...
My thoughts on the subject are that maybe your hubs just do not appeal to them anymore, I am mainly interested in psychology, cooking and body languages and stuff of that nature, if you only have 2 or 3 hubs of that kind written once I might read them and find you have stopped doing them. I will also find myself bookmarking something to read later, once I have read it I am done...until the next time.
For that reason I might remove you, as the email spam gets a bit much after a while :p
The important thing is that you do not take it as a personal insult, just silent feedback ;)
Also I think you are following way too many people personally, I certainly would not care to be a number that you only want to follow in order to get a follow back; did you even read my hub?, did you care what I said?, do you have anything to add of your own?
When following anyone I make sure I have read at least 3-5 of their hubs and often 10 or more...that way I can find out if I really do think they have something of value to share :p
These questions and more are what I ask of people who add everyone they can find.
I would much rather have someone following me because they choose to than simply because I decided to follow them in the hopes they would reciprocate.