A few times ( a VERY few times - like three times, maybe) I've unfollowed someone after I first followed them, thinking they would be writing about subjects in which I had interest - only to discover that they start writing on, for example, one subject that didn't interest me. (People sometimes start new here, writing a couple of one kind of Hub; and then they'll decide to write in a single niche.) I just figured there's no point in pretending I'm interested in a niche when I'm not. It's no reflection on the unfollowed Hubber - just the change in Hub style/subject.
I'm pretty slow to follow in the first place these days, because there are only so many Hubs a person knows he'll be reading over the course of a week; so now I know not to follow unless I get the idea that the person has written several Hubs that may interest me. That eliminates the situation where someone new writes two or three Hubs and then switches to a niche that's completely different from those first Hubs (and then being faced with deciding whether to unfollow or not.
I don't follow any questions/answers or Hub comments. I go by memory and go check to see if I'm interested in whether anything else went on somewhere. I don't want to be bombarded by notices every time someone comments on someone's else's Hub (that I commented on too). Again, it's no reflection on the person's Hub or even on the comments others make.