I don't follow too many Hubbers as I think I just wouldn't be able to keep up with what everyone has written. When I do follow someone, I follow them because not only do they have great hubs and an excellent writing style, but I also think they have interesting views and a unique way of viewing the world. I always tell someone why I'm following them, I think if I can't think of a reason then I shouldn't bother to follow them.
It does kind of... not really annoy me, but frustrate me a little when people follow me without leaving fanmail or commenting on any of my Hubs. Of course I appreciate the gesture, but I would appreciate knowing even more what they think works with my Hubs and areas they think I could improve upon.