I think I've only de-followed one person so far, that I originally followed back. The reason why I did such, was because this particular person kept asking about 10 inane, senseless questions every day, day after day, and I got tired of seeing them in my feed.
Other than that, people may de-follow others because they no longer like them anymore (ha!) or they feel that they are completely inactive and have left the site, or perhaps people get tired of the notification e-mails that they get whenever somebody they follow starts a forum thread, asks a question or writes a hub, etc.
Personally, I'm not here for a popularity contest. Besides, if there was a such thing as a "I hate you" list instead of a fan club list, I think I'd have loads of people following me; ha-ha!