No, I dont think so.. To be honest, I believe your hubber score will only be affected if you do not participate in anything that has to do with Hubpages.
For example. If you only write hubs, and dont do anything else. That can affect your hubber score. I am constantly writing, answering questions, and voting up other hubs, and my score stays the same... the more I write though, the more higher my hubber score will be.