You can but you should not. If you do it by accident once and only once it will not hurt your account, but if you do it as a habit then your hubberscore and hubscore will suffer.
Remember that HP records each vote and knows who vote and how they vote. The system has to track this for abuse. It's the same with people who constantly vote down other hubbers hubs, they eventually are affected by their actions.