The "vote up" or "vote down" affects both your hubscore and author/profile scores. The more up votes you get the higher both scores get. The feedback only affects the hub itself and is an addition to your actual comment. The feedback is a rating rather that a vote. The feedback also determines how people see your hub, that's why HP created an accolade for it.
On the other hand, if someone doesn't vote on the hub, the effect is almost similar to voting it down because over time the scores will begin to fall. Hubs with lesser feedback also have lower scores.