Disillusioned with MTurk Hub Hopping

1. 95

   PaulGoodman67posted 11 years ago

   I must admit, I was pretty excited when HP announed that they were going to pay people to hub hop on Amazon MTurk.  The HITs don't pay much, only 5 cents a time, but you can make more money through bonuses, if you have a high accuracy rate. 
   
   It all went pretty well for me at first and with only a few minor setbacks, I managed to reach over 95% accuracy and a good bonus rate.  But then it all went very crappy, and I sank and sank, despite me assessing hubs in what I though was the same fashion.  Nowadays, I earn so little in bonuses, there isn't much incentive.
   
   The problems are twofold, as far as I can see.
   
   Firstly, there is very little feedback from HP, so it is difficult/impossible to know where you are going wrong.  Yes, there are 4 basic examples to refer to, but that seems more than a little inadequate when you consider all the many different types of hub that people write and the different sorts of spammer tactics.  (You do also get occasional one off feedback hubs too, but they aren't that helpful.
   
   Secondly, as far as I am aware, the hopper's accuracy rate is assessed against what other hub hoppers have given.  That means that you can assess something correctly but still be marked down (and you won't get any feedback, so won't know exactly where the problem occured).  I consider myself pretty diligent, but it doesn't seem to help me.
   
   I consider myself motivated and keen to learn, so am mystified as to why I am doing so badly at hopping after doing so well initially...  Am I missing something?

   replyreport
   1. 75

      WriteAngledposted 11 years agoin reply to this

      And that is the problem in a nutshell! Majority vote-based quality control. However, most of the "voters" over at MT are people, who could not care less whether HP sinks or swims and, due to the pittance they are being paid, are probably gaming the system as much as possible, and thus using the simplest possible criteria to make their judgements.
      
      Going by my rating experience when I was on Helium, it is not even necessary to read an article in order to rate it. I never read a word of what I was rating on Helium, yet managed to have a rating accuracy of 95% or higher most of the time, while needing only 10-15 seconds to make each rating decision.
      
      If you are actually taking the trouble to read an article and assess it on the basis of content, I am not in the least surprised your ratings do not tally with those of the rest.

      replyreport
   2. 85

      Marisa Wrightposted 11 years agoin reply to this

      That is a concern, and I hope HubPages is paying attention.   Your experience tells me there is something wrong with the way rating is judged - and if there's something wrong with the way rating is judged, that means some Hubs are being rated unfairly.

      replyreport
2. 0

   EmpressFelicityposted 11 years ago

   Paul, I'm sorry you're not getting the results you wanted.
   
   But I have to say, my first reaction on reading your post was "Well, what did you expect?"

   replyreport
3. 83

   psycheskinnerposted 11 years ago

   I think more will be achieved over the long term by expecting competence and being disappointed than by giving up entirely. At least for those of us still participating in this site.

   replyreport
   1. 95

      PaulGoodman67posted 11 years agoin reply to this

      I think HP should give more feedback on how they expect certain types of hub to be assessed.  Otherwise, I just feel like I'm thrashing around in the dark.  It's shocking how quickly your accuracy rate can go down.  My bonus rate halved overnight.

      replyreport
4. STAFF

   Paul Edmondsonposted 11 years ago

   We know it's not easy to rate Hubs accurately.  There are multiple types of assessments.  One thing to look out for is rating a spun/translated Hub highly.  These Hubs can look really good at a glance, but the content is spun.  Sometimes, when people go quickly, they don't pick up on the spun/translated content which impacts substance and grammar substantially.

   replyreport
   1. 95

      PaulGoodman67posted 11 years agoin reply to this

      You've highlighted an area that's difficult, Paul.  More sophisticated spammers will produce something that's competent, in terms of grammar and appearance.  However, there will often be a couple of links to commercial sites that imply that the page has been set up as purely a backlink/advert page.  Are these to be marked with low grades?  If we mark them with low grades, how do we know that other hub hoppers won't just glance at the page and mark them higher - then it looks like we've made a mistake?  I do think that the current guidance, examples and feedback are too scant at present - certainly for me, at least.  I want to follow whatever guidlines HP sees as necessary and identify where I am making mistakes, but would appreciate more detailed instructions...
      
      Is it possible that we could have more of our previous 'mistakes' highlighted so that we can attempt to avoid them in future?  I know that this already happens to a small degree, but for me it isn't enough.

      replyreport
5. 83

   psycheskinnerposted 11 years ago

   If the accuracy score is based on simple agreement, it is also valid to point out that will make 100% "accuracy" impossible.  And may depress your "accuracy" severely based on things not under your control.

   replyreport
6. 95

   SmartAndFunposted 11 years ago

   Like EF already said, what did you expect? Yeah, sorry, Paul, but this thread is almost funny. Since it is our work that is being graded by this flawed system, however, it is not funny at all. It is downright infuriating.

   replyreport
   1. 59

      tussinposted 11 years agoin reply to this

      Oh I read between the lines your implication that MTurk is for suckers who think $1 for two hours work is a great boon.
      
      How do you know Paul is not living in a mud hut subsisting on a spoonful of rice a day?  I'll have you know his MTurk earnings allowed him to generously adopt a family of rats this Chirstmas and upgrade his chamber pot to an empty Folger's can.  I for one am very concerned that the downswing in MTurk will force Paul to make lifestyle adjustments that could cost him his engagement to a toothless bag lady he has been calling on of late. Let us pray for an MTurk miracle this season!

      replyreport
      1. 95

         SmartAndFunposted 11 years agoin reply to this

         

         replyreport
      2. 70

         galleryofgraceposted 11 years agoin reply to this

         No truer words were ever spoken!

         replyreport
7. 0

   EmpressFelicityposted 11 years ago

   @tussin:
   
   For some reason, I am reminded of this:
   
   http://www.youtube.com/watch?v=Xe1a1wHxTyo
   
   "In them days we were glad to have the price of a cup of tea. Without milk. Or sugar. Or tea."

   replyreport
8. 95

   PaulGoodman67posted 11 years ago

   I was satisfied with the income on the higher bonuses.  A couple of hours a day for some extra cash.  Not worth it when the bonus rate falls though.

   replyreport
   1. 59

      tussinposted 11 years agoin reply to this

      HP probably has their MTurk assignment set up so that if you actually start to make decent money from the task, they pull the rug out from underneath you.

      replyreport
   2. 59

      tussinposted 11 years agoin reply to this

      HP probably has their MTurk assignment set up so that if you actually start to make decent money from the task, they pull the rug out from underneath you.

      replyreport
9. 89

   Learning in Lifeposted 11 years ago

   My accuracy kept dropping until I wasn't allowed to do HP hits. They told me I was rating them different then they would when I asked. Ok, so then why are you paying people to do it if they get banned for not sharing your opinion? I did however make around 30 bucks before this happened.

   replyreport
10. 61

    LindaSmith1posted 11 years ago

    All people have to do is file a complaint with MT Turk, and they may ban HP

    replyreport
    1. 95

       PaulGoodman67posted 11 years agoin reply to this

       Well, I certainly won't be complaining to Amazon.  I am kind of hoping that HP will improve the hub hopping on MTurk, however.  If not, I will drift off and do something else.  I've just been a bit disappointed, that's all.

       replyreport
       1. 95

          SmartAndFunposted 11 years agoin reply to this

          I'm disappointed that this is how our articles are being graded.

          replyreport
          1. 91

             Jean Bakulaposted 11 years agoin reply to this

             This is in reply to Write Angled,
             I did the same thing for the short time I rated articles on Helium. I gave them several articles I worked really hard on. I thought the agreement was that they "owned" them for a year. Can I erase the URL can take them back, do you think? I thought it sounded good at first, but then realized I had to win a contest to make $1.00!  And I got great reviews on crappy stuff too.

             replyreport
             1. 97

                theraggededgeposted 11 years agoin reply to this

                Question: Would the fact that HP pay for views via MTurk look suspicious to Google? Might it be the reason for the latest traffic drop?
                
                S'cuse if that's a stupid question but being in the UK, I have no idea how MTurk works.

                replyreport
                1. 58

                   Mark Knowlesposted 11 years agoin reply to this

                   That is an interesting point. It could well be seen as an attempt to artificially inflate the US viewing figures. I wonder how bad the recent fall in traffic actually is when you take out the MTurk visitors.

                   replyreport
                   1. 95

                      PaulGoodman67posted 11 years agoin reply to this

                      I find it difficult to see how a few hundred daily visits to unpublished hubs would have any significant effect.  But who knows?

                      replyreport
                      1. 58

                         Mark Knowlesposted 11 years agoin reply to this

                         Is it only a few hundred? I guess 300 a day = 2,100 a week = 8,400 a month = minimal input, so not a big deal if that is the case.

                         replyreport
                   2. 61

                      SunSevenposted 11 years agoin reply to this

                      Now this rating is open to all non-US MTurkers as well. So, the visits will be from all over the world now!

                      replyreport
             2. 75

                WriteAngledposted 11 years agoin reply to this

                
                You cannot remove your articles from Helium.
                
                I am sorely tempted to set up a website simply to reproduce all my Helium articles and then try to get the ones on my site higher in the SERPS than the ones at Helium!

                replyreport
11. 75

    WriteAngledposted 11 years ago

    I wonder if the new rule that only people in the USA can work on MT derives from the fact that this is the population who are willing to accept the lowest rates for the job, as shown in this experiment:
    http://www.thedailybeast.com/articles/2 … -hour.html

    replyreport
12. 95

    PaulGoodman67posted 11 years ago

    I see that the hub examples on MTurk have more detail now.  I will carry on for now and see if I can raise my accuracy level.  Earnings are reasonable when accuracy is 90 something percent.  But when it falls, it starts becoming less worth it.

    replyreport
    1. 95

       PaulGoodman67posted 11 years agoin reply to this

       I understand that there will never be a perfect system, by the way.   I just think there is room for more tweaking.  Improving accuracy improves things for everyone.

       replyreport
13. 88

    Will Apseposted 11 years ago

    You have some patience, Paul. The clutter of nonsense you have risen above in this thread (including some of the nastiest posts I have ever seen) should get you reincarnated as a luminous light being of infinite wisdom. Well, that is assuming that any of the stuff my wife believes is actually true.

    replyreport
    1. 0

       EmpressFelicityposted 11 years agoin reply to this

       I'm not seeing much in the way of nastiness on this thread. Certainly none directed at the OP.

       replyreport
       1. 95

          SmartAndFunposted 11 years agoin reply to this

          As far as my posting here goes, I'm glad Paul is reporting the problems he's encountering on MTurk. These are things we need to know. I'm not upset with Paul, I'm upset that our hubs are being graded in a way that leaves so much room for inaccuracy, and by people who likely are not qualified. I'd be relieved to learn that Paul is grading my hubs, but I have a feeling he is the very cream of the crop there at MTurk, and those with his qualifications are few and far between.

          replyreport
          1. 0

             EmpressFelicityposted 11 years agoin reply to this

             
             
             I couldn't agree with you more, SaF.

             replyreport
          2. 85

             Marisa Wrightposted 11 years agoin reply to this

             +1

             replyreport
    2. 95

       PaulGoodman67posted 11 years agoin reply to this

       To be brutally honest, I think some people who are commenting in this thread don't really have any experience of hubhopping at MTurk and don't understand the bonus system - it is very clear from some of the comments.  My main purpose of opening the thread was to give some constructive criticism to the folks at HP.   I was also interested to see if there were any other hubbers in the forums who were turking and hear their experiences.

       replyreport
14. 61

    LindaSmith1posted 11 years ago

    So those who do not even have a handle on spoken English, let alone the written English is going to judge hubs on MTurk.  What a bad joke that is.  HP has the Hub Hopper, guess that isn't good enough,  I wonder if the pay they pay out to MTurk is going to affect earnings here.

    replyreport
15. 61

    LindaSmith1posted 11 years ago

    So those who do not even have a handle on spoken English, let alone the written English is going to judge hubs on MTurk.  What a bad joke that is.  HP has the Hub Hopper, guess that isn't good enough,  I wonder if the pay they pay out to MTurk is going to affect earnings here.

    replyreport