Mechanical Turk (mturk) Tips and Tricks

If you are just starting with Mechanical Turk (mturk), you may want to read my review here, which gives a good overview of the site and system.

Tip 1: Watch out for scams

There are scams on Mechanical Turk. Generally, anything that is over $3.00 per HIT is suspicious. There are academic surveys, writing HITs and transcription HITs, in this range, but pay closer attention. If it is a survey hosted on qualtrics.com, or a .edu URL,or from a well known requester you are probably okay. Turkopticon (more on this later) can also help with this. If it has anything to do with auto insurance quotes, credit ratings, signing up for free gift cards, or other similar 'promotional' offers, leave it alone, or better yet report it to Amazon. If you view a HIT, whether it is accepted or not, in the bottom right are two “report” links for broken HITs and policy violators. Use them as often as necessary, but be careful not to report legitimate HITs, and here is one reason why.

Technically, any HIT that asks for your email address or requires you to download software is in violation of Amazon's policies for Mechanical Turk, and most should be reported. However, there are legitimate academic surveys that will ask for your email for a follow-up or for information about future surveys. There are also a few academic surveys that require a piece of software called inquisit that runs on Java. Some of these surveys are the best paying around and reporting them gets them removed, even if they are legitimate. So, use common sense, and check the Turker Nation forums (more on this later as well).

Tip 2: Use Turkopticon

Turkopticon is a website and extension for Firefox and Chrome, that allows turkers to rate requesters and write reviews for them. The extension places a little red or blue triangle next to requester names on the Mechanical Turk HIT lists and on your completed HITs page. When you mouse over this triangle, it will show ratings for the requester on the following metrics:

• Communicativity (how well the requester communicates and deals with problems)
• Generosity (how well the requester pays for the task)
• Fairness (how fair the requester is compared to others)
• Promptness (how fast the requester pays)

The extension also shows how many people have rated the requester, the more that have, the more likely the scores are valid. If the triangle is blue, that means there are no ratings yet for this requester. There is also a link available in the dropdown to submit your own reviews and ratings. You have to sign up on the site to be able to read and write reviews, but not to see the ratings from the extension.

The site and extension can be slow to respond, but are an extremely useful tool in identifying good and bad requesters. The system works when you participate, so please leave reviews and ratings, but be fair. If you have a problem, try to contact the requester and give it a day or two, then leave a review. Some requesters also take a few weeks to approve HITs so please do not give a requester a poor rating on generosity if they did not pay in 20 minutes! You can always leave a rating blank and only rate on the categories you know about.

A few final notes about Turkopticon that also apply to Turk Alert. Requesters can change names, but their requester ID stays the same. Because of this, you may look up a requester on Turkopticon or Turk Alert and see a name you do not know for the requester. As long as the ID is correct, this is acceptable and only means that the requester changed names. Some get bought out, etc. To find a requester ID to search for in Turkopticon, simply click on a requesters name in a list of HITs. This will search for HITs by this requester. Also, you can click on a completed HIT from your dashboard. In either case, look at the URL in your address bar, it may be hard to read, but somewhere in there is requesterID=xxxxxxxxxxxxxx. Simply highlight and copy the ID number from right after the equal sign to the end. Do not include the ampersand (&) symbol if it is at the end, that is not part of the ID. Now you can paste that number into the search box at Turkopticon.

Turkopticon is at http://turkopticon.differenceengines.com

Tip 3: Use Turk Alert

Turk Alert is very simple. It is a free service (donation requested) that allows you to receive email when your favorite requesters post new HITs. All you do is sign up, add the requester ID and your email and as soon as that requester posts HITs, you will get an email. It is fairly useful, especially for requesters that post lots of HITs. Sometimes there is a delay in the alert, whether it is from my email provider and client, or from Turk Alert, or both I do not know. This lag can cause problems if it is a type of HIT that disappears quickly. It is also not terribly useful for academic surveys, as each one typically posts under a different requester. There are a few academic survey requesters that pay well and regularly post new work so it can help with those.

There are limitations on Turk Alert though. There is a limit to how many requesters you can have alerts for at one time. Also, you will only receive one alert per hour per requester, even if they post several times that hour. Finally you can only receive a maximum of 20 alerts in in one day.

Turk Alert can be found at http://www.turkalert.com

Tip 4: Keep your accepted HIT percentage high

Many HITs, especially the more lucrative ones, require a high percentage of accepted HITs. This percentage is easy to mess up, especially in the beginning. The less HITs you have completed, the more one accepted or rejected HIT affects this percentage. Consequently, if you mess up a few HITs early in your Mechanical Turk career, it takes more work to fix it later.

If you have never worked for a requester before, only do a few HITs from them, pay attention to the instructions, and wait and see if they are rejected. It is far better to only do a few, than several hundred only to get large amounts rejected. Getting rejections removed is almost impossible, the requester has to do it and many will not as Amazon makes it difficult for them. Also, there are requesters (a type of scammer) that will get work done, then reject it all so they do not have to pay. These are not too common, but it does happen. Some requesters also use 'majority rules'. This means they put the same question/task to many workers, and the most common answer wins, all others are rejected. This can cause many bad rejections if there are many scam workers completing that group of HITs, so be careful. Both Turkopticon and the Turker Nation forums can help you gauge how to treat a given requester.

Tip 5: Join and contribute to the Turker Nation forums.

I have saved the best for last. The Turker Nation forums are a fantastic resource for conscientious turkers. There are boards there for many major requesters, some of whom actually participate in the discussion. There are rants and ratings about particular requesters and batches of HITs. There are discussions about how best to do certain types of HITs and what tools can help, and much more.

One of my favorite areas is the Survey Area. There are two main threads, about 'Cool Surveys', which are those that pay more or are 'cool' in some way (very easy, high bonus possible, etc.), and 'Piggy Bank' surveys which are quick and lower paying (not necessarily less per hour though, just shorter surveys). Members of the board post surveys as they find them, often reporting the time to complete and other issues. Many people have these threads set for instant email notification, so they can see surveys as soon as someone posts it if they are busy doing other types of HITs. I do this and it is very handy.

When you join the forum you will NOT have access to most of the boards. There is some useful information, but in order to access it all you will need to be a member for a short time, and make an unspecified number of (non-spam) posts to the boards you have access to. It is well worth the wait. Sign up, read the rules, they are pretty lax but there are some things to be aware of, and make some posts. I recommend writing an introduction, and looking at others and responding to them, and also checking out some of the boards that are available and contributing to them. It doesn't have to be spectacular, just report your experience in an existing thread about a requester, say hi to a few people in their intros, etc, and soon enough you will be a full member. You will be very glad you did.

Turker Nation can be found at http://www.turkernation.com

Summary:

Mechanical Turk can be a great way to make some extra cash, but it takes a little work to get there. If you can type fast, transcription is a great way to make top dollar, writing can be lucrative too, but you can also sit and do the 'mind numbing' type HITs while sitting lazily on the couch listening to soap operas. Go easy at first and be careful of rejections, use the tools to find good HITs and avoid scams, and you could find a bit of extra money in the bank. Comments and questions are more than welcome!