Crowd Sourcing Quality Control Measures

You’ve decided to start crowdsourcing to save money, allow thousands of workers to complete your tasks far faster than your in-house employees could and gain access to a 24x7x365 workforce. However, you have discovered that chasing the lowest cost option can come at the cost of quality.

In some cases, the crowd doing the work isn’t human at all but robots designed to complete tasks automatically. The solution is to build in crowd-sourcing quality measures into your crowdsourcing tasks.

What crowd sourcing quality measures can you use, and when are the most appropriate? If you want to apply quality management to your crowd-sourced projects, what can you do?

Crowd sourcing forums are popular venues for giving surveys. Many graduate level students are using crowd sourcing surveys to get hundreds of responses in a short period of time for far less money than would have to be paid to the general public.

Consumer research groups, political activists and small businesses have all used crowd sourcing websites to ask for public opinions on everything from website names to political candidates to rating ads. However, the human desire to maximize income per hour results in many crowd-workers to click through the task as quickly as possible.

This problem can be partially solved by setting a minimum time limit. This could be done by preventing the “next” button from appearing until a minimum time has passed. This can frustrate workers who are actually done, and not all tasks are amenable to this option. For example, if your task involves filling out a single page questionnaire, this method simply isn’t feasible.

Another solution is simply not paying for tasks completed in an unreasonably short period of time. This solution is simple to implement and offers protection against robots as well as humans who are rushing through assignments. This can take the form of captchas, demanding answers to spelled out math problems and other tests robots can't do.

A third option is testing the test taker’s comprehension by asking questions about a scenario to ensure it was read. A common version of this check is asking, “What was the subject of this survey?” Then survey takers must select from a series of check boxes or radio buttons to indicate the subject of the survey. If the user selects the wrong subject, then the survey could be rejected.

When putting together surveys or questionnaires, a concentration check asks questions that are the opposite of an earlier question. For example, if you’ve asked someone if the website at the link to be verified is a shopping page, a later question could ask if it is another type of page. Contradictory answers, such as marking that it is both a shopping page and not a shopping page, reveal a person who is not paying attention or a robot.

When you are creating tasks to resist robots, the automated scripts that make selections on crowd source tasks to make money for the programmer, there are several protective steps you can take. One option is to add a text field in the task that requires data entry. You can ask questions such as “What is your favorite color?” or “What is your favorite sport from this list?”

Robots cannot answer this question, and if you make the task dependent upon an entry in this field, you prevent the robot from consuming your crowd sourced work. Another option is asking a question that a robot could not answer. For example, you could ask, “What is 5+8?” An advanced robot may be able to recognize this and answer it.

Using a text based question with similar reasoning will stop a robot. Ask questions such as “What is the square root of four?” and require a text based entry for the task to be accepted.These tasks have the added benefit of acting as concentration checks for human crowd-source workers.

IT quality management can be partially crowd-sourced. One example of this is sending the same task to three or more crowd-source workers. The correct answer is then determined by a majority vote.

If someone consistently votes against the majority, that user’s answers can be reviewed by a quality checker to determine if the person is guessing at random, answering based on a different personal classification system, requires more training or is working so quickly that quality suffers. You can offset the tendency of users to speed through tasks by automatically rejecting those completed too quickly.

Another option for IT quality management is using the crowd for QC. The review of other crowd-sourced tasks is also done by the crowd, though these tasks may be limited to those who pass a test to prove that they understand the acceptance criteria to be used when evaluating crowd-sourced tasks. An example of this is when the editor is paid to review the written content or editing of another worker.