How To Create a Survey Properly
Why is it Important to Follow Rules When Creating Surveys?
In order for any measure to be of use, it has be both valid and reliable. Validity refers to whether you are measuring what you say you are measuring, and reliability refers to whether you will get consistent results using that same measure. A number of factors can affect the reliability and validity of a survey. If you pay close attention to the 12 factors below you will have a much better chance of creating a survey that will not only measure what you want it to measure, but will prevent creating more work for yourself when eventually analyzing the results.
Example of Non-Mutually Exclusive Survey Categories
46 and older
Making Survey Categories Mutually Exlusive
Mutually exclusive categories are those for which the user answering the question does not fit into more than one category. The most typical example of violating this rule occurs when survey writers use numbers and one or more of that series of numbers occurs in two different categories. Consider the example to the right: Although the intention is that a person can only choose one answer, someone who is 25 years old fits into more than one category. For each person who is exactly 25 who answers your survey, you have no way of knowing which category they actually fit in and so any conclusions you draw based on this demographic are of limited validity. You must make sure that people taking your survey can only make one choice for each question unless your intent was to have them select all that apply.
Proper Use of an Exhaustive List in a Survey Question
With Which Race Do You Primarily Identify Yourself?
Making Survey Questions Exhaustive
Another common error that occurs in surveys is the presence of questions that do not include an exhaustive list of answers, in other words the person answering the question doesn't fit into any of the categories. I have attempted to respond to a number of surveys that asked for my state of residence, some of which I did not complete because "District of Columbia" or "Washington, DC" were not options. In each of those cases I was lost as a participant because I chose to stop taking the survey rather than provide an invalid answer. If you are only interested in a few demographic groups it is okay not to provide an exceptionally long list in order to make the choices exhaustive. The best way to then make your list of answers exhaustive is to provide an "other" category. Do be careful though that you have included all groups you may want to study before resorting to "other".
You should also consider whether you are capturing a true range of possible answers in your survey question. Using a sliding scale is a good way to ensure that you are truly capturing how the respondent feels rather than forcing them into a category. For example, you my want to know how strongly people feel about a customer service experience. If you only include "satisfied" or "unsatisfied" you will be missing a potential range of responses. Instead you could start with "completely unsatisfied" and create a range up to "completely satisfied". Be sure that you have an equal number of responses on both sides of the range and include in the middle "neither satisfied nor dissatisfied".
When is a Survey Long Enough?
This question is not an easy one to answer but it should be answered by anyone who is writing a survey. Go back to the question that you want answered and think through what information you need to answer that question. Any information that is key to that answer must be included, and your questions should include hypotheses other than your own. For example, let's say you are interested in what factors contribute to people being pro-war or anti-war. You may believe that the main factor is gender, but other possibilities include race, socioeconomic status, country of origin, past experience with war, etc. Keep an open mind and develop enough questions that you can test not only your own theory but competing causes.
When is a Survey Too Long?
Just as you must include enough questions to be sure that you can test or explore a hypothesis, you have to be careful not to make the survey too long. The longer it takes to answer a survey, the more likely it is that participants' minds wander, or they start choosing answers just to be done. A good way to test for this is during your pilot survey, discussed below.
Making Your Survey Fit Your Audience
It is very important that you develop survey questions that are easily understandable to your intended audience. If you are developing a survey about climate change, your questions would be different if you were surveying scientists as opposed to non-scientists. Stay away from any words or phrases that are not common knowledge. If such words or phrases are key to your survey, provide an explanation of what they mean.
Check for Spelling and Grammatical Errors in Your Survey
This is a very important step in creating a survey. As the researcher, you must eliminate anything you can that might affect the reliability and validity of your results. A poorly written survey can affect both as it will bring up emotions and actions in people that may affect their responses. I recently completed a survey in which the researchers used the phrase "what was your rational for making that decision?". They used this phrase three times in regards to three different scenarios that I had responded to. Rather than focusing on the survey, I found myself focusing on the fact that they should have used the word "rationale" instead of "rational", and was increasingly annoyed each time it was used. This could well have affected my responses. Do not rely just on spell check for your survey as you may have used a word that is spelled correctly but not in the right context. As with any writing, have someone who has not seen your survey before read it over. As the writer you may see words as you intended them, and hear the proper word in your head even though it is not what you wrote.
Limit Open-Ended Questions in Your Survey
In order to eliminate extra work on your end once your data collection is complete, limit as much the number of questions you include in which the user has to write in their answer as opposed to choosing from a list of options. Doing this properly will increase the reliability and validity of your responses as you eliminate potential errors on both the user end and your own. For example, if you are interested in where someone lives, it is much better to provide a list of countries from which someone can choose (don't forget to include "other"), rather than have them write in their answer. People in the United States could write in United States, United States of America, US, USA, or spell it incorrectly altogether. Making a list to choose from eliminates the burden of trying to quantify these answers after the data collection is complete.
Make Sure Your Survey Sample Size is Big Enough
There are two areas of concern:
Overall Sample Size - Your overall sample size will depend upon the particular field of study, though there is some disagreement even within fields as to proper survey sample size. To be on the safe side, you should plan to have at least 100 valid surveys once your data collection is complete. This means you should plan on finding more than 100 people to take your survey as you may lose some because they were incomplete, it was clear the person was answering questions randomly etc. Do quality checks as you go along to see what percentage of surveys you are losing so you can compensate for that.
Sample Size Within Key Demographics - Pay careful attention as you are collecting your survey responses that you are capturing a good mixture of people across key demographics. If you are studying war sentiments and you want to compare gender of respondents, you must be sure that you are getting enough people within your sample from each group. You will not have valid results if you end up with 98 male responses and 2 female. Demographics can also determine the overall sample size you need. If you have a key demographic with four categories, your overall sample size should reflect how many people you need in each of those groups. If you decide based on your field that need at least 50 people in a four group demographic, your overall sample size should then be at least 200.
Avoid Skewing Your Results Due to Sampling Error
Sampling error can include having too many people in one demographic as mentioned above, and may occur because of where and how you choose to give your survey. When a survey contains too many respondents in one group it is considered "skewed" and your overall results are invalid. Please be careful as results can be skewed even if the demographics you asked questions about seem covered properly in your results. I made this mistake in my first survey in college, when asking students about how university funds should be distributed. I stood in front of the library asking for volunteers as there was a lot of student traffic there. I had the correct number of males and females, captured people across race, etc. However, as my professor correctly pointed out, I did not measure student attitudes, I measured students who go to the library (who well could have a different view than the "average" student on how to disperse funds).
People have different views of the quality of news programs on talk radio. To ensure that you are paying attention, please select "Always"
Add Elimination Questions to Your Survey
Particularly in longer surveys, it is important to include questions that indicate the person taking the survey is doing so seriously. The example to the right is an elimination question, and if someone answers with anything beside "Always", you should throw out their entire survey. You have no way of knowing if any or all of their previous and subsequent responses are valid.
Make Your Survey User-Friendly
If a survey is not user-friendly, you will have difficulty judging the validity of the responses. If you are printing a survey for people to complete by hand, be sure that the font is large enough for them to read the survey easily and that there is enough space between responses that it is clear the respondent is answering another question, and to cut down on respondents skipping questions. If respondents have to write in a response rather than choosing an answer, be sure that there is enough room for that response. If you are developing a survey that will be completed online, be sure that the type color can be easily seen against the background you chose.
Do a Pilot Survey
This is a very important step in creating a survey that will give you the valid and reliable responses you are looking for. You should find people that are similar to the group you will be giving the survey and have them complete a test run for you. You can then get feedback regarding spelling errors, confusion regarding the how the questions were phrased, whether the survey is too long or too short, etc.
© 2014 SusanPlant