ArtsAutosBooksBusinessEducationEntertainmentFamilyFashionFoodGamesGenderHealthHolidaysHomeHubPagesPersonal FinancePetsPoliticsReligionSportsTechnologyTravel
  • »
  • Business and Employment»
  • Human Resources (HR)

Test Item Validation

Updated on December 20, 2017
Carolyn M Fields profile image

Lifelong learner, musician, author, world traveler, truth enthusiast, and all around bon vivant.

Bell Curve

An Idealized Bell Curve
An Idealized Bell Curve | Source

The Bell Curve

When you were a student, in any grade up to or even including College, you probably ran across a teacher (or two or three) who graded “on the curve.” This method was used to distribute grades across certain thresholds, and provide a “balanced” distribution of scores. This works fine for academic results, but not so much when an employer is trying to determine the competency of their workers. I guarantee you that the next time your car is in the shop, you don’t want your mechanic to have passed his/her training because the instructor “graded on the curve.” I am certain that you would prefer that this person had demonstrated an established level of competence in training before working on your vehicle.

Further, I’m sure you would prefer it if your mechanic had gotten 100% on all of the performance and knowledge tests in auto repair school. Or to take a more extreme example, wouldn’t you want your Doctors or Dentists to have gotten perfect scores on all of their tests when they were in school? Unfortunately, this is not a reasonable expectation.

You can, or course, demand high performance, but you can’t require perfection when it comes to learning new skills or knowledge. Which begs the question: how is an appropriate level of competence established when becoming proficient at a new job? If 100% is too high, what about 90%? Now it seems rather arbitrary to pick a percentage out of thin air. Fortunately, there is a process that can help test designers set an appropriate “cut score” as it is called.

What is Test Item Validation?

Dr. William Angoff wrote about his test item validation process way back in 1971. It is a means of certifying that the tests given in conjunction with a training course or class are a reliable and fair indication of how well learners have mastered the material being tested. By using a panel of experts (preferably five, but you can use three or four) known as Subject Matter Experts (SMEs), test writers are able to develop questions that are clear, reliable, and valid. These SMEs are knowledgeable and competent in the content area being tested. Plus, they are able to determine what an acceptable “passing score” might be for a newly trained, yet competent trainee.

What is a Minimally Competent Person?

You can’t expect a newly trained person to be an expert on the job. You may want that, but it’s not realistic. But what you can expect is the following from trainees:

• Knows basic functions required for the job – can do low level tasks

• Willing and able to learn

• Knows basic fundamentals

• Able to build on concepts taught

• Can pull from resources

• Knows where to find things

• Has initiative out of school

• Practices safe working habits

What are SMEs Responsible For?

Each SME on the panel will read through the test that has been developed, as if they were a student in the training course. They are instructed to consider a “minimally” competent person (newly trained) when making their determinations. The question then becomes: Out of 100 minimally competent people, how many do you think should be able to answer the question correctly? A super easy question would get a 100. A tougher question might get 70, 80 or 90.

In addition to establishing cut scores, SMEs must also evaluate three additional attributes for each question:

  1. Linkage (or Validity). Is it a good question to ask? Is it relevant to the job? Does it test the behavior in the training objective?
  2. Clarity. Is it clear? Does it make sense? Can a student understand what is being asked?
  3. Reliability. Can you expect all students to give the same answer once they understand the material?

If the test questions fail any of the above attributes, then the Facilitator will lead a discussion on how to “fix” the test item. If it can be re-worded easily, then it can be saved. Otherwise, the item will need to be deleted from the test.

What Else does the Facilitator Do?

In addition to leading the discussed described above, the Facilitator (person running the validation session) has the following responsibilities:

  • Poll the SMEs for their cut scores, and enter ratings into a spreadsheet
  • Check the standard deviation for the scores on each individual item. Wide extremes between ratings are not acceptable, and must be discussed.
  • Watch the clock so the meeting doesn’t get bogged down.

More About Cut Scores

Cut scores are established for each individual test item (question). It is the average of the ratings from all of the SMEs. The average of all the scores for all the items on a test becomes the cut score for the test. For example, if there are 10 questions that each have a cut score of 90, the learner must get 9 out of 10 questions correct to pass the test. Cut scores must be reasonable. The should never be so high that no one can pass the test. They should also not be so low that anyone off the street could pass the test.

Benefits of Using The Angoff Method

The Angoff Method of Test Item Validation is a systematic and well documented approach. Using this method will ensure that your tests are reliable, valid, and fair. In addition, they will be legally defensible as a means to determine employee competency. Finally, it has been a widely used and accepted approach to test validation for over 40 years.

How About You?

Do you currently use a validation process for test items?

See results

Comments

Submit a Comment

  • MsDora profile image

    Dora Weithers 3 years ago from The Caribbean

    On one of my latest jobs, there were SMEs but I wasn't really aware of all their responsibilities. Thanks for these explanations on the validation process.

  • Carolyn M Fields profile image
    Author

    Carolyn Fields 3 years ago from South Dakota, USA

    Good question - tsadjatko. Subject Matter Experts must be selected and vetted very carefully, in order for validation results to be accurate. For more information, please see:

    www2.education.ualberta.ca/educ/psych/crame/files/RickerCSSE2003.pdf

  • tsadjatko profile image

    TSAD 3 years ago from https:// www.consumeraffairs.com/ online/ hubpages. html

    This is interesting, I've never heard of it, but when you say 'They are instructed to consider a “minimally” competent person (newly trained) when making their determinations." Isn't the ability of the SME to consider a minimally competent candidate an extremely subjective task and subject to serious variation depending on the SME? So wouldn't that be a potential problem in effective test validation of the Angoff method?

working

This website uses cookies

As a user in the EEA, your approval is needed on a few things. To provide a better website experience, hubpages.com uses cookies (and other similar technologies) and may collect, process, and share personal data. Please choose which areas of our service you consent to our doing so.

For more information on managing or withdrawing consents and how we handle data, visit our Privacy Policy at: "https://hubpages.com/privacy-policy#gdpr"

Show Details
Necessary
HubPages Device IDThis is used to identify particular browsers or devices when the access the service, and is used for security reasons.
LoginThis is necessary to sign in to the HubPages Service.
Google RecaptchaThis is used to prevent bots and spam. (Privacy Policy)
AkismetThis is used to detect comment spam. (Privacy Policy)
HubPages Google AnalyticsThis is used to provide data on traffic to our website, all personally identifyable data is anonymized. (Privacy Policy)
HubPages Traffic PixelThis is used to collect data on traffic to articles and other pages on our site. Unless you are signed in to a HubPages account, all personally identifiable information is anonymized.
Amazon Web ServicesThis is a cloud services platform that we used to host our service. (Privacy Policy)
CloudflareThis is a cloud CDN service that we use to efficiently deliver files required for our service to operate such as javascript, cascading style sheets, images, and videos. (Privacy Policy)
Google Hosted LibrariesJavascript software libraries such as jQuery are loaded at endpoints on the googleapis.com or gstatic.com domains, for performance and efficiency reasons. (Privacy Policy)
Features
Google Custom SearchThis is feature allows you to search the site. (Privacy Policy)
Google MapsSome articles have Google Maps embedded in them. (Privacy Policy)
Google ChartsThis is used to display charts and graphs on articles and the author center. (Privacy Policy)
Google AdSense Host APIThis service allows you to sign up for or associate a Google AdSense account with HubPages, so that you can earn money from ads on your articles. No data is shared unless you engage with this feature. (Privacy Policy)
Google YouTubeSome articles have YouTube videos embedded in them. (Privacy Policy)
VimeoSome articles have Vimeo videos embedded in them. (Privacy Policy)
PaypalThis is used for a registered author who enrolls in the HubPages Earnings program and requests to be paid via PayPal. No data is shared with Paypal unless you engage with this feature. (Privacy Policy)
Facebook LoginYou can use this to streamline signing up for, or signing in to your Hubpages account. No data is shared with Facebook unless you engage with this feature. (Privacy Policy)
MavenThis supports the Maven widget and search functionality. (Privacy Policy)
Marketing
Google AdSenseThis is an ad network. (Privacy Policy)
Google DoubleClickGoogle provides ad serving technology and runs an ad network. (Privacy Policy)
Index ExchangeThis is an ad network. (Privacy Policy)
SovrnThis is an ad network. (Privacy Policy)
Facebook AdsThis is an ad network. (Privacy Policy)
Amazon Unified Ad MarketplaceThis is an ad network. (Privacy Policy)
AppNexusThis is an ad network. (Privacy Policy)
OpenxThis is an ad network. (Privacy Policy)
Rubicon ProjectThis is an ad network. (Privacy Policy)
TripleLiftThis is an ad network. (Privacy Policy)
Say MediaWe partner with Say Media to deliver ad campaigns on our sites. (Privacy Policy)
Remarketing PixelsWe may use remarketing pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to advertise the HubPages Service to people that have visited our sites.
Conversion Tracking PixelsWe may use conversion tracking pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to identify when an advertisement has successfully resulted in the desired action, such as signing up for the HubPages Service or publishing an article on the HubPages Service.
Statistics
Author Google AnalyticsThis is used to provide traffic data and reports to the authors of articles on the HubPages Service. (Privacy Policy)
ComscoreComScore is a media measurement and analytics company providing marketing data and analytics to enterprises, media and advertising agencies, and publishers. Non-consent will result in ComScore only processing obfuscated personal data. (Privacy Policy)
Amazon Tracking PixelSome articles display amazon products as part of the Amazon Affiliate program, this pixel provides traffic statistics for those products (Privacy Policy)