Navigation
Search
New Comments
Handy Links
Syndication
« Using Video Games to Enhance Training Effectiveness | Main | Front loading testing into the applicant tracking process »
The Utility of Unproctored, Internet-Based Testing
On May 16, 2006 by Jamie Madigan
I like the Internet and I think it’s going to be HUGE some day. So another one of the better SIOP symposiums that I saw this year was “Unproctored Internet Testing: What do the Data Say,” which dealt with unproctored, Internet-based, pre-employment testing. It’s been a hot topic for years now, but what was noteworthy about this talk was that it dealt with actual data from a number of different research projects. It was pretty interesting to see what actual data had to say on the topic, though some of the presentations elicited a few “well, duh” moments.
The short version of most of the presentations was that mean test scores don’t go up over time when you offer unproctored testing. I’m not really sure why this would be surprising if you’re offering non-cognitive tests like personality, biodata, or situational judgment tests where there’s no objectively correct answer that you can get from your calculator, a web search, or your nerdy roommate. And most unproctored testing programs these days seem to omit a cognitively loaded test out of (probably legitimate) fear of cheating.
My favorite part of this symposium, then, was when a couple of guys from Sprint and Previsor/ePredix used utility analysis to decloak the massive elephant in the room: when you drop cognitive ability tests so that you can go unproctored, the overall validity of your selection system suffers, as does the return on investment. The presenters made some modest estimates of the drop in validity (say 15% or 20%) and then calculated the utility for a proctored version of a test battery that included cognitive ability test, and an unproctored, Internet-based version without the cognitive test. Guess which came out ahead? Oh, I’ll tell you: the proctored one that had the cognitive ability test.
Now I know that utility analysis has its own problems. Mainly the wonky nature of SDy, the standard deviation of a worker’s productivity in dollars (I think that’s what it is, it’s been a while so somebody correct me if I’m wrong here). Even minor overestimates can result in utility values that rival the gross domestic product of, say, the Western hemisphere.
But given all else we know about cognitive ability testing, I believe the trend if not the specific numbers. I’m not sure I would ever recommend just yanking any cognitively-loaded tests if they’re valid and job related just so you can sell the idea of unproctored web-based testing more easily. A better approach that I’ve seen some companies adopt is to give the tests without proctoring, screen out the people who are neither capable enough to pass on their own nor unscrupulous enough to cheat. Sure, you may get some cheaters who would otherwise fail, but what you do is bring in the people who passed and test them again with an alternate form of the test in a proctored environment. The people who can’t pass without cheating get screened here. You’re testing fewer people overall on-site (and thus saving money), but you still get rid of the cheaters. I’d have liked to have seen a utility analysis of this approach, but unfortunately the presenters didn’t address it.
Existing comments:
Post your own comment:
