Alternative Test Validation Approaches

On May 4, 2007 by Jamie Madigan

At the 2007 SIOP convention in New York last week, I did manage to drag myself into several symposia, despite the alure of forty foot neon signs and cheap "Romex Brand watches" being sold by a shifty looking guy at the corner of 7th and Broadway. And while not quite as visually stunning as The Lion King production going on a block away, several of them were pretty interesting.

For example, on Saturday I made sure to go to a panel discussion entitled "Validity Generalization in the Workplace." This was of interest to me because the description promised to discuss the use of alternative test validation techniques, such as job component validity, validity transportability, and meta analysis. In fact, I have in the past or present worked directly with three of the four panelists (John Weiner from PSI, Wanda Campbell from the Edison Electrical Institute, and Ryan Ross from Hogan Assessments) and used products of theirs, the use of which had been validated using one of these alternative approaches. And the fourth panelist was Nancy Tippins from Valtera, whose work I was also familiar with.

So there was a lot of expertise sitting up there and it unfolded along the lines you'd kind of expect. The panelists talked about when they'd want to use these validation techniques (when you can't do a traditional criterion-related design), what the legal risks are (nobody knows for sure yet), and what you need to take into consideration (equivalence of job analysis tools, the quality of the original studies you're trying to generalize from). What was kind of disappointing, though was their response to a question about how close is close enough when determining equivalence. For example, if you're trying to transport the validity of a test from one location to another for what should be the same job, you should go through some research to determine that the jobs are actually the same in terms of what they require and involve. You might, for example, do dual job analysis and compare the end results in terms of how similar the KSAO lists are and how closely they match in terms of frequency and importance. But how close is close enough? And if you're looking at something like distance scores across profiles on a standardized job analysis instrument and arranging jobs into families, how do you know when a job's profile is close enough to a family's profile to warrant bringing it into the familial fold? What's the RULE, man?

The panel's answer, in a nutshell: "We don't know. That's for you to figure out in your professional judgment."

That's kind of an unsatisfying answer. In fact, I did this myself a couple of years ago. In my case I was lucky in that the KSAO lists were almost identical and the average frequency/importance ratings only differed by a few tenths of a point on a 5-point scale. So I looked around, shrugged, and felt confident in calling them equivalent. But where's the line? Like with most things, there's probably not a line per se, just your professional, experienced, judgment and your ability to back up your decisions and convince a judge or auditor that you didn't act arbitrarily. I think the benefits of these alternative approaches are too attractive to pass up and that case law will eventually catch up and provide some more concrete standards, but in the meantime I can still see why criterion-related validation is seen as the gold standard.

Navigation

Search

New Comments

Handy Links

Syndication

Alternative Test Validation Approaches

Existing comments:

Post your own comment: