Navigation
Search
New Comments
Handy Links
Syndication
« Teach the children to cheat | Main | Changes to I/O Licensing in California »
Testing the significance of corrected validities
On January 26, 2006 by Jamie Madigan
Put the following in that file folder labeled "Technical Stuff I Probably Should Have Known Coming Out of Grad School" along with "how to change that bulb thingie in the back of the toilet tank."
Some colleagues and I were recently debating whether or not to conduct significance tests on corrected validity coefficients. For example, say we're doing a validation study and have a huge number of test-takers and a less-than-huge number of employees with job performance data. Once you match up the two, the variation in test scores amongst those test-takers is truncated, maybe because only the brainiacs with higher test scores got hired. Because of the way God made math, that restriction in range is going to hold down your validity coefficient like a pair of cement shoes.
So what to do? Whip out your Bag of Magic Pixie Dust and correct for restriction of range, of course. If the magic dust doesn't work, apply this formula:
Where ru is the unrestricted validity coefficient, r is the observed validity, S1 is the unrestricted standard deviation, and s1 is the observed standard deviation.
So all this I already knew, minus having to look up the actual formula. The question that me and my chums were debating, though, is what to do with that shiny new corrected validity coefficient. My first inclination is to answer the question "Is it significantly different from 0?" In other words, do a significance test on it. Seems natural enough.
One of the other folks I was talking to, though, said he had vague recollections that such tests weren't appropriate on corrected coefficients and that making claims that a corrected validity coefficient is "significant" was naughty at best and nonsensical at worst. So I did some digging, and sure enough, he was right.
To quote SIOP's Principles for the Validation and Use of Personnel Selection Procedures:
That last sentence is key. The Principles pretty much carry enough weight to stop the debate right there, but I decided to get a second opinion from the American Psychological Association's Standards for Educational and Psychological Testing. Sure enough in Standard 14.5:When range restriction causes underestimation of the validity coefficient, a suitable bivariate or multivariate adjustment should be made when the necessary information is available.
...When adjustments are made, both the unadjusted and adjusted validity coefficients should be reported. Researchers should be aware that he usual tests of statistical significance to not apply to adjusted coefficients such as those adjusted for restriction of range and/or criterion unreliability.
Statistical significance tests for uncorrected correlations should not be used with corrected correlations.
Both works provide references for explanations of why this is the case, but suffice to say I don't want to get into the uber-technicalities now. In the meantime, maybe this is something new you didn't know. And knowing is half the battle! (The other half is actually doing stuff.)
Existing comments:
Posted by David Morris at January 30, 2006 8:40 PM:
I say down with significance testing and report p values and confidence intervals. Sorry Sam got sick and you had to miss Thursday's meeting. It was good and in a cool room. By the way, did you know the "Old Indian Dude" passed away last September?
Posted by Stephen at January 31, 2006 5:46 AM:
I have been testing the significance of corrected validity coefficients since 1991 using resampling techniques. Like any other statistic, all you have to do is build a reference (or sampling) distribution then compare your observed result to what happens under repeated random sampling. For a primer on resampling, check out the following 2005 IPMAAC presentation at http://ipmaac.org/conf05/: Using Simple Computer Simulations to Address Complex Assessment Problems. Bootstrapping, Monte Carlo, computer simulation (whatever you wish to call it) is quite simple using Resampling Stats software.
The most recent article on testing the significance of corrected validities is:
Chan, W. & Chan, D. (2004). Bootstrap Standard Error and Confidence Intervals for the Correlation Corrected for Range Restriction: A Simulation Study. Psychological Methods. Vol 9 (3), 369-385.
Posted by Jamie at February 3, 2006 4:27 PM:
Awesome, thanks, Stephen! I think I'll take a look at those and see what use I can make of them. Don't those techniques suppose a huge sample from which to bootstrap and draw subsamples?
David: Yeah, confidence intervals are assumed. I'm getting to where I report those with just about every significance test.
Posted by Stephen at February 6, 2006 10:36 AM:
Resampling uses the same sample size as the original analysis. If the original data set is small (less than 25), then no procedure is likely to produce stable results. In testing the significance of corrected correlations, I'm specifically referring to a randomization (AKA permutation) procedure that tests whether the population correlation (rho) is different from zero. Basically, one is merely shuffling (randomizing) the X and Y values, calculating Pearson r, storing the result, and then repeating 10,000 times or more. The observed correlation is compared to this randomized (sampling) distribution to see what the chances are of getting the observed result by chance alone.
For a clear explanation of how to do a randomization test on a correlation coefficient using resampling, go to David Howell's Statistical Home Page: http://www.uvm.edu/~dhowell/StatPages/StatHomePage.html, click on Randomization and Bootstrapping, the Randomization Procedures, and Correlation Between Two Variables. Testing corrected validities follows a similar procedure except the statistic is a corrected, not a raw correlation coefficient.
Post your own comment:
