Navigation
Search
New Comments
Handy Links
Syndication
« Playing World of Warcraft could get you a sweet job | Main | The Nihilist's Job Resume »
Using Lake Wobegon to set cut scores
On April 4, 2006 by Jamie Madigan
The official Google Research blog has an interesting blurb about their "Lake Wobegon" hiring strategy --that is, hire only people who are of above average ability. They went so far as to put up a Googletastic graph showing how this approach is, theoretically, better than the "hire if they're better than the worst person here" approach.
I've written before about how the rules change out of necessity when your business moves at insane speeds. I have first-hand experience with this, too. But the Google Research entry strikes me as a little nonsensical. What exactly is being graphed? What "ability" is being discussed here. Knowledge of computer science? Intelligence? Fit with the Google culture? Given that businesses of any complexity have a division of labor that creates diversity in its jobs and thus diversity of the skills, knowledges, and abilities needed for those jobs, trying to plot "ability" on one axis seems more like a bizzare exercise than proof that your particular hiring strategy works.
That being said, I do wonder what the medium-term and long-term results would be for applying this kind of reasoning to test cut scores if you were talking about a single ability or a composite of several abilities for a particular job or job family. Say you knew that customer service orientation, as measured by a personality test, was important for a job. The job of Customer Service Representative (CSR), for example. And the pass/fail decision for each applicant was made by comparing his/her score to the mean score of the existing CSRs. Would the trend in job performance look like what the Google Research post describes?
I'd guess so. It'd be pretty easy to find out by creating a Monte Carlo type of experiment with hypothetical data given a number of different starting conditions and assumptions about test validity. In fact, this is close to what the Google Research blogger says he did. I also wonder if this would be a defensible cut score strategy --improving the ability of the workforce is often cited as an adequate reason for setting cut scores higher than the average ability of your current job incumbents. Where things would probably break down, though, is when you can't find anyone with a high enough customer service orientation to hire after your group's mean passes some threshold. So you have to either alter your cut score strategy or not hire anybody. Still, interesting to consider if your goal were to improve the quality of your incumbents over some temporary period of time.
Existing comments:
Posted by Bryan at April 5, 2006 1:39 PM:
In lieu of the Lanning decision, I wonder how this would hold up in court (in the 3rd circuit at least). You raise really good issues; I'm generally uncomfortable with any cut-score method that is not based on the requirements of the job. Plus, I wonder about the type of test--this would only work if it's a well drafted, well scored, well administered, job-related test. Is the test so good that we can throw all our eggs into it? And, as you point out, what's being measured? Is it based upon good job analytic data? Is it a good idea to focus so much on one "skill"? Will this result in a diverse enough (in all senses of the word) workforce? End rant.
Posted by Stephen at April 7, 2006 6:45 AM:
Thanks for this shrewd object lesson in what NOT to do in establishing hiring standards. Your devastating critique of the so-called Google research leaves little need for further comment. I find the example disturbing because I am a big proponent of using statistical simulation in assessment and have used it extensively in my work. But holy bootstrap, I have always made some attempt to validate the simulation results against real-world outcomes! The WHAT-IFS were piled so high in the Google example as to make the analysis worse than useless.
The Lake Wobegon Strategy is completely foreign to us--and for good reason. Its sliding-scale approach is so arbitrary, impractical, and blatantly lacking in job relatedness as to be plainly unlawful. To those consultants with no training in selection science who nevertheless insist on providing assessment services, I say: DO NOT TRY THIS AT HOME, AT WORK, OR ANY PLACE ELSE.
Posted by Michael C at August 7, 2007 12:29 PM:
In review of the Google Research Blog, I do not see any reference to how the job-relatedness of the test was established. However, I don't believe the point of the research blog was to suggest that selecting from the mean test score would be effective regardless of construct being measured. Further, I don't know that the blog researcher was suggesting that only one construct/test scale should be used. Rather, I think one should assume that job-relatedness/validity is a prerequisite and that the method could/should be applied to all measures meeting this prerequisite. Given these assumptions, the method suggested is practical, easy to establish, and has been used by employment testing for decades (e.g., US DOL GATB). The best selection processes will, within reason, be used to hire those with the highest probability of being successful. Hiring from the top of the distribution(s) of scores will accomplish this for validated tests. Using the method of selecting above the mean is a practical approach to establish a consistent passing profile. However, in addition to setting scores to select the best candidates, there are some other important issues that need to be taken into consideration (e.g., adverse impact, salary requirements, time-to-hire).
Post your own comment:
