>
a weblog covering personnel selection, assessment, and employment testing in the news

« My Column in TIP | Main | Best Buy HR Manager: "Don't hire him, he sucks" »

Practitioner Periodicals vs. Academics: Round One, FIGHT!

On January 24, 2008 by Jamie Madigan

There's a great article in a recent issue of Academy of Management Journal titled "The Very Separate Worlds of Academic and Practitioner Periodicals in Human Resource Management: Implications for Evidence-Based Management." It's a mouthful, but basically what the researchers did was pick certain facts/issues about which academics feel mighty confident and then examine how those issues were covered by three more mainstream Human Resource Management publications:

  • HR Magazine (the publication of the Society for HR Management, or SHRM)
  • Human Resource Management (a "bridge" publication that theoretically appeals to both practitioners and researchers)
  • Harvard Buisiness Review (which admittedly isn't HR specific, but which is definitely read by a lot of HR managers)

The researchers started with a medium-sized list of findings from the academic research that they felt (probably rightfully so) that are de facto true at this point, but pared it down to three:

  1. The importance of intelligence or general mental ability for job performance
  2. The importance of goal setting and feedback for job performance
  3. The validity of personality measures (including integrity tests) as predictors of job performance

With two of those factors directly related to many selection tests, you can see why this piqued my interest. The researchers wanted to know how much these topics were covered, and when they were covered how in line with academic knowledge they were.

The long and the short of it is that these publications VERY RARELY mentioned any of these topics --at most, just over 1% of the articles mentioned anything related to any of the three. HR Magazine --the official publication of the largest professional society for HR Professionals in the world and with a circulation of over 200,000-- didn't mention cognitive ability testing AT ALL and barely mentioned personality or goal setting.

Let's think about that: If the researchers had also included in their sample The Archie and Jughead Double Digest and the assembly instructions that came with that end table you bought from Ikea, HR Magazine would not have scored any better than these.

The few times that these three periodicals did mention cognitive ability, personality, or goal setting, what they said was sometimes in line with academic research, but it was also often wrong, incomplete, or just bizarre. My favorite was from one of the very few articles on personality testing that recommended not that you use a scientifically developed and scrupulously validated personality test to screen applicants, but rather suggested this:

You can pick up a multitude of clues about a person's character by simply having a restaurant meal together. You'll see how they interact with the waiter or the people sitting at adjacent tables. I sometimes say, "Gee, how much tip do you think we should leave?" Then, based on whatever percentage they suggest, I ask why. I want to see how they make those decisions. A lot of it bears on how they view the world in a more general sense.

The authors of the Academy of Management article say, quite diplomatically, that "This quote represents a selection tactic that is low in validity and utility but high in exposure to potential legal liability." Personally, it reminds me of the time I interviewed for a job and later found out that one interviewer had dinged me because I had somehow chosen the wrong chair to seat myself in.

The article goes on, describing the different things they found in this audit of the 3 journals, and by the end the finding is clear: mainstream journals don't cover this stuff very frequently (if at all), and when they do they often get it wrong. They go on to explain some reasons for this and some ways to fix it, which I'd encourage you find out about if you have access to the AMJ article. There are also other articles that respond to this study and the topic in general, but I haven't had a chance to read them yet.

I think a good follow-up to this kind of research would be to do an audit of not HR magazines, but general HR textbooks. If education is the problem, this seems like a good place to start and an easier problem to fix.


  Existing comments:

Posted by BryanB at January 26, 2008 4:57 AM:

And that right there is why I don't belong to SHRM.

Posted by David Morris at January 27, 2008 3:25 PM:

Jamie forgot to point out that the person he interviewed with and sat in the wrong chair was hired in as a senior level HR Generalist in charge of recruiting. She worked with the company 6 months, went on maternity leave for 9 months, came back and worked for 3 more months and then quit. Not that there is anything wrong with taking maternity leave, but come on. Also, an interesting side note. I was once helping her recruiters do some testing for an Administrative Admin job. One day I tested 5 or so people on a battery of PSI's basic skills tests. Each test is speeded and test takers only have 5 minutes to get through a million items. The tests are designed that anyone rarely gets through the entire test in the given time. There was one guy in there that would put his pencil down after about 3 minutes or so and sit there and smile. I thought he was just blowing off the test. Later, when I scored the tests, he scored about 5 standard deviations above our normative cut score. He actually aced the computation test, which during a conversation with the vendor at SIOP, they hand only seen a handful of times during the 50 years or so the test had been in place. To cut to the quick, I followed up with the HR Generalist (described above) about whether or not we hired this guy. I can't remember the exact response, but it was something equally as lame as, "sitting in the wrong chair". I also think she was a member of SHRM and am pretty sure her business card had SPHR on it.

Posted by Jamie at January 30, 2008 10:44 AM:

David, yeah I remember you telling me that story. Good times!


  Post your own comment:




Remember My Info?





all this copyright until the sun explodes, jamie madigan