>
a weblog covering personnel selection, assessment, and employment testing in the news

« January 2006 issue of TIP | Main | Teach the children to cheat »

Things I wish I knew about surveys

On January 17, 2006 by Jamie Madigan

Survey

I was talking to a co-worker this morning about ways to optimize responses to surveys. I use job analysis surveys from time to time, and I'm always fretting over details in the hopes that they'll help me get more responses and cleaner data. And it's easy to come up a long list of possible factors that affect all this, but by and large I'm not sure of any kind of research to back them up.

For example, I want to know how much the following affect response rates and data integrity:

  • Sending out reminders
  • Who the reminders come from (e.g., me vs. a Vice President)
  • The timing of the reminders --day of the week, how far apart, etc.
  • 5-point vs. 7-point vs. 9-point response scale for Likert-type items
  • Whether or not to use a response scale with a "neutral" option vs. forcing people to one side of the fence
  • Font sizes, color, and general design rules
  • Personalizing the survey with the respondent's name or other info
  • Offering incentives for completing the survey
  • Print vs. e-mail vs. Internet-based surveys
  • Survey length
  • Complexity/length of directions (is shorter better or worse?)
  • Randomization or items or grouping them together by scale?
  • Annual vs. monthly surveys for recurring metrics?
  • "Survey fatigue" --that is, getting too many surveys too close together
  • Putting "WON'T SOMEBODY THINK OF THE CHILDREN?" in bold across every page?

Now, I've seen anecdotal evidence for some of these questions and I think I've seen research-based discussions for others (in fact, a co-worker and I addressed the "annual vs. monthly" question in a SIOP presentation a couple years ago) but what I'd really like to see is all this information scientifically studied and put together in one place for easy reference. While it may never find its way to the "Best Sellers" rack at your local Barnes and Noble bookstore, a book like this entitled "Survey Hacks: How Little Changes Improve Your Surveys" would be quite useful.

If anyone knows of such a tome, let me know. The world needs to know.


  Existing comments:

Posted by Karen M at January 17, 2006 2:54 PM:

Jamie,
funny you should ask that as I recently had done research for a client regarding surveys.. there are some good books out there - check
http://www.advancedsurvey.com/articles/recommended_survey_books.asp

Also if you use outlook here is a great tool for generating results and creating an access table for you - http://office.microsoft.com/en-us/marketplace/EM011329651033.aspx

Also see http://www.cit.cornell.edu/atc/itsupport/surveydetails.shtml for great info

Posted by Jamie at January 17, 2006 3:14 PM:

Great post, Karen!

I've added several of those books to my wish list. The Outlook plugin looks interesting, too. I may investigate that, though we use a fairly powerful program called eListen at my current employer.

Posted by Bryan at January 17, 2006 3:58 PM:

This research article addresses many of the issues you raise:

http://jom.sagepub.com/cgi/content/abstract/24/1/97

Posted by David Morris at January 21, 2006 9:51 PM:

Jamie,

I haven't checked out the links for the above posts, but really good book to look at is Questions & Answers in Attitude Surveys by Schuman and Presser (ISBN - 0-7619-0395-3. It has some great chapters on topics like question and response order, measuring the middle position, tone of wording. The book is chock full of empirical research instead of 'best practices'. Steve had a copy on his shelf, he recommended it. I liked it so much I bought a copy myself.

Posted by Spencer Stang at January 23, 2006 5:40 PM:

As a survey designer and taker here are the answers to your Qs (n=1).

Tight deadline with no reminders until after the deadline has passed.
Higher authority, higher participation.
One reminder immediately after deadline, no more than one per week thereafter (for relationship maintenance) or one per day (for effectiveness).
Five point scale
Neutral scale is more accurate in the sense that an average rating is often neutral. But, a neutral option may hinder your ability to get people off the fence on touchy subjects. A neutral option also might require you to gather more data to get meaningful results.
Clean & readable
Personalized
Incentive--sure
e-mail or online link
Length = short
Short directions--Questions should be self explanatory.
Randomization--Group by topic unless your goal is to trick your subjects into revealing the truth.
Annual strongly preferred to monthly
Survey fatigue happens fast
It's not about children, it's about old people.


  Post your own comment:




Remember My Info?





all this copyright until the sun explodes, jamie madigan