 |
||||||||
Organizational Effectiveness Through Feedback Management
|
||||||||
>> The Importance of Measuring Importance
|
||||||||
Review our
Read our latest article on
and learn the dramatic shortcomings of a pooled service delivery design -- and horrific Servcie Recovery.
|
|
Learn how to conduct a survey project from questionnaire design
to survey administration
to data analysis.
 March 2-4, 2010 DubaiMarch 15-19, SingaporeMarch 23-24, Kuala LumpurApril 21-23 2010, Wash DCJune 21-23, 2010, Chicago~~~
TBA Summer/Fall 2010~~~
Learn how to design a
~~~
View a short online presentation
summarizing the concept of surveying and outlining our courses.
|
Provides targeted assistance on your survey questionnaire and project while being mentored on good survey practices.
|
 Contract Holder GS-02F-0126U
|
Need more than
targeted survey consulting?
Great Brook specializes in
the design and execution of
Custom Survey Research Projects to meet
your individual needs.
|
Get an experti's review of the feedback management processes in your organization for customers or other stakeholder groups --
or get a deep dive assessment of your survey program.
|
|
Have a question about surveying you'd like to have answered in a future Insight article?
Please contact us.
|
Self-help guidance for your survey project
 |
Private On-site Survey Workshops are available at your company. This may prove more cost effective and deliver immediate value to your survey program as we build your survey instrument in class.
Please contact us for details.
|
For a comprehensive list
of survey automation tools
for your survey program.
|
-- Correctly -- on Customer Surveys
Recently, I heard a colleague present the results of a research survey. One area of the survey addressed the importance of various factors in determining marketplace actions. At this juncture he said, “Everything looks important.” It was apparently true. For each of the three factors questioned, over 50% of the respondents used the top box score of 5 (on a 1 to 5 scale), indicating Extremely Important. More than 80% of the respondents gave a 4 or 5 rating.
So what did we learn from this? Not much about the criticality of different factors, but we can learn an important lesson about survey instrument design practices.
This article is from our Service Insights Newsletter.
|
Most any customer research is likely to touch on the topic of measuring the importance or value of some set of factors. Perhaps the focus is the importance of aspects of services or products to the overall business relationship. Perhaps it's the importance of alternative future actions the company could take. Our real interest lies in the importance of each factor relative to the other factors. This is fundamental Pareto Analysis: separate the critical few from the trivial many. If everything is important, then where do you focus your energies?
The challenge lies in how the question is asked. Most survey designers will follow the practice shown above -- asking respondents to evaluate each factor individually on an interval rating scale. Few factors will get low- or mid-level ratings. For example, what if you were surveyed about the importance of various aspects of a recent flight you took. What wouldn't get a very high rating? Flight schedules? Price? Security processes? Speed of check-in? Handling of checked baggage? Seat comfort? On-time arrival? Flight attendant actions? Meal quality? (Well, okay… airlines seldom serve meals…) Perhaps the movie selection would get low importance scores. Yet, one or two of those items is truly more important to you if you were forced to perform an explicit trade-off analysis
So what's the solution? Six research approaches can be used to garner importance from respondents. Here's a quick review of each approach.
Interval Rating Scales. This is the more typical approach just described. Respondents are presented with a rating scale and asked to rate the importance of items. (It's called an interval rating scale since the distance between each adjacent pair of points on the scale cognitively must be equal intervals.) Since other questions on the survey likely will use an interval scale, it's natural to apply such a scale to importance questions as well. I recently heard a discussion among some people who conduct surveys professionally. One person said, "We are currently measure importance and performance - but we are having trouble getting people to answer the importance aspect - they say that it is all important!" The result of this approach is that the survey generates unactionable data. Everything is important -- so nothing is important.
Forced Rankings. An alternative is to ask the respondent to rank order a set of items. For example, six factors might be presented and the respondent is asked to place a “1” next to the most important factor, a “2” next to the second most important factor, and so on. Sounds great. We force the respondent to think of the factors as a set and rank them. But there are shortcomings to this approach.
First, we'd be tempted to take an average of the ranking scores respondents assigned to each factor. However, the question did not generate interval data; it generated ordinal data. That is, even spacing does not exist between each pair of ranked items. (For example, some respondents may consider the items ranked 1 and 2 as almost equal, but the item ranked 3 is a distant third.) You can take an average, but it would be statistically improper and could be misleading. Your analysis is really limited to cumulative frequency distributions for each factor. For example, “60% of respondents ranked Item A in the top two rating choices.”
Second, and more importantly, forced ranking questions are heavily prone to respondent error. Some people may interpret it as a rating question. Many will use the top and bottom rankings more than once. Others will rank the top and bottom items but skip the middle ones. How can you stop the errors? On a telephone survey, you could train the interviewers to prompt the respondent for full and correct answers. On a web-form survey, you could provide error feedback and not let the respondent move through the survey without completing the question correctly. These practices are as likely to annoy the respondent as to get real answers. Many respondents will quit the survey. You could also “correct” the errors - that's all you can do on a hardcopy survey. But what are the correct answers? (It's like interpreting a dimple on a chad.) You will be introducing error into the survey data because they've become your responses, not the respondents'.
Third, you can only ask the respondent to rank order a limited set of items. Ranking even six items is asking a lot of the respondent.
Multiple-Response, Multiple-Choice Questions. One viable solution is to seemingly step back in the sophistication of the question structure by using a multiple-response, multiple-choice question format. Ask the respondent to select from a list of items the 3 (or 2 or 4) most important items to them. Your analysis will then be the percent of respondents who selected each item in their top 3. The respondent burden - the amount of work you're asking the respondent to do - is much less, and the results are still meaningful. While some respondents won't check exactly the number specified, this is a minor weakness to this approach.
The number of items you should ask people to check is driven in good part by the number of items from which they have to choose. Two or three is a reasonable number of items, but if you have one item that you know everyone is likely to select, then you might want to ask for an additional choice. For example, price would likely be a top-three choice for everyone regarding factors that affect a purchase decision. True, you could not ask price, but people would write it in.
To enhance your data, you can also pose a follow-up question asking the respondent which of the three choices they just checked would be their number one choice. Then, you could pose a second follow-up question about the remaining two choices. Some web-form survey software will perform a threaded branch in which the selections from the first question are carried forward to the subsequent questions. In essence, you're asking respondents to rank order their top three choices - without presenting a set of instructions that would likely confuse many people.
Fixed-Sum Questions. A question format that combines the interval scale with the forced ranking is the fixed-sum or fixed-allocation question format. Here, you present the respondent with a set of items and ask them to allocate 100 points across the items. The respondent has to make a trade-off. More points for one selection means fewer points for another. The depth of thinking required means this question format is high in respondent burden, but there's real value in that thinking.
A key decision is how many items to present. Four, five or ten are optimal since they divide evenly into 100. Some data cleansing will be necessary since some responses won't total to 100 - it helps to have software that provides a running total to the respondent. This cleansing is likely worth the very rich data the format can generate.
Regression Analysis. One way to measure importance is to not ask it at all! Instead, importance can be derived statistically from the data set. Consider the scenario where you have questions measuring the satisfaction with various aspects of products and/or services and you want to know how important each aspect is to overall satisfaction. Include a summary question measuring overall satisfaction, which you probably would anyway, and skip any questions about importance. Through regression analysis, you can determine which factors contributed most to the overall satisfaction.
Conjoint analysis. A final method involves a more complex statistical technique, conjoint analysis. This technique is particularly useful in constructing product offerings where various features are combined into bundles, and the researcher wants to know how important each factor is in driving purchase decisions. Conjoint analysis requires a special research survey where the respondent is presented with pairs of factors asking the relative importance of one over the other. Subsequently, the respondent is presented with sets of product features based upon the previous responses and asked the likelihood of purchasing each feature set. The findings show the relative importance of each feature, and the tool allows posing many “what if” scenarios.
In conclusion, measuring importance doesn't have to lead to frustration with useless data. Think about these alternatives when constructing the research program and the survey instrument, and you can generate useful, meaningful data for business decisions.
-- Fred Van Bennekom, Dr.B.A., Principal Great Brook Consulting
Please contact us with feedback on this article or other topics that you might find interesting.
Click here to add your name to our Service Insights newsletter which highlights newly published articles.