You would probably think that mistakes in survey design would be made by small companies with limited resources and knowledge. Yet, some of the best examples of bad survey design practice I find in big companies. In this article, I'll illustrate the mistakes made in the IVR survey to measure Sears customer satisfaction. (Home Depot's store visit survey is another glaring example of bad survey design.) Read my related article regarding to learn the strengths and weaknesses of IVR survey administration method.
In a nutshell, the Sears IVR customer satisfaction survey:
Didn't allow me, the customer, to fully express my feelings about my entire transaction with Sears
Forced me to answer irrelevant (to me) survey questions
Force fit a suvey question to a scale to the point where the data generated are not interpretable (that is, they generate garbage data)
Asked me for comments -- and then ignored me.
Most importantly, lead to greater customer dissatisfaction and destroyed any customer loyalty I had to Sears
The last bullet should startle you. Customer surveys are typically meant to provide customer measurements to increase customer satisfaction. So, how could a survey program design actually lead to greater customer dissatisfaction? The answer lies in a poorly conceived survey project and program design that leads the customer to come away with a more negative attitude toward the company in addition to generating survey data of questionable value.
Before I explain all of these survey design problems in detail and how to avoid them -- hint: the answer lies in having a good customer survey program design -- some background on my interaction with Sears customer service is needed.
Soon after the technician's visit, I received an automated phone message asking me to take an IVR survey. I should have transcribed the survey, but I can do it good justice. The 6 or 7 question survey -- it was one question longer than stated in the introduction -- posed all its questions on a satisfaction scale, where 1 represented Very Dissatisfied and 5 represented Very Satisfied. All of the questions pertained to the technician's visit to my home, which, I'll argue, was a glaring mistake. But the biggest mistake was what Sears did with my survey data.
What Lessons Can We Learn from the Sears IVR Survey?
Keep the Survey Design Focused. I teach in my Survey Design Workshop to keep a tight focus on the survey contents and not let other departments meddle their way into the survey design process. This I believe happened at Sears. The first 6 questions were about the technician's performance, but the last question, asked about my “satisfaction with the technician making me aware of Sears' products and services.” That's not something about which I give two hoots, and I suspect the operational folks in Sears service don't either. But the marketing folks do. Clearly, the technician is tasked to promote Sears products, which is one of those strange -- and inappropriate roles -- for service technicians. Survey focus is achieved by having a good Statement of Purpose or Research Objectives -- and sticking to it.
Word Questions to the Scale. The question mentioned immediately above also displays a classic problem in survey question design: wording a question to fit the scale. Look at the question. It was posed on a 1 to 5 satisfaction scale. I had no idea how to respond. The technician didn't make me aware of any Sears products. I didn't know he was supposed to, and I was glad he didn't. So, does that mean I should have scored it a 5 indicating Very Satisfied with the technician, but not because he nicely made me aware of Sears products but because he didn't waste my time with a sales pitch that I would at best find annoying? Or should I score it a 1 indicating Very Dissatisfied because... well, I don't know why I would. But I am sure the marketing folks who pushed for this question would want me to score it a 1. I didn't. I did the cop-out, middle-of-the-road, 3. Regardless, the marketing folks at Sears are interpreting the results from this survey question that is simply generating garbage data.
Develop Comprehensive Response Options. That last question really needed a Not Applicable option. I tried just not answering, but the system would not let me do that. So, as said, I entered a 3. I had a sense I would soon get the opportunity to answer an open-ended question, so I had to get beyond this question.
Develop Comprehensive Question Set. While we want to keep any survey, but especially an IVR transactional survey, short and sweet, you still need to pose all the needed questions. Usually, my criticism of surveys is that they go too far a field. The Sears survey was too narrow. The technician gave me factually incomplete information about how to get my replacement heater, and I was charged improperly. Those critical flaws in the transaction went unaddressed in the survey questions. Trust me; those factors have made the deep impressions on me about Sears on-site service.
Think Beyond the Interaction to the Transaction. A service transaction is a chain of interactions, and the weak link needs to be identified. The Sears survey only asked me about my experience with the technician. My transaction with Sears went well beyond the technician's visit, including my interaction with the service center and the store. No one ever asked me about those interactions, and thus did not allow me to express my full feelings about my experience. (You're probably thinking, “but wasn't there an open-ended comment question?” Hold that thought!) Yet, I'm sure those who designed this awful survey think it's great. I, as the customer, came away very frustrated since I had had a poor experience and did not get to say my piece.
Notice how this issue relates to the need for a good Statement of Purpose. While the typical customer (survey respondent) won't think in these terms, they will know if a survey is comprehensive and well designed. The Sears IVR survey wasn't. Why? My guess is that Sears has very stovepipe functional groups that do not interact.
Need for a Service Recovery Program. The above mistakes are bad but not egregious. The most disastrous error in the Sears survey is what they did NOT do with my data. The last question presented the opportunity to make a free-form comment about my experience. I was waiting for that opportunity. I even developed notes to be sure I covered everything. I listed the litany of mistakes that Sears made. You would think that the person who analyzes that textual data would flag me for a follow-up call. A well-run company with a well-designed survey program would have done that. Sears did not. They ignored my cry for help, making me angrier than I already was. Yes, the survey -- designed to measure customer satisfaction and identify needed operational improvements -- actually made the situation worse!
Need for an Integrated Feedback Management Strategy. Even if someone had read and responded to my comments, it might not have mattered. I recounted in the background on my Sears transaction that Sears headquarters basically told me to get lost. Sears sees no ethical or legal obligation to honor the verbal contract established by its service agents. This is a pretty startling indictment of the strategic role of customer service at Sears. But why does Sears go to the bother and expense of having this one feedback survey that sits like an isolated island disconnected from its potential business value?
Will I ever shop at Sears again? You know the answer. That $69 savings is quite the Pyrrhic victory for Sears.
-- Fred Van Bennekom, Dr.B.A., Principal, Great Brook
Please contact us with feedback on this article or other topics that you might find interesting.
Here's some of the emails we've received from people thinking we're Sears... unedited but anonymous
sears tool #9 43664 rachet wrench reversable angle 13mmx14mm, "NO ONE" can find this tool!!.. please call
hello my name is XXX we have been along time customer of sears but these days i really wonder if you really do think about taking care of your customers and i dont mean just a single store i have been in contact with the store and every single phone number i can possibly find on or about sears you people just seem to want to give lip service and nothing else i guess when sears changed owners you really have a problem helping customers with their problems or you just run a big scam how your customers are really number one well im here to tell you the word of mouth is still a time tested and very effective way of comunicating even in this day of age and believe me i know a lot of people common and business men and women alike an old man once told a car dealer rather large that did a friend of his wrong he said to them if you do people right i will tell everyone i know and if you do people wrong i will tell everyone i know and he did the business went bank rupt after on
I was in the sdtore shopping and on my slip is says to go to your web site and enter for a 4,000 sears gift card, i did not liket he site all it did was ask me question about college and buyin other things, this is faluse addvertising. now all these people are calling me. what about the gift cert you said about, never did get any information on it. I am a sears customer not all this other junk.
Today, I called the number for Sears repair and service in my local phone book. The automatic answering machine did NOT give me any option to speak with the people at the service center in Melrose Park, where I left my vacuum cleaner for repairs a week ago. And there is no contact number on the receipt I received when I dropped off the appliance. So can you help me reach this repair center???? The woman to whom I was transferred merely hung up on me when I told her what I need.
Great Brook Home | Products & Services | Events | Clients | Partners | Resources | News | About Great Brook | Contact Us | Site Search & Map
Customer Satisfaction Surveys - Survey Workshops - Survey Consultant - Product Supportability Planning - Survey Software