Confusing Questionnaires

The new Disney movie Chimpanzee is out in theaters, and it got raves from CinemaScore, a market research firm that rates films based on feedback from opening night viewers. This approach ostensibly helps the studio decide how much additional money to put into advertising.

I saw a funny quote in a news article recently about the film:

On a curious note, 5 percent of CinemaScore participants said a main reason for attending the film was its "lead actor." Were they referring to the film's two lead apes? Or narrator Tim Allen? Even stranger, 1 percent listed "lead actress" as their reason for buying a ticket -- and that 1 percent gave the movie a harsh "B-" grade. Clearly those individuals were upset by the documentary's lack of actresses.

My take is this: this is a questionnaire problem, not a viewer confusion problem. Take a look at the CinemaScore questionnaire card as shown at Wikipedia. It reveals a very simple, paper-based form, the major features of which is a grade from "A" to "F" a la a student report card. From this card, I conclude the following things:

• The focus of the card is on the overall rating, suggesting that the other data will be less than perfect. This approach is appropriate for the purpose of the card but also subject to misinterpretation by uneducated interpreters. Conclusion: always be wary of the potential misinterpretation of your data once it gets out of your hands.
• The form of questionnaire and sampling technique (paper-based intercept survey) does not allow much flexibility for the interview, resulting in some strange question choices--hence the problem in the quote above about "lead actor." Conclusion: take survey results through the lens of how well the survey actually matches the customer behavior.
• The CinemaScore system purportedly does a good job of its primary purpose: predicting the box office success of films. Conclusion: don't necessarily change your market research approach because the data look skewed.

I learned this last lesson in spades when I helped to revise the Paint Consumers Research Program questionnaire a few years ago. The previous questionnaire had asked "brand purchased" as an open-ended question, resulting in some people saying they purchased Behr paint at Lowe's, where the brand is not currently available. We tried to fix this problem by prompting respondents to answer the store first and then showing only brands that were available through that store.

The new approach helped, but I only realized after we launched the survey that we failed to add a "don't know" option to both the store list and the brand list. Thus, if you chose "Lowe's" when you really shopped at Home Depot, you would not see "Behr" and potentially have some of the same confusion the original survey had. My take-away was to take care in the future not to dismiss automatically the results of a survey just because some of the results were skewed. Because sometimes the "fix" can cause new problems as well.

No More Holiday Bonus

What does 99.6% accuracy mean in distribution centers? To the casual observer, it would mean that on average, 1 of 250 orders have errors. In market research, however, we have to look at the sample frame, or the source of the data compared with the total census of all instances. In this case, the sample frame is often customers (or other ship locations downstream of the DC) who complained or otherwise adjusted the order when it arrived.

Customers who did not complain could have been of three types: 1) customers who did not notice or otherwise care about the error; 2) customers who got the right amount of product, or 3) customers who got too much product and kept the overshipment for themselves. There might be lots of reasons for customers to keep over-shipments, including the cost of sending them back, the desire to make up for lost profits elsewhere, or even good old-fashioned five-finger discount (aka shrink). Nevertheless, the fact that these customers don't complain means that actual error rates are likely upwards of 1 in 250.

Hence the story my lead generation guy tells about a checking in on a customer who implemented Vocollect(R) Voice: his DC's downstream customers were very pleased with the improved accuracy, but they asked the DC manager, "What happened to all the extra stuff you used to send us?" The answer: the DC didn't mean to send it in the first place.

Improving accuracy means decreasing largesse for the downstream parts of the supply chain. In this case, that's a holiday bonus that isn't good for business.