Confusing Questionnaires

The new Disney movie Chimpanzee is out in theaters, and it got raves from CinemaScore, a market research firm that rates films based on feedback from opening night viewers. This approach ostensibly helps the studio decide how much additional money to put into advertising.

I saw a funny quote in a news article recently about the film:

On a curious note, 5 percent of CinemaScore participants said a main reason for attending the film was its "lead actor." Were they referring to the film's two lead apes? Or narrator Tim Allen? Even stranger, 1 percent listed "lead actress" as their reason for buying a ticket -- and that 1 percent gave the movie a harsh "B-" grade. Clearly those individuals were upset by the documentary's lack of actresses.

My take is this: this is a questionnaire problem, not a viewer confusion problem. Take a look at the CinemaScore questionnaire card as shown at Wikipedia. It reveals a very simple, paper-based form, the major features of which is a grade from "A" to "F" a la a student report card. From this card, I conclude the following things:

• The focus of the card is on the overall rating, suggesting that the other data will be less than perfect. This approach is appropriate for the purpose of the card but also subject to misinterpretation by uneducated interpreters. Conclusion: always be wary of the potential misinterpretation of your data once it gets out of your hands.
• The form of questionnaire and sampling technique (paper-based intercept survey) does not allow much flexibility for the interview, resulting in some strange question choices--hence the problem in the quote above about "lead actor." Conclusion: take survey results through the lens of how well the survey actually matches the customer behavior.
• The CinemaScore system purportedly does a good job of its primary purpose: predicting the box office success of films. Conclusion: don't necessarily change your market research approach because the data look skewed.

I learned this last lesson in spades when I helped to revise the Paint Consumers Research Program questionnaire a few years ago. The previous questionnaire had asked "brand purchased" as an open-ended question, resulting in some people saying they purchased Behr paint at Lowe's, where the brand is not currently available. We tried to fix this problem by prompting respondents to answer the store first and then showing only brands that were available through that store.

The new approach helped, but I only realized after we launched the survey that we failed to add a "don't know" option to both the store list and the brand list. Thus, if you chose "Lowe's" when you really shopped at Home Depot, you would not see "Behr" and potentially have some of the same confusion the original survey had. My take-away was to take care in the future not to dismiss automatically the results of a survey just because some of the results were skewed. Because sometimes the "fix" can cause new problems as well.