Poll Watchers Beware

Every presidential election year, I find myself re-addicted to an awesome source of polling data, pollingreport.com. These guys aggregate the raw results of various independent polls and post them in a mostly unexpurgated format. I only wish I could do cross-tabs to break down the results further (e.g., by number of Democrats versus Republicans, age, sex, income, and so forth). Frankly, I find the raw data much more enlightening than much of the terrible commentary. [One notable exception to the usual polling pablum was today's excellent Dianne Rehm show with two experts breaking down the polling into the necessary detail.]

Particularly telling is the number of people who are "unsure" or "refused" as reported in some of these polls. The numbers are as high as 8% in some polls, suggesting that a lot of people are either still undecided, are dedicated to the old-fashioned privacy policy about politics, or are just sick of being asked. Nevertheless, one sees that Obama has quite a lead in a number of these polls when voters are given the option to be unsure.

I often find that business executives want to ignore the "don't know" responses in survey data. I believe they think the results are somehow less meaningful if a lot of respondents don't know the answers. On the contrary, I think executives can learn a lot when people are given the "don't know" option.

For example, when I was on the Paint Consumers Research Program board, we changed the survey to allow respondents to say "don't know" when asked what price they paid for paint. Not only did we get much more accurate results, we discovered that almost half of respondents don't know what they paid, even when the purchase was a month ago or less. From this, I learned that price is a lot less important than I think most paint industry executives think it is. In fact, I believe that price point (low, middle or high in the store's assortment) is probably much more critical in paint buyers' decisions than actual real price. This effect could explain in part why consumers are willing to pay $50 per gallon at Sherwin-Williams when they can get decent paint at $35 per gallon at Lowe's or Home Depot.

Some of the most important decisions in new product development fall to market research interpretation, so I believe everyone involved needs to take a closer look at the results. Surprisingly, for example, the products most likely to succeed are often the products with the most positive responses and the most negative responses. When respondents rate new product ideas, the lack of a strong visceral reaction usually indicates disinterest whereas a strong negative reaction can mean that they have a real interest in the product but are not willing to buy it themselves. A number of market research startups have popped up recently to capitalize on this idea by having respondents design products "for other people" instead of making decisions with themselves in mind.

Perhaps this could be good news for Mitt Romney, whose negative ratings have been going through the roof lately. But not if you subscribe to the idea that real money markets can predict presidential elections. If that is true, our next four years will be Obama's second term.

Confusing Questionnaires

The new Disney movie Chimpanzee is out in theaters, and it got raves from CinemaScore, a market research firm that rates films based on feedback from opening night viewers. This approach ostensibly helps the studio decide how much additional money to put into advertising.

I saw a funny quote in a news article recently about the film:

On a curious note, 5 percent of CinemaScore participants said a main reason for attending the film was its "lead actor." Were they referring to the film's two lead apes? Or narrator Tim Allen? Even stranger, 1 percent listed "lead actress" as their reason for buying a ticket -- and that 1 percent gave the movie a harsh "B-" grade. Clearly those individuals were upset by the documentary's lack of actresses.

My take is this: this is a questionnaire problem, not a viewer confusion problem. Take a look at the CinemaScore questionnaire card as shown at Wikipedia. It reveals a very simple, paper-based form, the major features of which is a grade from "A" to "F" a la a student report card. From this card, I conclude the following things:

• The focus of the card is on the overall rating, suggesting that the other data will be less than perfect. This approach is appropriate for the purpose of the card but also subject to misinterpretation by uneducated interpreters. Conclusion: always be wary of the potential misinterpretation of your data once it gets out of your hands.
• The form of questionnaire and sampling technique (paper-based intercept survey) does not allow much flexibility for the interview, resulting in some strange question choices--hence the problem in the quote above about "lead actor." Conclusion: take survey results through the lens of how well the survey actually matches the customer behavior.
• The CinemaScore system purportedly does a good job of its primary purpose: predicting the box office success of films. Conclusion: don't necessarily change your market research approach because the data look skewed.

I learned this last lesson in spades when I helped to revise the Paint Consumers Research Program questionnaire a few years ago. The previous questionnaire had asked "brand purchased" as an open-ended question, resulting in some people saying they purchased Behr paint at Lowe's, where the brand is not currently available. We tried to fix this problem by prompting respondents to answer the store first and then showing only brands that were available through that store.

The new approach helped, but I only realized after we launched the survey that we failed to add a "don't know" option to both the store list and the brand list. Thus, if you chose "Lowe's" when you really shopped at Home Depot, you would not see "Behr" and potentially have some of the same confusion the original survey had. My take-away was to take care in the future not to dismiss automatically the results of a survey just because some of the results were skewed. Because sometimes the "fix" can cause new problems as well.

Beware Vendor Metrics

I was reading about the end of the TV show House earlier today, and I saw this little tidbit in the article:

House‘s current eighth season ratings have remained solid, particularly for a drama airing at 8 p.m. The Monday night show averaged 9.8 million viewers and a 3.9 rating in the adult demo this season through early January when including seven days of DVR playback.

Since when should advertisers consider DVR playback? The DVR portion of the of audience adds only 16% to raw in-time viewing Gross Rating Points (GRP) according to this Nielsen study of DVR usage (as quoted in the New York Times). Let's say for the sake of argument that this particular show, like others, has 40% of the audience using a DVR. If that's the case, the real viewership was:

9.8 million * (1 - 40%) = 5.9 million * (1 + 16%) = 6.8 million

If (as an advertising buyer) you based your effective cost per thousand viewers (CPM) on the 9.8 million, you were over-paying by over 44%.

The misrepresentation probably stems from the network's presentation of their own overblown statistics. I have learned over the years to be highly skeptical of any vendor's own statistics, and in my own work for Vocollect, Inc. I try hard to provide our own customers an independent validation of statistics we quote on our truly superior products. The article mentioned above offers just one more reminder to smart market research analysts and marketing data consumers: examine the sources and rely on your own brain when using external data.

No More Holiday Bonus

What does 99.6% accuracy mean in distribution centers? To the casual observer, it would mean that on average, 1 of 250 orders have errors. In market research, however, we have to look at the sample frame, or the source of the data compared with the total census of all instances. In this case, the sample frame is often customers (or other ship locations downstream of the DC) who complained or otherwise adjusted the order when it arrived.

Customers who did not complain could have been of three types: 1) customers who did not notice or otherwise care about the error; 2) customers who got the right amount of product, or 3) customers who got too much product and kept the overshipment for themselves. There might be lots of reasons for customers to keep over-shipments, including the cost of sending them back, the desire to make up for lost profits elsewhere, or even good old-fashioned five-finger discount (aka shrink). Nevertheless, the fact that these customers don't complain means that actual error rates are likely upwards of 1 in 250.

Hence the story my lead generation guy tells about a checking in on a customer who implemented Vocollect(R) Voice: his DC's downstream customers were very pleased with the improved accuracy, but they asked the DC manager, "What happened to all the extra stuff you used to send us?" The answer: the DC didn't mean to send it in the first place.

Improving accuracy means decreasing largesse for the downstream parts of the supply chain. In this case, that's a holiday bonus that isn't good for business.