The Value of Market Research

My old market research professor Anand Bodapati at UCLA Anderson School of Management used to spend a lot of time warning against doing market research without understanding the goals of the project and the business decision that the research will help to answer. This emphasis counts as one of the reasons that I consider my time in Professor Bodapati's class some of the best spent hours of my academic career.

When should a company do market research? The answer to this question ought to depend on the value of the research versus the potential profitability. Many business schools teach a traditional method of evaluating this value based on Bayes' Rule, a technique that has some support and lots of criticism, some of the criticism going back thirty years or more.

What's a marketing manager to do? I say, apply a few rules of thumb if you don't have time to do the full analysis: Do market research when...

1. ...the outcome of your business decision is truly in doubt. This rule implies avoiding market research both when the outcome is almost certain success and when the outcome is almost certain failure. See more on the "expected value of perfect information" for the reasoning behind this rule.
2. ...the magnitude of expenditure (or the cost of failure) is at least several times that of the market research (I often use a 10x multiple). This guesstimate ensures that you spend money where you can expect the greatest return from learning. The method also preserves focus on the projects that are most important, a critical factor when you need to communicate results to the wider organization. Wide communication of market research results is, in my opinion, too often shortchanged in large companies.
3. ...you have the possibility of really upsetting your customers by messing up. See my recent post for more on this one.
4. ...senior executives, salespeople, or other individuals with a lot of influence start making questionable statements that could contradict reality in strategically important ways.

And why am I thinking about this topic today? My boss reminded me about #2 today...proving that even the best of Data Geeks sometimes forget their own rules in the excitement of gaining new customer intelligence.

Check the Facts

Vice President Joe Biden recently got in trouble with the conservative media for claiming that cities like Camden, New Jersey and Flint, Michigan would see a rise in robberies, rapes and other violent crimes because they have had to cut police forces in half due to budget woes. Of course, Biden was arguing for Obama's new jobs bill, but I immediately thought of the more interesting data question.

The problem with determining the relationship between police force size and crime rates is "simultaneity bias." This term refers to the two events you wish to study tending to happen at the same time regardless of causality. In this case, governments tend to increase the police force when they notice a rise in crime or even in anticipation of this rise. Therefore, it is hard to see which comes first, the chicken (crime rates) or the egg (police force size).

One can overcome this problem by looking for individual events in which one factor changed for exogenous reasons--reasons outside the normal timing. There is an excellent summary of some of this research here:

http://www.majorcitieschiefs.org/pdf/news/more_policing_does_matter.pdf

The upshot is that cutting or raising the police force does have both an apparent short-term and an apparent long-term effect on crime. Nevertheless, the range of response varies significantly in the available research, some suggesting that a 50% drop in police force could mean only in the range of 10-15% rise in crime.

In a business context, simultaneity bias comes up quite a lot when looking in the rear-view mirror of one's business. For example, an increase in size of sales force is often accompanied by increase in marketing spend. This simultaneity confounds examining the effects of each investment. That is why I often recommend running experiments changing only one variable, or changing different variables in different territories or regions, before making such investments throughout the company.

Too bad Joe didn't find a source for his facts before he met the press. I hope he finds my blog.

When Did You Last Click Through To An Internet Ad?

Rule one for advertising: break through the clutter. That applies to B2B companies, too. Their buyers are humans, just like everyday consumers.

Rule two: differentiate from competitors by demonstrating why your solution works better. Yes, this matters for B2B companies as well as B2C brands. Perhaps even more.

That's why this terrific Verizon ad works so well. It might be the only B2B advertisement online that I have clicked on for the past year.

Netflix Did the Wrong Thing

The flood of articles alternately condemning the stupidity of Qwikster and praising the Netflix reversal missed the point entirely. Comparing Qwikster to New Coke would be like comparing eBay to garage sales circa 1985. Reed Hastings had an entirely different set of tools with which to make decisions than The Coca-Cola Company did twenty-five years ago. And he missed them. Entirely.

The first tool is online research. Netflix has some 25 million subscribers in the U.S. and Canada, which means that the marketing team could have used an Internet omnibus panel survey to complete their market research in about a week and still expect to get in the range of 300 responses (20 million estimated U.S. subscribers divided by 112 million households times 2000 respondents per panel fielding).  They would have spent perhaps $8000 for a short set of questions.

Ones I would have asked:

1. Do you currently subscribe to Netflix?
2. Do you get your movies or shows from Netflix by mail, by downloading or streaming movies from the Internet, or both?
3. If Netflix were to separate the DVDs-by-mail Website and service from the download/streaming Website and service, how much would you like or dislike this change?
4. If Netflix were to separate the DVDs-by-mail Website and service from the download/streaming Website and service, would you:
1. Subscribe to the DVDs-by-mail service
2. Subscribe to the download/streaming service
3. Subscribe to both
4. Cancel your subscription to Netflix

I'm guessing by the flood of criticism and defections that this research would have easily identified this idea as a big, fat loser.

The second tool is online media monitoring. Many of the Honomichl 50 offer this kind of analysis, such as Synovate's BlogBase. It would have been pretty straightforward to comb the Web for information on why people like Netflix these days as opposed to why they hate it. And Netflix would have found (duh) that a major reason people like the service is the one Website for both mailed DVDs and streaming.

The third tool is customer advisory panels. Many new and exciting vendors (I'll mention CrowdScience and Vovici as just two) allow discrete and relatively unobtrusive research into a company's own customer base, often the best source on which to try these ideas. It helps to have an active website with lots of customers visiting this site for this approach to be extremely successful...

Hello, Reed! You have a website Use it!

Thank you, Advertising Age, for pointing out how critical it is to listen to your customer and for reminding us why tech companies don't listen very often.  R.I.P. Steve Jobs indeed.

R.I.P. Steve Jobs (But Not Your Management Style)

I surprised myself by my strong reaction to Steve Jobs's passing. Maybe having had an Apple II as my first computer prompted the youthful nostalgia. Poking around in the Google News archives, I found this old article about Mr. Jobs returning to Apple:

http://news.cnet.com/2100-1001-256947.html

The author reflects the ambivalence about the management style of someone who was, by many accounts, a royal a******. Nevertheless, the Stanford graduation address shows a different side of Mr. Jobs:

http://www.ted.com/talks/steve_jobs_how_to_live_before_you_die.html

The TED video is worth watching in its entirety to remind each of us what passion really means. For me, it means being good to other people (unlike Steve), remaining passionate about bringing science to marketing (maybe like Steve), and trying to live a life I want rather than the life I think someone like me ought to have (definitely like Steve).