A/B Testing For Everyone

The folks over at the phenomenal Marketing Experiments Blog had yet another post about A/B testing that reminded me of some consulting work I did in the past. Often, I have found that organizations think you have to be a gigantic company to do A/B testing. The reality is that a company of any size can A/B test just about anything, sometimes to dramatic effect. And a small company can apply very sophisticated marketing analysis very inexpensively in this age of free, high-powered statistical languages.

When I worked for Strategic Energy, management believed we couldn't just send our customers a contract and re-sign them for three years of electricity usage. I said, "What's the harm in trying?" We sent a hundred customers a thank-you for letting us serve them along with a new contract for service. About 35 of them sent us back a signed contract. How much did that test cost? About $300 and half a day of work. After that experience, Strategic Energy started sending every customer under a certain size a renewal contract, saving tens of thousands in sales costs per year for those that responded.

We then sent out postcards to the remaining customers plus about 200 more asking them to contact us about their contract renewal. On one postcard, we put an existing customer photo and an inspirational message about saving their business money. On the other postcard, we put a funny beach photo and a message to the effect of, "Wouldn't you rather be spending your time on the beach than renewing an electricity contract?" We assigned customers randomly to one or the other. To our surprise, the beach one got a statistically significantly better response. Simple A/B test done. Learning learned.

I applied this kind analysis to the funding solicitation work of the Jewish Federation of Greater Pittsburgh to equally powerful effect. In this case, some simple linear regression showed that of the greatest factors influencing the size of the gift was whether the gift was given online (even when holding donor age constant). Pushing customers to the website to donate increased the size of the gifts, and some tweaking to the website itself increased gift sizes even further. All that we needed to complete this analysis was a history of donations and some basic information about the donors and when they responded.

The barrier to basic A/B testing usually lies in company culture, not in cost or capabilities. Companies need to get wired for a "learning culture" that emphasizes marketing science over gut feel. This change must emanate from the senior executive team, and they have to understand how powerful data management and analysis can be to improve marketing response rates, revenues and profits.

As analysis professionals, we need to bring these smarts to the executive team so that they can bring culture change to the rest of the company. I try to remind myself of this goal periodically when I find myself tiring of yet another explanatory meeting with the VPs. Although sometimes repetitive and tiresome, the meetings to explain what we are planning to do after we test result in the executive support necessary to internalize the learning from the testing over the long term.

Inspiration Versus Perspiration

You've heard that genius is 1% inspiration and 99% perspiration? Well, I was interested to read a few months ago an insightful article on Wells Fargo and their success in retail and commercial banking. The relatively new CEO John Stumpf states that a good strategy flawlessly executed will always win versus a brilliant strategy poorly executed.

I would personally modify that statement a bit. I believe that a good strategy enables flawless execution but does not ensure it. In other words, a good strategy is necessary but not sufficient to win.  I would say that company success is 20% strategy and 80% execution. But you can't get the 80% right without the 20%.

I advocate the concept of "employee bandwidth" in management. The executive team has only a certain amount of time in the day, so anything that distracts their focus from work critical to the future of the company will ultimately help to sink the company. Having a single strategy, with elements that are mutually reinforcing and move towards a common goal, enables everyone to use their limited bandwidth to drive towards greater customer insight and profitability.

Where does market research come in to this equation? Done properly, the market researcher stands at the vanguard of understanding customer value. When communicated properly to executive management and the company at large, the market researcher has the unique responsibility to explain how to break value compromises that customers have endured in the past.

Take Southwest Airlines as an example again. The market researcher should have explained that pleasure travellers are willing to give up many perks of flying to get a better price. They are willing to give up free food, assigned seats, flight attendants in uniform, first class seating, entertainment options, non-stop flights, but not on-time arrival. Southwest Airlines could therefore orient their "value" offering to eliminate most perks as long as turning around the plane quickly (a key to their strategy) did not result in late departures.

Most of Southwest Airlines' approach helps to ensure that they can turn planes around quickly and still achieve one of the best on-time records in the industry. Nevertheless, their strategy has been devilishly difficult to implement. In fact, Herb Kelleher repeatedly has taunted his competitors to try his approach because he knows how difficult it is.

Difficult-to-execute strategies are not bad; in fact, they are excellent. "Difficult to replicate" equals "long-term competitive advantage." The history of companies attempting to copy Southwest Airlines is filled with failures, and I can only think of one partial success (Alaskan Airlines).

The great moment for the market research professional is the moment at which the strategy has been set, and the company is desperate for more information on what the customer is or is not willing to give up to get the benefit your company now offers. If you're offering a complete ecosystem of products that work seamlessly together, is the customer willing to give up in-person service? If you're offering the same product as competitors for half the price, is the customer willing to order direct instead of going through a distributor? If you're offering unparalleled service, is the target customer willing to pay a premium price and still give up ever going into a physical store? Market research can and should be spending money to find out these secrets.

What Is Strategy?

Having just finished presentations for Vocollect's strategic planning efforts this year, I am reminded of one of my all-time favorite business articles, Michael Porter's "What Is Strategy?" (You can find a free copy here apparently.) In a nutshell, Porter argues that strategy is a set of inter-related and mutually reinforcing decisions about what to do and not to do. Companies that try to execute two strategies at once often pay a "straddling penalty" because the two strategies compete for resources and detract from each other. Take a look at the diagrams, in particular, which I have found wonderfully instructive for explaining how good strategy works.

Back when I was in the paint industry, Benjamin Moore had an excellent strategy:

1. Target the residential repainter.
2. Sell through dealers only (no home centers, no company-owned stores).
3. Market to consumers as "high-design, high-fashion, color-forward."

Each of these decisions had implications and mutually reinforcing benefits. Targeting the residential repainter meant making the paint easy to apply, high coverage, and fast to dry. In fact, Benjamin Moore's highest-end paint dries so fast that regular consumers can't even use it because they paint too slowly. The company seems to have skimped a bit on the qualities that consumers value such as ability to wash the walls without leaving marks, but residential repainters don't care about these qualities. Selling through dealers enables full support for the design aspects that consumers value. The high-design positioning justifies the higher price at the dealer as well as making consumers tend to ignore the less appealing functional qualities of the paint in favor of the design knowledge.

I often look at technologies in our industry and wonder why nobody has tried to own a strategy in RF scanning devices. They are really just sold as commodities, but the supply chain market has room for a number of potential strategies:

• The "Dell" strategy: go direct, reduce cost and inventory, become a "fast follower" on technology, customize orders prior to shipment, compete on price.
• The "Apple" strategy: sell through proprietary network of high-value consultants, work really hard on the user interface, make the devices intuitive for users, solve problems in the supply chain with easy-to-download "apps" for the devices, provide a robust ecosystem of matching applications (printers, device management software, etc.), limit inter-operability to require the entire ecosystem and provide benefits for using all one vendor's ecosystem.
• The "Sub-Zero" strategy: go super up-market, provide extraordinary value for a premium price, sell through a network of carefully chosen partners with only the best knowledge base, focus only on niche applications that cannot take a commodity scanner.

I would be hard-pressed to explain any RF scanner's strategy in this space although many have a positioning. Consequently, none of the vendors seem to be making lots of money with RF scanning. Fortunately, I believe that our parent company Intermec has some of the best technology and thinking in the field and has the ability to create a break-out strategy that could win some serious market share and profits. Maybe I'll send them the article.

The Research Nobody Hears

I just returned from Gartner's 2012 supply chain conference, one of 63 of these kinds of events they will hold this year alone. It's a three-day slog that extends to five days when I added a Sunday night meeting with the analysts (requiring me to leave Pittsburgh at 7:00 a.m. Sunday morning) and a Thursday return flight to avoid the red-eye. Highlights: I met lots of interesting executives, talked to some potential prospects and danced to the hotel band's excellent renditions of Lady Gaga and the Black-Eyed Peas.

As for the meeting content itself... I was less than impressed. I averaged about two insights per hour, which does not nearly meet my goal number. Even the session on results of the "2011 End-User Wants and Needs Survey" underwhelmed me. Come on, Gartner analysts, you have an entire study with enough sample size to do interesting cross-tabs, and all you can produce for me is four data slides? Ultimately, the lack of data, combined with the persistent "talk at me" format of which Vocollect is as guilty as Gartner, got me to thinking about market research.

I have produced many studies with great insights over my ten years in market research and five years in consulting. Sometimes, I couldn't get people together to discuss the results. Other times, I couldn't interest the key decision-makers in implementing the clear recommendations from the research even when those same decision-makers requested the study in the first place. Still other times, someone else presented the results of the research in a way that obscured some significant findings.

In each of these cases, we were almost better off having not done the research at all because market research that does not get communicated or used is essentially just empty spending. The results stay hidden, so the company ends up failing to act based on customer wants and needs as opposed to acting on instinct and experience. Keeping those failures in mind (my own and others'), I humbly offer several suggestions for getting research noticed:

1. Schedule several times to present the research to different audiences. One research presentation is not enough. Finance wants different insight from sales who want different insight from marketing. Cut the data several different ways and show it to each audience with cross-tabs that answer their particular questions.
2. Bring up the research at every opportunity. Schedule meetings. Send out blast e-mails. Write internal white papers. Invite yourself to meetings on the same topic as the research even when you might not be welcome. Talk about the research at staff meetings. Promise custom cuts of the data based on hallway discussions. Be unavoidable until the insights stick.
3. Help to operationalize the findings. Although it will be more difficult to present a case study or working group format, you will see much better implementation. Turn the research into a project in your meeting (e.g., "Let's break into three groups and brainstorm changes we're planning to make based on the findings. In 20 minutes, we'll regroup and discuss which ones we can implement right away and which ones we can change over the long term.")
4. Revisit key questions. Did the research raise additional questions you can answer through informal qualitative work? Go do it yourself, use your staff or salesforce or current customers to do so, or leverage other past research to answer these questions. This strategy has the added benefit of gaining buy-in for future spending if necessary.
5. Pester the senior executives. The division VP missed your briefing? Schedule one with her yourself as a one-on-one. If she's game, I guarantee that 30 minutes will go further to result in operationalizing results than many of the suggestions above.
6. Archive the work in a public place. Make the research easy to refer back to, especially by the people who will be doing the groundwork later. If you can get them to attribute the reasoning by quoting or sourcing your study, it will be even better.

Speaking of which, I think I owe our new SVP and General Manager a call. I don't think he's seen my past year of work yet.

Steve Jobs Revisited

Okay, I might have to take back some my earlier post on Steve Jobs. Although he is still reportedly a poor people manager, this story about Apple's legendary customer service does change my view. Look in particular at the very end of the article.  Are you listening, Andrew Mason?

R.I.P. Steve Jobs (But Not Your Management Style)

I surprised myself by my strong reaction to Steve Jobs's passing. Maybe having had an Apple II as my first computer prompted the youthful nostalgia. Poking around in the Google News archives, I found this old article about Mr. Jobs returning to Apple:

http://news.cnet.com/2100-1001-256947.html

The author reflects the ambivalence about the management style of someone who was, by many accounts, a royal a******. Nevertheless, the Stanford graduation address shows a different side of Mr. Jobs:

http://www.ted.com/talks/steve_jobs_how_to_live_before_you_die.html

The TED video is worth watching in its entirety to remind each of us what passion really means. For me, it means being good to other people (unlike Steve), remaining passionate about bringing science to marketing (maybe like Steve), and trying to live a life I want rather than the life I think someone like me ought to have (definitely like Steve).