A/B Testing For Everyone

The folks over at the phenomenal Marketing Experiments Blog had yet another post about A/B testing that reminded me of some consulting work I did in the past. Often, I have found that organizations think you have to be a gigantic company to do A/B testing. The reality is that a company of any size can A/B test just about anything, sometimes to dramatic effect. And a small company can apply very sophisticated marketing analysis very inexpensively in this age of free, high-powered statistical languages.

When I worked for Strategic Energy, management believed we couldn't just send our customers a contract and re-sign them for three years of electricity usage. I said, "What's the harm in trying?" We sent a hundred customers a thank-you for letting us serve them along with a new contract for service. About 35 of them sent us back a signed contract. How much did that test cost? About $300 and half a day of work. After that experience, Strategic Energy started sending every customer under a certain size a renewal contract, saving tens of thousands in sales costs per year for those that responded.

We then sent out postcards to the remaining customers plus about 200 more asking them to contact us about their contract renewal. On one postcard, we put an existing customer photo and an inspirational message about saving their business money. On the other postcard, we put a funny beach photo and a message to the effect of, "Wouldn't you rather be spending your time on the beach than renewing an electricity contract?" We assigned customers randomly to one or the other. To our surprise, the beach one got a statistically significantly better response. Simple A/B test done. Learning learned.

I applied this kind analysis to the funding solicitation work of the Jewish Federation of Greater Pittsburgh to equally powerful effect. In this case, some simple linear regression showed that of the greatest factors influencing the size of the gift was whether the gift was given online (even when holding donor age constant). Pushing customers to the website to donate increased the size of the gifts, and some tweaking to the website itself increased gift sizes even further. All that we needed to complete this analysis was a history of donations and some basic information about the donors and when they responded.

The barrier to basic A/B testing usually lies in company culture, not in cost or capabilities. Companies need to get wired for a "learning culture" that emphasizes marketing science over gut feel. This change must emanate from the senior executive team, and they have to understand how powerful data management and analysis can be to improve marketing response rates, revenues and profits.

As analysis professionals, we need to bring these smarts to the executive team so that they can bring culture change to the rest of the company. I try to remind myself of this goal periodically when I find myself tiring of yet another explanatory meeting with the VPs. Although sometimes repetitive and tiresome, the meetings to explain what we are planning to do after we test result in the executive support necessary to internalize the learning from the testing over the long term.

Revenge of the Data

I have been following with relish the story about Elon Musk's war with the New York Times over a negative review of their Tesla S electric vehicle. What I loved about Musk's retort to the New York Times story is how Tesla Motors managed to use device data to refute the story. The war ends up being a debate between the hard data in the device and the reporter's notes.

I take away three conclusions from this episode:

Reporter's vehicle log as annotated by an angry Elon Musk

1. Data is power. Companies that think about information they could or already do have available and then exploit that data create sustainable competitive advantage through their installed base. I learned this first hand at PPG Industries, where we were able to use tint machine data to examine paint color usage by region. I only wish that PPG had been more open to using the color chip rack to collect data (discretely and privately) about user interactions with the display. At Vocollect, we are exploring a wide variety of ways to aggregate data from our wearable devices to enhance the user experience.
2. Companies should get data in the hands of users. I see this war in part as a problem stemming from the New York Times reporter's inability to get all of the information he could have had available...information Tesla then gathered from the log files. Perhaps giving this information to the user in the first place in a snazzy interface could have prevented some of the reporter's frustrations. Heck, a number of device manufacturers give the data to users in an API and end up getting cool tools for their other users for free, created essentially by fans of the brand.
3. Don't get into a pissing match in public. Elon Musk, known for his huge ego, could have been more diplomatic and apologetic to the reporter. Abusing customers or potential customers does not position the brand for success. And essentially accusing a reporter at one of the most prestigious papers in the world of journalistic fraud qualifies as abusing potential customers in my book. Tesla Motors might have gotten a better response from the Times and better publicity by working with them to diagnose what had happened rather than by working against them. Unless you believe that all publicity is good publicity, in which case Musk did the right thing by making this story huge.\

I will anxiously await the innovations from car companies and any other company that has direct interaction with the actual consumer, enabling us to understand and improve our own behavior. As you know if you read this blog regularly, I hope to be at the forefront of that user empowerment given my sincere belief in the power of some Major Data Geekitude to improve our collective future.