Why Doesn't Facebook Know My Friends?

I went to Facebook today to add someone, and the list of people who I might know was shockingly terrible. I knew about four of the first ~50 suggested, and they were all friends with my wife.

Why is this analysis so difficult? I suspect it has to do with one of three problems:

1. Algorithm development: Turns out it is harder than you would suspect to come up with a good "lift" formula based on disparate data. To me, this problem seems closest to market basket analysis, the classic formulation of which is "people who buy [item] are also likely to buy [item]" such as bananas and milk. Facebook may just not have spent as much effort on this area as, say Netflix did.
2. Computing power: Market basket analysis is really computationally intensive, so it sucks up a ton of processing time. Perhaps Facebook has decided it is not worthwhile to spend the money on this element of its business? I am a bit surprised, as it would seem that Facebook users are only as happy as the involvement and extent of their network, but the cost answer is possible.
3. Lack of knowledge: This is another way to say "stupidity." Actually, that's a bit unfair. I believe that many data analytics groups in companies are approaching the problem wrong. They are trying to find "people closest in the network to this person" rather than "people with whom this person might be most interested in connecting." For example, I see a lot of people on the suggested list who are friends with my wife. That's reasonable, but an even smarter way to approach the problem would be to see that I have lots of connections to college classmates and suggest more of them who are closer to my network or to notice that I just switched jobs and suggest people at my new workplace.

Some of these algorithm suggestions in #3 would involve "tweaking" the algorithm, which data scientists sometimes object to doing. They want purity so that every special group does not need his or her own algorithm. That's where the marketing folks come in.

The Product Manager's job ought to be thinking deeply about what makes people happy on Facebook and then challenging the data scientists to move towards that goal. If more than one algorithm is required (e.g., one for "currently working" and one for "not employed"), the calculation ought to be about cost versus long-term benefit. Those long-term benefits include customer loyalty, a calculation I have discussed previously and which lately is coming into doubt for me and Facebook. Sorry, Mark Zuckerberg.

Find Your Co-Data

"Co-data" is my term for data that goes well and augments your core data set. I particularly like that The Weather Channel has found consumer behavior data predicting what you will buy depending on the weather. You don't need The Weather Channel's giant data set to find this data set. It could be as easy as looking at your fellow local businesses' websites.

Let's say you're a cab driver. You want to minimize wait times and maximize distance driven. How about finding out when colleges in your area start up again? Or checking out when a particular bar closes? Or finding out the time a particular show (preferably one with drunken attendees interesting in safe-cabbing it home) gets out?

I tried this simple method when I worked at PPG Industries. Of course, our sales of exterior paint increased when the weather got pleasant. Pulling free data off the NOAA Climate Data Center enabled me to do some rudimentary comparisons between our past sales by region and temperature. I found that people start painting more at about 50 degrees F, and that over about 84 degrees F the amount they paint starts to drop off (too hot out).

Using such basic data and simple correlation, I was able to optimize the load-in for our largest retailer's stores so that we had enough exterior paint early in the season... but not too early. I also found that using last year's sales to predict when we should ship this year was a lousy measure; better to use the average over the past three years and then build back two weeks for safety.

At Vocollect, we're discovering lots of cool ways to use the information we have to make our workers' lives easier. We're helping by giving simple suggestions such as prompting the user to access a feature when we notice the feature could be used to solve a problem we deduce the worker is having. The next phase will be to combine this user data with simple information we have from other sources to help suggest, for example, how two coworkers can avoid each other in a distribution center aisle to ease congestion delays.

All this work goes back to my feeling about big data: you don't need it if you have plain old "data" that you're not using in the first place.

A/B Testing For Everyone

The folks over at the phenomenal Marketing Experiments Blog had yet another post about A/B testing that reminded me of some consulting work I did in the past. Often, I have found that organizations think you have to be a gigantic company to do A/B testing. The reality is that a company of any size can A/B test just about anything, sometimes to dramatic effect. And a small company can apply very sophisticated marketing analysis very inexpensively in this age of free, high-powered statistical languages.

When I worked for Strategic Energy, management believed we couldn't just send our customers a contract and re-sign them for three years of electricity usage. I said, "What's the harm in trying?" We sent a hundred customers a thank-you for letting us serve them along with a new contract for service. About 35 of them sent us back a signed contract. How much did that test cost? About $300 and half a day of work. After that experience, Strategic Energy started sending every customer under a certain size a renewal contract, saving tens of thousands in sales costs per year for those that responded.

We then sent out postcards to the remaining customers plus about 200 more asking them to contact us about their contract renewal. On one postcard, we put an existing customer photo and an inspirational message about saving their business money. On the other postcard, we put a funny beach photo and a message to the effect of, "Wouldn't you rather be spending your time on the beach than renewing an electricity contract?" We assigned customers randomly to one or the other. To our surprise, the beach one got a statistically significantly better response. Simple A/B test done. Learning learned.

I applied this kind analysis to the funding solicitation work of the Jewish Federation of Greater Pittsburgh to equally powerful effect. In this case, some simple linear regression showed that of the greatest factors influencing the size of the gift was whether the gift was given online (even when holding donor age constant). Pushing customers to the website to donate increased the size of the gifts, and some tweaking to the website itself increased gift sizes even further. All that we needed to complete this analysis was a history of donations and some basic information about the donors and when they responded.

The barrier to basic A/B testing usually lies in company culture, not in cost or capabilities. Companies need to get wired for a "learning culture" that emphasizes marketing science over gut feel. This change must emanate from the senior executive team, and they have to understand how powerful data management and analysis can be to improve marketing response rates, revenues and profits.

As analysis professionals, we need to bring these smarts to the executive team so that they can bring culture change to the rest of the company. I try to remind myself of this goal periodically when I find myself tiring of yet another explanatory meeting with the VPs. Although sometimes repetitive and tiresome, the meetings to explain what we are planning to do after we test result in the executive support necessary to internalize the learning from the testing over the long term.

Revenge of the Data

I have been following with relish the story about Elon Musk's war with the New York Times over a negative review of their Tesla S electric vehicle. What I loved about Musk's retort to the New York Times story is how Tesla Motors managed to use device data to refute the story. The war ends up being a debate between the hard data in the device and the reporter's notes.

I take away three conclusions from this episode:

Reporter's vehicle log as annotated by an angry Elon Musk

1. Data is power. Companies that think about information they could or already do have available and then exploit that data create sustainable competitive advantage through their installed base. I learned this first hand at PPG Industries, where we were able to use tint machine data to examine paint color usage by region. I only wish that PPG had been more open to using the color chip rack to collect data (discretely and privately) about user interactions with the display. At Vocollect, we are exploring a wide variety of ways to aggregate data from our wearable devices to enhance the user experience.
2. Companies should get data in the hands of users. I see this war in part as a problem stemming from the New York Times reporter's inability to get all of the information he could have had available...information Tesla then gathered from the log files. Perhaps giving this information to the user in the first place in a snazzy interface could have prevented some of the reporter's frustrations. Heck, a number of device manufacturers give the data to users in an API and end up getting cool tools for their other users for free, created essentially by fans of the brand.
3. Don't get into a pissing match in public. Elon Musk, known for his huge ego, could have been more diplomatic and apologetic to the reporter. Abusing customers or potential customers does not position the brand for success. And essentially accusing a reporter at one of the most prestigious papers in the world of journalistic fraud qualifies as abusing potential customers in my book. Tesla Motors might have gotten a better response from the Times and better publicity by working with them to diagnose what had happened rather than by working against them. Unless you believe that all publicity is good publicity, in which case Musk did the right thing by making this story huge.\

I will anxiously await the innovations from car companies and any other company that has direct interaction with the actual consumer, enabling us to understand and improve our own behavior. As you know if you read this blog regularly, I hope to be at the forefront of that user empowerment given my sincere belief in the power of some Major Data Geekitude to improve our collective future.

Simple Modeling

For all you people who thought I was going to talk about supermodels, you can stop reading now.

Today's post is about the kind of model you use to determine your forecasted sales or the effects of a future rebate or the effect of a new product introduction. I have been thinking a lot about this kind of modeling lately because of Nate Silver, the statistics genius who accurately predicted the election results two nights ago. Today, the Guardian had an awesome explanation of the likely content of Nate Silver's model which is worth reading in its entirety.

Although Silver apparently uses an advanced statistical technique called hierarchical modeling to perform his analysis, a manager needn't have a degree in statistics to use something more basic but still useful. I put together a similar but simpler model at Strategic Energy using Crystal Ball, an Excel spreadsheet plug-in now owned by Oracle. The software allowed me to build inputs that had an effect on energy prices and then run a series of simulations describing what would happen to electricity prices if my various inputs fluctuated. I chose how each input would fluctuate (for example, natural gas prices might fluctuate in a normal curve by plus or minus 10%) over a period of time, and the model told me the statistical likelihood that the electricity price would get into the range at which we could compete against the regulated utility price.

It's relatively easy to use this kind of modeling in all sorts of applications. I used it again at PPG to help forecast exterior paint sales, using simple inputs we knew to affect our sales such as temperature, rebates, competitor rebates, advertising, and price competition. This analysis helped to show how unprofitable our existing rebate program was and how dramatically temperature spikes increased our paint sales, both of which led to savings and greater on-shelf inventory at our retail customers.

Amidst all this usefulness, I'm constantly amazed when managers prefer to use experience and judgement rather than data to make decisions. Crystal Ball costs all of $995. Why leave your decisions up to chance when you can get fairly accurate help from a fairly simple model for a fairly cheap price (or free if you're willing to learn the R statistics package)? Alternatively, you could spend hundreds of millions of dollars and just ignore the models like this guy did. Good luck with that.

Benchmarking Performance (AKA Why Vocollect Rocks)

I spent the first five years of my career at a company that purports to do benchmarking and best practices. Really a lot of what they do is profile innovative case studies of leading companies and then help other companies (their clients) understand how to replicate these practices. Nevertheless, I also did some real benchmarking as well, and I can tell you: it's a gigantic pain.

There's the problem of getting consistent responses over time. There's the problem of different customers not measuring the same things. There's the problem of customers measuring the same things differently. There's the problem of accounting properly for exogenous variables, such as equipment depreciation cost. There's the problem of aggregating the data in a way that's meaningful for the customers who gave you the data in the first place while still hiding specific participants' performance.

Often I have found that benchmarking is extremely valuable despite all the problems. Back when I worked on bank operations benchmarking, for example, we found out that productivity at check processing operations starts to decline somewhere around 250 million checks per year, probably due to the dominance of the three evil C's of operations (chaos, confusion, congestion) after that volume. We also found no practical limit to cost improvement in ACH operations at any scale, explaining why Norwest Bank (later Wells Fargo) dominated the ACH processing business.

I mention this value because I found out this week how Vocollect's superior speech recognition for distribution centers has been effective in replacing our competitors in a number of installations in Australia. Although the engineering team has been resisting putting our speech engine up to a benchmark test versus the competition due to the difficulty of benchmarking, I believe it's time to do so. We are really the only speech recognition engine that works in a loud distribution center environment. Perhaps it's time to prove it despite the pain.

Beware Vendor Metrics

I was reading about the end of the TV show House earlier today, and I saw this little tidbit in the article:

House‘s current eighth season ratings have remained solid, particularly for a drama airing at 8 p.m. The Monday night show averaged 9.8 million viewers and a 3.9 rating in the adult demo this season through early January when including seven days of DVR playback.

Since when should advertisers consider DVR playback? The DVR portion of the of audience adds only 16% to raw in-time viewing Gross Rating Points (GRP) according to this Nielsen study of DVR usage (as quoted in the New York Times). Let's say for the sake of argument that this particular show, like others, has 40% of the audience using a DVR. If that's the case, the real viewership was:

9.8 million * (1 - 40%) = 5.9 million * (1 + 16%) = 6.8 million

If (as an advertising buyer) you based your effective cost per thousand viewers (CPM) on the 9.8 million, you were over-paying by over 44%.

The misrepresentation probably stems from the network's presentation of their own overblown statistics. I have learned over the years to be highly skeptical of any vendor's own statistics, and in my own work for Vocollect, Inc. I try hard to provide our own customers an independent validation of statistics we quote on our truly superior products. The article mentioned above offers just one more reminder to smart market research analysts and marketing data consumers: examine the sources and rely on your own brain when using external data.

Check the Facts

Vice President Joe Biden recently got in trouble with the conservative media for claiming that cities like Camden, New Jersey and Flint, Michigan would see a rise in robberies, rapes and other violent crimes because they have had to cut police forces in half due to budget woes. Of course, Biden was arguing for Obama's new jobs bill, but I immediately thought of the more interesting data question.

The problem with determining the relationship between police force size and crime rates is "simultaneity bias." This term refers to the two events you wish to study tending to happen at the same time regardless of causality. In this case, governments tend to increase the police force when they notice a rise in crime or even in anticipation of this rise. Therefore, it is hard to see which comes first, the chicken (crime rates) or the egg (police force size).

One can overcome this problem by looking for individual events in which one factor changed for exogenous reasons--reasons outside the normal timing. There is an excellent summary of some of this research here:

http://www.majorcitieschiefs.org/pdf/news/more_policing_does_matter.pdf

The upshot is that cutting or raising the police force does have both an apparent short-term and an apparent long-term effect on crime. Nevertheless, the range of response varies significantly in the available research, some suggesting that a 50% drop in police force could mean only in the range of 10-15% rise in crime.

In a business context, simultaneity bias comes up quite a lot when looking in the rear-view mirror of one's business. For example, an increase in size of sales force is often accompanied by increase in marketing spend. This simultaneity confounds examining the effects of each investment. That is why I often recommend running experiments changing only one variable, or changing different variables in different territories or regions, before making such investments throughout the company.

Too bad Joe didn't find a source for his facts before he met the press. I hope he finds my blog.