Join us at The Whalies • April 10th
Get Your Ticket
The Pitfalls of Being Data Driven

The Pitfalls of Being Data Driven

Last Updated:  
March 18, 2024

One of the major themes in business over the past two decades has been the importance of data. It’s almost become a meme at this point.

Walk through any airport in a major city and you’re bound to see advertisements from consulting firms touting their ability to unlock the power of data. Pundits say that “data is the new oil” and obsess over “the cloud”. Companies claim that being data driven is a key component of their culture, and screen for the same quality in their hiring processes.

So, can data really solve all your problems? That question reminds me of a story I heard from a friend of mine–a software engineer and stats obsessive. He made a living for some time by betting on basketball games. And he did this by designing a predictive model powered from publicly available data sets about the NBA season.

This person is now an entrepreneur; he doesn’t bet on basketball anymore. The reason? The model worked really well…until it didn’t.

Another sports-related story to illustrate the potential pitfalls of data: ratings for Major League Baseball games have declined sharply over the last five years. Viewership for the World Series and other MLB games is dropping much faster than cable TV viewership as a whole.

The reason? Many fans complain that baseball games have gotten more boring over time. Games are, on average, a half-hour longer than they were 40 years ago. And the most exciting elements of the game–players stealing bases and running the bases while fielders from the opposing team attempt to tag them “out”–are also happening less frequently.

Data eliminates risk

As baseball teams have become more data-driven, they’ve stripped out the elements of the game that made it compelling entertainment. To increase your odds of winning, you eliminate risk. But risk is also what put butts in seats. Risk produced the kind of once-in-a-generation outcomes that fans would brag to their friends about.

Of course, if a team consistently lost games, that produced a different kind of monotony. Teams that had a meaningful shot at making the World Series tend to attract more fans to the stadium. For those who couldn’t afford the best talent, a stats-based strategy was the best opportunity to become a winning team.

This was the plot of the popular film “Moneyball” starring Brad Pitt, which premiered more than ten years ago. But when your “top secret strategy” becomes a blockbuster film, anyone can use it. The outcome–the decline in baseball’s entertainment value–is the “law of shitty clickthroughs” applied to sports.

Each individual actor in the MLB ecosystem is optimizing toward its ideal outcome: a World Series win. But the net outcome is something that no individual actor expected: lower total profit for the entire league.

Data driven examples

1. Your data-driven strategies will be more sustainable the more you understand the underlying ecosystem.

Yes, it’s possible to build a predictive model by testing every data point available until something “works”. But that won’t tell you why your solution is working. And it will leave you high and dry when underlying conditions inevitably change and the model stops working.

The same goes for optimizing towards a specific KPI metric in your marketing programs without considering the broader context of your business goals. For example: I can make my web conversion rates go up immediately by turning off Facebook prospecting campaigns. But my daily sales will tank. Why? Prospecting traffic is less likely to convert, but it’s essential for maintaining sales momentum.

2. Many data-driven strategies are essentially arbitrage opportunities, which are rarely sustainable.

Does anyone remember when exit-intent email popups were invented? I do–it was around 2014, almost ten years ago at this point. What used to be innovative and pattern-breaking is now so commonplace that few of us even notice it.

If a tactic is public-facing and it appears to be effective, it will inevitably be copied. And even if a tactic is hard to understand from outside an organization, it will inevitably leak out. You can make more money teaching a tactic than you can implementing it, so the inventors of successful tactics are incentivized to talk. That’s exactly what happened to the stats-based baseball strategy outlined in Moneyball.

3. Humans are not great at anticipating the second- and third-order consequences of our decisions.

In the baseball example, individual teams harmed the league in their attempts to make it to the World Series. Something similar happened to the retail ecosystem–mall foot traffic and retailer profitability started to collapse when individual stores began pushing their shoppers to buy online.

For every customer that drove to the mall to visit a specific store like A&F, Victoria’s Secret or Macy’s, there were five to ten other customers who wandered in after visiting another store. Malls worked financially because of shared foot traffic. When one brand converted an in-store shopper to an online shopper, they also harmed their fellow mall tenants.

If humans have a spotty record of predicting how our decisions will play out in six months to a year, we’re almost incapable of viewing our decisions within the context of a broader ecosystem. And for most of human history that didn’t matter, because no single group of people had the capabilities to impact the ecosystem on a broad scale. In the era of “big data” and global economies, that’s no longer true.

The Right Way To Use Data

What I’ve covered up to this point might be considered pretty “heavy”. After all, the decisions you make today in Facebook Ads Manager are unlikely to crater the global economy or start a nuclear war. But those decisions do have the potential to make it much easier–or much harder–for your business to succeed in the future.

Digital marketing, as a discipline, has a tendency to focus on specific KPIs at the expense of overall business performance. Some marketers focus on hitting a ROAS or CPM target at any cost. If profitability or customer lifetime value suffer, well…that’s someone else’s problem.

Uncle Ben was right (RIP): with great power comes great responsibility. So here’s how you can use data more responsibly and increase your odds of making sustainable wins.

Start With First Principles

Data describes what’s going on in a given business or marketing campaign. But if you don’t understand how the parts of the business work together, or the mechanics of the platforms you advertise on, you’ll never know the whole story.

The fundamental unit of a business is the customer. Customers are people, and people’s core psychological motivations haven’t changed much over recorded history. This means that customers’ behavior is at least partially predictable at scale. It also means that running marketing strategies that attempt to change consumer behavior will always cost you more.

Similarly, the platforms you use to run ads are their own businesses. They provide a valuable service, but they’re also attempting to extract as much profit from you as possible. Align your objectives with the platform’s objectives as much as possible if you want consistent performance.

Have A Long Term Vision

Just to clarify: “make money” is not a long term vision. A long term vision usually starts with an outcome for your brand–do you want to go public, get acquired, or keep your brand in the family for generations?

This outcome determines the rate of growth you’ll seek and the level of risk you’ll find acceptable. And those parameters will shape your business’ strategy. Without those guiding principles, it’s hard to operate consistently. Data won’t help you make better decisions because, without a clear goal, there are no “right” decisions.

Use KPIs Responsibly

Sometimes, as marketers, we get so zoomed in on specific KPIs that we lose the forest for the trees. Every KPI should be connected to a big-picture business outcome. And there are always ways to max out a KPI that will move you further away from the big-picture outcome you’re trying to achieve.

For example: lots of marketers are obsessed with web conversion rate. What’s a surefire way to boost your conversion rate instantly? Turn off your Facebook prospecting campaigns. Of course, that will also tank your sales–the big picture outcome you’re trying to achieve. This is a very simplified example, but marketers make this mistake in more subtle ways every day.

Using KPIs effectively requires strong leadership and good storytelling in addition to a fancy analytics dashboard. Junior team members often zoom in on specific KPIs because their managers and company leadership fail to communicate the context behind the KPI. Don’t be that leader! Help your team understand the full story.

Don’t Get Analysis Paralysis

Sometimes access to data can become a double-edged sword: we can see so much that we become afraid to act until a situation has been analyzed from every angle. But sometimes the only way to get more information about a situation is to move forward–and potentially be wrong.

If you’re spending more time arguing over the definition of a “loyal customer” than you are developing strategies to increase loyalty, you’ve taken a wrong turn.

And if you want an easier way to manage and analyze your data, give Triple Whale a spin.

© Triple Whale Inc.
266 N 5th Street, Columbus OH 43209