This is the Data Wheel of Death:
Data Isn’t Constantly Maintained -> Data Becomes Irrelevant / Flawed -> People Lose Trust -> They Use Data Less
If the above looks familiar, you’re not alone. I estimate that greater than ⅔ of data efforts at companies fail.
This is trouble because data plays a key horizontal role in the growth process and mindset. Without good data, it’s not possible to run a legitimate experimentation cycle.
Today, I’ll take a look at 4 reasons why well-meaning data efforts fail so often, and what you can do about it.
Issue #1: Project Mindset vs Process Mindset
Most companies that want to get more serious about data approach it as a project. Something with a definitive start and a definitive end.
When you think about data as a project, you get the Data Wheel of Death.
The project wraps up, but at some point someone finds a piece of data that doesn’t look quite right. As a result, they don’t trust it and then they stop using the data. Because no one is using the data, the data isn’t maintained -- which leads to more distrust.
In the above scenario, treating data like a project is at fault. In reality, data is an ongoing, never-ending project, similar to building a product.
Build data collection -> Measure the impact of data outputs -> Learn from how data is or isn’t used
Your data needs to be refined and updated via an ongoing process for a few reasons:
1. Your product will change
Features will change on an ongoing basis. As your product evolves feature by feature, your data also needs to keep pace or it will become irrelevant / flawed, people will lose trust, and the Data Wheel of Death will continue.
2. Your understanding of the business will change.
Data should lead you to insight that might lead you to prioritize some things and de-prioritize others.
Andrew Chen likes to say, “Your data and KPI’s should be a reflection of your strategy.” Your strategy will change over time and what you track and analyze needs to evolve along with what it.
3. New answers expose new questions.
As you gain fresh insight from your data, it opens the door to new questions. As you have new questions, you need to update your instrumentation and analysis. Saying the process is “done” is saying you understand every thing there is to know about your users, product, and channels.
4. Shit breaks.
It’s that simple.
Data is never done
People spend more time on analyzing what tool to use than they do instrumenting and updating their data.
The project mindset is what’s behind this, and is tied to a mindset of “I have to get this right the first time.” The problem - there is no perfect tool and you end up in analysis paralysis.
If you view data as an ongoing process -- there’s no defined finish -- it’s easier to jump in knowing that you’re bound to iterate as new needs emerge.
What to do instead
Instead of a one-and-done, project-based approach to data, you should assign dedicated resources for both data collection and analysis.
In the earliest stages, this might be part of a couple of engineers’ or PMs’ time, but no matter how many or how few the number of hours it is, it needs to be seen as critical part of their role.
In later stages as the company grows, you will most likely need a dedicated team to maintain the data process -- both building and maintaining the data infrastructure, and promoting data usage.
It’s important to keep in mind that getting a data process off the ground isn’t just about instrumentation. Your other job is to build trust with the consumers of your data within the organization. You must spend time verifying data in at least some of the reports and working with the greater team around you to make sure they both understand and trust what they’re seeing. If they don’t trust it, they won’t use it.
Learn About Growth, One Email Per Post.
Issue #2: Misalignment of Incentives
Even if you treat data as a process vs a project, that doesn’t necessarily translate into success. There are companies that spend endless time on the infrastructure, tooling and instrumentation but then miss something extremely important.
Getting an organization to use data requires behavior change among the individuals.
Changing behavior is extremely difficult. There is typically one culprit behind the lack of change: a misalignment of incentives and rewards.
Teams and individuals will ultimately do what they are rewarded for. So if you want to change behavior, you have to make sure that the behavior is part of what they are rewarded for. There are multiple types of rewards:
- Financial Rewards (bonuses/raises/equity)
- Progression Rewards (promotions)
- Authoritative Recognition (good job from boss/superior, etc)
- Peer Recognition (“good job” from peers, etc)
When you look around your team, how often are you delivering one of those rewards related to the use of data?
Here’s the problem:
If the use of data is not rewarded, then the work that it takes to instrument and analyze data will be seen as friction to doing what is rewarded, or what’s perceived as a “good job” within the company.
If the use of data is rewarded, then the work it takes to incorporate data will be seen as an ally to doing a good job.
Then ask yourself a second question: How big / often is that reward in relation to other things you reward for?
How big/often the use of data is rewarded compared to other rewards signal importance/priority. Managers need to be careful how they weight different factors as part of rewards.
What to do instead
1. Every team needs a KPI
Every team needs a KPI as part of their measure of success. Having a KPI for each team aligns the use of data to their work vs positioning data as a friction point.
Just setting the KPI isn’t enough, though. These 3 things need to happen:
- The team needs have a sense of ownership over the KPI.
- If they don’t feel like they own it, they will view it as someone else's problem.
- Every single person on the team needs to understand the KPI and have easy access to viewing it.
Often times only the PM or a couple people on the team will truly understand it.
The KPI needs to be part of the four types of rewards, but not the only thing the team is rewarded for.
There are other important factors for product teams like shipping velocity, product quality, etc. Like most things, it is a balance.
2. Design systems for each type of reward
Go through each of the four reward types and design systems to reward for the use of data. What do I mean by systems?
Here’s a quick example around a system for Authoritative Recognition Reward: the best managers I know have checklists they use to prep for every 1:1. An item on the checklist might be “Did this person display the use of data in their work?”
If yes, make sure to give recognition for it. It is too easy to forget these types of things unless you have specific prompts.
3. Communicate the rewards clearly
Don’t assume that team members know how they get promotions, bonuses, praise. You need to put it in writing, make it explicit, and reinforce. You have to over communicate in order for it to get through.
When you promote someone, don’t just announce “Congrats to Jane Smith on her promotion to Sr. Product Manager.” Back the announcement up with examples of the type of work and behavior that the person has displayed which led to the promotion.
Issue #3: Data team becomes the bottleneck
If you solve process vs project and incentives and data starts to become valuable in the company it creates a couple of new problems.
The first is that the data team can end up becoming a bottleneck. This stems from a the data team taking an “ownership” mindset to the data, i.e. “We own the data.”
But ownership thinking leaves out one important point:
Every team has a “customer.”
The data team’s customers are the other people using the data internally: data analysts, product managers, engineers, marketers, etc.
In service to these internal customers, the data team needs to act like any other product team:
- They need to define their customer segments
- They need to understand customer needs
- They need to deliver the most compelling solution
- And they have to iterate!
In other words, their output must be able to enable other teams’ output, rather than them being the exclusive owners.
Issue #4: Brilliant Answers, Useless Questions
There’s a second issue that arises once data has become valuable and recognized, and that’s when people start noodling on data for data’s sake. Whether it’s because they find the data process intellectually fascinating, or just to seem smart, productive, etc. in the end, it’s data masturbation :)
Data for data’s sake is an alluring trap, but only creates “brilliant answers to useless questions,” as Ken Rudin (Head of User Growth & Analytics at Google, former Head of Analytics at Facebook) says.
Rudin reminds us that even though it’s alluring to increase knowledge, insight alone isn’t enough -- results and impact are the true goal of analytics. To that end, data teams should make sure they’re asking the right questions, not just coming up with more and more answers.
Rudin has two suggestions for how to create a data process that actually serves business needs:
1. Hire analysts that are business savvy, not only academic about data or its tools.
When you interview candidates, don’t focus on just “How do we calculate this metric?” Ask them “In this business scenario, what metrics do you think would be important?”
2. Make data everyone’s thing.
Rudin’s team at Facebook ran a two week “Data Camp” for not just the data team but for everyone at the company. It gives the wider organization a common language to frame questions that data can answer.
(Some great additional talks by Ken on data and impact here and here.)
A solid data process will be informed by asking good questions that result in a real business impact (Rudin also makes a great point that data teams must be accountable for not just “actionable insights” but the fact that those necessary actions get taken).
As data provides answers and actions, it helps shape better and better questions.
As data’s answers translate to impact, people will be incentivized to maintain and even improve data systems, leading to more data capacity, and stronger answers to better questions.
Comments