How to improve your product using data?
How to improve your product using data?
Key Learnings:
Start using data to improve the product as early as possible in the product development.
When improving the product, identify one most important goal and focus your efforts on that goal only. Split the goal in smaller subgoals if needed.
Define metrics for the (sub)goal, and set a target state.
Design a hypothesis to improve a metric, and test that hypothesis with a simple and cheap implementation. Continue with full implementation only if your test shows positive results.
Once reaching the target metric, move to a new goal to improve.
Digital products today are becoming a necessity for almost any business. And customers are not only expecting you to have a digital product, but they do expect a very intuitive and seamless experience when interacting with the product. This bar is being raised higher and higher.
However, creating such a product that is intuitive and that resonates well with the users is not straightforward. And it's not a one-off activity. It is a matter of validation and continuous improvement over time. And this is where data and analytics can play a big role.
Product analytics is a data driven approach that allows one to observe how the product is used by the end user. Using product analytics, you can get detailed insights in how the users engage with the product. Are they using the new feature you've just created? Are they following the user journey as you expect?
And while in theory the potential of product analytics sounds exciting, using product analytics in the right way is not that easy. And the key challenge is identifying what to measure, and drawing conclusions based on the data without being overwhelmed with hundreds of metrics.
A 5 steps framework to improve your product continuously
How do you improve the product using product analytics? In this post we are going to describe a five steps framework that you can use to improve your product.
Step 1. Define your key goal
Needless to say, having clarity in your goal is the one most important thing. Once you are confident in your goal, the rest will follow much easier.
But how do you define the goal? Think about what is the key bottleneck that pulls your product back at the moment. Is it that the customer churn is high and engagement is low, or do you want to focus on growing the number of users?
Your goal depends very much on the stage of your product. If you have already proven market demand for your product, but your product is not yet optimal, you should optimise user engagement. If engagement is good, you can focus on improving growth. In their book, Benjamin Yoskovitz and Alistair Croll explain the 5 stages of lean startup growth: Empathy, Stickiness, Virality, Revenue and Scale. It's worth reading and trying to apply this thinking to your business.
Step 2. Select metrics to capture that goal
Next step is defining a metric that capture your goal. If you are already keeping a dashboard of various metrics, then identifying the goal and the goal metric would be much easier. That is the metric that is most critical, and that once improved, would make a big difference. If you don't have a dashboard of metrics, trust your knowledge and intuition but begin measuring as soon as possible.
You should be careful when selecting a metric. When the wrong metric is chosen, it can have a counter-effect.
For example, if you have a social media platform, a metric like average engagement time is a positive metric, because a higher value of the metric would indicate that the platform keeps the user more engaged. However, the same metric for an online grocery application might have a negative meaning. Higher value of the rating might be a sign of bad user experience as the user spends a long time to find the right products.
There is a theory about using a single metric only OMTM (One metric that matters). While the key idea behind this theory is valid, and that is - "focus is critical", following blindly this definition and oversimplifying can be risky. Because one metric only can give only one perspective of your business goal, and even more it is often too broad and not actionable.
So I would recommend to select one goal at a given moment and to focus on that goal. But to be able to make improvements towards that goal, you can choose multiple metrics to measure.
Step 3. Define a target
What is your target metric? If the average user engagement time at the moment is 10min per day, what is the target value you want to achieve? It can be a bit tricky to define a target, if you are new to this process or when you are building a new product.
You can research to find good targets for common metrics, like conversion rate, or retention rate in businesses similar to yours.
But even if you don’t have any best practice value, just give it a start. Set a target and once you start monitoring the metric, you’ll learn more and you’ll get a pretty good intuition on what is good and what is achievable for your specific product.
Step 4. Design a hypothesis
Having a goal without taking an action will not bring any changes. The next step is to formulate initiatives that you think will improve the metrics you’ve set. But remember that these initiatives are just hypotheses. You assume that these initiatives would move the needle, but you need data to prove that they work.
Best is to follow the Lean Startup approach. Think about how you can build the cheapest possible implementation of any hypothesis that would allow you to test that hypothesis.
Step 5. Test the solution and repeat
The best approach is to conduct an A/B test. Deploy the test to a subset of users, and measure if there is any difference between the old vs the new solution.
Observe and analyse the results. Does the change you implemented improve the desired metrics? If the results are positive, that's excellent, then continue with the full implementation of that change. If not, try to come up with another idea for a solution, develop a minimum solution and test the results again.
If at some point none of the initiatives work, you should rethink the goal you've set initially, and potentially pivot to a new goal.
Example:
With this approach, we can create a hierarchical structure. We begin by establishing the initial overarching goal, and then break it down into smaller, more manageable subgoals.
For instance, let's consider an online piano learning platform. The initial goal could be set as: "Increase the average time users spend on learning to play piano."
If that is the main objective, the next step is to determine how to improve this metric. How can we engage students and motivate them to learn more? To answer this, we could define two subgoal metrics:
Increase the average time per session by X%
Increase the average number of sessions per week by X%
For each of these subgoals, we can define initiatives.
For example, to improve the first subgoal, an initiative might be “shorten the songs the user should play". To improve the second goal, we can think of “set a learning schedule and send notifications to the user".
Whether these initiatives will indeed improve the engagement, we don't know, they are hypothesis. So we need to test first to validate if they really work. And to conduct the test, we need to be creative and find an easy and cheap way to validate the idea before making the full and expensive implementation.
Trust the process, it will get easier over time
This process might feel discouraging and challenging at the beginning. You might have questions like: How to select the key goal? Am I measuring the right things? Why don't I see any patterns in the data? Why should I spend time on testing the hypothesis instead of directly implementing it?
The process requires patience. Over time you'll learn more about the users, you'll define more accurate hypotheses, you'll have much more data so observing patterns will become much easier. So trust the process and enjoy the journey, in a few months you'll be glad you've made product analytics an integral part of your product development.