In usability testing, numbers don’t lie—but misinterpreting them can lead to costly mistakes. Imagine launching a major redesign based on test results, only to realize later that your conclusions were skewed.

One critical factor that’s often misunderstood or overlooked? Margin of Error—the key to knowing how much you can truly trust your usability insights.

In this article, we’ll break down:

🔹 Why no usability testing results is 100% accurate?
🔹 What is Margin of Error (and why it matters)?
🔹 How to balance precision and cost in usability testing?
🔹 Practical steps to get the most accurate usability insights

Your Data Is Never 100% Accurate

Let’s say you’re in the vehicle leasing business and want to find out what percentage of Amsterdam residents have leased a car in the past two years.

Will you survey every single resident in the city?

No. You’ll take a sample—a smaller, random group representing the whole population.

And because you're only working with a portion of the population, your results can never be 100% accurate. They’re always an estimate of the true value, with some degree of uncertainty—known as the Margin of Error. A margin of error tells you how much your data could be off.

Key Statistical Concepts in Usability Testing

To make sense of your usability test results, you need to understand three key statistical concepts: Confidence Level, Margin of Error, and Confidence Interval.

1. Confidence Level

The confidence level is a value you select that reflects your tolerance for uncertainty.

• A higher confidence level means you want to be more certain, so you’re less willing to risk being wrong.

• A lower confidence level means that precision is less critical, and some uncertainty is acceptable.

There a few standard confidence levels used:

• 95%: This is the most common choice in usability testing and research. It balances certainty and practicality.

• 99%: Used in high-stakes situations where errors could have serious consequences. An example is testing the effectiveness of a new drug in medical research.

• 90%: Used for exploratory or early-stage testing, where early insights are more important than precision.

2. Margin of Error

The margin of error is a percentage that shows how much your sample results might differ from the true value.

Unlike the confidence level (which you choose), the margin of error is derived from your data and depends on factors like the sample size, variability in the data, and your chosen confidence level.

• A smaller margin of error means your results are more precise.

• A larger margin of error means your results are less precise.

For example, if the result of your testing is 60% (let’s say 60% of the surveyed participants have answered ‘Yes’), with a margin of error of 3%, it means the true value is likely within ±3% of your result (between 57% and 63%).

3. Confidence interval

Once you have the margin of error, you can easily calculate the confidence interval—the range within which the true value is expected to fall.

The confidence interval is expressed as:

[X-margin of error - X+margin of error].

In the example above, if result is 60% with a margin of error of 3%, the confidence interval would be [57% - 63%]

4. Putting It All Together

Let’s see how confidence level, margin of error, and confidence interval work together in real scenarios.

Example 1: Vehicle Leasing Survey

• Chosen Confidence Level: 90%

• Survey Result: 60% of participants reported leasing a vehicle.

• Calculated Margin of Error: ±3%

• Calculated Confidence Interval: [57%, 63%]

How to Interpret This:
With 90% confidence, we can say the true percentage of people in the overall population who leased a vehicle is between 57% and 63%.

This means that if we repeated the survey 100 times, the results would fall within this range 90 times out of 100.

Example 2: Usability Testing—Task Completion

• Chosen Confidence Level: 95%

• Testing Result: 80% of participants successfully completed the given task.

• Calculated Margin of Error: ±2%

• Calculated Confidence Interval: [78% - 82%]

How to Interpret This:
With 95% confidence, we can say the true percentage of users who would successfully complete the task is between 78% and 82%.

This means that if we conducted the usability test 100 times, the true result would fall within the range [78% - 82%] 95 times out of 100.

Factors That Influence Margin of Error

Let’s dive into the components of the margin of error formula to understand what drives it and how it behaves in different scenarios.

The error margin is a function of three parameters:

• Z (Z-Score):. This is fixed number calculated directly from the selected confidence level. For the most commonly used confidence level, we have

  • At a 95% confidence level, Z=1.96.

  • At a 90% confidence level, Z=1.65.

  • At a 99% confidence level, Z=2.58.

• n (Sample Size). This is the number of participants in your usability test or survey.

• p (Sample Proportion): The percentage or fraction of people in the test who meet the specific condition or answer the researched question positively.

  • Example: If you're testing a feature with 100 users and your goal is to observe how many will complete a certain task, and 60 of them successfully complete it, the sample proportion is 60% (p = 0.6).

Now with the formula in mind, this is what influences the margin of error.

📌 Higher Confidence Level increases the Margin of Error

If you choose a higher confidence level (e.g., 99%), the margin of error increases.

For example, if your result is 60%, and at a 95% confidence level, the margin of error is ±2%. This means you can be 95% confident that the true value falls within the range [58%, 62%].

However, if you want to express the same result with 99% confidence, you’ll need a wider range to account for the additional certainty. This results in a higher margin of error, meaning the range might expand to something like [57%, 63%] to include the true value with greater confidence.

📌 Higher Sample Size means lower Margin of Error

A larger sample size reduces the margin of error because it provides a more accurate representation of the target population. With more participants, the results are closer to what you’d get if the entire population were tested.

On the other hand, a very small sample size increases the margin of error, making the results less reliable.

📌The Variability of the Result Affects Margin of Error

The variability in your results, represented by the sample proportion (p), directly influences the margin of error. The value p*(1−p) is highest when p is around 50% and decreases as p moves closer to 0% or 100%.

For example, if 50% of users complete a task successfully, there is an equal chance of success or failure, leading to higher variability and a larger margin of error. However, if the result is more extreme—say, 90% or 10%—the outcome is more predictable, reducing the margin of error.

How to Improve Your Usability Testing Accuracy?

How can you achieve high precision in your results and reduce the margin of error?

While some factors are outside your control, like the confidence level (determined by the context of your work) or the variability in your data (the sample proportion p), there are two key steps you can take to improve your results:

1. Eliminate Biased Data

The margin of error formula assumes your data is unbiased.

What does that mean?

If you want to find out what percentage of Amsterdam residents have leased a car, it’s a bad idea to only survey people working at a car leasing company.

Why? The data would be biased, as this group is much more likely to lease cars than the general population.

Select your sample randomly so that every individual in the population has an equal chance of being included. Make sure your sample isn’t skewed toward a specific group that doesn’t accurately represent the entire population.

Without a representative sample, even a large sample size or high confidence level won’t give reliable results.

2. Adjust the Sample Size

The second important factor is the sample size.

A larger sample size reduces the margin of error and narrows the confidence interval, making your results more reliable.

Let’s say you’re working with a population of 10,000 users, a confidence level of 95%, and a sample proportion of 80%. Here’s how the margin of error decreases as you increase the sample size:

But there’s a trade-off: larger samples require more resources, time, and cost.

Testing the entire population would provide perfectly accurate results, but it’s unrealistic and unnecessary.

Balance resources and accuracy.

There isn’t a one-size-fits-all answer, but here are some tips to guide you:

• For exploratory research or initial usability tests, aim for a margin of error around 7-10%.

• For key decisions like product launches or user satisfaction surveys, aim for a margin of error between 2-5%.

• For high-stakes research, where precision is critical (e.g., medical testing or compliance studies), aim for a margin of error of 1-2%.

Practical Tips for Usability Testing

When conducting usability testing, these practical tips can help you maximize reliability and make the most of your resources:

Use a Sample Size Calculator: Many online tools are available to help you calculate the margin of error and determine the ideal sample size for your study. Here is an example of such tool. Use these calculators to experiment with different confidence levels, sample sizes, and error margins to understand how they impact your results.

Report with Transparency: Always include error margins and confidence intervals in your reports. Providing stakeholders with this information ensures they understand the reliability and limitations of the results, avoiding overconfidence in findings.

Prioritize Key Metrics: If resources are limited, focus on the most critical metrics only and ensure they are supported by adequate sample sizes.

Leverage Iterative Testing: Start with smaller, exploratory tests, identify major issues, then scale up with larger tests to refine your findings. Iterative testing saves time and resources while helping you address the most significant problems early.

Key Take Aways

If your usability tests ignore the margin of error, it's time to rethink your testing strategy.

You don’t need to be a statistician, but understanding a few principles you will significantly improve your outcomes:

📌 Use a representative sample—Ensure it's unbiased and large enough for reliable results.

📌 Confidence comes at a cost—A higher confidence level increases the margin of error, meaning you’ll need more data for precision.

📌 Balance accuracy with resources—Prioritize key metrics ensuring their accuracy.

📌 Report transparently—Always include margin of error and confidence intervals to avoid misleading conclusions.

A version of this article was originally published at https://www.logrocket.com/ on Feb 18 2025.

Enjoyed this read? Subscribe to Lean Product Growth for regular updates on building and scaling a successful product organization.

Subscribe now

Can you imagine a company without insights into its financial data? For many, unfortunately this is where the data journey ends. Beyond financial metrics, data is often superficial or non-existent in other crucial areas.

How data mature is your organization? Think of answers to these questions:

• Can your product team quickly determine how well the new feature was adopted by users?

• Can your platform team identify the key components in the infrastructure that drive high cloud costs?

• Do teams across the organization have metrics that align with the higher organizational goals?

• Are you confident that your product can handle the upcoming spike in users?

If your answer to these questions is no, it’s time to advance to a more data-mature organization and leverage the benefits of data.

What is Data Maturity?

Data-mature organizations effectively leverage data to make critical business decisions. They rely on objective insights rather than gut feeling or the opinion of the highest-paid person in the room.

In a data mature organization, data is not confined to the finance and executive team. While financial data is crucial, it alone is insufficient for other teams to take meaningful action. For instance, if financial data indicates a need to boost revenue, a product manager or a product development team must track more specific metrics to make informed decisions on how to contribute to this goal.

In a data-mature organization, data and insights are embedded throughout every part of the organization, guiding decisions both big and small. Additionally, data aligns closely with the goals that teams set for the upcoming quarter, ensuring that all actions are data-driven and goal-oriented.

Benefits of Data Maturity

The benefits of data maturity are undeniable. Numerous studies highlight the advantages of data. For instance, IDC’s Digital Product Analytics Maturity Study conducted in 2022 analyzed 600 product companies and found that data-mature organizations achieve 2.5 times better business outcomes compared to their less mature counterparts.

The study revealed that leaders in data maturity outperform laggards in every aspect, including revenue, operational efficiency, customer satisfaction, and product adoption. Here are some key findings:

These results demonstrate the significant advantages of becoming a data-mature organization, highlighting the importance of integrating data into all aspects of business operations.

Factors that Influence Data Maturity

Becoming a data-mature organization is far from simple. It's not just about adopting a tool and expecting instant results. Achieving data maturity involves a significant shift, requiring many elements to align effectively:

Culture: Organizational culture and data mindset are crucial. Leadership must foster a culture of data experimentation and encourage departments to set measurable objectives.

Access to Data: Effective data collection and accessibility are vital. For instance, if an organization lacks comprehensive customer data, it can hinder insights and decision-making.

Tools: Proper tools are essential for leveraging data. They impact the quality of insights, as well as the efficiency and cost of data analysis.

Data Skills: The data skills of the teams and their ability to extract valuable insights can significantly impact how well the organization harnesses the potential of data.

Five Stages of Organisational Data Maturity

The journey towards a leader in data maturity goes through multiple stages. Let’s explore.

1. Data Apathy

This is the initial stage where data is either non-existent or largely underutilized. At this point, there is no awareness of the value or benefits of data, no leadership support for the use of data, and decisions are made almost entirely based on intuition. This stage is often seen in small businesses that rely solely on the owner’s experience or informal feedback from customers.

Consider a local restaurant where decisions about menu changes are made based on the owner's personal experience rather than real quantitative data.

2. Data Experimentation

At this stage, data awareness increases, and data starts to be used across the company. However, it is carried out in an ad-hoc manner, with no formal strategy or standardized tools in place.

Data is gathered in silos, with different teams using disparate tools and metrics, leading to inefficiencies and a lack of cohesion. For example, while a sales team might track basic financial and sales data, they may lack insights into customer behavior or operational performance.

Although this stage is not very advanced, moving beyond it is a significant step forward. It signifies that the organization is beginning to recognize the value of data and is starting to incorporate it into decision-making processes. If managed effectively, this stage can set the stage for a crucial mindset shift, proving that data can drive significant business value and motivating leadership to invest further in this direction.

However, there is a notable risk associated with this stage. When the motivation to accumulate data arises without a strategic approach, it can lead to data overload. The sheer volume of data becomes overwhelming, expensive, and counterproductive. In such cases, organizations may struggle with inefficiencies and increased costs without deriving meaningful insights.

Therefore, it is essential to use this phase as an experimentation stage and a stepping stone toward the next, more advanced stage.

3. Data-Informed Strategy

At this stage, the true potential of data begins to shine. Organizations start to extract high value from their data as it becomes democratized across the company; everyone has access to it, and everyone understands its importance.

Data-informed decision-making becomes a fundamental part of the organizational culture. Success initiatives, whether at a high or low level, are now closely tied to measurable results and supported by data.

Teams are empowered to identify and address bottlenecks through data insights. Their performance is directly linked to the success metrics defined by broader organizational goals. This alignment helps in understanding how their contributions impact overall success.

Everything becomes much more predictable. Teams gain greater confidence in committing to certain targets thanks to the clarity provided by data. They know which direction to choose to maximize the value of their efforts.

Problems become more transparent and actionable at an early stage. This predictability reduces frustration and dissatisfaction among employees. With clearer insights into challenges, teams and leaders can implement timely and effective measures, leading to a more streamlined and efficient operation.

4. Data-Driven Culture

At this stage, data becomes deeply ingrained in the organization’s DNA, evolving from a support function to a core component of operations and decision-making.

Everyone is aligned on the key metrics that measure the company's success for the quarter, focusing on the One Metric That Matters (OMTM) most.

Teams prioritize experimentation as a crucial step before embarking on new initiatives, relying on data to validate their direction.

A data-oriented mindset becomes essential in the hiring process, and advanced data skills are considered valuable.

Dedicated data teams refine data practices and reporting mechanisms, ensuring high data consistency across the company. Every department uses standardized dashboards and accesses a centralized database.

Mature business intelligence tools, strategies, and processes are employed for regular reporting. Innovative predictive analytics techniques are explored and implemented to unlock even greater value from the data, helping the organization stay ahead of trends and competition.

5. Data as Strategic Asset

Some organizations see their data warehouses as a strategic asset with the potential to unlock significant business opportunities beyond traditional uses.

They advance to the next phase by leveraging data to enhance not only internal operations but also to explore new ways of getting revenue. By creating innovative business models and products based on valuable data insights, they turn data into a powerful competitive advantage.

This approach helps them generate new revenue streams and stand out in the market. Data becomes a cornerstone of their strategic vision, driving both operational excellence and innovative growth.

For example, a retail company might use data patterns to launch a subscription service that offers personalized product selections, gaining a significant competitive edge and creating a new revenue stream.

Increase Data Maturity Gradually

Moving from the initial data apathy stage to achieving the highest level of data maturity is a gradual process that requires significant changes in mindset, especially among leadership. This transformation often takes longer in larger organizations due to the scale and complexity involved.

If you are at the initial stage, focus on small-scale data initiatives to prove value. Start by engaging with data on a manageable level and demonstrate its benefits through targeted experiments. Show some early successes first, and then it’ll become easier to promote the broader adoption of data within the organization.

As awareness grows and others begin to recognize the value of data, the virus starts to spread. Capitalize on this momentum. Organize training sessions to educate teams on effective data usage and support small-scale projects to use data for informed decision-making.

When the impact of data becomes evident more broadly, the organization will naturally progress toward more advanced data practices. Investing in centralized databases, adopting sophisticated tools, establishing robust processes, and hiring data-savvy professionals will be a high priority.

When high-quality data practices and advanced methodologies are already in place, transitioning to the last maturity stage is no longer a miracle. It is a natural and expected progression for a mature, data-driven company.

Conclusion

Data maturity is essential for organizations of all sizes, whether you're a startup or a large enterprise. While the practices will differ based on your company's stage, prioritizing data is crucial for maximizing business value.

For startups, engaging with data from the beginning provides a significant edge. Implement lean, data-driven practices early on that can scale with your growth. Cultivating a data-focused mindset now will position you advantageously as your organization expands.

For established organizations still developing their data maturity, it’s never too late to start the transformation process. Though it requires time and dedication, improving your data practices gradually will yield substantial benefits. Embrace the journey and leverage the incremental gains you achieve along the way.

Originally published at https://blog.logrocket.com/ on Sep 2 2024.

Enjoyed this read? Subscribe to Lean Product Growth for regular updates on building and scaling a successful product organization

Data matters.

It should be part of every segment of your organisation: to get insight in the quality of the product you deliver, the usage patterns of your product or the effectiveness of your HR processes.

While many organizations struggle with a lack of data, others face the opposite challenge — an overabundance of data. And when teams are overwhelmed with data, they can’t make decisions because they’re stuck in analysis paralysis.

So, how can you recognise and address this challenge, and how can you integrate effective data-informed processes into your organization? Let’s find the answers in this article.

How Does Metric Overload Happen

Metric overload occurs when an organization collects and tracks excessive amounts of data without a clear focus or strategy. Important insights are buried in a sea of irrelevant metrics, causing teams to struggle with analysis paralysis and make no change to improve.

Metric overload is a common issue for organizations transitioning to a more data-mature level but still operating immaturely.

Initially, immature data organizations have no data collection or analysis. Decisions are made based on intuition or anecdotal evidence. For example, a small business might rely solely on the owner's experience and word-of-mouth feedback.

Over time, as data awareness grows, data tools are integrated across various departments. However, problems arise when this integration occurs without a clear strategy. Teams become overenthusiastic about tracking every possible data point, leading to metrics overload.

Examples of Metrics Overload

I have observed this in many situations and with various teams. Teams collect excessive metrics without clear prioritization and without connecting the metrics to a higher strategic goal. This lack of focus leads to data overload, making it challenging to extract meaningful insights or take effective action. 

Here are a few examples.

• A platform team tasked to implement observability to monitor system performance. Excited by the potential, the team incorporates a plethora of metrics — CPU usage, memory usage, server response times, latency, memory page faults, etc. 

  However, without a clear strategy, the sheer volume of data becomes unmanageable, critical insights are lost in the noise, and the team struggles to identify the metrics that truly matter. Consequently, the company faces high costs from unnecessary data storage and processing, and the team is unable to justify the expenses or make informed decisions.

• A product development team integrating a quality check tool to improve software quality. This tool floods them with hundreds of metrics and violation reports for each piece of code — size of the methods used, complexity of the code, metrics about the test coverage, and metrics counting different types of security vulnerabilities in the code, among others. 

  Of course, these are all great insights, but without a clear goal and a strategy to prioritize, the team becomes overwhelmed and ends up ignoring the metrics entirely. Yes, they may superficially check the box and claim they’re using an advanced quality check tool. In reality, however, no improvements were made — they continue releasing software with unresolved issues while bearing the costs of an underutilized tool.

• A team required to meticulously log work hours and track how employees spend their time on a project to measure productivity. So, employees must fill out exhaustive forms with data. However, many employees fill in the data hastily, often inaccurately, and merely to comply with mandates. 

  As a result, the collected data yields skewed insights, leading to misguided decisions and unhappy employees, negating the intended purpose of improving productivity.

The Symptoms of Metric Overload

If you notice these signs, it might be time to rethink your data strategy and simplify things.

• Ignored dashboards: Are there dashboards in your organization that no one looks at? Metric overload could be the reason. When dashboards are crammed with data but fail to provide clear insights or highlight key areas, it's inevitable that they will be ignored.

• No action, no change: Metrics should drive change. If your team isn’t making new decisions or if the business continues as usual with stagnant or declining metrics, it’s a red flag. Metrics are meant to highlight the path to improvement, not just collect dust.

• Wrong focus: Is your team making changes based on data, but those changes don’t impact your organization's success? This indicates an immature metric strategy. Even if it’s not metric overload, it’s a sign your data efforts need realignment.

• Resource drain: Are your team members spending a lot of time discussing data collection and challenges they encounter with it? This is a sign that your metrics are not effectively integrated. If too much effort goes into gathering and analyzing data, it’s time to streamline.

Strategies to Combat Metric Overload

So how do you avoid getting into the metric overload trap. Let’s review some effective strategies.

Strategic alignment

This might be the most crucial aspect. You don’t want your organization to just collect data; you want to drive change with it.

Before implementing metrics for any team or the broader organization, start from a higher strategic perspective.

• What is the objective of the team you are implementing metrics for?

• What insights do you want to gain by implementing these metrics?

• What actions should these metrics drive?

The objectives and the metrics you implement must align with the overall company strategy. The key pain points the company wants to tackle in the coming quarter should guide your teams when implementing metrics.

Define KPI’s to track the health of your team performance and OKRs where you want to make a difference. Derive the metrics from these objectives.

Some organizations use a single metric as a north star (the theory about One Metric That Matters - OMTM)). For instance, for a company like Netflix, a north star metric could be the number of videos watched per day.

What your team measures should align with the organization’s north star. While not all your KPIs will directly align with this metric (think of tech debt for a development team), there should be a clear rationale for how improving these areas contributes to the broader organizational purpose.

Ensure the metrics are actionable and focus on driving meaningful change. Every metric should serve a purpose and help your organization move closer to its strategic goals.

Keep It simple

If you’re just starting, keep it lean and simple. Begin with a few high-level, important metrics. As you gain insights and become adept at using these metrics, you’ll naturally refine this view with more detailed ones.

Starting with a simple set of metrics helps gain momentum. As you progress, you’ll discover what additional metrics you need.

Same holds for dashboards. Keep them clean and simple.

It can be tempting to create rich dashboards with various charts, especially if the tool you use makes it easy. While creating these dashboards is fine, focus on what is most important and consider the rest as supplementary information.

Refrain from presenting a dashboard with 15 charts to stakeholders every week. This will be overwhelming and will lose value without a clear narrative. Instead, agree on a few—3 or 4 key insights that provide a comprehensive view of the team's performance and focus discussions around them.

Make it motivating for the team

Metrics are just numbers if people don’t connect them with their own values, goals, and daily work. That’s why selecting metrics that matter to everyone is paramount.

Engage with your team, educate them, and make sure everyone is on the same page. When people understand and value the metrics, they become more than just numbers—they become a driving force for improvement.

When everyone is aligned, reviewing the dashboard can ignite meaningful discussions.

Team members should feel passionate about improving the metrics, and stakeholders should trust the data to make informed decisions. This alignment makes it easier to discuss key actions, such as increasing investment in an initiative that shows positive results or improving a process indicated by a declining metric.

This is the way you create a data-driven culture where informed decisions drive success and continuous improvement.

Balancing metrics and intuition

Metrics are essential, but they are just one piece of the puzzle. To gain complete and relevant insights, you need a blend of various data points. Both qualitative and quantitative.

Data should not completely replace human intuition and experience. We can’t draw conclusions about employee performance or make significant decisions based solely on collected data.

Engage with those directly involved, whether it's your team, customers, or users. Understand the real "why" behind the data. Encourage teams to use metrics as a guide, not as the sole determinant.

By balancing metrics with human insight, you ensure a more nuanced and effective approach to decision-making.

Originally published at https://blog.logrocket.com/.

You've set an ambitious goal for the next quarter: improving your team's productivity. You've launched several initiatives to hit this target. But as the quarter ends, the big question looms: should you keep funding these efforts?

It's tough to decide.

Opinions within the team vary—some see progress; others don't notice any change. You're eager to continue, but convincing stakeholders to invest more without clear data evidence of success is tough.

Many organisations underestimate the importance of data and metrics, dismissing them as costly extras. They prefer to trust their gut. However, operating without data is like trying to navigate to a mystery destination without a map. It's likely to take more time, effort, and emotional energy than you'd expect, and the hidden costs can add up without you even realising it.

There are, however, other extremes. It’s not uncommon to see organisations use metrics closely tied to employees incentives, or make big decisions following only numbers. It’s not a surprise that in such cases metrics can damage the overarching strategy or even the organisation culture.

In this article we’ll explore:

• 🎯 The Importance of Success Metrics and Practical examples.

• 📊 Frameworks for Measurement using KPIs and OKRs.

• ⚠️ Tips to Avoid Common Mistakes In metric implementation.

What are Success Metrics?

Success metrics are measurable parameters used to measure progress, effectiveness, and ultimately, success. They encapsulate core objectives and enable you to track your trajectory toward achieving these objectives.

Whether you're launching a new product, expanding into new markets, or streamlining internal processes, metrics are necessary if you want to objectively assess progress, and make effective decisions along the way. They help you understand what's effective, what requires adjustment, and where to focus efforts for optimal outcomes.

Let's explore some real-world examples of success metrics.

E-commerce Product. When managing an e-commerce application, it’s helpful to keep an eye on key metrics that give you insight into how effectively you’re converting visitors into buyers and retaining customers. Some metrics to track may include:

Customer Service Center. The main goal of a customer service is to resolve customer issues swiftly to boost customer satisfaction. Key metrics here include:

Software Development. In the context of software development, there are number of metrics that measure every aspect of the software development process and the quality of the software as output. A useful metrics framework that captures a few key aspects is the DORA framework:

What are OKRs vs KPIs

Two common methodologies used in business to measure performance and guide decision-making are KPIs and OKRs. There is often confusion about the differences between them and which one is more effective to use in practice. In fact, both methodologies are different; they are complementary and serve different purposes.

KPIs (Key Performance Indicators)

Imagine a dashboard showcasing various metrics, such as customer satisfaction scores, revenue growth rates, and employee turnover rates. KPIs can be many, each of them giving different perspective of the health of the domain we are measuring.

Based on the health of this KPI dashboard, certain metrics may stand out, signalling areas needing attention or improvement.

OKRs (Objectives and Key Results). When there is area for improvement identified, OKRs come into play. They represent the objectives an organization aims to achieve (O=Objectives), along with the key results (KR = Key Results), the specific, measurable outcomes that indicate progress towards achieving the objectives. Unlike KPIs, which measure the ongoing performance (business as usual), OKRs focus on setting ambitious, outcome-oriented goals and driving change.

Success Metrics Framework Using KPIs and OKRs in Practice

OKRs and KPIs serve distinct purposes, and they can complement each other when used together strategically.

Let’s explore how you can leverage KPIs and OKRs in a structured approach, taking customer service center as an example.

1. Define Objectives: Begin by understanding what is important for the success of the team and its primary purpose. What does success look like for the team? How does this align with the company’s strategic goals? In the example of customer service center team, objectives should emphasize key areas such as customer satisfaction and operational efficiency. These goals are crucial for maintaining high levels of service and ensuring smooth operations, ultimately contributing to the company's overall success.

2. Define KPIs: Select KPI metrics that directly reflect these objectives and serve as indicators of the team's performance baseline. You can include more metrics, but keep it simple and ensure that gathering these metrics does not become burdensome. Examples include customer satisfaction score, first response time, issue resolution time, or the percentage of manual issue handling. These KPIs provide valuable insights into the team's current health and performance.

3. Identify Areas for Improvement: What are the red flags in the KPI dashboard? Analyze KPI data to identify areas requiring improvement. For instance, if prolonged first response times are affecting customer satisfaction, delve deeper to understand the underlying causes, such as inefficient workflows or resource constraints.

4. Set OKRs: Based on identified areas for improvement, set specific key objectives that address these issues. Make objectives measurable and actionable. These objectives provide clear targets for performance enhancement. An objective can be improving first response time by 50% within a defined timeframe.

5. Define Initiatives: Next step is to develop initiatives or strategies to achieve the defined objectives. This may involve implementing automated response systems, optimizing routing and triage processes, setting clear service level agreements (SLAs), or introducing response templates and scripts to streamline agent workflows. Each initiative should align with the overarching objectives and address identified improvement areas.

6. Prioritize Initiatives: Prioritize initiatives based on their expected impact and feasibility of implementation. Test solutions iteratively, starting with those that offer high potential benefits and can be implemented quickly. Remember, these initiatives would often be just hypotheses, and you should validate their effectiveness through testing and refinement.

7. Test the Solution and Repeat: Monitor the impact of implemented initiatives on selected KPIs and overall performance. If the desired targets are not met, iterate on strategies and adjust initiatives as necessary. Continuously test and refine solutions until objectives are achieved.

Challenges and Pitfalls in Defining Success Metrics

Integrating metrics into business practices is not always easy, and many organizations give up when faced with the initial challenges, often finding excuses that it's not worth the effort. Let’s explore some common challenges and pitfalls:

• Difficulty to get data: Some metrics are challenging to measure or they may require manual effort, leading to administrative burden and potential inaccuracies.

• Too few metrics: While a selected few metrics can provide focus, having too few may offer a limited perspective, leading to erroneous decisions.

• Too many metrics: Overloading with metrics can lead to analysis paralysis or having dashboards that nobody looks at, diminishing their effectiveness.

• Time investment: Data analytics often gets sidelined due to a lack of time and competing priorities, making it challenging to prioritise amid urgent tasks.

• Overemphasis on data: Relying solely on data can be problematic as you may overlook important factors that cannot be quantified, leading to incomplete decision-making.

• Data reliability issues: Manual data entry, such as recording hours worked, can introduce reliability concerns, undermining the credibility of the metrics being used.

Despite these challenges, the benefits of leveraging data outweigh the negatives. To address these challenges effectively, it's essential to:

• Keep it simple: Start with simple metrics that are easy to implement and gradually refine them over time. Focus on metrics that are easy to derive automatically to minimize manual tracking.

• Select a few but key metrics: Choose a small set of metrics but make sure they provide a holistic view of the problem.

• Promote data-informed decision-making: Encourage a culture that values data while recognizing the importance of intuition in decision-making.

• Regularly review and refine: Continuously evaluate and adjust metrics based on changing priorities and feedback to ensure relevance and effectiveness.

Key Takeaways

Integrating metrics as a regular practice in your organisation may initially feel daunting and challenging. However, the benefits far outweigh the initial hurdles. Patience is key in this process.

Over time, the process becomes more manageable, and you'll develop a robust system for capturing data effortlessly. With a well-equipped system in place, identifying patterns and trends becomes much easier, enabling you to make informed decisions. And as you gain experience, you'll improve your ability to formulate hypotheses effectively.

While the beginning may be the most difficult phase, in a few months' time, you may find yourself wondering how you ever operated without the insights provided by data.

If you found this guide helpful, don't miss out on more valuable insights and tips. Subscribe to to stay updated with the latest strategies and tools.

Subscribe now

Based on an article originally published at https://blog.logrocket.com on 23rd April 2024.

In today’s rapidly evolving digital landscape, the pressure on product leaders to build superior products is on the rise. User expectations for modern digital products are continuously increasing as well. Seamless user experience, smooth application performance, and high reliability are today’s baseline. But there is a deeper layer to this.

Users seek more and more easy access to data that provides them with meaningful insights and helps them make better data-informed decisions. This is where embedded analytics comes into play.

In this article you will learn what embedded analytics are, their key benefits, and the common challenges for implementing them within your product.

What is embedded analytics?

Embedded analytics enables users to gain data insights directly within the application they are using. This is done by providing data dashboards or comprehensive reporting functionalities to users.

Contrary to the traditional business intelligence (BI), embedded analytics solutions are integrated into the business applications that the users are already using. Essentially, the data analytics tool is an inherent part of the user’s primary application. This improves users’ experience, as the users do not need to login in a separate BI software — they can continue to use the same application they are familiar with.

Embedded analytics solutions have also found their way into news institutions, perhaps without you even realizing it. You have probably already come across websites that seamlessly embed data analytics solutions.

One example is The New York Times, which during the height of the COVID pandemic, employed embedded analytics to present data insights about Coronavirus cases, hospitalizations, and related data within their online articles.

Financial Times is another example. They use data visualizations to present the latest market data.

Importantly, the data analytics reports and visualizations are dynamic — the data presented is up to date, and is also interactive, allowing users for example to filter and explore the data themselves.

In essence, embedded analytics have quietly transformed the way websites or applications present and engage with complex information, enhancing the reader’s understanding and interaction with the content:

The role of embedded analytics in product management

This embedded analytics trend opens up new opportunities for product managers. The direct integration of analytics into your product offers a powerful way to enhance the value proposition of your product and elevate the experience of your users.

However, it’s important to recognize that while this technology solution is undoubtedly beneficial, it alone shouldn’t be the primary driver. The users and their experience should remain at the heart of the solution.

The role of the product manager remains centered on the user. This involves understanding user needs and expectations, the data insights they need, the decisions they confront, and understanding how data-driven insights can facilitate their decision-making process.

Once these insights are researched, you can consider embedding data insights into the product interface to empower users with real-time, actionable information. This goal of choosing embedded analytics as a solution should be to bridge the gap between raw data and meaningful insights, thus empowering users to make well-informed decisions within the natural rhythm of their tasks.

Key benefits of implementing embedded analytics

The advantages of incorporating embedded analytics are multifaceted, benefiting both users and the overall business. Embedded analytics aid:

• Data-driven decision making

• Enhanced user experience

• Competitive advantage for the product

• New revenue streams

Data-driven decision making

Leveraging data analytics, visualizing information, and generating insightful reports gives users valuable advantages. Embedded analytics bridges the gap between data collection and analysis, providing quick access to valuable insights. Users can promptly identify trends, patterns, and anomalies, and make timely actions and decisions informed by data.

Enhanced user experience

While leveraging analytics alone is a great advantage, embedding these analytics within the core business application takes it a step further. As this analytics is “embedded,” it’s integrated directly into the primary user interface. This allows users to effortlessly access real-time information without the need to toggle between disparate tools and without interrupting their workflow.

Competitive advantage for the product

Embedded analytics can bring a fresh look to your product, making it even more valuable and bringing a new competitive advantage. Empowering users with actionable insights becomes a compelling draw, attracting both new and retaining existing customers.

New revenue streams

Implementing embedded analytics can be a strategic choice to not only improve the product but also become a new revenue stream. For example, the new data-analytics module can be offered as a new add-on to the product with premium features.

Common challenges of implementing embedded analytics

While the benefits of implementing embedded analytics are substantial, they do come with some considerations. These are the most common challenges encountered during implementation:

• Integration complexity

• Data quality

• User experience design

• Security and privacy

Integration complexity

The process of embedding analytics requires technical adjustments within your application. This involves integration with external data providers and analytics components that must smoothly align with your application’s user interface. This technical implementation demands dedicated effort, potentially even affecting your product’s underlying architecture.

Data quality

Another common challenge is data quality. Embedded analytics heavily rely on accurate and reliable data. If the data used is incomplete, outdated, or inconsistent, the analytics results can be misleading. This damages the user experience and undermines the credibility of the embedded analytics feature. Maintaining data quality requires ongoing effort, such as data validation and ensuring that data sources are up to date.

User experience design

While embedded analytics can offer valuable insights, they might also introduce a learning curve for users who are not familiar with interpreting data visualization. Balancing the addition of new features without overwhelming users requires careful design and user testing to ensure that the analytics component enhances the product’s value without compromising its usability.

Security and privacy

Introducing embedded analytics involves handling new streams of data. Ensuring that the analytics process is compliant with data protection regulations can be challenging. Proper data encryption, access controls, and clear communication about data usage are essential to build and maintain user trust while leveraging the benefits of embedded analytics.

Strategies for integrating embedded analytics into a product

Embedded analytics is not just another product feature. Instead, product managers should view it as a strategic component that has the potential to reshape the product’s strategic trajectory. It can serve as a distinct competitive advantage, open doors to new revenue streams, or attract new and retain existing customers.

Integrating embedded analytics into a product requires a well-thought-out strategy to maximize its benefits and ensure effective implementation. Here are a few key strategies for successfully integrating embedded analytics in your product:

• Start with the users

• Choose the right analytics tools

• Select the right data visualization techniques

• Address data security and compliance

• Continuous data-informed improvement

Start with the users

Start the process by establishing clear objectives for embedding analytics within your product.

What are the specific goals and benefits you intend to deliver to your users? What is the strategic significance this solution will bring to your product? What are the types of data insights users require and the decisions they seek to inform? What visualization methods would be comprehensible for them? What is the desired level of interactivity within the visualizations?

Engage in user research, surveys, and feedback sessions to gain a thorough understanding of your target audience’s needs and preferences. Conducting this research is necessary before delving into the implementation.

Choose the right analytics tools

Once you know that embedded analytics is the way to go and you have a clear goal in mind, you need to select the right analytics tool and platform that aligns with your product’s goals and user requirements.

If you have unique and highly specific requirements that off-the-shelf solutions may not fully meet and the analytics is a big asset for your product, building your own embedded analytics tool can be the way to go. But in most cases, purchasing an existing embedded analytics platform is the right solution, as it significantly reduces development time and is more cost effective.

There are different analytics platforms on the market: Tableau, Sisense, Qlik, Looker, and Yellowfin are just a few.

Before selecting the right tool, it’s important to evaluate a few considerations:

• The ease of integration with your existing product and technology stack

• The connectivity the tool offers with other data sources and databases that you need to connect to

• The scalability and performance capabilities of the tool, especially if you expect high load for your product

• The security and compliance the tool guarantees

• The quality and variety of data visualization options; their user-friendliness, ease of use and customisation

Select the right data visualization techniques

Determine the data visualizations that will best convey insights to your end-users. Explore ways to align dashboard design with your brand identity for a cohesive look and feel.

You can provide users with the ability to personalize their analytics according to their preferences. For example, configurable dashboards, filters, and visualization settings, can be used to empower users to concentrate on the data that they find most important.

Selecting the appropriate visualization techniques predominantly fall within the scope of the UX designer. It should become integrated in the ongoing feature development process, incorporating feature refinement or user testing. However, establishing a set of guiding principles as a baseline can streamline the overall process.

Address data security and compliance

As your product gains access to new data, the potential benefits are accompanied by new security and privacy risks. Security should never be compromised. It’s crucial to ensure that the embedded analytics solution aligns with industry regulations and adheres to best practices in data privacy, security, and compliance.

First, choosing a reliable vendor is crucial. But the commitment to protect data extends within your organization as well. You need to set strict rules for how data is managed, track where the data comes from, who can access it. By following these rules, you make sure that any sensitive information is safely protected.

This careful approach not only keeps your embedded analytics working well but also makes users feel secure and confident.

Continuous data-informed improvement

To enhance your product effectively, start with small steps and refine over time. Instead of overwhelming users with complex data features at once, start with a simple approach and build on it gradually.

This approach allows users to become familiar with the new data tools and gives you the chance to gather feedback for ongoing improvements based on their input.

Integrating product analytics into newly introduced data visualization features is a wise move. It helps you understand how much users are adopting these features, how they are benefiting from the analytics, and where enhancements can be made.

Looking ahead: Future trends in embedded analytics

Embedded analytics is not a novel technology, it’s becoming the new norm today. As technology continues to evolve, we are likely to witness new advancements in this area.

One direction involves improved user-friendly interfaces and even more interactive and engaging visualizations. These enhancements are aimed at making data interpretation accessible to everyone, especially non-technical users. Furthermore, with AI playing an increasingly significant role in embedded analytics, users will be empowered to interact with data in an even more intuitive and conversational manner.

Another notable area of development is integration and connectivity. Seamless connectivity is becoming the new norm. Embedded analytics platforms will be easier to integrate with other tools and platforms, fostering an ecosystem where data flows effortlessly.

Given the growing prominence of AI in today’s technological landscape, predictive analytics is also on the rise. This entails analytics that not only showcase historical data but also predict trends and provide valuable insights into the future. Users will gain the ability to anticipate future trends, identify potential issues, and receive actionable recommendations based on data analysis.

Final thoughts

Embedded analytics is changing how we use data. It goes beyond being just a feature – it’s a new way of making data valuable for users. As technology keeps evolving, embedded analytics will continue to enhance user experiences by providing accessible and intuitive data-driven insights.

For product managers, this trend requires finding a balance between technical feasibility and the user needs. By embracing embedded analytics, product managers can make products smarter, more competitive and most importantly, can help users make better data-informed decisions.

Originally published at https://blog.logrocket.com on August 7, 2023.

Featured image source: IconScout

• Start using data to improve the product as early as possible in the product development.

• When improving the product, identify one most important goal and focus your efforts on that goal only. Split the goal in smaller subgoals if needed.

• Define metrics for the (sub)goal, and set a target state.

• Design a hypothesis to improve a metric, and test that hypothesis with a simple and cheap implementation. Continue with full implementation only if your test shows positive results.

• Once reaching the target metric, move to a new goal to improve.

Digital products today are becoming a necessity for almost any business. And customers are not only expecting you to have a digital product, but they do expect a very intuitive and seamless experience when interacting with the product. This bar is being raised higher and higher.

However, creating such a product that is intuitive and that resonates well with the users is not straightforward. And it's not a one-off activity. It is a matter of validation and continuous improvement over time. And this is where data and analytics can play a big role.

Product analytics is a data driven approach that allows one to observe how the product is used by the end user. Using product analytics, you can get detailed insights in how the users engage with the product. Are they using the new feature you've just created? Are they following the user journey as you expect?

And while in theory the potential of product analytics sounds exciting, using product analytics in the right way is not that easy. And the key challenge is identifying what to measure, and drawing conclusions based on the data without being overwhelmed with hundreds of metrics.

A 5 steps framework to improve your product continuously

How do you improve the product using product analytics? In this post we are going to describe a five steps framework that you can use to improve your product.

Step 1. Define your key goal

Needless to say, having clarity in your goal is the one most important thing. Once you are confident in your goal, the rest will follow much easier.

But how do you define the goal? Think about what is the key bottleneck that pulls your product back at the moment. Is it that the customer churn is high and engagement is low, or do you want to focus on growing the number of users?

Your goal depends very much on the stage of your product. If you have already proven market demand for your product, but your product is not yet optimal, you should optimise user engagement. If engagement is good, you can focus on improving growth. In their book, Benjamin Yoskovitz and Alistair Croll explain the 5 stages of lean startup growth: Empathy, Stickiness, Virality, Revenue and Scale. It's worth reading and trying to apply this thinking to your business.

Step 2. Select metrics to capture that goal

Next step is defining a metric that capture your goal. If you are already keeping a dashboard of various metrics, then identifying the goal and the goal metric would be much easier. That is the metric that is most critical, and that once improved, would make a big difference. If you don't have a dashboard of metrics, trust your knowledge and intuition but begin measuring as soon as possible.

You should be careful when selecting a metric. When the wrong metric is chosen, it can have a counter-effect.

For example, if you have a social media platform, a metric like average engagement time is a positive metric, because a higher value of the metric would indicate that the platform keeps the user more engaged. However, the same metric for an online grocery application might have a negative meaning. Higher value of the rating might be a sign of bad user experience as the user spends a long time to find the right products.

There is a theory about using a single metric only OMTM (One metric that matters). While the key idea behind this theory is valid, and that is - "focus is critical", following blindly this definition and oversimplifying can be risky. Because one metric only can give only one perspective of your business goal, and even more it is often too broad and not actionable.

So I would recommend to select one goal at a given moment and to focus on that goal. But to be able to make improvements towards that goal, you can choose multiple metrics to measure.

Step 3. Define a target

What is your target metric? If the average user engagement time at the moment is 10min per day, what is the target value you want to achieve? It can be a bit tricky to define a target, if you are new to this process or when you are building a new product.

You can research to find good targets for common metrics, like conversion rate, or retention rate in businesses similar to yours.

But even if you don’t have any best practice value, just give it a start. Set a target and once you start monitoring the metric, you’ll learn more and you’ll get a pretty good intuition on what is good and what is achievable for your specific product.

Step 4. Design a hypothesis

Having a goal without taking an action will not bring any changes. The next step is to formulate initiatives that you think will improve the metrics you’ve set. But remember that these initiatives are just hypotheses. You assume that these initiatives would move the needle, but you need data to prove that they work.

Best is to follow the Lean Startup approach. Think about how you can build the cheapest possible implementation of any hypothesis that would allow you to test that hypothesis.

Step 5. Test the solution and repeat

The best approach is to conduct an A/B test. Deploy the test to a subset of users, and measure if there is any difference between the old vs the new solution.

Observe and analyse the results. Does the change you implemented improve the desired metrics? If the results are positive, that's excellent, then continue with the full implementation of that change. If not, try to come up with another idea for a solution, develop a minimum solution and test the results again.

If at some point none of the initiatives work, you should rethink the goal you've set initially, and potentially pivot to a new goal.

Example:

With this approach, we can create a hierarchical structure. We begin by establishing the initial overarching goal, and then break it down into smaller, more manageable subgoals.

For instance, let's consider an online piano learning platform. The initial goal could be set as: "Increase the average time users spend on learning to play piano."

If that is the main objective, the next step is to determine how to improve this metric. How can we engage students and motivate them to learn more? To answer this, we could define two subgoal metrics:

1. Increase the average time per session by X%

2. Increase the average number of sessions per week by X%

For each of these subgoals, we can define initiatives.

For example, to improve the first subgoal, an initiative might be “shorten the songs the user should play". To improve the second goal, we can think of “set a learning schedule and send notifications to the user".

Whether these initiatives will indeed improve the engagement, we don't know, they are hypothesis. So we need to test first to validate if they really work. And to conduct the test, we need to be creative and find an easy and cheap way to validate the idea before making the full and expensive implementation.

Trust the process, it will get easier over time

This process might feel discouraging and challenging at the beginning. You might have questions like: How to select the key goal? Am I measuring the right things? Why don't I see any patterns in the data? Why should I spend time on testing the hypothesis instead of directly implementing it?

The process requires patience. Over time you'll learn more about the users, you'll define more accurate hypotheses, you'll have much more data so observing patterns will become much easier. So trust the process and enjoy the journey, in a few months you'll be glad you've made product analytics an integral part of your product development.

Product analytics and customer interviews are both product validation techniques. But how are they different? What are the pros and cons of these techniques? When do you use each of them and is using one of them sufficient?

Product validation techniques

When building a product, we rely on a lot of assumptions. We assume that our idea works, we design the product assuming that it is user friendly for the user, we select the features that we think users would value, etc. But if one of these assumptions is wrong, there is a high risk that the product will not be accepted by the end user.

This is why it is crucial to validate the assumptions continuously throughout the whole process, starting from the ideation phase and continuing even after the product is live in production. And during this process, we rely on different validation techniques.

Christian Rohrer categorises these validation techniques as follows.

First, we have Quantitative versus Qualitative techniques. A Qualitative validation technique involves collecting and analysing non-numerical data via conversational communication or observations. With this technique, we draw conclusions based on judgement and not on data. A Quantitative technique on the other side is a validation technique where analysis is based on data. This technique allows one to collect a higher number of inputs in order to get some statistical insights, analyse the results per user segment, or observe some trends in the data.

Another important categorisation of the validation techniques is in Behavioural vs Attitudinal techniques. With Attitudinal type of validation, we are not observing the actual behaviour of the user, but we ask the users to give their opinion "What would they do in a certain scenario". The result is their opinion, and not the actual behaviour. With a Behavioural technique on the other side, we observe the actual behaviour instead of asking the user for an opinion.

Customer Interviews

A customer interview (or user interview) is a discovery technique where we get insights about the customers by interviewing them. We ask questions to understand the Why. This makes an interview a Qualitative technique.

It’s true that we could quantify some of the results from an Interview, but the whole point of the interview is getting Qualitative answers. We typically ask open-ended questions and we go deep in answering the Why. So we can get a solid understanding of the user's intentions or needs, but as a drawback, it is difficult to apply this method on scale.

Another important characteristic of an Interview is that it is an Attitudinal approach.

We don't observe the actual behaviour of the customer, but we ask them to give their opinion about their behaviour. For example, we can ask them “would you be interested in buying a product?”. The answer is their opinion, but not their actual behaviour.

This is the drawback of the interviews. People’s opinion about themselves is not always their real behaviour. When we give our opinion, we would often answer in a way that we want to be perceived by the interviewer, or in a way to make the interviewer feel good. People’s opinions rely more on emotions and not on factual data.

For example, if you ask someone. “Do you exercise regularly?” The answer might be “Yes” because the person has trained twice in the past week and they feel good about it at the moment. But if you look at the data from the past few months you might see that they have exercised only twice per month on average.

Product Analytics

Product analytics is all about using data and getting insights out of this data. This makes Product Analytics a Quantitative technique. It allows us to collect a higher number of inputs, so we can conduct the validation on scale, get some statistical insights or observe trends in the data.

A Product Analytics is a Behavioural technique. This means that we set up a realistic scenario and we observe the real behaviour of the user. For example, we could measure how many of the users who are visiting the page click the “Buy” button, instead of how many think they would buy the product (as it was the case with Attitudinal techniques).

Product Analytics vs Customer Interviews - Pros and Cons

So let’s summarise these techniques.

Product Analytics is a Quantitative and Behavioural technique. This makes it easy to scale. It allows more advanced statistical analytics. Results are fact-based and not opinion-based. However, to trust the data, it is very important that the data collection is carefully designed and well thought out. Another important disadvantage of Product Analytics is that with data we cannot always capture everything, especially the deeper Why questions. Although data techniques are becoming extremely powerful, there are still limitations.

An Interview is a Qualitative and Attitudinal technique. As such, they allow us to go deep into understanding the Why and the mind of the customer, and get the answer on very specific questions that are otherwise difficult to capture answer by data. But the limitation of Interviews is that they are opinion-based, and they are difficult to conduct on a large scale.

Do you need both Interviews and Product Analytics

Can one of these techniques replace the other one? Do you still need Product Analytics if you are already interviewing? Or vice versa?

The best approach is to combine diverse validation techniques. Using both Product Analytics and Interviews is much more powerful, because these techniques complement each other.

You should use Product Analysis to continuously monitor and observe the behaviour of the user. But data can sometimes be misleading or confusing. This is when you want to involve interviewing, where you select a targeted group of users, with whom you go deep in understanding the Why behind the data. So you apply Product Analytics on scale, and you apply Interviewing periodically to address specific questions to a targeted audience.

Best decisions are not data-driven but data-informed, they use a blend of both data and intuition.

• Product analytics is an integral component for the success of the product. It is about using data to understand and improve the way how the users engage with the product.

• Product analytics helps product teams understand better the users and their behaviour, measure progress, validate if new ideas work, and make decisions of better quality.

• Product analytics should be applied as early as possible in the product development, even before the new product or feature has been built.

• Best product companies make data-informed and not data-driven decisions.

Product teams are making so many decisions every single day. Some are more strategic and determine the direction of the product, for example which new product or feature to build. Others are more tactical, for example a decision of how to design that new feature, or which colours or layout to use, etc. The success of the product is largely determined by the quality of these decisions.

If these decisions are based only on assumptions, there is a huge risk that the product will end up with a failure. We build the product with the assumption or the hope that it will bring value to the users. But would that actually be the case? Will the users find the product attractive and engaging? Will they be using the features in a way that you expected?

Wouldn't it be great if we can get such insights and read the minds of the users, so we can make better decisions on how to improve the product? This is where product analytics comes into play.

What is product analytics?

Product analytics is an integral component for the success of the product. It is about using data to understand and improve the way how the users engage with your product.

Product analytics can provide all kinds of information you need to understand how users use your product. Which features are they using, how do they interact with the product, what obstacles do they face when using the product.

Let’s take a user journey of an e-commerce application as an example. The user creates an account, signs in, then searches for a product, uploads a photo of a desired item, then searches a similar product, buys the product, and then leaves the application.

Using product analytics you could not only measure how many users buy a product at the end, but you could go in detail to learn how they engage with the product throughout their journey.

• What products are most searched? Do they maybe use the price filter mostly so price matters most to them?

• Are they using the new feature you’ve just created - searching by uploading a photo of a product similar to the desired one?

• Do they spend a lot of time searching for the right product?

• Do they find a product they like?

• And you could answer such questions even for different segments, like user age, or user location.

Product analytics helps you get data for each of these touch points in the user journey. It enables you to maintain a dashboard of important metrics, so you can visualise the performance of your product, identify bottlenecks in the journey and work on resolving them.

Product analytics before the product or feature is live

Product analytics is valuable to understand how users are using your product features, so you can evolve and improve the product continuously. But it is even more valuable to use this technique to test new features before they are actually built. This is the key to successful product development. Testing and validating as early as possible, and failing fast before you spend time and money on building a feature that is not going to be adopted by the users.

This is how most successful companies and product teams work. Experimentation is the integral part of their culture and their processes. New ideas are tested first. Only if the data shows positive results, a solution for these ideas is built. Because most ideas don’t work when applied in practice. Even the ideas of the most successful product managers or business leaders are often wrong. This is why experimentation is crucial.

So how to involve product analytics during the product development? The process looks roughly like this.

1. The product team defines a clear measurable hypothesis.

2. Instead of building a complete solution, they would think creatively to find a simple way to experiment and test this hypothesis.

3. They would compare the results of the current product vs the product including this idea by using A/B testing.

4. If results are positive, they would continue implementing the full solution.

Should you start using product analytics for your product?

Yes, you should. Actually, you should have already started. Because otherwise you are targeting a goal with closed eyes.

It’s true that you can get much better data insights when product analytics is applied on a scale, when the product is more mature and the number of users is higher. But the earlier you start the more you can benefit. Think about how to use product analytics as early as possible, even from the Ideation phase, before you start with the actual delivery of the product feature.

Make data-informed and not data-driven decisions

While it's critical to use data and support decisions with data-driven techniques, following blindly the data can also be a risk. We also need intuition to design a data-driven test, to interpret the results of that test, or to know if we can trust the results. We need the opinion of our users, who can share their thoughts or emotions about why they are using the product in the way they are using, insights that not always can be captured by data.

Best decisions are not data-driven but are data-informed, they rely on a blend of both data and intuition.