Are You Trusting Faulty Data? The Margin of Error in Usability Testing
What Is Margin of Error and How to Apply It for More Accurate Usability Testing
In usability testing, numbers don’t lie—but misinterpreting them can lead to costly mistakes. Imagine launching a major redesign based on test results, only to realize later that your conclusions were skewed.
One critical factor that’s often misunderstood or overlooked? Margin of Error—the key to knowing how much you can truly trust your usability insights.
In this article, we’ll break down:
🔹 Why no usability testing results is 100% accurate?
🔹 What is Margin of Error (and why it matters)?
🔹 How to balance precision and cost in usability testing?
🔹 Practical steps to get the most accurate usability insights
Your Data Is Never 100% Accurate
Let’s say you’re in the vehicle leasing business and want to find out what percentage of Amsterdam residents have leased a car in the past two years.
Will you survey every single resident in the city?
No. You’ll take a sample—a smaller, random group representing the whole population.
And because you're only working with a portion of the population, your results can never be 100% accurate. They’re always an estimate of the true value, with some degree of uncertainty—known as the Margin of Error. A margin of error tells you how much your data could be off.
Key Statistical Concepts in Usability Testing
To make sense of your usability test results, you need to understand three key statistical concepts: Confidence Level, Margin of Error, and Confidence Interval.
1. Confidence Level
The confidence level is a value you select that reflects your tolerance for uncertainty.
A higher confidence level means you want to be more certain, so you’re less willing to risk being wrong.
A lower confidence level means that precision is less critical, and some uncertainty is acceptable.
There a few standard confidence levels used:
95%: This is the most common choice in usability testing and research. It balances certainty and practicality.
99%: Used in high-stakes situations where errors could have serious consequences. An example is testing the effectiveness of a new drug in medical research.
90%: Used for exploratory or early-stage testing, where early insights are more important than precision.
2. Margin of Error
The margin of error is a percentage that shows how much your sample results might differ from the true value.
Unlike the confidence level (which you choose), the margin of error is derived from your data and depends on factors like the sample size, variability in the data, and your chosen confidence level.
A smaller margin of error means your results are more precise.
A larger margin of error means your results are less precise.
For example, if the result of your testing is 60% (let’s say 60% of the surveyed participants have answered ‘Yes’), with a margin of error of 3%, it means the true value is likely within ±3% of your result (between 57% and 63%).
3. Confidence interval
Once you have the margin of error, you can easily calculate the confidence interval—the range within which the true value is expected to fall.
The confidence interval is expressed as:
[X-margin of error - X+margin of error].
In the example above, if result is 60% with a margin of error of 3%, the confidence interval would be [57% - 63%]
4. Putting It All Together
Let’s see how confidence level, margin of error, and confidence interval work together in real scenarios.
Example 1: Vehicle Leasing Survey
Chosen Confidence Level: 90%
Survey Result: 60% of participants reported leasing a vehicle.
Calculated Margin of Error: ±3%
Calculated Confidence Interval: [57%, 63%]
How to Interpret This:
With 90% confidence, we can say the true percentage of people in the overall population who leased a vehicle is between 57% and 63%.
This means that if we repeated the survey 100 times, the results would fall within this range 90 times out of 100.
Example 2: Usability Testing—Task Completion
Chosen Confidence Level: 95%
Testing Result: 80% of participants successfully completed the given task.
Calculated Margin of Error: ±2%
Calculated Confidence Interval: [78% - 82%]
How to Interpret This:
With 95% confidence, we can say the true percentage of users who would successfully complete the task is between 78% and 82%.
This means that if we conducted the usability test 100 times, the true result would fall within the range [78% - 82%] 95 times out of 100.
Factors That Influence Margin of Error
Let’s dive into the components of the margin of error formula to understand what drives it and how it behaves in different scenarios.
The error margin is a function of three parameters:
Z (Z-Score):. This is fixed number calculated directly from the selected confidence level. For the most commonly used confidence level, we have
At a 95% confidence level, Z=1.96.
At a 90% confidence level, Z=1.65.
At a 99% confidence level, Z=2.58.
n (Sample Size). This is the number of participants in your usability test or survey.
p (Sample Proportion): The percentage or fraction of people in the test who meet the specific condition or answer the researched question positively.
Example: If you're testing a feature with 100 users and your goal is to observe how many will complete a certain task, and 60 of them successfully complete it, the sample proportion is 60% (p = 0.6).
Now with the formula in mind, this is what influences the margin of error.
📌 Higher Confidence Level increases the Margin of Error
If you choose a higher confidence level (e.g., 99%), the margin of error increases.
For example, if your result is 60%, and at a 95% confidence level, the margin of error is ±2%. This means you can be 95% confident that the true value falls within the range [58%, 62%].
However, if you want to express the same result with 99% confidence, you’ll need a wider range to account for the additional certainty. This results in a higher margin of error, meaning the range might expand to something like [57%, 63%] to include the true value with greater confidence.
📌 Higher Sample Size means lower Margin of Error
A larger sample size reduces the margin of error because it provides a more accurate representation of the target population. With more participants, the results are closer to what you’d get if the entire population were tested.
On the other hand, a very small sample size increases the margin of error, making the results less reliable.
📌The Variability of the Result Affects Margin of Error
The variability in your results, represented by the sample proportion (p), directly influences the margin of error. The value p*(1−p) is highest when p is around 50% and decreases as p moves closer to 0% or 100%.
For example, if 50% of users complete a task successfully, there is an equal chance of success or failure, leading to higher variability and a larger margin of error. However, if the result is more extreme—say, 90% or 10%—the outcome is more predictable, reducing the margin of error.
How to Improve Your Usability Testing Accuracy?
How can you achieve high precision in your results and reduce the margin of error?
While some factors are outside your control, like the confidence level (determined by the context of your work) or the variability in your data (the sample proportion p), there are two key steps you can take to improve your results:
1. Eliminate Biased Data
The margin of error formula assumes your data is unbiased.
What does that mean?
If you want to find out what percentage of Amsterdam residents have leased a car, it’s a bad idea to only survey people working at a car leasing company.
Why? The data would be biased, as this group is much more likely to lease cars than the general population.
Select your sample randomly so that every individual in the population has an equal chance of being included. Make sure your sample isn’t skewed toward a specific group that doesn’t accurately represent the entire population.
Without a representative sample, even a large sample size or high confidence level won’t give reliable results.
2. Adjust the Sample Size
The second important factor is the sample size.
A larger sample size reduces the margin of error and narrows the confidence interval, making your results more reliable.
Let’s say you’re working with a population of 10,000 users, a confidence level of 95%, and a sample proportion of 80%. Here’s how the margin of error decreases as you increase the sample size:
But there’s a trade-off: larger samples require more resources, time, and cost.
Testing the entire population would provide perfectly accurate results, but it’s unrealistic and unnecessary.
Balance resources and accuracy.
There isn’t a one-size-fits-all answer, but here are some tips to guide you:
For exploratory research or initial usability tests, aim for a margin of error around 7-10%.
For key decisions like product launches or user satisfaction surveys, aim for a margin of error between 2-5%.
For high-stakes research, where precision is critical (e.g., medical testing or compliance studies), aim for a margin of error of 1-2%.
Practical Tips for Usability Testing
When conducting usability testing, these practical tips can help you maximize reliability and make the most of your resources:
Use a Sample Size Calculator: Many online tools are available to help you calculate the margin of error and determine the ideal sample size for your study. Here is an example of such tool. Use these calculators to experiment with different confidence levels, sample sizes, and error margins to understand how they impact your results.
Report with Transparency: Always include error margins and confidence intervals in your reports. Providing stakeholders with this information ensures they understand the reliability and limitations of the results, avoiding overconfidence in findings.
Prioritize Key Metrics: If resources are limited, focus on the most critical metrics only and ensure they are supported by adequate sample sizes.
Leverage Iterative Testing: Start with smaller, exploratory tests, identify major issues, then scale up with larger tests to refine your findings. Iterative testing saves time and resources while helping you address the most significant problems early.
Key Take Aways
If your usability tests ignore the margin of error, it's time to rethink your testing strategy.
You don’t need to be a statistician, but understanding a few principles you will significantly improve your outcomes:
📌 Use a representative sample—Ensure it's unbiased and large enough for reliable results.
📌 Confidence comes at a cost—A higher confidence level increases the margin of error, meaning you’ll need more data for precision.
📌 Balance accuracy with resources—Prioritize key metrics ensuring their accuracy.
📌 Report transparently—Always include margin of error and confidence intervals to avoid misleading conclusions.
A version of this article was originally published at https://www.logrocket.com/ on Feb 18 2025.
Enjoyed this read? Subscribe to Lean Product Growth for regular updates on building and scaling a successful product organization.