Customer Interviews or Product Analytics

Product analytics and customer interviews are both product validation techniques. But how are they different? What are the pros and cons of these techniques? When do you use each of them and is using one of them sufficient?

Product validation techniques

When building a product, we rely on a lot of assumptions. We assume that our idea works, we design the product assuming that it is user friendly for the user, we select the features that we think users would value, etc. But if one of these assumptions is wrong, there is a high risk that the product will not be accepted by the end user.

This is why it is crucial to validate the assumptions continuously throughout the whole process, starting from the ideation phase and continuing even after the product is live in production. And during this process, we rely on different validation techniques.

Christian Rohrer categorises these validation techniques as follows.

First, we have Quantitative versus Qualitative techniques. A Qualitative validation technique involves collecting and analysing non-numerical data via conversational communication or observations. With this technique, we draw conclusions based on judgement and not on data. A Quantitative technique on the other side is a validation technique where analysis is based on data. This technique allows one to collect a higher number of inputs in order to get some statistical insights, analyse the results per user segment, or observe some trends in the data.

Another important categorisation of the validation techniques is in Behavioural vs Attitudinal techniques. With Attitudinal type of validation, we are not observing the actual behaviour of the user, but we ask the users to give their opinion "What would they do in a certain scenario". The result is their opinion, and not the actual behaviour. With a Behavioural technique on the other side, we observe the actual behaviour instead of asking the user for an opinion.

Customer Interviews

A customer interview (or user interview) is a discovery technique where we get insights about the customers by interviewing them. We ask questions to understand the Why. This makes an interview a Qualitative technique.

It’s true that we could quantify some of the results from an Interview, but the whole point of the interview is getting Qualitative answers. We typically ask open-ended questions and we go deep in answering the Why. So we can get a solid understanding of the user's intentions or needs, but as a drawback, it is difficult to apply this method on scale.

Another important characteristic of an Interview is that it is an Attitudinal approach.

We don't observe the actual behaviour of the customer, but we ask them to give their opinion about their behaviour. For example, we can ask them “would you be interested in buying a product?”. The answer is their opinion, but not their actual behaviour.

This is the drawback of the interviews. People’s opinion about themselves is not always their real behaviour. When we give our opinion, we would often answer in a way that we want to be perceived by the interviewer, or in a way to make the interviewer feel good. People’s opinions rely more on emotions and not on factual data.

For example, if you ask someone. “Do you exercise regularly?” The answer might be “Yes” because the person has trained twice in the past week and they feel good about it at the moment. But if you look at the data from the past few months you might see that they have exercised only twice per month on average.

Product Analytics

Product analytics is all about using data and getting insights out of this data. This makes Product Analytics a Quantitative technique. It allows us to collect a higher number of inputs, so we can conduct the validation on scale, get some statistical insights or observe trends in the data.

A Product Analytics is a Behavioural technique. This means that we set up a realistic scenario and we observe the real behaviour of the user. For example, we could measure how many of the users who are visiting the page click the “Buy” button, instead of how many think they would buy the product (as it was the case with Attitudinal techniques).

Product Analytics vs Customer Interviews - Pros and Cons

So let’s summarise these techniques.

Product Analytics is a Quantitative and Behavioural technique. This makes it easy to scale. It allows more advanced statistical analytics. Results are fact-based and not opinion-based. However, to trust the data, it is very important that the data collection is carefully designed and well thought out. Another important disadvantage of Product Analytics is that with data we cannot always capture everything, especially the deeper Why questions. Although data techniques are becoming extremely powerful, there are still limitations.

An Interview is a Qualitative and Attitudinal technique. As such, they allow us to go deep into understanding the Why and the mind of the customer, and get the answer on very specific questions that are otherwise difficult to capture answer by data. But the limitation of Interviews is that they are opinion-based, and they are difficult to conduct on a large scale.

Do you need both Interviews and Product Analytics

Can one of these techniques replace the other one? Do you still need Product Analytics if you are already interviewing? Or vice versa?

The best approach is to combine diverse validation techniques. Using both Product Analytics and Interviews is much more powerful, because these techniques complement each other.

You should use Product Analysis to continuously monitor and observe the behaviour of the user. But data can sometimes be misleading or confusing. This is when you want to involve interviewing, where you select a targeted group of users, with whom you go deep in understanding the Why behind the data. So you apply Product Analytics on scale, and you apply Interviewing periodically to address specific questions to a targeted audience.

Best decisions are not data-driven but data-informed, they use a blend of both data and intuition.