March 9, 2016

How many users should you interview?

In studies, the principle is simple: the less you know, the less information you need to learn to reduce your uncertainty and thus improve your decisions. However, the more information you have (thanks to regular testing, for example), the more you will need new data to learn something new about your site or mobile app.

Paradoxically, a magic number comes up regularly when talking about user tests. That number is 5.


Is five users enough for user tests?

More than 20 years ago, Jakob Nielsen demonstrated that it was entirely acceptable to simplify the methods of the time and he laid the foundations for less expensive user and simpler to organise tests – in short, accessible to more decision-makers.

In his benchmark article, he explains how interviewing only 5 users allows you to identify 85% of problems (or, more precisely, 85% of the problems which users have a 1/3 chance of encountering).



Source: Nielson Norman Group

Instinctively, we might well perceive that this figure of 5 users as insufficient in an increasingly complex mobile environment. How could 5 people possibly reveal 85% of the problems that a site or app could have on a PC, smartphone, tablet, iOS, Android, Windows Phone, etc.?

In fact, numerous researchers have demonstrated that this figure of 5 users is a simplification and not a universal response. How can we know how many users to use in tests?


A practical guide to the number of users to research

To go beyond the 5 users principle, here are a few thresholds that are both simple and applicable to various stages of your project:

Your situation

How many users?

Why is this number better than the previous limit?

What type of solution is needed to get it started?

I’m just kicking off, it’s a blank page.

1

You will learn something from every user.

“Get out of the building”. Speak to users yourself.

It’s my first test.

5-15

You can identify 85% of the problems encountered.

Qualitative research institution

I need to specify my optimisations.

30

You can identify these issues*

Crowdtesting platforms

I know a lot – a gain of just 0.1% would be beneficial.

100-1000

You can measure your known issues precisely (but you lose the exploratory aspect)

Quantitative research institution / AB test

I’m constantly perfecting

Everyone!

You are Facebook or Twitter!

Big data


* For this, we use the normal law, 30 individuals allows you to calculate frequency and percentage. You can therefore see the problems that arise more often than others.


The Ferpection user tests, data in qualitative research

A user testing platform like Ferpection gives you the flexibility to tackle all of the stages of qualitative research in total peace, from 1 to 100 users.

When your projects are just kicking off, a very small number of users, between 1 and 3, is enough to improve your knowledge. It is important to interview them yourself. People speak of great revelations when they observe users. The Ferpection platform, , thanks to its technology, allows you to gather this feedback and assist in your analyses.

As the organisation becomes too cumbersome internally and you want to take advantage of expertise in user tests, you can entrust the entirety of this testing process to a research company or a crowdtesting platform. At the heart of this second category, Ferpection stands out by allowing you to test samples of 5-15 people quickly and flexibly, as well as going beyond these benchmarks.

We suggest to our clients to regularly conduct research among 30 users per segment as it allows them to go from a list of problems to a list of problems ordered by importance. Why settle for a list when you could prioritise your issues and make the data talk?

To compare various surface areas (PC vs. mobile, men vs. women, etc.) as in the graphic below, it is important that each area includes at least 30 individuals.


While the explosion of mobile technology has led us to reconsider the magic number of 5 users, crowdtesting brings us exciting possibilities to adapt and improve our ability to make decisions and attract users.



| |

En savoir plus sur les tests utilisateurs
Author
Thibault Geenen

Thibault Geenen

Thibault est fan de la méthode lean start-up d'Eric Ries (qu'il applique avec beaucoup de sérieux) et faire des blagues que ses jeunes protégés ne comprennent pas toujours. Grand amateur de Science Fiction, il a récemment écrit un article faisant un lien entre entreprenariat et le film 'Seul sur Mars'.