Sample size and confidence: how to get your team to trust qualitative research

Originally published on the GDS User Research Blog

If you’re doing qualitative design research, don’t worry about sample size. Sample size and statistical significance don’t matter*.

The only thing that matters is how confident your team is about the next decision they need to make** 

You might get that confidence from watching 2 people.

You might need to watch 20.

You might need to watch 20, then check your hypothesis at a larger scale with a survey or analytics.

It all depends on the confidence of your team.

There are only 2 things you need to do to start building confidence in your next decision:

  1. Start watching end users using the thing you’re making. (Do research in every iteration.)***
  2. Make sure enough of your team observe the research. (Research is a team sport, help your team get their exposure hours.)

As soon as you start doing this, you’ll be so busy working out what to do with everything you’re learning, you’ll have no time to worry about statistics.

You’ll know what is broken and what is not, and you’ll know what you need to spend more time learning about.

It’s as simple as that.


* Obviously it’s better to see a lot of people over very few

We recommend you see a handful in each iteration – keep iterating, and you’ll gradually get a larger and broader sample. Also, the ‘Is 5 people enough?’ question often discussed by usability people is irrelevant when you’re seeing 5 people in every iteration.

** Your next decision might be…

  • How do we fix the question in this form so that people can answer it better?
  • Is this service meeting the user needs?
  • What is the next most important thing we should prioritise in the backlog?
  • Does this work better than what we’ve currently got live?

*** Who do you watch?

It’s tempting to launch into a complicated segmentation exercise. Don’t do that. There will be some obvious segments – just start with those. Start talking to people and the segmentation that really matters will emerge from those discussions.

The important segments will sometimes be things you couldn’t have guessed at before you started meeting people. Start talking to the easiest people. Pick the people who are most likely to be able to (and want to) use your service and test with them. If your service doesn’t work for them, it won’t work for anybody.

As you begin to solve problems, make it harder for yourself by inviting people who are going to be more difficult – those with edge case and the complex user needs and those with lower (or no) digital literacy.