Comments on: Embracing the Un-Science of Qualitative Research Part One – Small Sample Sizes are Super

By: JC

JC — Thu, 11 Jun 2009 07:29:33 +0000

Hi Leisa
I totally agree upon the idea of small sample groups when conducting qualitative research. As you said, the rest will become repetitive and redundant, and more time should be spent on further research that complements the previous one.
The reason I am reading your blog is that it’s a requirement for my Master’s program in Educational Technology at MSU, but I’m enjoying it nevertheless:-)

By: photo48 » Embracing the Un-Science of Qualitative Research Part One - Small …

Mon, 22 Oct 2007 15:37:24 +0000

[...] sourced here [...]

By: disambiguity - » Embracing the Un-Science of Qualitative Research Part Three - Improvising is Excellent

Sat, 11 Aug 2007 21:36:24 +0000

[...] Use a relatively large sample size (which we destroyed here) [...]

By: magia3e

magia3e — Sun, 05 Aug 2007 23:16:39 +0000

As a former lecturer in statistical methodology for psychology it’s interesting how many people make mistakes in the choice and use of tools like statistical analysis.

When testing children for developmental problems, like autism or ADHD, giving a child an intelligence test like the WISC will give good indications of where (statistically) the problems lie. What it won’t tell give you an understanding of the child and his problems — only the tester’s observations can do this. And when it comes to telling the parents, handing them numbers won’t relate the issues — this is where the qualitative measures are critical.

Only qualitative measures are going to give the numbers any real meaning. Only qualitative and quantitative measures together are going to give you the whole picture.

By: Christopher Schmitt

Christopher Schmitt — Wed, 25 Jul 2007 20:30:15 +0000

[...] Embracing the Un-Science of Qualitative Research (tags: research statistics testing Usability experience user) [...]

By: Qualitative Research » UIE Brain Sparks

Qualitative Research » UIE Brain Sparks — Wed, 25 Jul 2007 17:05:07 +0000

[...] You can read part 1 here: Embracing the Un-Science of Qualitative Research Part One – Small Sample Sizes are Super [...]

By: Alex Nisbett

Alex Nisbett — Wed, 18 Jul 2007 19:44:00 +0000

I’d say the numbers are bang on Leisa. We recently did some fairly early user feedback on service propositions, articulated as user scenarios, in two distinct locations and we reported back on exactly six participants from each session. To be fair, we were specifically testing ‘desirability’ through usefulness but I figure the same guidelines apply.

What often makes for even better client buy-in (or, to put it another way, de-risking the whole process) is if you can meet with the same ‘users’ you saw at the beginning of the project – if you’re lucky enough to have carried out empathic research.

Key of course is in selecting your participants – not to slant the results by only choosing ‘friendlies’, but a balanced group of differing skills and abilities, a ‘cross-section’ of society if you will.

Not that I’m an expert on usability – I didn’t make it anywhere near David Armano’s Top Ten Names in User Experience, unlike some people around here!

By: Michael Clarke

Michael Clarke — Mon, 16 Jul 2007 15:42:26 +0000

Actually, I have issues with applying statistics to something as rich and variable as user behaviour and meeting user needs and expectations. Aside from that (yes, I’m a convinced relativist most days of the week), I’d say that you’re more likely to pick up small but potentially disruptive-in-the-long-term problems from detailed work with small samples than large surveys. I’d contend that large qualitative surveys frequently only tell you what you’ve already decided is likely to be the case if you look at how hard it is to avoid questions that telegraph or construct the ‘appropriate’ answers.
Now ‘positivist’ data does have it’s place – and that may well be in the area of working with user stats (another advantage of “release small and release often”).

By: Michael Zuschlag

Michael Zuschlag — Mon, 16 Jul 2007 13:26:41 +0000

For the purpose identifying common usability problems, such small sample sizes are perfectly valid from a statistical perspective. One approach to convincing a skeptical client of that without going into the math is to point out that if problem exhibits itself in a large portion of your users, you’re very likely to see it in a small sample; if the problem exhibits itself in a very small portion, you’re very unlikely it see it in a small sample. Small samples, in other words, are a wide-mesh net, convenient for likely catching only the big problems –which are often the ones the client primarily cares about.

Now you can point out to the client that if one out of a mere five test users have a certain problem, you can’t say with high certainty whether that means 10%, 20%, or even 50% of the total population of users will have that same problem. But as Reichelt implies, in many situations it’s more important to spend your time and money on detailed qualitative interviewing and observing of that user to find out _why_ she or he is having a problem than on getting a more precise fix on the size of the problem.

By: disambiguity

disambiguity — Sun, 15 Jul 2007 15:00:35 +0000

[...] Embracing the Un-Science of Qualitative Research Part One – Small Sample Sizes are Super [...]