Embracing the Un-Science of Qualitative Research Part One – Small Sample Sizes are Super

If you’re into qualitative research at all, it wouldn’t have taken long before you had someone ask you about the statistical significance of your research and how you could back your findings with such a small sample size, or to find others out there trying to make qualitative research look more scientific by trying to extract hard data from it.

There are three main ways that you can try to make qualitative research look more scientific, being:

  1. Use a relatively large sample size
  2. Ensure that your test environment doesn’t change
  3. Ensure that your test approach doesn’t change (don’t change the script, and stick to it)

Now, there are some times when one or more of these tactics is appropriate, but conversely, in many instances it has been my experience that by breaking these rules, you are able to get much greater insight into the research question(s) you have set yourself.

There are many different kinds of qualitative research study, so in the interests of clarity, let’s pick one just like I’ve been working on this week – a lab based combination of interview & a wee bit of usability which is intended to ensure that my client’s proposition is sound, that it is being well communicated, that the users understand what the service is and how it works, and to weed out any critical usability issues.

In the interest of not making you read an enormous post, I’ve divided this into three parts. So, let’s start with part one – a large sample size. Now… to the best of my knowledge there is no scientific way to determine the correct number of participants in a qualitative research study. Now, I’m no statistician (if you are, please feel free to weigh in here), but it is my understanding that the likelihood of reaching a statistically significant result using the methodology I’ve described above, is pretty much nil. Not that it’s impossible, but you’d have to do a heck of a lot of interviewing.

And here’s one golden rule of qualitative research that always holds true – if the research is going to take too long or be too expensive, it will not happen. You can count on that one.

As a result, sample size for qualitative research is often driven by the time and budget available – and that’s not necessarily a bad thing. In fact, this is one subject upon which Jakob Nielson and I actually quite agree. Jakob says that most of the time elaborate usability testing is a waste of time and that you should test with no more than five users. He has a natty little graph that illustrates why this is so:

Problem Finding Curve

As you can see – by the time you’re up to five or six users, you’ve gotten to the bottom of most of the usability issues, and from then on you spend more time repeatedly seeing what you’ve already seen before and uncovering very few new findings. In my experience – this is as true for other aspects of research as it is for usability.

I would add a caveat which is that if you have user groups that are quite divergent in their attitudes, experience, or requirements/goals etc. you will want to ensure that you apply this rule to each of those groups. So, for example, if you have an audience of ‘buyers’ and an audience of ‘sellers’ you’ll want to get no more than five each from each key audience. One final caveat – when I say no more than five, I also say no fewer than three (and, what do you know, so does Jakob). You need at least three to identify what are actually patterns from those things that are just personal quirks – because that’s what you’re looking for here – the patterns.

Is it scientific to use such a small user group? If you want to make it look that way, you can look to Jakob for some algorithms and graphs. In my experience – it doesn’t matter whether it is scientific or not. The richness of the information and insight you receive even from this small sample size makes the return on investment enormous – and the small sample size makes it an activity that almost any project can incorporate into their timeline and budget. At the end of the day – those things are far more important than scientific validity.

Is it worth doing qualitative testing with only a small sample size? Absolutely yes. In fact, in many ways, this is the best way to do this research. Qualitative research is not about numbers, it is about the richness of the information and insight you can get access to by spending time with the people who form your audience (or potential audience), and looking for patterns in their reactions and responses.

In many cases, increasing the size of your sample so that it seems more ‘valid’ is a waste of time and money as the later interviews become more and more a repetition of finding you’ve already identified and confirmed. This time and money could be much better spent improving your product and conducting another round of research.

If it’s numbers you’re after – go do a survey. I say embrace and defend the small sample size of qualitative research.

What say you?

(Coming soon: Part Two – Ever-Evolving Prototypes are Ace)

Innies and Outies


I’ve been thinking on this a little bit lately. I’m just about to make some more changes at work, and I have to admit, for a while there I was toying with becoming an ‘innie’.

There is something quite seductive about having the access to resources that innies have that outies never really get. Especially if you work somewhere really big. You might also get to work on stuff that *really* matters, that makes a difference to lots of peoples lives every day. This is pretty powerful stuff.

But… at the end of the day… you only get to work on more or less the same stuff, month after month. You can look forward six months into the future and have a fairly good idea of what you’ll be doing. This, for me, is the downside.

That, and I was never really sure that I wanted to back one company hard enough to commit to them with full time employment. (And they say men are the ones with commitment issues!)

Yes, I’ve been an outie almost all my career. I started off as an innie, but that job ended with a retrenchment (which I think was a good thing and I’m pretty sure didn’t scare me off the innie thing). And I’ve just chosen, again, to be as outie as you possible can be – to freelance.

(For those of you *completely* lost by this innie/outie business – an ‘innie’ is someone who works within a company as an Information Architect, Interaction Designer, User Researcher, etc. An ‘outie’, on the other hand, consults or works for a consultancy, and does IA, IxD etc. on a project by project basis usually as contracted by the large company.)

Are innies and outies different kinds of people? I think, perhaps, they are.

Innies must surely have a lot of patience for internal politics. In many cases, they have a slow moving ship they need to gradually turn around. They need to work very hard, often, to make the business aware of their presence and their importance, and to be able to get involved with the decision making early enough to do their job. They need persistence. They need to be happy to work with the same people for long stretches, even if those people cause them great frustration. They need to be able to deal with bureaucracy that often flies in the face of what they are trying to achieve.

Where innies must manage lack of change, many outies have the opposite problem – never ending flux. They not only have to adapt to new projects but often whole new industries every few months. New teams of people to work with, new sets of politics, new priorities, new objectives, new obstacles, new challenges. Often times they have the same challenges as innies, but they know that these will go away as this project ends and the new one commences. And they can always just fire the client if it gets extreme. Outies, as a rule, require more developed ‘consultancy’ skills – the ability to ‘manage’ clients, to sell ideas, to gain confidence in their abilities and their approach.

I was recently at the Enterprise 2.0 conference in Boston where I’d been invited to speak. There were a lot of ‘innies’ at that conference. A lot of big company innies. I have to say, by and large, it made me happy to be an outie. I felt that being an outie makes me more agile, more connected, more responsive – I feel a drive to keep in touch with what the rest of the world is doing, where I got the distinct feeling that there was a lot more navel gazing (sorry) going on amongst the innies, and that when they did look outwards, they never looked too far.

Sure, their projects may be the really large important ones. They may might be building a space shuttle. But perhaps some part of the innovative work that I was able to do on a much less ‘important’ project will one day feed into the design of a very important project. Who knows… maybe one day they’ll outsource the space shuttle? :)

It is my suspicion that, even if you have worked as both an innie or an outie, you know which one of these you *really* are. That you’re more one than the other.

I suspect that having the experience on both sides is a very valuable thing. Perhaps I should do more ‘innie’ work. But, at the end of the day… I’m most definitely an outtie, and that’s how I do my best work.

How about you?

Image Credit: Mr Truffle @ Flickr

Remind me … what’s so great about Omnigraffle?

For many years, as the groundswell towards Mac has gathered pace, I’ve had to endure many of my colleagues scoffing at the fact that I continue to use Visio when they’ve seen the light and made the move to Omnigraffle.

I got my first Mac in more than a decade last week, so I’ve left behind all my Visio skills for the time being and am trying to ‘level up’ in Omnigraffle as quickly as possible!

But I don’t get what’s so special about it. Can someone remind me?

Making the switch to Mac has been a fascinating experience. I’ve had so little experience with OSX and Mac applications, that I really feel like a beginner. And, no. It’s not as easy and idiot proof as those of you who’ve been using Macs for a while seem to think. Sometimes, really basic tasks like trying to save a document into a particular folder, seem completely impossible to me (there is a lot of functionality hidden behind little black triangles, I’ve come to discover).

I miss knowing all the shortcuts desperately. And knowing how to diagnose problems. I have to learn entirely new patterns and ways of interacting.

I’m a beginner. And it’s really frustrating, and disempowering. It makes me feel pretty dumb.

It also makes me think that I wish that I could have this experience about once a year to REALLY bring home what the experience of using the interfaces that I design must be for very many people. It lifts the ‘Curse of Knowledge, or the The Curse of Expert Ennui as Anne Zelenka might describe it.

We know so much about making our computers work and so much about how they are designed… it’s impossible for us to forget enough to really empathise with novice users.

Which, of course, is why it’s so important to regularly, carefully and empathetically observe users of all levels of expertise and familiarity using your product. You might *think* you know what they understand, but you’re probably wrong. Design expertise is incredibly important, but it only goes so far. Regular observation of real people interacting with technology is a really important input to good design, and becoming a good designer.

Meanwhile. I’m taking any tips on how to become an expert in all things Mac. Let me have ‘em.


Yes, you should be using personas

Personas seem to go in and out of fashion. Not long ago, people were advocating hyper-researched personas done in painstaking detail, these days designers seem more inclined to leave them out of the process.

So, are personas actually useful or should we stop wasting time and ditch them?

I first came into contact with personas in an academic context. They seemed like a nice idea but I tended to use them to justify my design rather than to guide design, which seemed kind of back to front.

Since then, I’ve worked in places where personas are more or less embraced as part of the process, and then longer I use them, the more I’ve come to the opinion that personas are incredibly valuable, but not for the reasons that many people think they are.

If I’m working on a UCD project (and thankfully, these days that is pretty much every project I do), then I would much prefer to include persona development in the process than not.

But, having said that – I find personas virtually useless when it comes to design, and I very rarely reference them in making design decisions. For me, personas aren’t about design, but that doesn’t mean they’re not incredibly powerful in other ways.

Personas communicate the user centred process like no other method

Having your clients view user research and testing is incredibly powerful in helping them realise that there is a problem in the way they’ve been approaching things to date (if you’re not encouraging stakeholders to actively participate in observing research and testing you’re missing out on a lot). But to get them to actually understand what user centred design is about – you need personas.

Personas should always be developed collaboratively with key stakeholders – as many as possible. They can often be derived from existing marketing personas or profiles (but, don’t be mistaken that personas that the marketing department gives you are personas you can use without any work). You should try to validate your personas with some kind of user research if at all possible – this can be in the form of some contextual interviews, lab based studies, or by talking to people in the company who interact with user directly on a regular basis (I’ve found people who work in call centres can often provide invaluable insight to what users are really like).

Personas should define the boundaries for which you will design. It’s a common misconception that personas are about creating a set of ‘typical’ or ‘stereotypical’ users. Much more useful is to use personas which incorporate edge cases behaviour or requirements. Including these kinds of personas means that you actually end up catering for a much greater portion of your user base, and it also offers the potential for ‘limiting’ personas (think Clevis McCloud if you’ve read Inmates – if he can use it, anyone can.)

Creating ‘edge case inclusive’ personas and then prioritising personas and their goals is much more useful in helping decide what functionality goes in and what doesn’t. And which of the tasks or outcomes are most common, or more important. Decisions which are critical foundations to allowing good design work.

Personas are powerful in guiding requirements definition.

Even before you start designing you need to know what the product is and what it does. What are it’s core feature and what are it’s advanced functions. Personas really help guide discussion and decision making on requirements.

It’s generally agreed that one of the biggest factors for bad user experience is featuritis. Products that do too many things, or that prioritise advanced functionality over core functionality.

Putting each function to the test using personas before letting it in the door is very helpful. Then giving features relative priority using the number and importance of personas who are using them helps guides the overall application or site’s design priorities.

Using this evaluation process makes putting the user at the centre of the design process habitual and natural, and is also very helpful in reaching concensus within a project team.

Personas kill the ‘elastic user’

If I’m managing to inspire you to use personas more or differently, and you haven’t read it recently, let me encourage you to take the time to read Allan Cooper’s ‘The Inmates Are Running The Asylum‘. It’s a classic text that is renown for making programmers angry because Allan isn’t particularly flattering to them, but it gives real insight into the original ideas behind personas and I think you’ll find yourself even more inclined to use personas regularly.

I re-read it earlier this year and I particularly took away the idea of the Elastic User. This is where stakeholders make statements about what ‘users’ want, what ‘users’ do, what ‘users’ prefer, and because the ‘user’ in that context is so undefined and broad, they are able to say almost anything they like and there is no real way to contradict that opinion.

The creation of personas means that user groups are much more defined, so broad sweeping statements about what users want can actually be tested against something. Rather than having a free pass to do anything to the requirements or design by just using the word ‘user’, these assertions can now be tested and validated against more closely defined user characteristics and goals.

Use what you’ve got to build personas

Sometimes you’ll have a whole lot of research, sometimes you’ll have practically nothing. Either way, personas can be useful. Obviously the more you know the better, but even just a quick verbal sketch can be extremely powerful.

Not long ago I used the following persona in a design workshop:

‘Betty is a pensioner who lives in suburban Newcastle and is in charge of the local Neighbourhood Watch chapter’

That’s it. But it made a huge difference because it created an edge case persona for a site that was originally targeted at statisticians. ‘Creating’ Betty allowed the team to see that designs would work perfectly well for people who understood the language of statistics would be completely useless for Betty. And we found that pages that were really *for* Betty hadn’t actually be designed appropriately for her at all.

Sometimes, a pencil sketch persona can provide massive bang for buck.

You should care what car your persona drives

One of the things that make personas easy to criticise is that sometimes they feel like an exercise in creative writing.

‘Clare is 28 and lives in Primrose Hill. She drives a Prius but only on the weekends and she has a puppy that she takes walking twice a day. Clare is single but dates frequently and likes to travel.’

For as long as you think personas are about design you’re right – what does this have to do with design? When you use personas predominantly as a communication tool, then these small details become all important. After all – what you are trying to do is humanise the ‘target audience’. As humans, it is exactly these little details we care about, it’s what helps us to relate to other people and what makes them real. Does it really *matter* what car Clare drives? Well… kind of. People choose things like cars to drive and places to live as a means of communicating who they are. It adds depth to Clare’s personality. But, I’d never advocate actually *researching* those details. They’re communication devices, if people don’t know what Prius *means* then there’s no point using it. Make sense?

Don’t design for personas

As I mentioned before – personas should include observed edge case behaviour and requirements. From your personas you can work out what functionality is included and what is the core functionality. Once you’ve got that information – and you’ve immersed yourself in understanding the users through the creation of the personas – put them away and use your skill as a designer.

If you use the personas to closely guide your design you will end up supporting a series of edge cases. This will invariably mean that your CORE functionality is compromised. That’s bad design.

Once you’ve designed the product, you can then use the personas to evaluate the design, and to help communicate to your stakeholders how and why your design supports the goals and tasks of your personas.

Don’t be tempted to use the goals and task lists you generate for your personas as a jigsaw puzzle to layout on a page. That way failure lies.

So. Personas. I’m a fan, but you have to know where their power lies and use them appropriately, and don’t get too caught up in the process of their creation – do what you can with what you’ve got, bring your stakeholders on board and teach them how to be user centred, and get on with the important business of designing great user experience.

(Need more background on personas. Austin Govella has collected a bunch of useful links)

Note: this post was edited on 30 April 07 to better clarify my position on ‘edge case’ personas in response to come of the comments received :)