in customer research

Embracing the Un-Science of Qualitative Research Part Two – Ever-Evolving Prototypes are Ace

So, earlier we were talking about whether you can or should attempt to make qualitative research more scientific, and that there are three ways you might go about doing this, being to:

  1. Use a relatively large sample size (deconstructed in Part One)
  2. Ensure that your test environment doesn’t change (which we’ll talk about now)
  3. Ensure that your test approach doesn’t change

One of the fundamentals of quantitative research is its systematic nature. It’s about measuring stuff. And, you don’t want that stuff to change as you’re measuring it for a number of reasons – not the least of which being that it makes it very difficult to plot on a graph :)

Qualitative research, on the other hand, is not about numbers so much. It is about the depth of insight that you can gain from having much greater and more flexible access to your research subjects. As you are seeking insight, not statistics, it matters far less whether whatever you are testing, say a prototype, changes a bit throughout the course of the study.

In my experience, some of the most fruitful research has occurred when the prototype has changed quite a bit from interview to interview – and sometimes even within an interview.

Here’s how it works (again, using the example study I described in part one: a lab based combination of interview & a wee bit of usability which is intended to ensure that my client’s proposition is sound, that it is being well communicated, that the users understand what the service is and how it works, and to weed out any critical usability issues).

On one side of the big brother mirror you have the researcher and the participants (sometimes known as ‘users’. Urgh). On the other secret side of the mirror you have your client including a member of their technical team (or, perhaps, a gun Visio or Omnigraffle driver, depending on what stage your prototype is at) with laptop at the ready.

As you proceed through the first couple of interviews, some really obvious stuff emerges. These are the things that you don’t really notice because you get too close to the project and you develop a kind of ‘design blindness’. Or they’re things that you never really thought about because you were concentrating on other more important or complex aspects of the design.

These findings are almost always crystal clear – the change required is obvious and rarely time consuming. What are your options? You can:

  1. spend time on and note the problem as it occurs in every single interview you perform in that study, or:
  2. fix the problem and spend your valuable research time learning about things you don’t already know about.

OK, so I might have biased that survey just a little, but the answer seems obvious to me. You get so little opportunity to spend with actual end-users – why spend that learning the same thing five times when you could spend that time getting more and richer insight, and exploring the more complex problems.

Because you can use this technique to explore the more complex problems that emerge from qualitative research.

With complex problems the solution is not so clear cut so often this means you end up doing some improvised A/B testing where you explore a number of potential solutions with participants to see which is most effective, or at least, which seems to be the correct solution to further explore.

(Interestingly that Wikipedia entry I linked to there suggests that you need an audience of statistical significance for A/B testing to be effective… of course, I disagree).

This approach to research is more demanding on the researcher than a typical ‘fixed script, fixed environment’ approach to testing. Using this approach, I can never quite be sure what will be on a page when it loads or whether the navigation items might have changed a bit and I need to explore that, or to completely change the flow of my script because the part of the prototype we were going to explore next is a bit broken and we’ll need to come back to it later.

These extra demands are well repaid, though, by the forward leaps that you will see your client being able to take even before the research is complete – and well before your analysis is done and presented. Not only this, but the value of undertaking the research is well and truly demonstrated even as it is being performed – which is gratifying to you and often very exciting (or relieving) to your client.

So, again I say – if it’s numbers you need – go do a survey. Or use one of the great online remote usability testing tools, or any number of excellent quantitative methods. In their own ways, they are fantastically valuable.

But if you want to quickly weed out problems with your site/application/prototype – then I recommend that you consider using this technique of the ever-evolving prototype. It will certainly keep you awake as you’re researching, you’ll get rapid return on investment and excellent bang for buck as far as research techniques go.

What say you?

  1. I like your general take on this but have a comment on “validation.”

    I think you don’t need statistical validity with observation-based testing because the validation is really the expertise of the expert. In other words, you observe something that is, like you said, a “big” or “obvious” issue. That assessment validates the finding.

    However, with A/B testing, there’s a big caveat: are you measuring preference or performance? In other words, are you banking on what people say they like or say they’d do? Or are you measuring what you observed them doing, and are comfortable that that’s what they’d do independent of any test effect?

    I’ve always found that hard to achieve, and have generally avoided using 121 testing to find preference for one design/solution over another.

  2. Ever-evolving prototype – I do like that. And if you’re trying to hit a tough deadline, it’s the only way to go.

  3. I agree with this 3 part series and look forward to the last part. But my question is, what about generalizations? Having rich data from a few participants is great, but what works for one doesn’t always work for another. Can you then make generalizations? And if so, how can you be sure you’re adequately representing the participant population? And, in a formal environment, how can you justify your generalizations?

Comments are closed.