Guerrilla Research – Recruitment

It’s been almost a year now that I’ve been doing predominantly ‘guerrilla’ design research. For me, this means testing in the field with a minimum of time, budget and fuss so that this kind of activity and the insight it provides is available to pretty much any client/budget/timeframe.

One of the first challenges for guerrilla design research is recruitment – the particular objective being to avoid using recruitment companies in order to reduce the cost of the project and also avoid the delay (often up to 2 weeks!) that is typically associated with recruitment.

A common approach to guerrilla recruitment is simply to rock up at a venue where your target audience is likely to congregate and to try to recruit on the spot. Typical venues might be a local Starbucks or a conference.

This is not an approach that I tend to use, for a couple of reasons – primarily because I really have to relinquish a lot of control over who I involve in my research… more than I feel comfortable with, given the responsibility that I have to my clients to provide them with useful insight. Also, sometimes the recruit briefs I need to meet are quite complex and require me to be quite selective when identifying participants. This is quite difficult to do on the fly and face to face… it can lead to either some awkward moments or spending time doing research with people who aren’t really quite right.

The approach that I have been using (and many of you are probably very aware of this!) is to use my online social network using tools such as my blog, Twitter, LinkedIn, and FaceBook. I have been pleasantly surprised by how successful this has been – on a number of levels.

Essentially how it works is that I work with my client to define a ‘call for participation’ that will be posted – the objective here is to get people interested in the research, to get more rather than less people to contact us, but hopefully mostly people who are at least close to right for the profile. I have to say, I am quite happy not to have to put together screeners any more (or review screeners received from recruitment companies). In fact… if I never saw another screener again I’d be perfectly happy :)

I do, up to a point, think about what ‘channels’ to use. As a gross generalisation, somewhere like Facebook is better for a less technical participant, where as Twitter is better for a more technically savvy participant… this is a gross generalisation though because more often than not, the people who I end up meeting are not directly in my network, but friends, family, colleagues of people in my network. This is the real power of ‘guerrilla’ recruiting, and a big reason why the ‘channels’ matter much less than having great people in your networks :) (Preferably great people with lots of friends, family and colleagues!)

To date, my experience has been that undertaking recruitment in this way has been cheaper (obviously – I don’t charge myself or my client ‘recruitment’ fees per participant), and faster (rather than weeks, recruits are sometimes filled within hours, definitely days of posting a call for participation). The most pleasant discovery has been that the quality of research participants is significantly higher than I had experienced when using recruitment companies.

In the past 12 months I’ve interviewed dozens of people and only had one ‘no show’ – and that was just a mix up with the dates as opposed to someone who just decided not to show. If you’ve done much research you know that it is not uncommon to have a day of six interviews lined up and only to get four participants to show up.

Some of the recruits I’ve had to fill have been fairly simple, but there have also been some incredibly complex briefs that I have no idea how I would have managed to effectively communicate to a recruitment company.

And the people who do turn up are fabulous – I’ve been amazed at how close to the ‘brief’ almost everyone I’ve met has been. They’ve been interested, enthusiastic, articulate and certainly not ‘professional research participants’ – the bane of any researchers existence!

Of course, you can always just take your design and show it to people in the office, take it home to your own family, show your friends – this can be very valuable. If you’re looking to do some research that is slightly more formalised, then perhaps you could consider using your own online social networks for this purpose.

It has certainly made me value even more the networks that I have in place and thankful for the great people who I interact with in these spaces.

(and, while I’m at it – if you’ve helped me with recruiting in the past year or so – thank you, thank you very much!)

Client-Centred Research

Word on the street is that the benefit of User Centred Design is hard to prove… I’m a fan of UCD as a process and I think that it is difficult to measure where process has been central to outcome, but today that’s a tangent. Part of the argument that is made is that a good designer can do almost as much by applying their design experience and expertise as what can be achieved through a UCD approach. In many cases, I heartily agree. Sometimes the process is more about the client than it is about the end users.

A project I’ve been working on recently is a case in point. This project is a pretty big deal – a lot of time and effort and, of course, money, have been ploughed into it already and there is plenty more to follow. Stakeholders have already been working on this project for ages when I get the call to come in and do my thing.

From a quick look at the proposed designs it is clear that there are some fairly significant issues that need to be resolved – both at the proposition level and at a more tactical, executional level. I don’t need research to tell me that and, because of my experience, I have a pretty good idea of what we need to do to fix this.

There is no WAY that I am going to be able to persuade all of the stakeholders who have invested so much in their current approach that they need to make the changes I’m suggesting… after all – I’m just one voice. One subjective, lonely voice… with my opinion alone, I have a long, lonely and difficult path ahead to try to get my clients to make the right decisions. Chances are, I’ll not be successful.

Cue user research. By conducting a quick round of user research I develop astounding powers of persuasion, because it is no longer my subjective, individual opinion that I am asking people to trust. Through the access I have to data accumulated in the course of research, I am able to validate my opinion with something much more substantial – the voices of the end users, the stories of my interactions with those end users, anecdotes, examples, tangible stuff.

Is this just a waste of project resource? Should I spend more time on trying to be more persuasive and less time on research? I think not.

In my experience, by talking to and observing your end users, by looking for patterns and themes, there is always something interesting to learn – almost always something that is directly applicable to your project, and if not, then certainly something that will help you develop in your ability to understand and design good user experience. If you’re interested in social interactions online, then staying in touch with how people are talking about, thinking about, using different tools is something that you should regularly be doing. Take this opportunity to throw in a few questions related to ongoing areas of interest.

At the end of the day – our job is to try to make sure that good decisions are made about the user experience of projects we are working on. If it were up to us alone, perhaps we wouldn’t need research. For as long as persuasion (of our clients, stakeholders etc) is a critical part of our role, research is an incredibly useful tool.

Are you using it?

Why collaborative research analysis rocks out.

dConstruct workshop - affinity sorting

(a quick definition, given that I’ve discovered that English is at least three separate languages: to rock out = to perform exceptionally well and give great satisfaction, as say, a rock band might ‘rock out’ on stage’.)

These days when I’m doing any kind of user research, rather than going to my secret consultant place and doing that consultant magic that results in a presentation of research findings, I much prefer to get into a big room with clean walls and several hundred sticky notes and my clients/project team, and to work out the research findings collaboratively.

Am I just being lazy and getting my clients to do my work? Well kind of…. but with good reason!

Why do it? Well, there are a few reasons.

Firstly, to combat what I think is probably the single most frustrating outcome of a research project – having your results either not accepted or immediately shelved, meaning that all of your work has come to pretty much nothing. By involving your clients in the process, they have a stake in defining exactly what the findings are, what is important, what is not. When you’re presenting the findings, you (or even better, the project team) are presenting the *team* findings, not just your own.

Secondly, to educate your client. To help them to understand that there is actually a rigorous process that occurs between the interviews or focus groups or whatever your research activity is, and when the findings magically appear in the presentation. To allow them to use the tools themselves when it is appropriate.

Thirdly, to get better results. Having your client with you will ensure that you apply appropriate rigor in reviewing research data. Not to say you don’t do this by yourself as well but it’s great to have the extra incentive.

Not only that but in this situation, three or four or five heads definitely are better than one. Take for example this study that Jared Spool shares in his article on the KJ Technique (which I’m referencing all the time!)

Back in the late 1970’s, the US government commissioned a study to look at effective group decision making. In the study, they asked 30 military experts to study intelligence data and try to construct the enemy’s troop movements.

Each expert analyzed the data and compiled a report. The commission then “scored” each report on how well it reported the actual troop movements. They found that the average military expert only got 7 out of a 100 elements correct.

Each expert then reviewed all of the other experts’ reports and rewrote their initial assessment. The average accuracy for these revised reports was 79 out of a 100.

What was different between the first report and the second? The experts didn’t have any new information. All they had were the perspectives of the other experts. When they added those perspectives to their own, their accuracy increased ten-fold.

It’s been my experience that if you can get your project team members (and their associated and diverse expertise) involved in the research analysis process, then you will most definitely get more accurate and more useful research findings.

So, how do you do it?

I’m sure there are a whole bunch of ways to do collaborative research analysis but I’ve gotten the most success from the following approach.

Firstly – encourage as many team members as possible to observe the research (if possible). Give them sticky notes and markers, give them the rules for writing sticky notes (one concept per sticky, clear handwriting in capital letters) and ideas about what kind of things should go onto the sticky notes. Don’t worry about the fact that you’ll have duplicates. Get them to write as many stickies as they can.

Then, when it comes time for analysis, you want a big room with lots of clean wall space. Plaster the walls with white or brown paper (whatever is easiest to get hold of) so you can move the stickies around en masse with ease. Then it’s time to get stuck into the process.

  1. start by defining the research question(s) – you should have done this before you undertook the research so this should just be a refresher. I like to get them written up and positioned somewhere highly visible in the room. This is what we’re trying to discover, the questions we’re trying to answer. They help maintain our focus.
  2. do a large scale affinity sort (follow steps 4, 5 and 6 from the KJ Technique). I know that this process looks completely chaotic at first… it is. Trust the process though, it actually does work. What happens is that you end up with lots of big groups with very vague names and some duplicates around the room. After you’ve done the very first sort, pick a big group and start dissecting it – look for groups within groups, and make sure that the group labels are actually meaningful in relation to your research questions. This is the tough part – you need to keep driving the group to keep seeking themes and meanings within the groups… and to sort and re-sort, and have lots of long, pedantic discussions – until finally the room full of stickies is completely sorted. (You can deal with the duplicate issue now by sticking duplicates one on top of the other so that they are not over-represented within groups).
  3. prioritise your findings. As a group – review all of the findings that you’ve come up with (each group is now a ‘finding’), and start grouping your groups together based on their relevance to your research question. You might have meta-group headings that are something like ‘Interesting but out of scope’ and ‘In Scope – High Priority’, ‘In Scope – Low Priority’ etc.
  4. then finally, go back to your research questions and work out what you’ve found. Based on the research you’ve done in this project, what are the answers to your questions?

Be sure to photograph all of your work, and then instead of the dreaded task of writing a ‘research report’, your job is then to gather all of this information into a digestible format for the team to use going forward.

And, of course, because they’ve actually been involved in the process, they’re much more likely to actually use it. Yay!

What do you reckon? Have you tried working like this? How did it go? Any other techniques that you’ve found work well? I’d be interested to hear what you think! :)

Who is the customer in Agile UCD?

Have you read the Agile Manifesto lately? If you’re doing Agile work then hopefully you’ve come across it. I always find it a really good touchpoint to come back to when thinking about what *is* and *isn’t* Agile – much more useful than looking at how any one particular flavour of Agile or one companies interpretation of a flavour of agile can be!

At any rate, here’s a refresher. The Agile Manifesto says:

We are uncovering better ways of developingsoftware by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Seems almost straightforward, doesn’t it :) Deceivingly so, of course. There’s all kinds of complexity wrapped up in that simple manifesto, some of which we touched on recently and which requires so much more exploration.

Add in the UCD (User Centred Design) aspects and the questions only multiply!

Let’s start with this really easy one – who is the ‘customer’ in Agile UCD?

I would hazard a guess that your response would depend on whether you’ve been doing more Agile or more UCD lately.

If you’ve been doing lots of Agile lately, your answer would quite likely be that the ‘customer’ is the person who is paying the bills. The company or department who has commissioned the work that you’re undertaking. And that, as such, they essentially get a seat at the table as the development goes on, to have their input as decisions are being made minute to minute, and to request changes as they see fit. With any luck, this person is actually a real decision maker within their organisation and their input won’t be completely over-ridden as soon as your work is presented to the wider client team.

If you’ve been doing lots of UCD lately, then your response will probably be quite different. You’ll probably be thinking that ‘customer’ means ‘end user’ – the people who will ultimately end up as the beneficiaries of all your hard work. The people who you’re observing in order to generate your design work and who’s needs and characteristics are influencing decisions about the product functionality.

They’re two very different types of customers, aren’t they. With very different demands and expectations and requirements of you as a designer or developer of a product.

So, what to do?

Well, frankly, in Agile UCD, you need *both* of these types of customers. A successful project takes into account both the business requirements and the user requirements and having direct input from both sources directly is incredibly valuable.

Is it really necessary and valuable to have a member of your ‘client’ (which is what we’ll call the people who pay the invoices you send for this work) as a part of your day to day project team? I don’t think so. I think that ‘clients’ should be highly involved at the beginning and end of each iteration/sprint and on call throughout this time, but I don’t think that either the project team or the client gets much out of them having a permanent seat on the development team. I’d much rather see the team being left to get the work done and the client doing whatever it is they do best – which is very rarely being part of a design or development team. (Hands up if you think I just committed some kind of Agile heresy?)
On the other hand, one of the key reasons for moving Agile to Agile UCD is to add in involvement from actual end users. Traditional flavours of Agile simply haven’t allowed for the type of observation that we do in UCD and that serves us so well as designers to be included in the Agile process. This is why we are seeking to building in scope for contextual research and usability testing activities as iterations occur – rather than just at the very beginning of the project and at the very end.

So, in fact, there are two ‘customers’ in Agile UCD – but perhaps we’re better calling them the ‘client’ and the ‘end user’. One of the challenges of working out how Agile UCD should work is to determine the best way to involve each of these parties in our cycles of work so that we get the best from them and they give us the information and insight we need to do the best work we can.

You’ve seen some of my ideas about how this might work… what are your thoughts, ideas and experiences?