When experience matters (and when it doesn’t)

A little while ago I wrote this, which turned out to be more controversial than I intended.

I wrote it after having several encounters, in close succession, where one of these had happened:

  1. Recruiting scenarios where people assumed that previous domain experience was more important than the overall research experience, and capability, of the researcher.
  2. Working with a researcher who had domain experience on a particular topic who kept shutting down team conversations based on experience they’d had in previous projects, which the rest of the team had not been involved in.

These are both anti patterns of domain experience in user researcher and should be avoided.

Recruiting user researchers

When you’re recruiting for a user researcher, there are three things that teams should be looking for:

  1. Firstly, ensure that the user researcher has experience in the methods of user research you expect you’ll be using on the project. For example, don’t hire someone who has a decade of experience doing usability testing when you need someone who is strong in contextual research to work on the discovery phase of your project.
  2. Secondly, be confident that the user researcher cares about the quality of the service that their team ultimately delivers, will be able to hold their own in the team and will be effective in communicating research findings in a way that compels the team to act on them. This is the hardest criteria to meet. Too many user researchers have had a great time doing research and making reports, then leaving the teams to do with it what they will. You want a user researcher who wants to have ‘skin in the game’, and wants to see their research valued and used in making service design decisions and in setting priorities for the team.
  3. Finally, if the researcher has previous experience in the subject domain of the project, this can be an advantage. But only once the first two criteria have been met. These first two criteria are much more important for user researchers than subject matter domain expertise, and they’re also pretty hard to find.

When you hire a user researcher, the domain they should be most expert in is user research.

In teams, diversity is a strength

For most projects, having some subject matter expertise is essential. In most cases, that would be at best inefficient and at worst dangerous to do otherwise.

What is equally problematic, though, is a team full of people who have extensive experience working on the problem space they are just about to tackle.

Teams like this have so many shared mental models and assumptions about how things work, what things mean, where the constraints are, and how people think and work, and, despite their experience, not all of these things are right.

The CivicPatterns website calls this the Clean Room pattern and says:

If your project operates within any bureaucratic system, ensure that the person responsible for its design knows as little as possible about how the existing system works.

Most people hate dealing with bureaucracies. You have to jump through lots of seemingly pointless hoops, just for the sake of the system. But the more you’re exposed to it, the more sense it starts to make, and the harder it is to see things through a beginner’s eyes. Therefore, when building a system that helps someone bypass bureaucracy, start by designing how the system should be, with as little pre-knowledge as possible, and then, when you need to add any complexity, work as hard as you can to hide that from the user.

Lack of diversity in experience-levels (and lack of diversity in general) in the team will reduce their ability to consider a full range of service design options that can streamline the experience for users. This will limit the potential for transformation.

There are some roles where experience the domain of the project is essential and teams would be foolish not to include them. Designers and user researchers are not those roles.

Design your teams so they have diversity, in gender, in age, in backgrounds and in subject matter expertise and you’ll design better experiences for your users.

Death to ‘it depends’

Lately I find myself on a mission for mass simplification. Possibly over simplification, but I’m not sure it matters.

It’s one of the things I care most about at the moment – how can we simplify what we are asking people to do so that there is nothing else they can do but start doing it, instead of following their natural inclination to make a list, hire a consultant, write a white paper, do anything but doing the thing.

It requires that I stop saying (or even thinking) one of the things I have probably said most in my entire working life – ‘it depends’. That’s hard, but I think it’s the right thing to do.

It depends is paralysing.

It is a stop sign. It tells people they can’t do anything yet. That it is complicated.

And it presents things as  complicated at a time when the person asking the question usually doesn’t have the capability to understand they subtleties you are trying to explain.

Of course, most things are complex and it usually does depend (on context), but knowing what to do in the face of all of that is less important than just doing something now. Ideally something that gets you on the right track to having the ability to ask more nuanced questions, to have a more sophisticated understanding of the thing you’re doing and the context you’re doing it in.

What does this mean? For me, at the moment, this means trying to create a massively simplified message for teams across government about how they should start doing user research.

Lots of different teams, different projects, different audiences – a whole world of it depends.

Forget all of that. What’s something massively simple that all of them can start doing right now.

Here’s an idea.

  1. Start doing some research in every iteration (we’re all doing agile)
  2. Start by getting 5 or 6 people in a lab, give them the thing you’re making and a task and watch them use it. Talk to them about what they find difficult. Maybe also ask them about how they use the internet in real life, what kind of interaction they’ve had with government before. Don’t explain your thing.
  3. Get as many of your team as you can to observe (hence the lab). Aim for 2 hours every 6 weeks for everyone as a minimum.
  4. Capture what you hear and see on post it notes. Do affinity sorting to analyse research. Write down what you learned (your insights) and what you’re going to do about it. Do those things, come back and do the same in two weeks.

Now lets all start saying this to everyone, every chance we get.

It won’t be the best thing for every project to be doing all the time, but it is way better than doing nothing.

As soon as you start doing this, you will start to learn about different and better things that you can do (if for no other reason than because you’ll need a user researcher to do this and they can tell you. Before you start this, you probably won’t have that person).

Now we just need to do this for all the things…

Related: I watched this video from the ever grumpy but rather clever Russell Davies the other day. He’s scaring the pants of people who are studying to become Planners in the advertising industry and telling them that the best strategy is not the most creative one but the one that actually gets anyone to actually do something.

There’s an interesting section on the value (or not) of research as well.

Strategy is being on message.

If you’re trying to implement strategy I reckon you need to spend a good 30% of your effort on communications. Probably more. You need to think more like someone who is trying to sell expensive trainers and less time making power points, spreadsheets and gantt charts, or doing whatever the thing is you’re officially paid to do.

This is the most important thing you can do to help, I’ve said to teams at GDS and people in other government departments just this week, the most important thing you can do to help is to say the same things that we say to anyone you get the chance to say them to. Of course, I am taking as a given that people will always do the best work they’re able to do in any situation, we need that too.

If you want to implement a strategy, do what the marketing people and the political advisors say – get on message and stay there.

This requires two things:

  • working out what your messages are (the fewer the better)
  • expressing those messages clearly (choose your words carefully)

Achieving both of these things takes time, effort and attention. It doesn’t tend to happen organically, without consideration.

Work out what your messages are

There might be lots of behaviours that you want to change, but you’ll only ever be able to get a couple of messages out to people – and only then if you say them over and over and over again.

Work out the one or two key messages that you can focus on – look for behaviour changes that are relatively easy to achieve (or that sound like they might be). Bonus points for finding behaviours that, when adopted, will naturally lead towards other behaviours you would like to see.

An example. Over the past year or so, we have tried to achieve a pretty significant change to the way that people in government use user research in digital projects. Mostly we wanted research to become a discipline that was as agile as the developers, product owners, designers and was able to be involved throughout the product lifecycle so that we could be more effective. We wanted to stop  people outsourcing a bit of research at the beginning and end of the project and complaining that it wasn’t very useful. We wanted user researchers to be able to be effective contributors to the success of the projects.

You have to do a lot of things differently to achieve that, but we’ve focussed on three main behaviour changes:

  1. project teams to do research more regularly throughout the project lifecycle
  2. researchers to be able to work in way that required less documentation and more light weight and regular communication with the team
  3. everyone in the team to participate more in the user research, for more people to actually see end users interacting with the service we’re making.

If we can achieve those three big things, then we can start to worry about the detail of exactly what happens in the team, who does what, when and how. Those will be good problems to have, and we’ll worry about having a clear position on those when we need to.

These are three very noble goals, but they don’t feel easy to implement and achieve.  So, we needed to look at our messaging.

Express your messages clearly

User research is a team sport poster

We’ve taken those three objectives and crunched them down into three soundbites that you hear more and more people starting to say around GDS and across government. They are:

  1. User Research is  team sport (often followed with, ‘we don’t do this for us (researchers) we do this for you (the team)). This doesn’t really tell you exactly what to do, but it tells you a lot about the new philosophy of user research. That it requires participation and it is open and collaborative.
  2. Do research in every sprint (often followed by, you need a user researcher in the team for at least 3 days every week). Making the call to action very specific is both motivating and rewarding. It helps teams know what they should be doing and for those who are doing it, confirms they’re on the right track.
  3. Everyone in the team should watch user research for at least 2 hours every 6 weeks. We make a lot of use of the UIE research around Exposure Hours. It continues the ‘team sport’ message, validates the need to do research each sprint, and is another clearly measurable goal.

We’ve learned that making your goals attainable is important, and that using specific measures (every sprint, 2hrs every 6wks) seems to be very powerful. This is a big change for people who are used to answering most question with the preface ‘it depends’.

By reducing our overall goals down into three simple soundbites it makes it easier for everyone in the user research team to say these things over and over again, but it also makes it easier for people who are less familiar with user research but who are integral in organising budgets, people, timeframes – the things that are critical to us being able to achieve this – to know what we want them to do and to ask for the same things without feeling they need research expertise.

Although we know that these three requests aren’t necessarily the best way to do research in every project, we think they are the best way to get an empowered researcher into a project – once we’ve achieved that, tweaking the program to better suit the project is relatively simple.

These are the words that will be adopted throughout the organisation and, as we know, words are powerful, so you should think carefully about the ones you choose.

This example that focusses on user research, but you can apply this to any area of your business or project where you are looking to get a strategy implemented.

Be on message every chance you get

Of course, once you work out your key messages, you then need to take every opportunity you get to repeat them. This is where ‘saying the same thing’ really kicks into action. Really, literally, say the same words. Use your soundbites.

After a while, you will start to feel a bit silly saying the same thing over and over again. Remind yourself that no one (except you) hears you say it every time, and use your key messages in conversations, presentations, blog posts… we’ve even made posters for ours and post them around GDS and government departments when we visit.

What success sounds like

One day, you’ll hear someone you’ve never spoken to say it back to you. Or even better, they’ll say it to a large crowd of important people in your organisation. You’ll only know that if you’ve carefully chosen your words.

When that happens, have a quiet celebration because, unlike many other strategies before it, yours is now making progress.


This is the seventh post in a series of rambles around the topic of strategy in the general vicinity of user experience which I’m posting as a kind of obituary to the book I almost finished writing then realised was pretty much completely wrong.

I’ve been writing this instead:

  1. Everyone is doing strategy right now
  2. Strategy doesn’t live in a silo (or there is no such thing as UX Strategy)
  3. Strategy fast and slow (or strategy is culture for breakfast, lunch and tea)
  4. Strategy is a team sport
  5. Good strategy is modular
  6. Why words matter (more on the relationship between culture and strategy)

Why words matter (more on the relationship between culture and strategy)


Sketchnotes by @YahnyInLondon of Gill Ereaut’s talk at Design of Understanding 2012

Words are pegs to hang ideas on – Henry Ward Beecher

It is not unusual for me to be involved in a debate about words. Words I am frequently pedantic about include user research, user testing, user experience and user centred design. I think that the words we use matter. They do more than just define what we see and do, they help us understand what we think about those practices.

It’s always interesting to me to see who finds these discussions useful and who doesn’t. I’ve found that people I’m working with who are less familiar with design, research, user experience find the definitions useful. Within our teams we have to correct ourselves often, change what we called things before, get rid of previous habits of language, but that’s a good thing – every time we remember, we remind ourselves of why we use the language we use (because we’re testing ourselves and not our users, for example).

Interesting, the people who get stroppy about the definitions and say that it doesn’t really matter tend to be people who have been involved in the ‘user experience’ community for at least five years. I guess they have a vested interest in not being seen to be wrong. Or, perhaps more generously, they are just so close to the subject that any word will always be an oversimplification of all the beliefs, attitudes and practices of their work.

If you want to know what an organisation really believes in, look at the language they use in their day to day work – in their meetings, in their documents, in what they call things.

There are people, like Gill Ereaut, who do this as their full time job – they do linguistic analysis to help companies try to join up what they ‘officially’ say they are trying to do (their official language) with what people actually say as they go about their day to day work (the surface language).

Language is the medium through which culture is enacted – Gill Ereaut

A mismatch between surface and official language is a signal of ineffectiveness in an organisation. There can also be lots of different surface languages in different parts of the organisation that makes it even more difficult for communication across the functions and increases the impact of silos (for example marketing speaks a totally different language to the tech team).

Organisations who are thoughtful about the words they use repeatedly are more able to have a more consistent culture that allows their strategies to come to life in the products and services they create and the way they interact with their audience (this works even if those words might appear artless, like ‘show the thing’, something people at GDS say all the time and even have a poster for)

Show the thing poster

Changing the words on the hymn sheet won’t create a whole new religion, but, as Gill says,

‘linguistic change at the surface level affects the assumptions held by the organisation’.

Especially when it comes to customers, making sure that the words we use in the organisation are empathetic to the people the organisation is serving and that it reflects the type of experience and interaction that the organisation wants to have with those people sends daily micro signals widely through the organisation that this is something with which it is genuinely concerned.

In her talk at Design of Understanding conference in 2012, Gill (yes, I am a bit of a fan) talked about an organisation where she’d done some linguistic analysis. This was an organisation that believed that they cared about good quality customer service, but analysis of the way the organisation communicated showed that actually, they were afraid of their customers – they were distant from and lacked empathy for their customers, making it almost impossible for them to really deliver great customer experience.

Every day, in the words that organisation used, in the names of their processes and documents, in the way they communicated with each other, they were entrenching this fear of the customer that made it impossible for any corporate level ‘Customer Experience’ strategy to be effective.

Words can remind us many times a day what we all care about and what we believe in.

Changing the words we use can help us to change our culture in tiny moments every day and help us to be more able to implement strategy effectively.

Calling it user research instead of user testing won’t change the way the moderator runs the session or the experience of the research participant. As the person who is running the sessions, it probably makes no difference to your competence what you call it. But deliberately deciding to call it user research even though your reflex is to call it user testing means that every time you choose to use the words we’ve agreed on, you are also agreeing, once again, to the reason we chose those words – because we’re testing ourselves, our work, our design, our services and not our users. Because we are in service to our users and if they don’t understand, that reflects poorly on us and not them.

Being thoughtful about words seems to be one of the simplest and least confrontational yet most powerful ways to transform organisations in tiny moments every day. Teeny, tiny moments of strategy everyone in the organisation can implement every day.

Thus, words being symbols of ideas, we can collect ideas by collecting words. The fellow who said he tried reading the dictionary but couldn’t get the hang of the story simply missed the point: namely, that it is a collection of short stories. – James Webb Young, A Technique for Producing Ideas

This is the sixth post in a series of rambles around the topic of strategy in the general vicinity of user experience which I’m posting as a kind of obituary to the book I almost finished writing then realised was pretty much completely wrong.

I’ve been writing this instead:

  1. Everyone is doing strategy right now
  2. Strategy doesn’t live in a silo (or there is no such thing as UX Strategy)
  3. Strategy fast and slow (or strategy is culture for breakfast, lunch and tea)
  4. Strategy is a team sport
  5. Good strategy is modular