Triple testing your survey

Sending a survey is a convenient way to gather data quickly. But, it’s very easy to inadvertently gather misleading and inaccurate data.

When was the last time you filled in a survey that let you actually express what your really thought about an organisation, experience or topic? Just because you have a reasonably large sample size and you can make graphs out if it doesn’t mean it is good data with which you could be making important decisions. Data quality matters.

A good way to make sure you’re getting reliable data (and making good use of your survey respondents’ time) is to do a triple test before you hit send.

Here’s what you do.

  1. Create your survey (this is actually not as simple as it may seem)
  2. Find someone who could be a potential respondent for your survey (matches the target audience, not people in your team or the people who sit closest too you)
  3. Ask them to complete the survey, watch them while they do it, ask them questions to see whether they understand what the question means and whether the way your are collecting the answers allows them to give the answer they want to give
  4. Adjust the form based on what you have observed (there are always adjustments you will want to make)
  5. Repeat steps 2,3,4 until you’ve seen at least three people complete the survey OR you’re certain there is no more you can do to adjust the survey so that people understand the questions and can provide meaningful (to them) responses.

I have never known someone who has tested their survey this way and who didn’t make changes that would result in a better experience for respondents and better quality data.

Guerrilla empathy (or why we should probably stop banging on about users all the time)

If you work anywhere near digital design, someone has probably talked at you about empathy recently. Or you’ve talked at people about empathy. Empathy is a buzzword du jour.

Now, you and i know empathy is important but – the reality is, most of the people we work with don’t really believe that. They’d don’t. They think they do real work and they think that we are like tree-huggers but for users.

Banging on about empathy while they are trying to do their ‘real jobs’ doesn’t help get them to care about users more. We need to take a more empathetic approach to empathy.

Guerrilla empathy perhaps.

Here are some things we know.

  1. If we work in multidisciplinary teams where research is done regularly and everyone (including senior stake holders) observe users for at least 2 hours every six weeks, we make better services. Our friends at UIE (thanks Jared) did research to evidence this. Everyone watching users use our stuff regularly helps us make better stuff.
  2. People (say they) love to use evidence to make decisions. They love data. User research provides evidence. If you take a methodical, hypothesis led approach to user research, your team learns about what works and what doesn’t so it can make things work better. (Qualitative evidence is data too).
  3. Businesses care about business outcomes – this might be policy outcomes, compliance rates, fraud reduction, members sign ups or sales. We know that making all of these things work better is always easier if end users understand what you are trying to tell them and can actually do the thing you are wanting them to do. Good usability helps achieve business outcomes.

So, in doing user research (which can build empathy) there are at least three things that business wants that we can supply: better services, evidence for decision making and achieving business outcomes.

We need to talk more about these three things.

And then we need to not let them forget the bit in part 1 that requires them to come observe real users regularly in order for it to work. And we need to make sure that we do the work in a way that really focusses on delivering these benefits for the business. And then you have a much more compelling reason for people to come observe users.

Get people to observe end users regularly in order to meet their business objectives and – unless you have an organisation full of sociopaths – empathy will naturally follow. Without you even mentioning it.

An empathetic team is transformational. But empathy is difficult to sell – especially to the senior stakeholders who need it the most. Business outcomes are not hard to sell.

Do empathy by stealth. Stop talking about empathy. Let empathy be the by-product of helping your organisation meet its objectives through user research and demonstrate this by taking a methodical, collaborative, hypothesis driven approach to your work.

Then stand back and wonder, yet again, at empathy’s power to transform teams and organisations.

Stop your team using technical terms and jargon

Most weeks I am ridiculed by someone for insisting on plain language – avoiding acronyms and technical language / jargon in particular. People tell me that I’m slowing the team down by making them use proper words, and that their end users or stakeholders expect them to use technical language.

These things are both true. You should still use plain language.

Technical language is exclusive.

If you know all the fancy terminology then you’re allowed to fully participate, if not, you’re at least partially excluded. This is unfriendly and can also severely limit the number and type of people who can fully contribute to the work your team is doing.

Recently someone insisted that they needed an incredibly experienced and excellent user researcher who was an expert in all the technologies associated with commercial banking. Do you know how many of those people their are in the entire world… not many.

Do you really want to make hiring that hard for yourself?

If you use plain English in your team, then more smart people can be involved and help you to think about ways that you can improve. That is a good thing.

Technical language bundles assumptions.

If you are in the business of transforming service design, then you are in the business of questioning assumptions. You are constantly asking – what is this really? how does it really work? is it actually the right thing? does it really need to be that way?

One of the best ways we can do this is to keep questioning things until they are broken down into their most basic parts.

We stop calling things by their name (‘we’re working on the registration form‘) and we start talking about what things do (‘we’re working on a way people can to access health benefits more easily‘).

We can recognise opportunities to design things better and differently by describing and understanding the real intent of each component.

Technical terms and jargon almost always bundle up assumptions about the necessity of a thing that – if unchallenged – allow teams to completely miss the real opportunities for transformation.

Plain language is slower.

Yes, switching to using plain language is slower for those who are proficient in the technical terms. It’s not slower for the rest of us. Anyway, there is no advantage in doing the wrong thing faster. (Or, at least, there shouldn’t be).

Plain language is unexpected.

Yes, your colleagues in banking, government, etc. will be surprised and maybe even scornful when you stop doing the linguistic secret handshake and start using plain language. This is no reason to revert to jargon. This is your chance to change culture, one word at a time. Really, it’s the easiest bit of transformation you can do in your organisation ever single day.

Starting now.

where does crazy come from?

You can imagine how this went down.

The lawyers decided that the privacy policy needed updating, so they updated it and put it on the website.

Then they decided that they had to make sure people knew about it. Who knows why… I can only imagine because they’re actually starting to do something that they know people probably won’t really be happy with (or they’re beginning to admit that’s what they’re doing), so they need to prove that a certain number of customers knew about it.

They say, ‘put a sign up in the stores, we have to do this because this is a risk to the business. It could damage our brand and cost a fortune in law suits’.

No one wants to be the person who is putting the company at risk and responsible for lawsuits, so they don’t ask any questions except for what is the cheapest and easiest way to meet the lawyers’ needs.

I imagine the email the lawyers sent (it would totally have been an email and not a meeting), the meeting to discuss what to do about the lawyers email, all the emails to argue about which type of signage they’d use, what would get bumped so this sign could exist, arguing over the wording. And finally, the meeting where someone approved this.

And so it is that, as I go to buy some paracetamol, I am informed that the privacy policy has changed and I can find out about this on the website.

I wonder – is this just the website privacy policy, or the Boots loyalty card privacy policy, are those the same things or different?

I wonder – how many people would pull out their phones or rush back to their desks to find out what changes have been made (I did… I couldn’t resist. There’s nothing to draw your attention to this information on their website and the section on privacy doesn’t make any reference to recent changes). Even if I do want to find out about the changes to this policy, I can’t see how on earth I might meet that need on their website.

I wonder – did they think about how it would make people feel or act to see this.

I wonder – did anyone fight for this not to happen? They probably did. Fear won.

I wonder – if a law suit did come up, could Boots actually prove that this has sufficiently informed people of the change to the privacy policy? Probably not.

I wonder – what opportunities were lost to actually understand the end user need and design a way of meeting that need in a way that benefitted the relationship between Boots and their customers.

You see these little things that companies do, you can see that all they’re doing is ‘ticking a box’, a token effort for the lawyers or whoever else, and you know that they’re pushing the burden onto the customer, and probably missing gigantic opportunities to be really great.

Fixing the crazy way things like this happen is at least as important as designing the shiny signs and the websites.