design by committee vs design by community (things we learned from the project)

Recently I presented a casestudy of things that we learned about designing with a community, the Drupal community, at the Interaction09 Conference in Vancouver. (I’m still trying to get my slides down to a reasonable size to post on Slideshare!) It was a short presentation so I thought I’d take some time to flesh out some of the ‘things we learned’ here for anyone who is interested. It certainly was an interesting, challenging and fairly unique project, and we’ll be doing more like this in the future, perhaps you will be too! This is the first post in a series of our learnings.

Often when I talk to other designers about the redesign project they can’t stop themselves from shuddering at the thought of having so many people involved in their design process.

It’s an understandable reaction – after all, how many of us have suffered design by committee, which is really it’s own special circle of hell, in which a group of somewhere between 3-12 (usually) stakeholders with various levels of authority (actual or effective) provide copious and detailed feedback to your designs – feedback that often conflicts either with itself, or with the objectives of the project, or just with the principles of good design. Usually these people are the people who are responsible for paying your salary or invoice. They can’t be ignored. As Whitney Hess tweeted and then blogged, they have itches that need to be scratched.

So, it seems logical that having thousands of people involved in the design process should be even worse right? Design by committee on steroids? Well, you might think so but, happily, you’d be wrong. It’s really a whole different beast with it’s own challenges and opportunities and – I’m happy to report – there is much more good than bad about design by community and it’s an approach that I’d encourage you to consider. (Unlike design by committee, which should be avoided at all costs.)

The main reason for the different experience is scale. Surprisingly, scale is your friend.

When you’re dealing with feedback from hundreds of people you don’t need to address every single issue raised. You’d be mad if you did and have no time for getting the design work done. Rather, what you’re looking for three things:

  1. emergent trends: what are the issues that multiple people are mentioning or agreeing/disagreeing with. If half a dozen people mention it, it’s probably worth looking at.
  2. unexpected comments: every now and then you’ll see something that takes you by surprise. (This doesn’t include comments like ‘your design sucks’ which you will get no matter how wonderful your design is – you have to learn to not be surprised by these!). When you get that ‘surprise’ feeling (you know the one) – pay attention, even if just one person mentions it.
  3. obvious pickups: – with a few thousand fresh sets of eyes, obvious mistakes, things you’ve just left off or misspelled for example, will get picked up quickly. Acknowledge those as quickly as you can so that they don’t turn into big (and often dramatic) conversations.

The absolute best way to a respond to an issue is in your design, rather than in responding to comments on a blog, messageboard, flickr posting, tweet or wherever you’re gathering your feedback (and I’d encourage you to keep it fairly messy and don’t just do it in one place – more on that in a later post!). You should stay in touch with the conversation and respond when appropriate (again, that’s a whole other post!), but the ratio of your responses to comments should be at least 1:10, if not closer to 1:50

This is quite a departure for most of us who are used to consolidated feedback lists and having to respond to every piece of feedback we receive, to begin with it almost feels a little naughty (at least, it did for me!) – but it is a really necessary approach if you want to maintain your integrity and not reliquish your responsibilities as the designer.

Remember – just because you’re working with a community doesn’t make this a democratic process. Design should never be democratic. We’re not voting on interface elements here, we’re working with a community to let them help us the best way they can – by telling us about their community and their product, in this case the website and what they use it for, and drupal itself of course. Communities aren’t designers – they can give you a lot of GREAT information to help you design well for them, but that’s the crux of the issue – you need to find ways to work with them so you can get from them what they do and know best, and so you can do what you do best – design great experiences.

A big part of your role on a project like this is facilitation and communication, but don’t let those roles waylay you from your most important responsibility, which is to do good design.

It’s a terrifying but exhilarating experience, this community design caper. If you have an appropriate project, I’d really encourage you to give it a try. I’ll be sharing more of what we learned soon! redesign – help usability test Iteration 6 next week!

As you may have read, we’ll be doing some usability testing on the 6th iteration of the prototype in London next week. It seems like a great time to also kick off some crowdsourced usability testing, as we’d talked about earlier, and for any of you who’d like to get involved to do so!

(UPDATED!) Iteration six is now live here. I’d like to encourage you to take part in our Crowdsourced Usability Testing Campaign by doing a few tests yourself, wherever you are in the world, and contributing your findings back to the project.

Here’s what you need to do:

  1. Find some participants to take part – we want a mix of people along the spectrum of Drupal involvement from those who don’t know much to those who know lots and are super involved. Some tips for recruiting can be found here (feel free to add any other tips you have to our wiki!)
  2. Take a look at the prototype and work out how you’re going to approach the interview – some interview tips and a sample script can be found here (again, feel free to add more!)
  3. Work out a way to record your interview – some ideas here. Personally, I’ve found remote testing more hassle than it’s worth and much prefer to do in person interviewing. My technology of choice is a MacBook with Silverback installed for audio and video recording (you can get a 30 day trial for free).¬†
  4. Do your interviews!
  5. Share your interviews and findings! I’ve been exporting and posting some interviews on Vimeo, which is my preferred video sharing site. You can put yours wherever you like, just link to them from the comments of this post once they’re posted (and/or add them to the wiki where mine are now) – if you have some time to write up what you’ve learned as a result of the testing that would be fantastic! (If not, don’t worry, we’ll take a look through the video ourselves!)

That’s it! Not so hard at all, is it!

If you have any questions at all, post them here (no matter how silly they may sound, chances are others have exactly the same question or it’s something I forgot to cover in this post or on the wiki!) – I or someone else helpful will get back to you ASAP.

This is a great opportunity to help out with the Drupal project and a great chance to get some usability testing experience under your belt – which is a really fantastic skill to have, whatever aspect of design or development you’re most into. I really encourage you to give it a try and look forward to seeing what you come up with! I’ll be sharing my videos as soon as I can export them after usability testing sessions on Monday 3/11

If you’re able to do some testing early next week and post your feedback mid-late next week that would be fantastic. If this schedule doesn’t work for you – don’t fret – more iterations are coming hot on the heels of this one and more testing will be required and welcomed! You can get involved in the next few weeks if that suits you better.

Good luck, thank you and yay! redesign – Participate in Usability Testing in London!

You’re going to be hearing more and more about usability testing in the coming weeks! As you know, we’ve been including users in the redesign process from before the first wireframe was sketched, and we continue to include both ‘insiders’ and ‘outsiders’ in the process – currently in the form of usability testing the prototypes as it moves from iteration to iteration.

We recently conducted some tests at the Web 2.0 Expo in Berlin which allowed us to talk to a bunch of European Drupal Users (and non-users too!). Now we’d like to do some usability testing in London (where I’m based).

If you’re in London on Monday 3 November and would like to participate – please email me [email protected] and let me know where you will be (I can meet you somewhere convenient) and when would suit you. I’ll need around 30-40mins of your time.

I’m looking for a mix of people who know a *lot* about Drupal (and are involved in the community) through to people who know not much about Drupal but who have some interest in content management systems for websites – whether for your own blog perhaps, or for a company or organisation you’re affiliated with.

Unfortunately, we don’t have any cash incentives for this project, but you will receive much gratitude and lots of good karma for helping build good user experience into an open source project.

I hope to hear from you soon!

User research for redesign – what we’ve done, what we’re doing

If you’re following the redesign project, you will no doubt have noticed that we’ve been busy getting to know the Drupal community. It’s been fun so far, and we look forward to doing much more.

I thought you might enjoy a little heads up of what we’ve done and what our plans are (keeping in mind, of course, that a project quite like this has never been planned before, so we reserve the right to switch course at any point in the future if another course looks better!)

‘Lies, damn lies and statistics’

Now, as you can probably tell, we place a lot more stock in talking to the community than poring over numbers (‘what’ is interesting, ‘why’ is much more useful!), but we’re looking at whatever statistical information is being thrown our way (none of it lies, I’m sure! I just really like that quote)

Here’s a little sample you may or may not be familiar with: has 300,000+ users and gains on average 300 new users per day. There are 375,000 discussion threads between the forums, mailings lists, groups, etc. Our community is active with a 5:1 response rate on general discussions, 4:1 response rate on support discussions and 10:1 response rate on development discussions. Our community is large, engaged and growing. (excerpt from the Redesign Project RFP)

We’re looking at search terms, paths through the site, and other bits and pieces that are somewhat interesting. But nowhere near as interesting as talking… so, onto that part.

Drupalcon Interviews

Mark and I were fortunate to be invited to attend Drupalcon in Szeged, Hungary a couple of weeks ago and we took the opportunity to meet some of the Drupal community whilst we were there. I think we ended up conducting about a dozen depth interviews, between 30-60mins each, with a wide range of participants from a well known ‘maintainer’, through experienced and less experienced Drupal developers, to complete newbies, to people who sell Drupal solutions to their clients – and people from all over the world!

During these interviews we talked a lot about each persons experience with Drupal, with, with the Drupal Community – online and in ‘real’ life. I also asked people to show me a little of how they used which was very enlightening – allowing me to better understand what are the really important bits of for different kinds of users, and which ones of these are all but hidden from the uninitiated!

This was an incredibly worthwhile exercise and really fast-tracked us into understanding both the key issues with the website as well as the dynamics of the Drupal community and the different types of experiences that people have of depending on where they sit on the ‘Drupal Learning Curve’.

Recruiting for research participants – the survey

No doubt you’re aware that we’ve had a ‘survey’ running over the past week or so – the purpose of this is to get in touch with people who are willing to help us out with some more structured feedback once we start getting into the design phase and we have a prototype that we’ll be iterating very regularly (at least weekly is the current plan).

If you’ve completed the form, you *may* get an email from us in the next few months asking for your help – this will involve us taking about 45 mins of your time to have a virtual interview – we’ll be using some screen sharing tools so that you can take a look at the prototype, and we can watch how you’re using it, and we’ll have a bit of a chat, ask you some questions etc. as well. It will be fun, and it will also mean that we won’t waste piles of time designing something that isn’t going to work well for the community OR for people who are new to Drupal because – happily, we have recruited lots of people who know little or nothing about Drupal to help us with this exercise – hurrah!

Want to know more about who responded? Here’s some snippets (apologies for the cut off text on the graphs, this is what Google is giving me and I don’t have time/inclination to remake them!):

1024 people have completed the survey so far. Respondents represent  78 different countries and speak about 56 different languages!! (Not including C++, en-au, English (UK), Java, PHP, E Bonics which some people offered as their native language Рwe take your point!)

10% of respondents don’t know Drupal at all. We’re looking forward to meeting them! Looking after these guys is one area we really need to do better with on

Of course, we also need to look after the existing community well – and there are lots of opportunities for improvement – fortunately we have 540 people from the Drupal community who have already put their hands up to help out.

My favourite thing about this chart though is the 359 people who answered ‘not yet!’ when asked if they were a part of the Drupal community. It might not be particularly scientific, but it really does give a sense of the potential that we’re working with here!

Personally, I was quite surprised with the spread of ‘roles’ that people said they were responsible for. Most respondents selected more than one ‘role’ in this survey and that maps with the interviews we did when we were at Drupalcon – for this reason, we’re leaning towards the view that ‘role’ isn’t really a very good way to break down the different types of people we’re designing for (into personas, perhaps) and we’re leaning more towards a breakdown based on ‘closeness’ to the community. More on that soon.

And what are people doing here?

12% seeking ‘overview’ information about Drupal and what it can do
5% looking to be ‘sold’ on Drupal
28% looking to use Drupal to build a website (introductory information, tutorials, modules, themes etc.)
9% comparing Drupal with other CMS options
20% looking for API-level details about Drupal as a content management framework
21% looking for themes, looking for information on how to make Drupal beautiful
40% looking for an answer to a particular problem I’m having with Drupal
51% looking for Drupal modules
22% participating in the Drupal community – I’m a regular
22% exploring the Drupal community – I’m a lurker
30% checking up on Drupal news
9% other (which was frequently to do with checking on projects, issues etc.)

Other channels

We’re also live on Twitter (, Flickr ( and Slideshare ( and taking feedback, suggestions, inspiration – whatever you can throw at us.

Thank you!

I want to take this opportunity to thank everyone who has participated so far – even after such a short time, I really feel as though we know so much more about this project, its rather long history, and some of the background to issues than we would without your input.

We have to work very quickly on this project, so we don’t have a lot of time to labour over all of this information, but we will continue to work openly so if you don’t see us paying attention in the work that we’re putting back to you, there are lots of opportunities to set us straight!

I look forward to working with you more!