Over the past few weeks I’ve been wrapping up and kicking off a bunch of projects. It is during both of these phases that I am reminded how incredibly valuable it is for me, as a UX practitioner, to proactively encourage my clients to clearly define the goal for their project and to create success criteria – ways that we can tell whether or not the project has been successful.
In my experience it is very often left up to me to make sure that the project goals are as clear as they need to be. Many clients come into projects wanting ‘to improve the usability of our website/application/etc’, in my experience that is most often too vague a project goal as to be useful.
The kind of project goals that are really useful go more like this:
We’re getting lots of traffic to the site but not many are joining/buying/contributing/coming back. We want to fix this.
We’re getting lots of calls from people who have visited our website but still don’t know what we do. We want to fix this.
People are coming to our site and doing X but we really want them to start doing Y, we want to find out why this is happening and what we need to do to address it.
As soon as you start defining more specific project goals you can immediately see the way that success criteria start to become immediately apparent.
Measuring success criteria
Some success criteria are immediately apparent and easy to measure, for example return visitors, increased membership, activity or sales. Ideally you want to put some numbers around what you’d consider would define this project as ‘successful’, but even just identifying the metrics that you will use to judge the success of the project is a good start.
Some success criteria are less easy to ‘measure’ but don’t let that discourage you. Often for these kinds of criteria I’ll use a round of research to determine whether or not we’ve been successful – those things that are difficult to quantify are often quite easy to examine using qualitative research. I find myself more and more using a last round of research to ‘check off’ the less quantifiable success criteria for projects.
What’s in it for me?
Failure is a built-in risk of success criteria, but don’t let that put you off. Failing knowledgeably is actually an incredibly useful learning experience for practitioner and client alike (and yes, I speak from experience). I would argue that by defining clear goals and success criteria you are going to do nothing but increase your chances of success in a project.
Clearly defined project goals allow you to make all kinds of good decisions on a project and it can impact everything from design decisions through to who you recruit to participate in design. Just the other day a project I’m working on managed to avoid wasting a number of research sessions on ‘people we felt we had to include’ because we were able to really clearly define why, for the purposes of achieving the goals for this project, they were not our target audience. Without the clear goal we would never have been able to come to this decision.
Clearly defined and measurable success criteria similarly guide you through decision making throughout the project, but they also continue to be useful well after a project has wrapped up. Of course, we use them to judge the success of the project, but we can also use them to communicate this success to others in the organisation and beyond. Clearly defined and measured success criteria give us something tangible to talk about and take much of the ‘touchy feely’ fuzziness out of User Experience and that makes it much easier for more people to understand and appreciate the value of the work we have done.
As you may know, I spent a few months this year working with Mark Boulton and the Drupal community to try to make Drupal 7 (their upcoming release) a Great User Experience. I’ve spent the past weeks reflecting on that experience and trying to understand what we learned from that project and with any luck this will be the first of several reflective posts.
It is all to easy to make excuses for why designing in an open source community can be tough. Certainly there are lots of communication challenges and we don’t necessarily have the right tools. Some people focus on the relationship between designers (minority) and developers (majority) in these communities – I think to do so is to focus on a symptom of the problem and not the cause.
We faced many challenges with the D7UX project, but the greatest of all was to convince the community that the changes we were suggesting were actually going to result in an improvement to their project. There are many who steadfastly resisted our approach and who continue to do so.
It would be very easy to take this personally and to argue about the details of the way the design works (and I most definitely include Information Architecture when I say design). This would also be a fairly typical Drupal thing to do. Actually, I think almost all the issues stem from a much more fundamental place – defining the real purpose of Drupal and it’s real primary target audience.
From the very outset our goal with D7UX was to make Drupal more accessible to people outside of the Drupal community and less technical people – people who didn’t even know what PHP was let alone how to code it. Verity and Jeremy who emerged as part of this definition embody the target audience that the design work that Mark and I were doing in this project. This approach is representative of the goal to make Drupal more of a ‘product’ – an out of the box CMS solution that non-technical users can drive. This is key to the goal to increase the reach of Drupal, as Dries outlined in his keynote at the recent Drupalcon.
There is one big problem with this approach, particularly if it touches the very core of Drupal. The changes that are required to the interface to really achieve the goal that we were tasked with – to really make Drupal understandable to Verity & Jeremy has the consequence of making Drupal a less efficient and enjoyable place for Drupal developers to build cool stuff.
Drupal developers (and some designers!) who want to build cool things with Drupal are the core of the Drupal community. They are the people who make Drupal the incredibly vibrant community that it is. Without these people, Drupal becomes a shadow of it’s current self. These people are more than an important audience, they are the most important audience. What this audience wants is not Drupal as a product that Verity & Jeremy can use out of the box, they want a developer toolkit that gives them more and more flexibility and capability to build cool stuff, and to push Drupal way beyond the realms of a simple Content Management System.
And so we have this tension. Drupal as a ‘Consumer Product’ and Drupal as a ‘Developer Framework’. Currently, the official direction is that the project is going to attempt to be both. I think this is a serious problem.
The target audiences for each of these objectives are so far removed from each other in terms of their tasks & goals, their capabilities, their vocabulary, their priorities. An attempt to devise an interface to suit both will result in an outcome that I expect we’ll see in the release of Drupal 7 – that is a compromise to both parties. Verity is still going to need a lot of support to get anything done in Drupal 7. If Drupal 7 had been designed primarily as developer tool, it would be a much more focussed and efficient tool for developers. As it is now, it is hopefully an improvement on Drupal 6, but certainly not the best that it could be for developers.
For Drupal to really succeed, a decision has to be made, and I think there is only one decision that can be made. Drupal 8 should be designed with developers as the primary target audience and the language, tools, interface should be designed to support them in their goals of building really amazing stuff using Drupal.
That doesn’t mean that there is not still a lot of work for the User Experience people in the Drupal community to do – there is still an enormous learning curve even for those new to Drupal who have great PHP and other technical skills. It’s going to be much easier to address that learning curve with a more finely targeted audience in mind – or more importantly, with the right target audience in mind.
So what of Verity & Jeremy? How will they ever come to know and love Drupal?
There are a number of ways that we can address the experience of non-technical users of Drupal and to ‘productise’ Drupal as a content management system. The most obvious is to design and develop an admin theme that is specifically targeted at these end users that can be applied by developers once the development work is done and the site is being handed over for administration.
I’m not sure where the incentive to design and develop this theme comes from (given that it doesn’t exist today it strikes me that there is a problem here incentive-wise).
There are also opportunities to be explored with installation profiles (see above though re: the fact that they don’t really exist now and no one seems motivated to work on them, where is the incentive?).
And, of course, we have the emergence of tools such as Buzzr from Lullabot and Gardens from Acquia, perhaps also products like Open Atrium from Development Seed can included in this list.
A tough decision but a necessary one
I know that for many people the idea of making a Drupal that Verity can love, making something that can actively compete from a UX perspective with the likes of WordPress, is a grand aspiration. So it is, but unfortunately I also think it is the wrong aspiration for Drupal core.
The sooner we focus on the core target audience of Drupal core – the developers – and commit to making a user experience that supports them in their use of Drupal, the sooner we’ll really have actually achieved a really Great User Experience for Drupal.
If that is really our goal, then let’s get focussed, let’s re-write the design strategy and principles, and let’s take the work we’ve done already and refocus it more tightly on the community we know and love. Focussing on the strength of Drupal and then looking for new and innovative ways to create and motivate the Drupal community to do a better job of looking after our Verity’s and our Jeremy’s.
The best way to a good end user experience is a program of iterative testing and design. There’s been quite a lot of design but not as much iteration as we’d have liked, so now is a good time to really kick off some testing of the D7 interface as it comes into existence so that we can potentially iterate it a little before it is set loose on the unsuspecting public.
In order to take best advantage of the enthusiasm within the community to participate in testing we should ensure that we are taking a generally consistent approach to the test process so that we can better compare our results and make decisions more easily. A way that anyone who wants to can conduct some tests and provide feedback, insight and recommendations, no matter where they are in the world, in a way that we can actually act upon.
Here are some initial thoughts on a framework for testing the D7 interface that can be adopted by anyone within or outside of the community and around the world.
Making use of previous usability test framework:
The Drupal Usability group conducted formal usability testing in Baltimore in 2009 (ref: http://drupal.org/usability-test-university-baltimore-community-solutions) which has informed much of the UX work that has been undertaken over recent months. It makes sense to capitalise on this previous effort as far as possible, not only to reduce workload at this point but also because it will provide a useful benchmark. There are some areas where it is appropriate to make some amendments to the approach taken in the Baltimore testing, these will be discussed below.
As with the Baltimore approach, our testing should target two specific user groups who are important to achieving increased ‘reach’ for Drupal, and who have been at the heart of the D7UX design effort. Baltimore identified these groups as ‘Beginner’ and ‘Intermediate’. This assumes that every end user is at some point on the same Drupal learning curve, which our research has shown not to be the case – this is demonstrated in the example users identified in ‘Who Is D7UX for’ (http://www.d7ux.org/who-is-d7ux-for/), where the users Verity and Jeremy are described.
Verity would equate most closely to the ‘Beginner’ audience used in the Baltimore testing, however the description of ‘Beginners’ it was specified that these people have some development experience, and the script implies that their role may involved building a website using Drupal. Verity, on the other hand, is the recipient end user of a Drupal site who is then tasks with maintaining the content on that website. This is a very typical end user role for Drupal and one which we hope that Drupal 7 will do a much better job of supporting.
Jeremy, however is quite a good fit for the Intermediate audience group recruited in Baltimore, although again, development skills should not be required in order for him to successfully complete the tasks in his test script.
I propose that went we are recruiting people to participate in our user research we compare them to the Verity & Jeremy sketches and if they are similar, they are good matches for testing. Depending on which they match, we will then use a discussion guide and set of tasks tailored to their role and objectives.
The scenario presented in the Baltimore testing was essentially to build and/or manage content on ‘a website for a conference where users will post information about the conference like the schedule, speaker bios and local bars’. This is a great scenario and is suitable for both our Verity and our Jeremy participants and I suggest that we re-use this scenario, the content materials prepared to support the scenario and used in the test, and as far as possible, the script that was devised for the previous round of testing.
The existing beginner script is, I think, almost entirely suitable for the Verity role, and comprises the following tasks:
publish some information about the conference (to existing content types/sections of the site) taking content from a text file provided.
create a navigation link (if we leave anything out of this script, I think this might be it… not sure if this is a real Verity task… could be tho’)
change the look of your site, remove the logo at the top left
creating a new role (speaker) with additional permissions (suggest that perhaps rather than creating a new role, assigning users to a the speaker role might be more appropriate for Verity)
create a new page (schedule) and navigation to that page
The existing Intermediate script is also very suitable for Jeremy, and comprises the following tasks:
publishing some content about the conference to the site
creating a new role (speaker) with additional permissions
using taxonomy to categories ‘workshops’
using a block to highlight content on every page of the site
customising the URL of a particular page
allowing speaker role to use images in their content creation
set up site search
There may be some additional tasks that we want to include, however these scripts are already quite lengthy so perhaps it is best to prioritise tasks and we can iterate the script over time as more testing is completed, to increase task coverage.
Test Materials (What to test on)
I think this is the biggest challenge, especially in the case of Verity, and it’s not something for which I have an immediate solution.
Here’s the problem – it is not reasonable to expect Verity to be able to perform the tasks outlined above using a straight Garland Drupal website, and it is not a reasonably approximation to the reality of this end user.
Garland looks far to much like Drupal and far to little like a contemporary and designed website. It introduces a raft of challenges to understanding the interface that Verity would never have to address in using the Seven admin theme over the top of a professionally designed website.
We need a theme for testing, however simple, that looks like a proper, designed website. And, in the case of Verity, we need a little structure and content already in place on the website.
Now, there are dozens and dozens of Drupalcamp websites out there… is there some way we can get one of those approximated in Drupal 7 in a reasonably short timeframe?
Ideally, Jeremy will have a theme other than Garland as his default Drupal 7 theme also. But, for me, this is less problematic compared to Verity.
Reporting Findings & Making Decisions
Our testing needs to be video recorded (screencast) – I’ll be using Silverback (because it’s cheap and I use a Mac) but there are many other solutions out there. In order for our findings and recommendations to be agreed and accepted it is important that we’re all able to check back to the source.
There is a Drupal UX Group on Vimeo (http://vimeo.com/groups/drupalux) which seems like the logical place to post these videos (to avoid the 10 minute rule on YouTube if nothing else) and it also makes sense to utilise the existing drupalusability.org infrastructure for reporting findings although it will need to be reworked to allow for the ‘crowdsourced’ and iterative nature of this test approach.
Assuming we agree that this is a good way forward, then there are three key tasks that have to be completed:
a) we need to finalise scripts and make them available to people who wish to conduct testing
b) we need to resolve the issue re: a testing theme, and
c) we need to ensure that drupalusability.org is ready to receive feedback from multiple sources
I know that within the UK there are at least two groups who are keen to make a start on doing some testing before the end of September. It would be great to have this framework in place so that we can make the best possible use of their time and expertise, so, time (as ever) is of the essence.
Today is Demo Day at The Economist, where all the various SCRUM teams will show and tell what they’ve achieved in their latest iteration. I thought I might see if I can get into the habit of pre-recording my demo so I can share it with you here and ask for feedback & advice! So, here we go!
In the past two iterations (this project runs on 1 week iterations) we’ve done a combination of research, design and testing the publishing tools that the editorials staff will use to administer what is known as the ‘Channel Index Pages’ – these are pages that you’d land on if you clicked something in the navigation that said, say, Business & Finance, or Science & Technology, and their job is to pull the most interesting current content to the top so that readers can access it more easily.
These pages are made up of a range of elements, most importantly for this demo are News Packages (consisting of a lead story and related stories/media files) and what is known as the Bloggy Chunk (an editable text area that section editors can freely input their own content, it is still very much a work in progress and is variously used as an aggregator for recent interesting stories that The Economist isn’t covering in depth or to highlight interesting reader comments, it may in the future show content from an RSS feed).
Although working within the SCRUM methodology, I am trying to take a systemwide approach to the design as far as possible, so there has been a whole range of questions that I’ve had to consider that aren’t strictly in the ‘stories’ for our sprint (logging on, activating editing, for example), and similarly, I’ve tried to devise interaction approaches that will be reusable across the other parts of the system that we have yet to design rather than just customising the design approach to this individual problem set.
The design approach shown in this demo is still quite lo-fidelity and ‘broad brushstroke’, it will become much more precise over time, at the moment my priority is to try to work through and communicate the ideas as quickly as possible – giving me more time to explore a range of approaches, and to iterate approaches which has been very valuable.
So, if you have a moment and are so inclined, I’d really love to get your thoughts on the approach we’re taking so far. We’re also about to embark on starting some development work on this – to make sure we can build it in Drupal without too much difficulty – so if anyone has any guidance on how best to approach this, modules we should take a look at, anything else you think we might need to know, I’d love if you could share it here!
Look forward to hearing from you and thanks in advance!
(apologies to anyone who saw the first version of this blogpost with the screwy audio synching)
My name is Leisa Reichelt. I am the Head of User Research at the Government Digital Service in the Cabinet Office.
I lead a team of great researchers who work in agile, multidisciplinary digital teams to help continuously connect the people who design products with the people who will use them and support experimentation and ongoing learning in product design.
If you're interested in working with me or would like to talk more please email me