*This article has been updated since it was originally published in 2016*
Tight budgets are nothing new in product development land. Going lean & mean to meet budget and time constraints happens all the time. The one thing that shouldn’t be avoided though is conducting a usability evaluation of your actual site or system before you go live. Here at PeakXD, our favourite method to evaluate the usability of a site or system is usability testing with real users.
As ‘expert’ user experience (UX) practitioners we are often asked to conduct an expert review of a site or system when budget or timing is tight. An expert review typically a desktop research method performed by an experienced UX practitioner, which keeps the costs low. This research method has its benefits but it also has drawbacks.
Let’s explore the pros and cons of expert reviews and usability testing (sometimes called user testing or UX testing), drawing on some work we did for a client which had some unexpected outcomes.
Exploring expert reviews
Sometimes known as a ‘discount usability method’, conducting an expert review involves using desktop usability evaluation methods (UEMs) to evaluate a site or system using heuristic checklists and design guidelines as well as cognitive walkthroughs using personas.
The main benefit of expert reviews are that they are a quick and affordable research method. They are best completed by an experienced UX practitioner who understands interaction design principles and has conducted a large number of usability tests with users.
We often find though, that business stakeholders don’t accept “just one person's opinion” as something worth making changes for, even when the evaluator has years of experience in the field of UX. At PeakXD we often identify UX issues before testing with actual customers but we know that we need the evidence to convince our clients’ cynical stakeholders.
Understanding usability testing
Conversely, usability testing with real users often provides our clients' project team with the evidence they need for issues they’ve known about for years but were unable to get buy-in from other stakeholders to address. It becomes very hard for business stakeholders to refute issues when they see real users (or customers) failing to complete tasks on their site or system and suffering a poor user experience.
The benefits of usability testing generally outweigh the benefits of expert reviews:
- less reliant on the evaluator’s knowledge (although experienced testers will pick up a lot more issues)
- are more accurate as they reflect real users attempting common tasks using your site
- are less subjective and results are likely to be accepted by business stakeholders.
Case study: Why usability testing is important
An insurance client approached us to assess a portion of a online quote that had just been launched. They wanted to get a better understanding of any usability issues in their new online quote. Our client only had a very small budget, so our first reaction was to conduct an expert review of the system. However, we had difficulty justifying the value of an expert review when, on initial inspection, there appeared to be very few usability issues. The online transaction seemed to be nicely designed, flowed well, and was mobile & tablet friendly.
So instead, we proposed an alternative approach of ‘guerrilla’ usability testing. Because of the widespread usage of this type of online transaction by most Australian adults, we were able to source a wide enough range of participants to conduct 10 X 15-minute usability test sessions.
Expecting the test participants to blitz through the online transaction, we were somewhat surprised to discover a number of usability issues. This was due to the wide range of the participants – each with their own unique situations, needs, and mental models.
A simple question asking “How is the property occupied?” proved difficult for landlords to answer without being taken down the wrong path.
Another question asking “Is there any business activity conducted from the home?” didn’t provide a way to distinguish between working from home for an employer and running a business entirely from a home.
This was then further misconstrued through the follow up question “How many employees (other than household members) work at the insured address?” as one participant considered how this related to his family-based café business, when they only did minor bookkeeping duties from home.
In total we identified 20 issues through the usability testing with real users, which was quite unexpected.
Had we simply conducted an expert review of the system, we don’t believe we would have identified most of these issues. It just shows that testing with real users, each with their own unique circumstances and mental models, is the most effective way to identify usability issues.
The right method for you
To decide which research method you should use for your project, you need to assess the project objectives and constraints. Think about the available budget, the required time frames, and also the expectations of your stakeholders.
To learn more about our usability evaluation methods including expert reviews and usability testing, attend one of our upcoming UX design or short courses - see all the dates and locations at www.peakxd.com.au/training.