We have recently had the opportunity to mentor several organisations who wanted to increase their internal UX capabilities. As part of our UX coaching service we work with our clients to plan, execute and analyse usability test sessions. There are some common trips and traps that new or inexperienced moderators tend to get caught up in, in response to different types of participants.

Different Types of Participants

All participants are different and unique. We often see:

• Chatty participants who babble about irrelevant things or the same thing over and over.
• Quiet participants who won’t think out aloud.
• Novice internet users who tend to blame themselves for everything.
• Hasty participants who blitz through everything in the 3 seconds you were writing.
• Rude, outspoken or abrupt participants who really want to give you an earful.
• The persistent participant who refuses to give up.

So how do we deal with these types of participants? To help us understand what NOT to do we have identified 5 Star Wars moderator archetypes to demonstrate how normally good qualities in people (e.g. being helpful) can actually harm the usability testing effort.

Moderator Archetypes

• Luke the Hero – wants to rescue the participant and jumps in to ‘save’ them. The problem with rescuing the participant too early is it can make the actual issue more difficult to determine – what were they about to do? At what point would they have discovered the fault themselves? Would it cause them to abandon the site? Often just giving the participant time to make and realise the error will uncover the actual issue or uncover what the participant was trying to achieve.

• Padme the Caregiver - wants to help the participant and ensure they have the correct information. It’s important to avoid telling the participant the correct answer during the test session as this may impact future task successes and may actually make them feel bad. For example, ‘A prospective international student wants to find out if there are any scholarships that might help them afford to study at a university in Australia’ …if they can’t find it, and you help them, they begin to understand the system, their own mental model will evolve, and they won’t have as many issues. You’ve unwittingly diluted their mental model. However, if after the test session there is time, feel free to be nice and show them – after all they’ve just helped you a great deal. it cause them to abandon the site? Often just giving the participant time to make and realise the error will uncover the actual issue or uncover what the participant was trying to achieve.

• Obi-Wan the Explorer – gives the participant too much freedom to explore or talk about an issue unrestrained – while it’s good to let the participant explore when you’re testing a website, too much exploration can also be harmful to the goal of the research (unless that is your goal). Talkative participants are great, but knowing how to get the right words out of them, keep them on task and politely move through the test plan is a skill that experienced moderators learn over time. Politely interject and show your respect for their time, “You’ve been very helpful in assisting me understand the issue here, but I don’t want to keep you here more than your hour, so let’s move on to the next task”.

• C-3PO the scribe – focuses too much on taking notes and documenting every click, navigation pathway, quote etc. and less time on observing behaviour and body language. When you’re writing, you’re looking at your notes, and not focussing on the participant so can miss key behaviours or reactions. If you have the luxury of having an observer to log notes you can focus on moderating but many of us do not. If you are moderating and note taking don’t try to write down everything. Come up with your own shorthand, observing their non-verbal cues, listening to what they are saying, and observing how they’re interacting with and using the test artefact. We tend to focus on the issues, successes and write down good quotes from participants.

• Darth Vader the leader – has a strong personality and tends to unknowingly lead participants using ‘the Force’ (leading questions and body language) to validate their existing view. They often only hear things they want to hear that support what they suspect or believe. Commonly known as confirmation bias. Avoid asking questions such as “Do you like that?” or “Would you use that feature?” which often compel participants to answer “Yes” even if they wouldn’t. Instead ask open questions such as “How do you feel about that?

Moderator tips

As moderators, we need to play 3 important roles:

• Friend – building rapport with participants so they are comfortable.
• Counsellor – being empathetic and getting participants to trust and open up.
• Polygrapher (lie detector) – knowing how to read participants body language and their ‘tells’ when they lie and get them to admit when they lied (in a playful or empathic way).

Here are some tips for being a good moderator:

• Build the participant’s confidence by telling them you know there are issues with the site but you need their help to find them.
• Ensure your body language is neutral - Don’t subconsciously provide feedback or your frustration.
• Acknowledge all participant feedback or criticism – if you defend the design they will just shut down.
• Encourage participants who are nervous and remind them that the issues they are experiencing were experienced by many others so participants do not feel stupid.
• Avoid leading participants through your questions, instructions or interface.
• Allow participants sufficient time to attempt the task by themselves - don’t provide assistance too early. Politely end the task and encourage them to move on to the next if you have the information you need and only provide assistance when the participant indicates they would abandon and you need to test the next step in the process.
• Ensure you acknowledge recommendations from participants (even if not viable).
• Finally, be patient. Participants can be very frustrating but you can’t let them know that.

In conclusion, in the beginning it’s always challenging to develop a new skill, it takes practice and you will probably feel angst, but you’ll pick up this new skill quickly. If you’re new to moderating test sessions or are about to embark on this great journey into understanding the user experience through usability testing, that is great! Don’t despair if you find yourself exhibiting the characteristics of these archetypes. Many of us have been there. You might find it useful to sit down and watch one of your test sessions and self-critique – or sit down with a colleague and talk with them about what you could do different/better next time then you’ll both learn.

If you want to learn more about usability testing and improve your moderating skill consider attending our 3 day UX Design course or contact us about our UX mentoring services and we can help you build your testing skills and capabilities.

UX Tip

Be extremely careful how you ask participants questions. The wording of questions can influence how participants answer a question. For instance, in an episode of Brain Games (Season 3 Episode 7), a group of participants observed a car hitting another car on a movie set. Half the group were asked “How fast was the car going when it smashed into the other car?” and the other half of the group were asked “How fast was the car going when it bumped into the other car?” The group that answered the question incorporating the word ‘smashed’ commented about 40-50mph whilst the group that answered the question incorporating the word ‘bumped’ commented that the car was going around 10-15mph. Participants weren’t lying but their perception of the event was altered by the suggestion and way the question was asked.