If you're new to mobile usability testing, fear not. It is not as hard as you might think but there are some key differences from testing a traditional website in a lab that you need to be aware of.
Over the past year I have done everything from testing a quit-smoking app with a woman whilst she breastfed her five-week-old on her sofa to testing a mobile car insurance website in a lab. I've conducted mobile usability testing with no recording technology out in the field as well as testing using a mobile sled system with webcams that feed into Morae usability test software.
In the hopes that I can save other UX professionals time and effort, here are 8 things that I have learned from all of this mobile usability testing. The following article was recently published in UX Magazine, however, we thought it may be of interest to many of our readers who may not have seen this article.
Lesson #1: How to test mobile paper prototypes
One of my favourite quotes is, " Fix it on the drafting board with an eraser or on the construction site with a sledgehammer " (Frank Lloyd Wright). Testing paper prototypes with users has its merits but how do you test paper prototypes of mobile screens that require scrolling?
Showing users a paper mock-up of a full screen does not really test what they would do when they only see the first screen. My solution was to develop a cardboard template of a smart phone with 2 slits through which a paper prototype can be inserted. This simulates what users would see on one screen. You can easily create your own or download our smartphone template for paper prototype testing and print or stick on cardboard.
Lesson #2: Limitations with screen recorders
The obvious benefit of a using a smartphone screen recorder is that it is unobtrusive and does not get in the way of testing. Users can use the device as they typically would in any context or environment.
Apple isn't big on approving screen-recording apps for sale in the iTunes store. There are screen recording apps such as Display Recorder (from Cydia) that you can use to record screen activity but personally I don't want to jail break my iPhone in order to record a mobile test session. There are a few Android screen recorder apps available, but even if you find a screen recorder app for your smartphone, there are a few issues.
- Screen recorders don't typically record users' gestures, such as trying to tap 10 times on a tiny link or a user attempting to use a slider on a touch screen.
- They often have limited recording times and capabilities. You will fill your phone up very quickly if recording one-hour sessions.
- They generally can't be used to record screen activity of another app so you may be limited to recording mobile web sessions only.
- You can't use the test participant's own device unless you want to install an expensive app on their iPhone, which they may not be too keen about.
- They don't integrate with usability testing software such as Morae, which allows observers to unobtrusively observe a session in real time.
- You can't typically record user's facial expressions and verbal comments like you can with standard usability testing software such as Morae.
There are two iOS app that I know of that partially address the last issue. One is called UX Recorder. It does a reasonably good job of recording screen activity, captures some user gestures (such as circles for taps and arrows for swipes), and records a user's voice and face (picture-in-picture) using the own iPhone camera. Its biggest drawback is you can't use it to test mobile apps. You can only use it to test mobile web.
The second is a relatively new iOS app is called Magitest, which in my view is more usable than UX Recorder. Magitest captures tap gestures only at this point (although they are soon to add swipe capture I believe), as well as user's voice and face using the iPhone camera (picture-in-picture). More importantly, using Magitest Native, you can also record screen activity using other iOS apps. It is also quite reasonable priced.
Magitest app and example output
Lesson #3: Benefits of using mobile testing sleds
The alternative to screen recorders is a mobile or tablet testing sled that will allow you to mount a small camera over the mobile device. There are a few key benefits in using sleds.
You can record users' gestures, which are really important for mobile and tablet usability testing. We once had a user trying about 10 times to use a slider on a tablet before giving up. You can also plug your sled webcam into your computer and input into testing software such as Morae. The great thing about Morae is that it lets you input two cameras so you can also record the user's face and voice using a second webcam. You can then record user's screen activity, gestures, face and voice and stream all of this live across your network to an observation room or create video clips later to effectively communicate issues to stakeholders.
Example output using a mobile sled for testing with Morae test software.
Nearly all the UX practitioners I know are using DIY sled systems that they have made themselves. We have investigated a number of mobile test sleds and developed a few of our own over the last 2 years. Here are some of the things to be aware of:
- The sled must be light and easy for test participants to hold.
- The sled should not slow down users or get in their way.
- The sled should accommodate different sized devices so that you can use test participants own device.
- The camera mount ideally needs to be adjustable to accommodate user preferences and different devices.
- The sled should allow users to easily switch device as well as orientation i.e. portrait to landscape view without having to move the camera position.
- The whole sled set up must still be stable & not move during testing.
Lesson #4: What cameras work best for mobile testing
When we started out with our first sled, we used two Logitech webcams: C910 and C615. These webcams offer good quality, high definition recordings, plug into Morae test software and have easy to use drivers. The main disadvantage is the size and weight. The Logitech C910 is also difficult to connect to a sled.
We found the Microsoft Lifecam offers good 720p resolution, works with Morae and is easy to connect to the sled as it can bolt on. The downside of this camera is still the size and weight, less flexibility in the camera mount, and camera software we find difficult to use.
An alternative to webcams is the IPevo document camera. What we like most about this camera is that it smaller and very light weight, has a very high resolution, plugs into Morae, and comes with a flexible light plastic arm. The down side is that is it difficult to attach to a sled and the frame rate is a bit slow and jerky as it is meant for recording documents.
Our favourite webcam is the Hue HD webcam. It is very light, offers good resolution and frame rates, comes with a flexible bendable arm and plugs directly into a PC and Morae. It even comes in pretty colours such as pink and green but as we don't want to draw attention to the camera we use plain old black.
Lesson #5: There are times when you should use no technology
So far I've mainly focussed on mobile testing technology but there are times when I advocate using no technology (with the exception of the user's phone of course). As I mentioned, I recently conducted testing of a quit smoking app designed for pregnant women. We ended up recruiting mothers with babies who had recently quit smoking or were attempting to quit. Given the sensitive nature of the topic (women who had tried to quit smoking when pregnant) and who I was testing with (new mothers with babies), I needed to conduct many of the test sessions in users' homes.
As a mother with young children, it was easy to build rapport with these women and for them to open up to me as I attempted to make the testing as casual as possible. One of the test sessions was even run on a participant's sofa whilst she was breastfeeding her 5 month old. She had baby in one arm and tested the app with her iPhone in her other hand. Obviously it was not appropriate to record these sessions in any way and the intrusion of any additional technology would have potentially affected the results. Sometimes it is better not to record at all.
Lesson #6: Context may not always be as important as you think
We hear that context is critical for mobile design but it is more important for some types of sites or apps than others. There are some interesting mobile web statistics from comScore that actually indicate that the highest use of mobile devices is actually early in the morning and after 8pm at night, presumably when most people are home.
My advice is that you need to consider where users are likely to use your app. For instance, many users of the quit smoking app for pregnant women said they would probably use the app alone at home when they have limited resolve and their phone is on hand. We also worked on a public transport journey planner for mobile, which would much more likely be used when users are out and about walking down the street. For that mobile site context is critical.
Lesson #7: How to simulate users' context and environment
In a lab environment, we may have a stable wireless connection and not be moving about but this doesn't happen in real life. People use their phone while on trains, buses, and waiting in a queue. Signals drop out. Activity often occurs in short bursts such as during TV ad breaks. So how do we test for this?
One of our insurance clients was interested in understanding users' expectations regarding what would happen in the event they lost their signal or if they left their session and returned to it 30 minutes later. Halfway through their task of getting an insurance quote, I actually stopped users and said "imagine you were doing this on a train and the signal dropped out as you went through a tunnel". I then asked them what they would expect to happen and showed them a screenshot to ask them what they would do. So even though we could not test on a train, I managed to simulate their experience.
When I conducted testing of the quit smoking mobile app, some users preferred to come into our test lab. Instead of using our lab test room with cameras, desks, and computers, I tried to recreate a "home" like environment and we ran the sessions on our sofa in our test observation room and didn't use any technology to record the session. Even though this wasn't the users' true context and environment, we simulated their home environment and the test outcomes were consistent with sessions I ran in other users' homes.
Lesson #8: Using users' own technology will discover more issues
When we conducted testing of a mobile insurance website, we found that Galaxy and iPhone users loved the pickers for date input. However, I had one user with a HTC phone that really struggled with the date picker for inputs such as years. The year values in the date picker were very slow on the HTC phone. When he scrolled quickly it scrolled the whole page.
When testing a public transport journey planner, we used users' own mobile carrier network connections and some screens/search results were really slow to load. This had an effect on the user experience and user satisfaction. This was good to understand and inform the client prior to launch.
So by using user's own devices and network connections we found issues that we would not have picked up if we had just used our iPhone and fast office wireless connection.
In conclusion
With nearly a third of all page views now coming from mobile and tablets in some western countries (and even more in many developing countries) UX practitioners and researchers need to embrace mobile. Mobile is no longer an area that a few UX people specialize in, and we need to start designing and usability testing everything for smartphones and tablets as well as computers.
Hopefully this article will save UX practitioners new to mobile some of the pain I have been through.
If you are interested in finding out more about our upcoming mobile test sled, which we are hoping to make available to the wider UX community, please contact [email protected].
Usability tip
When designing for mobile you need to design for distraction.
When users access their smartphone or tablet it may be whilst waiting in a bus line, during an ad break whilst watching TV, whilst talking to their mum on the phone or actually while they are watching their favourite TV show. As such, you need to assume that users will be distracted and will not be giving their full attention to your mobile website or app. They may also be accessing your site or app in short focussed bursts of activity then pause for 5 minutes as they get on a bus or their TV show resumes. So how do you design for this? Be sure to provide clear visual feedback regarding whose site they are on. Provide clear signposts e.g. home button and/or logo and clear page headings. If possible, don't timeout too quickly if users are inputting data or shopping online and protect users' work and data they have entered. If testing your site, simulate this experience by giving your users a random page and seeing if they can figure out where they are and whose site they are on.