Last week, I created a series of paper prototypes for The Shanghai Source mobile app. I enjoyed the act of constructing the paper prototypes and building the earliest and roughest version of what will eventually become my final app. But, as I sketched and built my paper prototypes, there was still an intense amount of uncertainty—I wasn’t sure if the structure and flow that I had created were the most beneficial for its users. This means it must be time for testing. Testing—which is the fifth stage of the design thinking process—is the opportunity to gather real user insights on your prototypes. As described by UX Designer Leow Hou Teng, “The most effective way of understanding if a design works, is to observe how people use it.”
The most effective way of understanding if a design works, is to observe how people use it.
Leow Hou Teng on Medium
What is usability testing?
Usability testing is a common form of testing in design thinking. It is an opportunity to evaluate the design decisions you have made in your prototype by putting it in front of users and observing how they interact with it. Prototyping and usability testing go hand-in-hand. Testing your prototypes throughout the design process—from paper prototyping to low-fidelity wireframes and high-fidelity comps—creates a continuous feedback loop between designers and users. Testing also saves time, money, and resources by gaining valuable insight into what’s working during the iterative design process rather than afterward.
User feedback is priceless; without an understanding of what users need in order to carry out specific activities and tasks, the iterative process will fail.
Interaction Design Foundation
Before Testing
Although usability testing helps save companies money in the long run, it is still an investment in time and resources. Planning ahead will provide the most value out of every testing session. Here are a few considerations before conducting a test.
1. Build Your Plan
A usability test plan outlines all of the details before any tests can occur. Plans help teams make sure their bases are covered, and they can prepare accordingly. Your plan should answer the following questions:
What are you testing?
The specific sections or flows of a product you need to gain insight on (for example, navigation, task flow, or the layout and information architecture structure).
How, when and where will you test?
Will you be using a paper prototype or digital version. Will, your test be conducted in person at a coffee shop or online from the office? When will you be hosting your tests?
How will you capture results?
Will you be video and/or audio recording your tests? Will a note-taker be present? If you will be recording the tests, do you have a user consent form prepared? Remember to only record if you have received consent from your users before testing begins. Explain why you are recording and where the resources may be used in the future.
Who are your users?
Based on what you are testing, identify who your users should be. Try to use people that fit your user segment and would be most likely to actually use the product. Like ___ says in an article written for Smashing Magazine, “There’s no point setting aside a considerable amount of time to undertake usability testing and then testing random strangers. Spend some time to identify and find the right kind of people for your test.”
What equipment do you need?
If you are recording, you may require a tripod and additional lighting, for example. You’ll also want extra paper prototyping supplies on hand to make revisions on the fly and test new versions based on user feedback.
What are the roles and responsibilities?
When testing, you want to make sure you have the right people in the room. Typically, you’d have a facilitator run the session, the user, and a note-taker who records relevant observations. In paper prototyping, there is usually a “human-computer” as well.
When a user “clicks” on the paper prototype, the “human-computer” will change the screen to imitate the computer interaction. If you are conducting a test on your own, try your best to record the session—you’ll definitely want to review and observe the session in more detail later.
2. Write Your Script
Your script will act as your guide during testing and help you facilitate the session clearly and consistently. Your script should include:
Introduction and Level Setting
Before testing, you should explain to the user how the test will be conducted. Before diving in, they should feel familiar with what will occur and understand completely that they are not being tested—instead, it is the product that is being tested. This is also a perfect time to explain to users that you will only be observing. Since this test should simulate a real-life scenario, you will avoid guiding them in completing tasks.
Tasks You Will Be Testing
The tasks are what you will be testing during your session. Writing a solid task statement will be important in determining the success of your testing. Keep in mind that a user testing session should only run between 30minutes-1hr—longer sessions can result in testing fatigue. With that in mind, 3-5 tasks are plenty.
Consider adding context to your task statements to make the actions feel more relatable; and avoid using the same terms as what you have in your app, if possible. For example, if you want your user to try to book a flight. Instead of asking them to “click on book flight,” you can say, “You want to visit your friend in Vancouver next weekend, use the app to explore options for doing so.”
Below are the tasks I created for my paper prototype testing of The Shanghai Source app.
Task 1
You just downloaded the Shanghai city app and would like to create a new account.
Task 2
You’re interested in finding out what’s happening this weekend.
Now, you want to learn more about a particular event, including how to get there by riding your bike.
Task 3
You are interested in finding out what attractions there are in the city.
Now, you want to find parks located in the district you live in, Jingan.
During Testing
With your plan and script prepared, you are ready to conduct your user tests! Consider the following when running a testing session.
Prepare the Space
Set up the room before users arrive. Ensure your cameras are in place and ready to record. Check the audio and visual quality beforehand—the last thing you want is to conduct a test and later find out the camera angle or audio quality ruined the recording! If testing online, make sure you have a strong internet connection and are in a non-disruptive environment.
Read Your Script Introduction
Once the user is in the room, begin with ice breakers and the introduction you prepared. Ask if there are any questions before getting started.
Show Don’t Tell
Begin working through the list of tasks that you have written. Ask the user to complete each one sequentially. While interacting with the prototype, try to ask the user a few questions to get a sense of their thought process. You can ask them to talk through their experience or what they think will happen next.
Stay Neutral and Observant
Remember, the purpose of usability testing is to gather as much authentic insight as possible, which means facilitators should avoid guiding the users or interfere with their experience. Embrace any criticisms and mistakes—they are a good thing! Every challenging experience for your user can improve the next iteration of the product and make their future experience much more pleasant and intuitive.
Ask Follow-Up Questions
Once your user has finished their tasks, ask them questions about their experience. How did they enjoy using the app? What did they like or dislike?
Conclude the Session
Once the session is complete, thank the user for their time and reward their participation if possible. Celebrate your successful act of testing, pack it up and head home! Later, review your notes and recordings and begin iterating the product based on the testing insights.
Usability Testing on The Shanghai Source Paper Prototype
This past week, I conducted two usability tests on my paper prototype. One was completed in person, and the other virtually, using the POP by Marvel app and Zoom. I found the POP app incredibly easy to use. In a short amount of time, my paper prototype became a digital version ready for testing with anyone in the world.
Each test provided valuable insights that I am glad to have captured, and that I know will make my app better. I’ve called out a few highlights below, but to read the full list of results in greater detail, please download the file linked below.
Virtually Testing using the POP app by Marvel
During the virtual testing with Mike and Val, I gained a ton of feedback in regards to the functionality of the prototype, as well as what was working and not working with the app. Mike and Val validated the inclusion of features like Events, Transportation, Maps, Filtering options, and the Gallery section. They also challenged the clarity of other content sections like News, Resident Resources, and Community Center. I was really happy with how the testing session went and amazed by how much great feedback I was able to collect, using such a low-fidelity prototype. Below is a short clip from our 30-minute testing session.
In-Person Testing using the Paper Prototype
The paper prototyping testing proved to be very insightful as well. It was exciting playing “computer” and moving the pieces around in response to David’s “clicks.” I’m glad that I recorded the session to review in more detail afterward because the session went by very quickly. There were certainly areas of confusion that were insightful to observe—several of which matched Mike and Val’s concerns. David also provided insight into the content hierarchy and which areas of the app he thought would be more relevant than others.
After both testing sessions last week, I’m now a firm believer in the immense potential this method provides for gathering a large amount of feedback for a low amount of effort. Both methods worked extremely well, and I would not hesitate to conduct either one again. I look forward to taking what I’ve learned and revising The Shanghai Source to make it better for users like Mike, Val, and David in the future.
Resources and References
Stage 5 in the Design Thinking Process: Test by Interaction Design Foundation
Writing good task scenarios for usability testing by Leow Hou Teng
A Comprehensive Guide To User Testing by Christopher Murphy on Smashing Magazine
Writings by Michelle Ovalle 