Design Thinking ICM517

Putting Paper Prototypes to the Test

Gaining highly valuable feedback using low fidelity prototypes.

Last week, I created a series of paper prototypes for The Shanghai Source mobile app. I enjoyed the act of constructing the paper prototypes and building the earliest and roughest version of what will eventually become my final app. But, as I sketched and built my paper prototypes, there was still an intense amount of uncertainty—I wasn’t sure if the structure and flow that I had created were the most beneficial for its users. This means it must be time for testing. Testing—which is the fifth stage of the design thinking process—is the opportunity to gather real user insights on your prototypes. As described by UX Designer Leow Hou Teng, “The most effective way of understanding if a design works, is to observe how people use it.”

The most effective way of understanding if a design works, is to observe how people use it.

Leow Hou Teng on Medium

What is usability testing?

Usability testing is a common form of testing in design thinking. It is an opportunity to evaluate the design decisions you have made in your prototype by putting it in front of users and observing how they interact with it. Prototyping and usability testing go hand-in-hand. Testing your prototypes throughout the design process—from paper prototyping to low-fidelity wireframes and high-fidelity comps—creates a continuous feedback loop between designers and users. Testing also saves time, money, and resources by gaining valuable insight into what’s working during the iterative design process rather than afterward.

User feedback is priceless; without an understanding of what users need in order to carry out specific activities and tasks, the iterative process will fail.

Interaction Design Foundation

Before Testing

Although usability testing helps save companies money in the long run, it is still an investment in time and resources. Planning ahead will provide the most value out of every testing session. Here are a few considerations before conducting a test.

1. Build Your Plan

A usability test plan outlines all of the details before any tests can occur. Plans help teams make sure their bases are covered, and they can prepare accordingly. Your plan should answer the following questions:

What are you testing?

The specific sections or flows of a product you need to gain insight on (for example, navigation, task flow, or the layout and information architecture structure).

How, when and where will you test?

Will you be using a paper prototype or digital version. Will, your test be conducted in person at a coffee shop or online from the office? When will you be hosting your tests?

How will you capture results?

Will you be video and/or audio recording your tests? Will a note-taker be present? If you will be recording the tests, do you have a user consent form prepared? Remember to only record if you have received consent from your users before testing begins. Explain why you are recording and where the resources may be used in the future.  

Who are your users?

Based on what you are testing, identify who your users should be. Try to use people that fit your user segment and would be most likely to actually use the product. Like ___ says in an article written for Smashing Magazine, “There’s no point setting aside a considerable amount of time to undertake usability testing and then testing random strangers. Spend some time to identify and find the right kind of people for your test.”

What equipment do you need?

If you are recording, you may require a tripod and additional lighting, for example. You’ll also want extra paper prototyping supplies on hand to make revisions on the fly and test new versions based on user feedback.

What are the roles and responsibilities?

When testing, you want to make sure you have the right people in the room. Typically, you’d have a facilitator run the session, the user, and a note-taker who records relevant observations. In paper prototyping, there is usually a “human-computer” as well.

When a user “clicks” on the paper prototype, the “human-computer” will change the screen to imitate the computer interaction. If you are conducting a test on your own, try your best to record the session—you’ll definitely want to review and observe the session in more detail later. 

Above is the plan I built in preparation for the testing of my paper prototypes for The Shanghai Source.

2. Write Your Script

Your script will act as your guide during testing and help you facilitate the session clearly and consistently. Your script should include:

Introduction and Level Setting

Before testing, you should explain to the user how the test will be conducted. Before diving in, they should feel familiar with what will occur and understand completely that they are not being tested—instead, it is the product that is being tested. This is also a perfect time to explain to users that you will only be observing. Since this test should simulate a real-life scenario, you will avoid guiding them in completing tasks. 

In the script, I tried to introduce the app itself, so users felt confident they understood the context of the tasks, and what they will be asked to do.
Tasks You Will Be Testing

The tasks are what you will be testing during your session. Writing a solid task statement will be important in determining the success of your testing. Keep in mind that a user testing session should only run between 30minutes-1hr—longer sessions can result in testing fatigue. With that in mind, 3-5 tasks are plenty. 

Consider adding context to your task statements to make the actions feel more relatable; and avoid using the same terms as what you have in your app, if possible. For example, if you want your user to try to book a flight. Instead of asking them to “click on book flight,” you can say, “You want to visit your friend in Vancouver next weekend, use the app to explore options for doing so.”

Below are the tasks I created for my paper prototype testing of The Shanghai Source app.

Task 1

You just downloaded the Shanghai city app and would like to create a new account.

Task 2

You’re interested in finding out what’s happening this weekend.

Now, you want to learn more about a particular event, including how to get there by riding your bike.

Task 3

You are interested in finding out what attractions there are in the city.

Now, you want to find parks located in the district you live in, Jingan.

During Testing

With your plan and script prepared, you are ready to conduct your user tests! Consider the following when running a testing session.

Prepare the Space

Set up the room before users arrive. Ensure your cameras are in place and ready to record. Check the audio and visual quality beforehand—the last thing you want is to conduct a test and later find out the camera angle or audio quality ruined the recording! If testing online, make sure you have a strong internet connection and are in a non-disruptive environment.

Read Your Script Introduction

Once the user is in the room, begin with ice breakers and the introduction you prepared. Ask if there are any questions before getting started.

Show Don’t Tell 

Begin working through the list of tasks that you have written. Ask the user to complete each one sequentially. While interacting with the prototype, try to ask the user a few questions to get a sense of their thought process. You can ask them to talk through their experience or what they think will happen next. 

Stay Neutral and Observant 

Remember, the purpose of usability testing is to gather as much authentic insight as possible, which means facilitators should avoid guiding the users or interfere with their experience. Embrace any criticisms and mistakes—they are a good thing! Every challenging experience for your user can improve the next iteration of the product and make their future experience much more pleasant and intuitive.

Ask Follow-Up Questions 

Once your user has finished their tasks, ask them questions about their experience. How did they enjoy using the app? What did they like or dislike?

Conclude the Session

Once the session is complete, thank the user for their time and reward their participation if possible. Celebrate your successful act of testing, pack it up and head home! Later, review your notes and recordings and begin iterating the product based on the testing insights.

Usability Testing on The Shanghai Source Paper Prototype

This past week, I conducted two usability tests on my paper prototype. One was completed in person, and the other virtually, using the POP by Marvel app and Zoom. I found the POP app incredibly easy to use. In a short amount of time, my paper prototype became a digital version ready for testing with anyone in the world.

Each test provided valuable insights that I am glad to have captured, and that I know will make my app better. I’ve called out a few highlights below, but to read the full list of results in greater detail, please download the file linked below.

Virtually Testing using the POP app by Marvel

During the virtual testing with Mike and Val, I gained a ton of feedback in regards to the functionality of the prototype, as well as what was working and not working with the app. Mike and Val validated the inclusion of features like Events, Transportation, Maps, Filtering options, and the Gallery section. They also challenged the clarity of other content sections like News, Resident Resources, and Community Center. I was really happy with how the testing session went and amazed by how much great feedback I was able to collect, using such a low-fidelity prototype. Below is a short clip from our 30-minute testing session.

Respecting the privacy of my testers, I have grayed out their screen. The clip above shows a task I asked Mike and Val to complete. Ending the tasks with a “task completion” notification helped my users understand that they’ve officially reached the end of the task.
In-Person Testing using the Paper Prototype

The paper prototyping testing proved to be very insightful as well. It was exciting playing “computer” and moving the pieces around in response to David’s “clicks.” I’m glad that I recorded the session to review in more detail afterward because the session went by very quickly. There were certainly areas of confusion that were insightful to observe—several of which matched Mike and Val’s concerns. David also provided insight into the content hierarchy and which areas of the app he thought would be more relevant than others.

This clip is taken from the in-person testing session I conducted with David. By observing areas in which the he didn’t know exactly where to click, I can now go back to my IA and wireframes and make necessary improvements.

After both testing sessions last week, I’m now a firm believer in the immense potential this method provides for gathering a large amount of feedback for a low amount of effort. Both methods worked extremely well, and I would not hesitate to conduct either one again. I look forward to taking what I’ve learned and revising The Shanghai Source to make it better for users like Mike, Val, and David in the future.

Resources and References

Stage 5 in the Design Thinking Process: Test by Interaction Design Foundation

Writing good task scenarios for usability testing by Leow Hou Teng

A Comprehensive Guide To User Testing by Christopher Murphy on Smashing Magazine

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s