Usability Basics: Running a Test, 1
The usability test introduction that we use at Novell is below. This introduction is designed as a guideline only; I usually glance at this introduction while I am beginning a test, and use it as an outline. The language of the introduction is very formal because we designed it for use in a corporate environment. You should feel free to change the language so that it feels comfortable to you.
“Hello. Thank you for taking part in this usability study.The name of the software I am testing is $APP. $APP is $SUMMARY_OF_APP. I am designing it to $PURPOSE. Your participation in this test will help me to make $APP better, easier to use software.”
The purpose of this paragraph is to give the test subject some context. It is important that she understands why her help matters to you. If you don’t remember to say anything else to the test subject before the test, let her know that you appreciate the favor she is doing for you. Because your test subject wants to perform well on your test, and because testing situations have the tendency to put some pressure on the person being tested, it is important that you make a concious effort to put your test subject at ease.
After you’ve explained what you are going to test, it is time to get some information about your test subject. I do this using the following questions:
“Before we begin, I would like to ask you few questions.
- First, what is your profession?
- Second, how much do you use computers, and what do you use them for? What kind of software do you use most often?
- What operating systems do you use?
- How would you describe your level of computer experience?”
These questions are designed to help you understand your test subject’s background. This gives you insight into her biases and expectations. It is important that you record this information as she is giving it to you. Whether you write down her answers, or tape record them, or capture them on video is not important– just don’t rely on your memory.
Sometimes users are confused by these questions; if they don’t understand what you mean by “operating system”, or if they aren’t sure what you mean by “level of computer experience”, give them concrete examples.
“This test should take around $TIME minutes. During this time, I’m going to ask you to complete $NUMBER_OF_TASKS tasks.”
Try not to let $TIME be more than 30 minutes. Most of the people I’ve tested have begun to get tired after 15 minutes. Remember that the more tired a test subject is, the less likely she is to be able to communicate well with you, and the less fun she will have. If you wear your test subjects out too much, they will not want to do another test with you. A corollary to this guideline is to limit $NUMBER_OF_TASKS to five or fewer.
“While you are completing the tasks, I’m not going to be able to answer any questions. I would be happy to answer any questions you may have when the test is over.”
You don’t want to influence your test subject during the test. After all, if she were using your software in the real world, she wouldn’t have you there to help her. There are a few exceptions to this rule. If you are testing a piece of software that requires specific vocabulary in order to function, then you can give your test subject definitions of the words that she doesn’t understand (though you should do this at the beginning of the test), before you read her the tasks, and only remind her of the definitions if she asks you to. To clarify:
When I was testing the XST Shares Tool, the majority of my test subjects had no idea what “NFS”, “Samba” and “mounted” meant. Since understanding these terms is essential to using the Shares tool, I briefly explained each, and wrote the definitions of them on a whiteboard in the testing room. Occasionally, during the test, a test subject would forget what these terms meant and would ask me. Since I’d only explained the terms once, and since understanding them was crucial to the test, I would remind the test subject of their meanings, and refer her to the whiteboard.
“It’s important that you think out loud while you are taking the test. Tell me everything that comes to mind when you are trying to complete the tasks. It’s okay to complain. I need to know what you like and what you don’t like– and what seems confusing to you. The more that I can understand what you think when you use the software, the more I can improve my design.”
The more your user talks during the test, the better. If you begin the test and she isn’t saying very much, ask her what she is thinking. You may need to do this several times during the test to get the input you need. During the test, record as many of her thoughts as possible. As with her past computer experience, don’t assume that you will be able to remember everything that she tells you.
“We will be using paper prototypes of the $APP for this test. To interact with these prototypes, pretend that your index finger is the mouse. Tell me whether you are clicking once, or twice on the prototype. Also, tell me out loud what you are typing into the interface. And, tell me if you want to use the right-click on something. ”
At this point, it is helpful to have a sample paper prototype to show the test subject. Take a screenshot of a widely familiar application (like Firefox), and print it out. Present it to the test subject, and pretend that you are the one taking the test. Show her how to click on things, how to use various widgets, and how to think outloud.
“Remember that any failure to complete a task is not your fault. I am testing this software because I know that it isn’t perfect, and I want to make it better. I know that some parts of the user interface are difficult to use. If you find that you cannot finish a task, let me know and we’ll move on to something else. ”
This marks the end of the introduction. It is important to emphasize now, before the test begins, that it is okay with you if the test subject wants to stop testing, or if she can’t finish a task. This serves the same purpose as thanking her at the beginning of the introduction; it makes her feel safe, and it takes the pressure to do well on your test off of the test subject. The information you can gather from her is equally useful whether or not she can complete the task. Lastly, before you begin reading you tasks to her, ask her if she has any questions.