Usability testing

Introductory articles

  • 5 keys to a successful usability test
    Let's suppose that the "usability" mantra you've heard over the past few years has finally convinced you that conducting user research with your target audience is an essential aspect of your company's website development efforts. Here are five things to consider that will help you plan a successful test, set realistic expectations with your colleagues, stick to a budget, and wind up with actionable results.

Discussion articles

  • 5-second tests: measuring your site's content pages
    "As the name suggests, the 5-Second Test involves showing users a single content page for a quick 5 seconds to gather their initial impressions. Five seconds may not seem like a lot of time, but users make important judgments in the first moments they visit a page. This technique unveils how those judgments turn out, giving the team insight into some essential information about the page. Using this technique, we've found the information we've gathered essential for making huge improvements to our clients' sites."
    (Christine Perfetti)

  • 8 guidelines for usability testing
    "In professional web design circles, the usability testing session has become an essential component of any major project. Similar to focus groups in brand development and product launches, usability testing offers a rare opportunity to receive feedback from the very people the website is aimed at - before it's too late to do anything about it. But how can you get the most from these usability testing sessions?"
    (Tim Fidgeon - Webcredible)

  • 90% of all usability testing is useless
    Usability testing for the web doesn’t require outside firms that hold themselves separate from the design process. Understanding context means embracing this research as part of the design process. It should be performed with the full participation of the design team, so that everyone can internalise the user’s perspective. Developing and leading usability tests should be a core competency owned by someone within your team (though many folks believe it ought not to be the original designer).

  • A proposal for evaluating usability testing methods: the practical review system
    The purpose of this article is to explain the Practical Review System (PRS). The PRS is an outline of 28 characteristics that can be used to understand any usability method, thereby allowing any individual to decide between methods. This solves many of the problems associated with understanding and explaining usability methods. 

  • Archiving usability reports
    "Most usability practitioners don't derive full value from their user tests because they don't systematically archive the reports. An intranet-based usability archive offers four substantial benefits."
    (Jakob Nielsen)

  • Authentic behavior in user testing
    "Despite being an artificial situation, user testing generates realistic findings because people engage strongly with the tasks and suspend their disbelief."
    (Jakob Nielsen)

  • Beginner's guide to moderating a usability study
    " Here are some considerations and steps I usually take when I’m moderating a usability test. For those who are experienced, I encourage you to add your thoughts. For those of you who are learning, I encourage you to ask questions of the other readers here. This article does not discuss the screening for candidates, setting up the test scenarios, nor the reporting of the data. "
    (Kevin Cheng)

  • Caroline's corner: estimating testing the plausible way
    How do you estimate the time it will take to do usability testing? Caroline Jarrett suggests two approaches to take.

  • Checklist on usability testing for accessibility
    A comprehensive checklist for planning, preparing, conducting and reporting on usability testing for accessibility.

  • Checkpoints for reviewing usability test reports
    Usability practitioners are called on, not only to conduct many research studies during their careers, but also to read, review, and advise on usability studies that have been conducted and reported by others. The ability to critically review the research of others, and to help stakeholders weigh up the merits or shortcomings of research data and conclusions, is an extremely valuable skill. These checkpoints will help you ensure your review covers the key issues.
    (Philip Hodgson)

  • Cleaning up for the housekeeper
    Once in a while a client will tilt their head and look at me with one of those smiles. “You want to do expert review and then also usability testing?” they say. “Is this one of those consulting tricks? Why would I need to do both?” Doing ER is like straightening up before the housekeeper gets there. If you conduct ER first, ER provides feedback that allows the developers to ‘tidy up’ the interface so that the usability testing can focus on cleaning. If you don’t straighten first, both the tester and the participant are distracted and waste time. However, if the right methods are applied at the right time, ultimate outcome is a really clean house….er interface.

  • Comparative usability evaluation
    Rolf Molich of Dialog Design has overseen extensive research into usability testing through his comparative usability evaluation series. Four separate research projects have now been conducted. The methodology and results are discussed on this page.

  • Conducting and using usability tests
    Usability testing encompasses a range of methods for identifying how users actually interact with a prototype or a complete site. In a typical approach, users--one at a time or two working together--use the website to perform tasks, while one or more people watch, listen, and take notes.

  • Conducting usability testing for accessibility
    This paper provides advice on conducting usability testing for accessibility. It covers the usual issues including obtaining informed consent, orienting the test participant to the environment, but also includes advice for test facilitators who have no experience in facilitating tests with people who have disabilities.

  • Cost of user testing a web site
    Jakob Nielsen argues that it takes 39 hours to usability test a website the first time you try. This time estimate includes planning the test, defining test tasks, recruiting test users, conducting a test with five users, analysing the results, and writing the report. With experience, web user tests can be completed in two work days.

  • CUE: a usability testing bake-off
    "In 1998, Rolf Molich held what we could call the first usability testing bake-off. Instead, he called it a Comparative Usability Evaluation or CUE. However, the goals are the same. In an apple pie bake-off, the chefs often compare their recipes to their own. In Rolf's sessions, participants compared their techniques, looking for interesting differences."
    (Jared M Spool)

  • Design choices can cripple a website
    "If you have some pages on a site which are critical to its overall success, instigate a program of A/B split testing. You cannot afford to guess; you have to know. Be aware that however strong the copy and text on a page, its performance is very much dependent on the way in which it is presented. In other words, design choices can enhance or diminish the power of the words."
    (Nick Usborne - A List Apart)

  • Do all web sites need usability testing?
    "Recently I've been working on sites where I've done little in the way of formal user testing. From what I can tell, while not perfect (what site is?) these sites have come through rather well in the absence of formal testing. It’s caused me to question whether, under certain circumstances, usability testing is needed for a web site. I don't know the answer, and in general I'd always recommend user testing if you have the ability and resources to do it, but I'd like to explore the idea a bit."
    (D. Keith Robinson)

  • Don't test users, test hypotheses
    "Observe your users", a maxim most user experience professionals subscribe to. But how do you "observe?" When testing websites, generating hypotheses about user behavior can help inform the observation process, structure data collection and analysis, and organise findings.

  • Eight guidelines for usability testing
    "In professional web design circles, the usability testing session has become an essential component of any major project. Similar to focus groups in brand development and product launches, usability testing offers a rare opportunity to receive feedback from the very people the website is aimed at - before it's too late to do anything about it. But how can you get the most from these usability testing sessions?"
    (Tim Fidgeon - Usability News)

  • Eight is not enough
    One of the most popular questions we hear from web designers and usability professionals is: "How many users is enough when conducting usability tests?" Until recently, we had believed--and told our clients--that it wasn't usually necessary to test with more than eight users.

  • Experience remote usability testing, part 1
    In this two-part article, Pervasive Computing specialists Velda Bartek and Deane Cheatham share the experience gained by conducting a number of remote usability studies using application-sharing technology. The first article provides a context for remote usability testing by detailing and describing the benefits and pitfalls of remote usability evaluations and the application-sharing tools that were evaluated. The second article describes some of the experiences and lessons learned as the authors planned for and conducted remote usability evaluations for software products.

  • Experience remote usability testing, part 2
    In this second part of the Remote Usability article, Pervasive Computing specialists Velda Bartek and Deane Cheatham share the experience gained by conducting a number of remote usability studies using application-sharing technology. The authors describe some of the lessons learned as they planned and conducted remote usability evaluations for software products. The first article provides a context for remote usability testing and describes the benefits and pitfalls of remote usability evaluations and the application-sharing tools that they evaluated.

  • Eyetracking studies: usability holy grail?
    "Eye-tracking studies are a type of usability test where user gaze concentrations are recorded in thermal-like "heat zone maps". The heat zone maps track user eye movements. Eye tracking tests make usability testing look really interesting, sophisticated, high-tech and scientific. The reality is that eye-tracking, while valuable, doesn't make usability testing any more powerful. It's what you do with the observations and the usability test data that counts."
    (Frank Spiller)

  • Eye-tracking web usability
    "User interface guru Jakob Nielsen of the Nielsen Norman Group is on the road, giving seminars based on a recently completed an eye-tracking study that indicates how users consume Web pages–such as where people start browsing on a page, whether they have banner and text link blindness, where users look for navigation, how they react to different text types, relative attention allocated to text vs. pictures and more. I caught up with Nielsen via phone in New York to talk about the eyetracking research."
    (Dan Farber - ZDNet)

  • Eyetracking: worth the expense?
    "Is eyetracking a valuable usability tool? I’m not sure. I do agree that it’s always good when your clients and developers become aware of how users behave. And an eyetracker is a great demonstration of fine grain behavior. In just mere moments, you can easily see how users gaze at the screen. So, I agree eyetrackers have demonstrative value. But do they have diagnostic value? Can we actually learn what to change in our designs from them? Well, after watching hundreds of eyetracking tests, I can tell you it’s still really hard to know what you can learn from them."
    (Jared M Spool - User Interface Engineering)

  • Finding the right users
    There is no such thing as sound user research without an airtight user-selection process behind it. No matter how good the observation and analysis, it’s all for naught if you’ve studied the wrong people.

  • 5 days, 5 heatmaps
    "Anyone who's done any research into Eye Tracking will know that there's very little information available on the subject. So we thought we would rise to your challenge and publish the preliminary results of an internal study we've been running. Over the next five days we'll publish heatmaps from five of the sites we've tested. We'll give you our thoughts on each and hopefully you'll give us your questions, comments and analysis."

  • Fly on the wall
    For developers to make products that delight customers, they need adequate information about who exactly the customers are and what their requirements are. The User-Centered Design (UCD) process provides numerous options for gathering both customer and user input, with wide variation regarding the time involved, labour required, overhead costs, and validity of the information collected. The Fly on the Wall (FOTW) technique is a low-cost, low-overhead method of collecting valid customer data. The method is illustrated here through a pilot study that used first-hand, unobtrusive observations by UCD practitioners to collect valid customer data in a timely, cost-effective manner in collaboration with development and marketing staff.

  • Formal usability reports vs quick findings
    "Formal reports are the most common way of documenting usability studies, but informal reports are faster to produce and are often a better choice. You can maximize user interface quality by conducting many rounds of testing as part of an iterative design process. To move rapidly and conduct the most tests within a given time frame and budget, informal reports are the best option."
    (Jakob Nielsen)

  • Get out of your lab and into their lives
    "The proliferation of usability labs is a sign of success for the field of user-centered design. Whether it’s a low-rent lab comprised of a couple adjacent conference rooms, a video camera, and a television, or a fully decked-out space with remote-control cameras, two-way mirrors, an observation room, and bowls of M & Ms--more and more companies are investing in such set-ups. Conducting user tests in labs is probably the most common means of getting user input on projects. That’s a shame, because standard user testing practice is remarkably out of sync with reality."
    (Peter Merholz - Adaptive Path)

  • Guerilla facilitation
    "Having facilitated a few hundred one-on-one usability tests and focus groups, I've come away with several thoughts about the art of facilitation. I've seen too much written about the right way to facilitate. Consider these thoughts my first attempt at what I’ll call 'Guerilla Facilitation.'"
    (Ronnie Battista)

  • Hotspots and hyperlinks: using eye-tracking to supplement usability testing
    "This article discusses how eye-tracking can be used to supplement traditional usability test measures. User performance on two usability tasks with three e-commerce websites is described. Results show that eye-tracking data can be used to better understand how users initiate a search for a targeted link or web object. Frequency, duration and order of visual attention to Areas of Interest (AOIs) in particular are informative as supplemental information to standard usability testing in understanding user expectations and making design recommendations."
    (Mark C Russell)

  • How many users should you test with in usability testing (latest research)
    "Usability research is largely qualitative, or driven by insight (why users don't understand or why they are confused). Qualitative research follows different research rules to quantitative research and it is typical that sample size is low (i.e. 15 or 20 participants). Usability research is behavior-driven: You observe what people do, not what they say. In contrast, market research is largely opinion-driven: You ask people what they think and what they think they think. You need big samples for market research because of this."
    (Frank Spillers)

  • How much interaction is too much?
    How much should usability test facilitators interact with test participants? Simply recognising that facilitation is a real, difficult, hard-to-acquire skill may help. This is particularly the case for beginners, especially when one realises that the skills involved are not natural in any sense of the word.

  • How to conduct usability evaluations for accessibility
    The report explains how to conduct usability studies with test participants who use assistive technologies such as screen readers. 40 guidelines to help plan and run usability studies with users who are blind, have low vision, or have motor skill challenges.Includes sample forms that were developed through several iterations and ended up in versions that proved to work well: screening questionnaire for recruiting test participants, facilitator script, satisfaction questionnaire, consent form. Note: report must be purchased.

  • How to start customer research
    "In our non-directed listening labs, we ask customers to use the Internet in the way they normally use it at home or work. While we do have a goal for the research, we try to let the customers lead us to the answer, rather than the other way around."
    (Mark Hurst)

  • If they don't test, don't hire them
    The single best indicator as to the overall competence of an interaction design team is their plan for user testing. If you are presented with no plan or a sort of vague "and we'll eventually do some user testing", you may want to back off and look at other resources. If, on the other hand, you are given a proposal outlining repeated design and test cycles, you are dealing with people who know exactly what they are doing.

  • Interview-based tasks: learning from Leonardo DiCaprio
    "In interview-based tasks, the participants interests are discovered, not assigned. Unlike scavenger-hunt tasks, the test's facilitator and participant negotiate the tasks during the tests, instead of proceeding down a list of predefined tasks."
    (Jared M Spool)

  • Iterative Usability Testing as Continuous Feedback: A Control Systems Perspective
    "This paper argues that in the field of usability, debates about number of users, the use of statistics, etc. in the abstract are pointless and even counter-productive. We propose that the answers depend on the research questions and business objectives of each project and thus cannot be discussed in absolute terms. Sometimes usability testing is done with an implicit or explicit hypothesis in mind. At other times the purpose of testing is to guide iterative design. These two approaches call for different study designs and treatment of data. We apply control systems theory to the topic of usability to highlight and frame the value of iterative usability testing in the design lifecycle. Within this new metaphor, iterative testing is a form of feedback which is most effective and resource-efficient if done as often as practically possible with project resources and timelines in mind."
    (Alex Genov - Journal of Usability Studies)

  • Making use of user research
    "By focusing on how a product performs in the lab without broader knowledge of the user's environment and goals, measurement alone may be misleading. To get the most value and meaning out of user feedback it is important to choose the appropriate method for conducting and analyzing user research."
    (Gretchen Anderson)

  • Measuring online experience: it's about more than time
    Most researchers studying online behavior use "experience" level as a way to categorise individual differences in using the Internet. For the majority of these studies, however, experience level has been defined by time and/or frequency measurements, such as How long have you been using the internet? Research shows that the amount of internet usage is leveling off, but that the breadth of demographic user characteristics are continuing to expand. Therefore, measuring experience level by frequency of use and longevity of use may not be truly representative.

  • Observer effects in usability testing
    The hallmark of modern particle physics centers on the "uncertainty principle". Namely, you can know either the position of a so-called wave-particle or its momentum, but not both. The reason for this conundrum is that the very act of observation changes "reality" from probability (the wave-particle might be here) to actuality (hah, gotcha pinned down). Could the same hold true for usability testing?
    (John Sorflaten)

  • Outliers and luck in user performance
    "6% of task attempts are extremely slow and constitute outliers in measured user performance. These sad incidents are caused by bad luck that designers can--and should--eradicate."
    (Jakob Nielsen - Alertbox)

  • Planning usability testing for accessibility
    Planning for a usability test that includes people with disabilities involves the following considerations: determining participant characteristics, recruiting participants with disabilities, compensating participants, choosing the best location and scheduling the right amount of time.

  • Practical usability: beyond user testing
    Academics say that usability testing is all about uncovering usability problems, and it is very good at that indeed. But we think it is about much more. After working with dozens of the biggest names in the industry, I have seen that user tests have a number of surprising beneficial side effects.

  • Practical usability testing
    "The most critical aspect of user-centered design, usability testing breaks down the wall between the designer and user, and allows us to see how real users do real tasks in the real world. There are many benefits of usability testing, including uncovering pitfalls in a current system before a redesign and evaluating the usability of a system during and after design. Usability testing should be an iterative practice, completed several times during the design and development life-cycle. The end result is an improved product and a better understanding of the users that we’re designing for."
    (Joshua Kaufman - Digital Web Magazine)

  • Preparing usability testing for accessibility
    Preparing for the usability test for accessibility involves some additional considerations beyond standard usability testing preparation. These include ensuring the facility is accessible, setting up and testing the participants' configurations, and becoming familiar with the assistive technology.

  • Professional website usability
    In this five part series, Lauren Kirby discusses website usability in terms of knowing how to give your users what they want. She goes from helping you make the decision to conduct a usability study to interpreting the results and producing a report.

  • Putting A/B testing in its place
    "Measuring the live impact of design changes on key business metrics is valuable, but often creates a focus on short-term improvements. This near-term view neglects bigger issues that only qualitative studies can find."
    (Jakob Nielsen - Alertbox)

  • Putting perfect participants in every session
    "Recruiting the wrong participant can have dramatic effects. It can slow down the research process, increase costs, and, in the worst-case scenario, create faulty results which waste valuable developer resources as they chase down the wrong issues. We recommend to our clients that they closely review their recruitment process to ensure they are executing best practices."
    (Jared M Spool - OK/Cancel)

  • Quantitative studies: how many users to test
    "When collecting usability metrics, testing 20 users typically offers a reasonably tight confidence interval."
    (Jakob Nielsen - Alertbox)

  • Recording screen activity during usability testing
    Recording what users do is a crucial aspect of usability testing. One of the most useful recordings you can make is a video of screen activity, recording everything on the screen, much like a VCR: the mouse moving, pages scrolling, clicking links, typing in the search terms, and so on. Recording screen activity doesn’t necessarily cost much. Three Windows-based software programs--Lotus ScreenCam, TechSmith Camtasia and Hyperponics HyperCam--range between $30 and $150 and all have free trial versions available for download so you can try before you buy. All three offer good performance, but unfortunately, I can only recommend two, since the third is no longer being actively developed by its maker.

  • Recruiting screener: questions for usability test participants with disabilities
    A usability test participant recruiting screener is used to determine if a potential participant matches the user characteristics as defined in the usability test protocol. When recruiting participants with disabilities, ask the usual questions about demographics, frequency of use, experience level, etc. Additionally, include questions that address the characteristics related to disability and accessibility defined in the specific usability test protocol.

  • Recruiting test participants for usability studies
    Easy test user recruiting is crucial to an effective usability process. The average per-user cost is $171, but varies greatly depending on location and the targeted profession.

  • Remote online usability testing: why, how, and when to use it
    Traditional, one-on-one usability testing is a great technique for uncovering usability issues on a website. Unfortunately, in-person usability testing isn’t always feasible due to tight schedules, tight budgets, and elusive target users. So what’s a usability crusader to do when in-person usability testing is impossible?

  • Remote usability testing
    "Conducting a remote usability test is not so different from taking a classical usability evaluation, but there some facts that have to be considered prior beginning the session regarding: participants recruiting, environment setup and test execution "
    (Menghini Calderón Fabrizio Antonio)

  • Restoring confidence in usability results
    Adding confidence intervals to completion rates in usability tests will temper both excessive skepticism and overstated usability findings. Confidence intervals make testing more efficient by quickly revealing unusable tasks with very small samples. Examples are detailed and downloadable calculators are available.

  • Seven common usability testing mistakes
    "What's the easiest way to conduct a usability test? Well, you could just sit a person down (it doesn't matter who) in front of your design and ask them to do something (it doesn't matter what).If this is so easy, why does a standard usability test contain all that other rigmarole? Because that rigmarole goes a long way to ensure that the test will produce quality results."
    (Jared Spool)

  • Six tips to ensure a successful usability test
    "Instead of thinking of success as a perfect design, you should think of success in a usability test as learning what you need to know. The goal of formative, diagnostic testing is to find the critical problems that prevent users from completing their tasks. Timing is critical. You want to learn about your design's problems during the usability test, not when it fails after it is released."
    (Ginny Redish)

  • Standards update: usability test reporting
    It’s a truism that even a bad usability test will help improve your software. But the findings from different usability tests are notoriously difficult to compare. This makes it difficult to track usability improvements or to see how you compare against an earlier product. An emerging international standard looks set to solve this problem.

  • Streamlining usability testing by avoiding the lab
    "The usability lab, with its fancy cameras, one-way mirrors, and comfortable observation suites, is often considered a can't-do-without necessity for conducting serious usability tests. Even those who feel it's not required will jump at the chance to use a lab when available. However, while studying successful projects over the years, we've found that usability testing can often be more effective when the team eliminates the lab from the process."
    (Jared M Spool)

  • Testing 1, 2, 3, 4, 5.. testing
    Usability can sometimes be more about belief than about evidence or engineering, with usability testing heading the list as a central tenet of the dogma of modern practice. One disgruntled participant in a recent conference even commented: 'It is unbelievable that an instructor at CHI would question the importance of user research and usability testing'. Yet, precisely because of its leading role, it is important for the profession to question the dogma of usability testing and for professionals to keep abreast of new developments and changing perspectives.

  • Test your designs - on people
    Forget about the jargon-laden disciplines of cognitive psychology and behaviorism. While they offer hard, scientific methodologies for understanding how people comprehend and process information and tools, you're just interested in common sense. Sitting users down in front of your designs and watching them use your site will uncover the countless mistakes you overlooked while putting your pages together. And, rest assured, you made them.

  • The internet and websites: why test?
    While random users may not be surprised by a web transaction going awry, they will not wait ten seconds to give your web site a second chance. That is the great advantage users on the Internet have--they can easily go somewhere else. That fact should give pause to project leaders as they prioritise web testing.

  • The problem with usability change recommendations
    " Contemporary user testing methods have proven highly effective at identifying problems in computer interfaces. By directly measuring users’ ability to complete key tasks, practitioners can expediently uncover what are often colossal failures of usability that are otherwise difficult to perceive. User testing, then, affords a strong empirical basis for recommending that designers make changes to resolve the problems found. Most test reports take the additional step of actually suggesting what those changes should be, and it’s at this point that they start running into trouble. While the existence of the problems is based on observational evidence, the efficacy of the proposed changes is not established by the test itself."
    (John Ferrara)

  • The road to recommendation
    "To get a recommendation for change, we need to slow down and go through the four steps: Observation, Inference, Opinion, and finally Recommendation."
    (Jared M Spool - User Interface Engineering)

  • Time budgets for usability sessions
    " Up to 40% of precious testing time is wasted while users engage in nonessential activities. Far better to focus on watching users perform tasks with the target interface design."
    (Jakob Nielsen - Alertbox)

  • Tips on moderating 'listening labs'
    'Listening labs' is the term Mark Hurst uses to describe a form of usability testing where the users' tasks are determined in an interview that takes place at the beginning of the test session. In this article, tips for moderating listening labs are discussed.

  • Towards the design of effective formative test reports
    "Many usability practitioners conduct most of their usability evaluations to improve a product during its design and development. We call these "formative" evaluations to distinguish them from "summative" (validation) usability tests at the end of development. A standard for reporting summative usability test results has been adopted by international standards organizations. But that standard is not intended for the broader range of techniques and business contexts in formative work. This paper reports on a new industry project to identify best practices in reports of formative usability evaluations. "
    (Mary Theofanus, Whitney Quesenbery - Journal of Usability Studies)

  • Two simple post-test questions
    "This week, I was reminded of a practice we conduct at the conclusion of a usability test. After the participant(s) have finished with their tasks and filled out whatever paperwork we give them to subjectively rate the design, our test facilitator will ask two questions: 1. What are two things about the design that you really liked? followed by 2. What are two things about the design that you didn’t like? "
    (Jared M Spool - User Interface Engineering)

  • UI guidelines vs usability testing
    This paper defines UI guidelines and describes the problems inherent in only following guidelines, as well as the danger in being too focused on consistency in your design. It also discusses how usability testing should be used to find out if your product meets the needs of your users and allows them to do their jobs effectively.

  • Usability at 90mph: presenting and evaluating a new high-speed method for demonstrating user testing in front of an audience
    "This article documents the authors' attempt to develop a quick, inexpensive, and reliable method for demonstrating user testing to an audience. The resulting method, Usability@90mph, is simple enough to be conducted at minimal expense, fast enough to be completed in only thirty minutes, comprehensible enough to be presented to audiences numbering in the hundreds, and yet sophisticated enough to produce relevant design recommendations, thereby illustrating for the audience the potential value of user testing in general. In this article, the authors present their user testing demonstration method in detail, analyze results from 44 trials of the method in practice, and discuss lessons learned for demonstrating user testing in front of an audience."
    (Paul F Marty, Michael B Twidale)

  • Usability Stockholm syndrome
    "People tend to be much less critical of the software designs they're testing than they probably should be. I think of this as a form of 'Stockholm Syndrome', in which a hostage becomes sympathetic to his captors."
    (Jensen Harris)

  • Usability test data
    People often throw around the terms "objective" and "subjective" when talking about the results of a usability test. These terms are frequently equated with the statistical terms "quantitative" and "qualitative". The analogy is false, and this misunderstanding can have consequences for the interpretations and conclusions of usability tests.

  • Usability testing for accessibility
    Usability testing provides quantitative and qualitative data from real users performing real tasks with a product. Usability professionals can evaluate accessibility by using standard usability testing protocols, with a few modifications for including participants with disabilities.

  • Usability test reporting
    It’s a truism that even a bad usability test will help improve your software. But the findings from different usability tests are notoriously difficult to compare. This makes it difficult to track usability improvements or to see how you compare against an earlier product. An emerging international standard (CIF) looks set to solve this problem.

  • Usability testing: myths, misconceptions and misuses
    "In this article, I identify and try to straighten out some common misconceptions about usability testing. These are my opinions, based on my interpretation of what usability testing is and how it works. I hope these comments are useful. My approach is to state a common misunderstanding and then discuss it."
    (Richard F Dillon)

  • Usability testing: you get what you pay for
    A valid and useful usability test takes time and expertise. Shortcuts and lack of rigorous controls invariably reduce the validity and usefulness of the data generated by the test. Here I offer a description of some of the key ingredients of a good usability test, and some examples of poorly designed or run testing.

  • Usability testing world wide websites
    Notes from a CHI '97 workshop on user testing

  • Users begin to demand software usability tests
    The Boeing Co. is changing the way it buys software and is making a product's usability--the ease with which end users can be trained on and operate the product--a fundamental purchasing criterion. It's a move the aerospace giant sees as an essential means of controlling IT costs.

  • User testing: how to find out what users want
    As a developer, you probably know your site's territory like the back of your hand. And you should--you built it, after all. But the same expertise that makes you an asset to a web project can also be a drawback when it comes to usability. Think of what it's like to use someone else's computer, drive their car, or stay at their house. They know exactly how things work and where to find things. You, on the other hand, don't.

  • User testing is not entertainment
    "Don't run your studies for the benefit of the people in the observation room. Test to discover the truth about the design, even when user tasks are boring to watch."
    (Jakob Nielsen - Alertbox)

  • Variability in user performance
    "When doing website tasks, the slowest 25% of users take 2.4 times as long as the fastest 25% of users. This difference is much higher than for other types of computer use; only programming shows a greater disparity."
    (Jakob Nielsen - Alertbox)

  • Web usability studies
    A presentation on how to do usability testing, by Keith Instone.

  • What happens after customer research?
    Direct customer research should be followed by a phase in which the team creates a customer experience strategy which organises all the findings of the labs, and creates, holistically, a much bigger and stronger vision of the improvements to the service.

  • What's in a number?
    Whereas 7 (plus or minus 2) is the mantra for structured writing and other methods for organizing information, 5 (plus or minus 2) is the mantra for the number of participants needed in a usability test. The "Magic Number 5"--five participants will yield 80% of the findings from a usability test--comes from research conducted in the 1990’s by Nielsen, Virzi, Lewis, and other human factors engineers. Of late, however, our confidence in this magic number 5 has been eroded when it comes to usability testing of web sites, especially large, commercial websites.

  • What is the Common Industry Format (CIF)?
    The Common Industry Format (CIF) for usability test reports specifies the format for reporting the results of a summative usability evaluation. The most common type of usability evaluation is formative, i.e. designed to identify usability problems that can be fixed. A summative evaluation produces usability metrics that describe how usable a product is when used in a particular context of use. The CIF report format and metrics are consistent with the ISO 9241-11 definition of usability.

  • When getting the job done isn't enough...
    "A study by Kelkar and colleagues looks at data collected from a usability test of a Web-based application (WebCT). It compares the relative impact of collecting performance data to collecting performance and process data on the resulting design recommendations. This work suggests that the various data streams focus designers on different opportunities to improve the user experience. If the goal is to increase both task completion and perceived ease-of-use, then the collection of process data is critical. This data provides direct insight into the users' expectations and mental task model."
    (Kath Straub - Human Factors International)

  • When to oursource the recruiting of test users
    For an intranet usability test, the "representative users" are your own employees , so it doesn't make sense to ask an outside agency to recruit them. Similarly, sites that aim at a narrow group of external users may be better off doing their own recruiting. For external websites that are aimed at the general population or at relatively broad groups (e.g. farmers or Perl programmers) it is easier to have users recruited by a company that specialises in recruiting.

  • Why ask 'why' in a usability evaluation
    If you've ever kept company with a 5 year-old you have experienced how their incessant desire to understand life results in an unending stream of "whys?" Over the years I've found myself becoming impatient with my children and grandchildren. So I was surprised to find that "asking why" in a usability evaluation had a more profound effect on the outcome than I had anticipated.

  • Why user testing is good
    If you put your product in front of people and make them use it, at least you'll have some idea of how the real world is likely to react to your brilliant idea. And that, in a universe where a week's delay could mean life or death, is a major advantage.

  • Why you don't need a usability lab
    It's time to get serious about testing the usability of your website. Now you just have to convince your boss to spring for an elaborate laboratory with up-to-date computer equipment on one side and video monitor screens on the other, divided by a soundproof wall and a one-way mirror. Right?

  • Why you only need to test with 5 users
    Some people think that usability is very costly and complex and that user tests should be reserved for the rare web design project with a huge budget and a lavish time schedule. Not true. Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users and running as many small tests as you can afford.

  • Writing memorable scenarios for usability testing
    "One of the most important aspects of running a successful usability test is getting the scenarios right. Making a mess of scenarios will, more than anything else, result in a usability test that is worthless or highly biased."
    (Donna Maurer)

Research articles

  • A comparison of two evaluation techniques for technical documentation
    "This study compared two evaluation techniques, usability testing and cognitive walkthrough, in their ability to identify errors in aviation maintenance documentation. The techniques were evaluated to see how much unique information they each produced as well as the type of errors identified. Results showed that the techniques were complementary in their findings and both are recommended in the development of technical documentation."
    (Bonnie Rogers, Chris Hamblin, Alex Chaparro)

  • Beyond the five-user assumption: benefits of increased sample sizes in usability testing (PDF)
    It is widely assumed that 5 participants suffice for usability testing. In this study, 60 users were tested and random sets of 5 or more were sampled from the whole, to demonstrate the risks of using only 5 participants and the benefits of using more. Some of the randomly selected sets of 5 participants found 99% of the problems; other sets found only 55%. With 10 users, the lowest percentage of problems revealed by any one set was increased to 80%, and with 20 users, to 95%.
    (Laura Faulkner)

  • Evaluating the 'evaluator effect'
    A few years ago, Niels Ebbe Jacobsen, Morten Hertzum and Bonnie John (1998) observed that the 'evaluator effect' had received little study. The 'evaluator effect' is when different evaluators evaluating the same system detect substantially different sets of usability issues. They conducted research that showed how insidious and potentially destructive the 'evaluator effect' could be in usability testing.

  • Pitting usability testing against heuristic review
    Empirical evaluations of the relative merit of usability testing and heuristic review outline both strengths and drawbacks for each. This article reviews research evaluating each method and concludes that in order to select the most appropriate method for your project you will need weigh the expertise of your evaluators, the maturity of your application, the complexity of the tasks, and possibly even the current status of your usability program.

  • The validity of the stimulated retrospective think-aloud method as measured by eye tracking (PDF)
    "Retrospective Think aloud (RTA) is a usability method that collects the verbalization of a user’s performance after the performance is over. There has been little work done to investigate the validity and reliability of RTA. This paper reports on an experiment investigating these issues using the method called stimulated RTA. By comparing subjects’ verbalizations with their eye movements, we found stimulated RTA to be valid and reliable: the method provides a valid account of what people attended to in completing tasks, it has a low risk of introducing fabrications, and its validity is unaffected by task complexity. More detailed analysis of RTA shows that it also provides additional information about user’s inferences and strategies in completing tasks. The findings of this study provide valuable support for usability practitioners to use RTA and to trust the users’ performance information collected by this method in a usability study."
    (Zhiwei Guan, Shirley Lee, Elisabeth Cuddihy, Judith Ramey)

  • Using eyetracking data to understand first impressions of a website
    "This study discusses the contributions of eye-tracking data to traditional usability test measures for first-time usage of websites. Participants viewed the homepages of three different websites. Results showed that eye-movement data supplemented what users verbally reported in their reactions to a site. In particular, the eye-tracking data revealed which aspects of the website received more visual attention and in what order they were viewed."
    (Mark Russell)

  • When 100% really isn’t 100%: Improving the accuracy of small-sample estimates of completion rates
    "Small sample sizes are a fact of life for most usability practitioners. This can lead to serious measurement problems, especially when making binary measurements such as successful task completion rates (p). The computation of confidence intervals helps by establishing the likely boundaries of measurement, but there is still a question of how to compute the best point estimate, especially for extreme outcomes. In this paper, we report the results of investigations of the accuracy of different estimation methods for two hypothetical distributions and one empirical distribution of p."
    (James R Lewis, Jeff Sauro - Journal of Usability Studies)

Case studies

  • Case study: Conducting large-scale multi-user user tests on the United Kingdom Air Defence Command and Control System
    "IBM was contracted to provide a new Air Defence Command and Control (ADCC) system for the Royal Air Force. The IBM Human Factors (HF) team was responsible for the design of the operations room, workstations and the graphical user interfaces. Because the project was safety-related, IBM had to produce a safety case. One aspect of the safety case was a demonstration of the operational effectiveness of the new system. This paper is an in-depth case study of the user testing that was carried out to demonstrate the effectiveness of the system. Due to time constraints the HF team had to observe five participants working simultaneously. Further, to provide a realistic operational environment, up to twenty-eight operators were required for each test. The total effort for this activity was four person-years. The paper will detail the considerations, challenges and lessons learned in the creation and execution of these multi-user user tests."
    (Elliott Hey - Journal of Usability Studies)

  • Using eyetracking to compare web page designs: a case study
    "A proposed design for the American Society of Clinical Oncology (ASCO) Web site was evaluated against the original design in terms of the ease with which the right starting points for key tasks were located and processed. This report focuses on the eye tracking methodology that accompanied other conventional usability practices used in the evaluation. Twelve ASCO members were asked to complete several search tasks using each design. Performance measures such as click accuracy and time on task were supplemented with eye movements which allowed for an assessment of the processes that led to both the failures and the successes. The report details three task examples in which eye tracking helped diagnose errors and identify the better of the two designs (and the reasons for its superiority) when both were equally highly successful. Advantages and limitations of the application of eye tracking to design comparison are also discussed."
    (Agnieszka Bojko - Journal of Usability Studies)


  • Usability test data logger
    Most people use Microsoft Excel to analyse the results of usability tests, but did you know you can use it to collect the data too? This spreadsheet allows you to measure task completion rates, analyse questionnaire data, and summarise participant comments. It even includes a timer so you can measure time-on-task.


  • Pete Gordon on portable usability labs
    Pete Gordon, who created Visual Mark, shares his thoughts on portable usability labs.

  • Usability testing best practices: an interview with Rolf Molich
    You may have never heard of Rolf Molich. Yet, if you’ve done any usability testing, design evaluations, or heuristic inspections, then you’ve been affected by his pioneering work. Since entering the field in 1983, Rolf has produced some of the most impressive and forward-thinking research on effective discount usability techniques. Two of Rolf’s more renowned contributions include the co-invention of the Heuristic Inspection process with Jakob Nielsen and the more recent CUE (Comparative Usability Evaluation) studies. We had the opportunity to ask Rolf about some of his thoughts on the best practices surrounding usability testing.