We are often asked about sample size when we talk about usability testing. Skeptics find it difficult to believe that we can achieve so much by talking with so few. Here’s our stance on usability sample sizes.
Lots of work has been done to determine the appropriate number of respondents required to gather usability test results. Nielsen Norman Group has talked about this for years and stands by their suggestion that 85% of critical usability issues can be uncovered by testing as few as 5 respondents. We have found this to be true in study after study. There are a few qualifiers that should be kept in mind. The key to gathering accurate and robust findings is to ensure that you’re talking with the right users. People who fit your target profile are the only ones who can provide you with an honest evaluation of your product. Your coworkers or family members, or even an expert review, are no substitute for testing with actual users.
As researchers, in addition to having an eye toward usability, we often recommend testing with a slightly larger sample than 5. We find that many of our clients have a user base that is segmented somehow, usually by one or more demographic factors. In those cases, we typically recommend conducting tests with 8 to 12 users, and sometimes more, in order to get a good representative sample. Even then, it’s rare that we uncover new usability issues after the 5th or 6th interview.
Another note on sample size – so much of usability testing is observational and this allows users to simply do what they do without explaining their reasoning. Other types of research delve deeper into the psyche, and would require more respondents to gain a clearer picture of their thinking and to be able to project that to the overall population. The beauty of usability testing is that we can talk with a few representatives to get that clear picture.
Finally, we live by the motto “Test early and test often.” Ideally, usability testing is part of an iterative design process, whereby the product is designed, tested, redesigned, tested again, tweeked, tested again then launched. So, it doesn’t make sense to talk with large samples of users in each of these stages because it’s not cost-effective and doesn’t get you a big bang for your buck. Instead, talk with fewer users at several stages throughout the design process. And, as we’ve said before, doing some usability testing is always better than doing none at all!