On the surface, remote usability testing can be very appealing. Online assessments are quick, easy and can lead to results similar to traditional lab testing. There are many advantages to online usability testing, with low cost being chief among them.  For example, “Usertesting.com” charges the low rate of only $39 per participant. This price, accompanied by the large recruiting capacity innate with online forums, can multiply the pool of participants tested.  Other advantages to online testing include a more geographically diverse group of respondents who can participate from anywhere in the world. Large data sets translate to statistical significance and higher validity of results.  The results may be more accurate if participants are being assessed in real time as they interact on a site chosen by free will. Online testing is more ethnographically sound, since users are being tested in their natural environment, using the product as they would in the real world. True intent testing allows participants to provide feedback as they interact with the site and uncover problems as they arise. However, this type of testing would not be possible for new prototypes or interfaces in the beginning stages of production. Overall, online testing seems ideal for those looking for a cheap, easy way to gather a large collection of data quickly on an established site.

However, those in need of a thorough, well-founded usability analysis should opt for traditional in-house testing.  There are many trustworthy and reputable facilities that provide the equipment and knowledge necessary to conduct successful usability studies.  Years of experience have led industries to develop, refine, and standardize usability testing to be as effective as possible.  Testing on-site allows analysts to control the environment. This helps ensure that participants feel relaxed, and that there is consistency in the test environment for users. Using an established facility eliminates fumbling with technological devices. They offer more control over the browser and functionality of the system. Relying on participants’ computers during usability sessions carry the inherent risks of weak signal strength and/or slow internet connection speed, and even whether personal distractions will interrupt the session. We all know technology can be unpredictable and, when depending on the participant’s devices, important data may be lost. The notes taken on site during a user session are kept as a hard record, and are accompanied by video footage stored on a secure system.

On-site testing is essential for those who want to keep their product, site or application confidential.  Online usability sites such as “usertesting.com” are not data sensitive. If a company is testing a Beta version of their application or site, that material could be open and available to the public. Facilities like ours are able to control this aspect and provide assurance to clients that their products will not leak outside facility doors. When participants are completing testing on their own computer, they have the ability to save or take screen shots of the product. There is also the risk of having the company’s competitors participating in the study as “respondents.”

Facilities employ recruiters whose sole responsibility is recruiting participants that fit specified criteria. Planned recruiting allows testers to customize who their participants will be. Recruiters have the opportunity to speak with the potential subject on the phone beforehand and deem how useful they feel their opinions would be to the study. Recruiters can also sift through their databases to reduce “testing junkies” or “focus groupies” who are only interested in receiving payment.  These “serial respondents” can be weeded out of the study by a seasoned recruiter. Online respondents are more difficult to vet.

Many might assume that the overall costs of conducting a usability study online would be drastically cheaper than doing the same study in person. However, a lot of the online services are priced on a monthly basis and require a monthly subscription. This may be costly if you only need to use the service once. Some sites charge per minute, such as WebEx, or will require you to purchase additional services that in the end may cost more than expected.

One of the primary advantages to facility testing compared to online testing is the physical interaction between the participant and moderator. Most of communication is non-verbal and thus the gestures and non-verbal cues that users provide can be very informative. The interaction between the tester and participant can be essential for deciphering the real impressions of the user.  A truly comprehensive analysis entails much more than just timing, clicks, and errors.  Eye patterns, body language, gestures and facial expressions are all valuable aspects of usability testing. Having the users come in to a lab guarantees that someone will be paying close attention to these subtle details, and will have the ability to understand what they imply.

Simply having usability testing tools is half the battle; you still need expertise to interpret results and make practical recommendations. Experienced usability specialists have the knowledge to not only gather data, but to make sense of it.  Heuristics can only take you so far. Analyst expertise can provide additional details that can only come from years of training and familiarity. During in-house sessions, participants are instructed to think aloud about the task they are completing, allowing the analyst to get inside the head of the user. Subjects have the opportunity to build rapport with the moderator and perhaps open up more than they would if performing the task alone.  Side conversations may inadvertently provide helpful information.  If errors occur while the user is exploring the website, there is someone there to guide the process and provide assistance. Interaction feedback is not possible with online testing.  Therefore, the prominent advantage facility testing holds over online testing is the ability to delve into the “why” behind user’s actions. The ability to understand the decision-making aspect of performance is perhaps one of the most valuable resources usability testing has to offer.

In the end, the choice of whether to use online or in-house usability testing really depends on the goals for the project and the purpose of the study.  If it is a simple, straightforward usability assessment testing a simple product, online testing might be best. However, for more in-depth, bigger projects where the results hold greater weight, on-site testing would be the better choice. Online testing may also be a preferable choice when traditional in-person usability testing is not possible for financial or logistical reasons.  Therefore, it is important to assess the goals and future implications of a study prior to choosing the testing format that will best suit your needs.

Jason Pentecost

About Jason Pentecost

As Principal and Co-Founder of Mediabarn, clients and associates rely heavily on Jason for his breadth of interactive experience and his keen perception of where the industry is heading. He works closely with our key clients and is never too far from anything that goes on at the ‘barn. Jason…
Posted in user research, uxlab.

Comments are closed.