On the surface, remote usability testing can be very appealing. Online assessments are quick, easy and can lead to results similar to traditional lab testing. There are many advantages to online usability testing, with low cost being chief among them.  For example, “Usertesting.com” charges the low rate of only $39 per participant. This price, accompanied by the large recruiting capacity innate with online forums, can multiply the pool of participants tested.  Other advantages to online testing include a more geographically diverse group of respondents who can participate from anywhere in the world. Large data sets translate to statistical significance and higher validity of results.  The results may be more accurate if participants are being assessed in real time as they interact on a site chosen by free will. Online testing is more ethnographically sound, since users are being tested in their natural environment, using the product as they would in the real world. True intent testing allows participants to provide feedback as they interact with the site and uncover problems as they arise. However, this type of testing would not be possible for new prototypes or interfaces in the beginning stages of production. Overall, online testing seems ideal for those looking for a cheap, easy way to gather a large collection of data quickly on an established site.

However, those in need of a thorough, well-founded usability analysis should opt for traditional in-house testing.  There are many trustworthy and reputable facilities that provide the equipment and knowledge necessary to conduct successful usability studies.  Years of experience have led industries to develop, refine, and standardize usability testing to be as effective as possible.  Testing on-site allows analysts to control the environment. This helps ensure that participants feel relaxed, and that there is consistency in the test environment for users. Using an established facility eliminates fumbling with technological devices. They offer more control over the browser and functionality of the system. Relying on participants’ computers during usability sessions carry the inherent risks of weak signal strength and/or slow internet connection speed, and even whether personal distractions will interrupt the session. We all know technology can be unpredictable and, when depending on the participant’s devices, important data may be lost. The notes taken on site during a user session are kept as a hard record, and are accompanied by video footage stored on a secure system.

On-site testing is essential for those who want to keep their product, site or application confidential.  Online usability sites such as “usertesting.com” are not data sensitive. If a company is testing a Beta version of their application or site, that material could be open and available to the public. Facilities like ours are able to control this aspect and provide assurance to clients that their products will not leak outside facility doors. When participants are completing testing on their own computer, they have the ability to save or take screen shots of the product. There is also the risk of having the company’s competitors participating in the study as “respondents.”

Facilities employ recruiters whose sole responsibility is recruiting participants that fit specified criteria. Planned recruiting allows testers to customize who their participants will be. Recruiters have the opportunity to speak with the potential subject on the phone beforehand and deem how useful they feel their opinions would be to the study. Recruiters can also sift through their databases to reduce “testing junkies” or “focus groupies” who are only interested in receiving payment.  These “serial respondents” can be weeded out of the study by a seasoned recruiter. Online respondents are more difficult to vet.

Many might assume that the overall costs of conducting a usability study online would be drastically cheaper than doing the same study in person. However, a lot of the online services are priced on a monthly basis and require a monthly subscription. This may be costly if you only need to use the service once. Some sites charge per minute, such as WebEx, or will require you to purchase additional services that in the end may cost more than expected.

One of the primary advantages to facility testing compared to online testing is the physical interaction between the participant and moderator. Most of communication is non-verbal and thus the gestures and non-verbal cues that users provide can be very informative. The interaction between the tester and participant can be essential for deciphering the real impressions of the user.  A truly comprehensive analysis entails much more than just timing, clicks, and errors.  Eye patterns, body language, gestures and facial expressions are all valuable aspects of usability testing. Having the users come in to a lab guarantees that someone will be paying close attention to these subtle details, and will have the ability to understand what they imply.

Simply having usability testing tools is half the battle; you still need expertise to interpret results and make practical recommendations. Experienced usability specialists have the knowledge to not only gather data, but to make sense of it.  Heuristics can only take you so far. Analyst expertise can provide additional details that can only come from years of training and familiarity. During in-house sessions, participants are instructed to think aloud about the task they are completing, allowing the analyst to get inside the head of the user. Subjects have the opportunity to build rapport with the moderator and perhaps open up more than they would if performing the task alone.  Side conversations may inadvertently provide helpful information.  If errors occur while the user is exploring the website, there is someone there to guide the process and provide assistance. Interaction feedback is not possible with online testing.  Therefore, the prominent advantage facility testing holds over online testing is the ability to delve into the “why” behind user’s actions. The ability to understand the decision-making aspect of performance is perhaps one of the most valuable resources usability testing has to offer.

In the end, the choice of whether to use online or in-house usability testing really depends on the goals for the project and the purpose of the study.  If it is a simple, straightforward usability assessment testing a simple product, online testing might be best. However, for more in-depth, bigger projects where the results hold greater weight, on-site testing would be the better choice. Online testing may also be a preferable choice when traditional in-person usability testing is not possible for financial or logistical reasons.  Therefore, it is important to assess the goals and future implications of a study prior to choosing the testing format that will best suit your needs.

There are numerous websites available that provide online usability testing, eliminating the need to use a usability testing facility. With so many possibilities, it is important to assess the options to make an educated decision on what service would best fulfill your needs.

UserTesting.com is a popular site that regards itself as “the fastest, cheapest way to find out why users leave your site.” UserTesting.com allows you to select the participants being tested, select the user tasks, test competitors’ websites, and receive feedback right away. These features stress the control customers will have in their testing. The site allows you to test a wide range of interfaces – from mobile apps to Facebook games – and choose the amount of users to test, for the low cost of $39 per user. Customers have the option to take time-stamped notes, edit the videos, and ask follow-up questions after receiving results.  As with all remote usability sites, the results may hold higher validity since participants are being tested in their natural environment and are more likely to behave as they would normally. Testing can be done at any time and respondents are drawn from a panel of users who have been pre-screened to meet certain qualifications. However, there are a few drawbacks when comparing this useful service to testing in a research facility. Chief among them is that the respondent demographic questionnaire includes only four questions, so finding users who meet very specific criteria is challenging. In addition, customers are unable to interact with participants during testing and cannot moderate the session to stay on course.

There are many sites that look at particular features of the user experience without including an all encompassing analysis.  For example, Chalkmark, ClickHeat, and ClickTale measure clicking actions, while Clixpy and Morae capture on-screen user movements. Pricing is usually calculated through monthly or annual plans ranging from $19 to $109 a month.  However, none of these forums allows for clients to understand the “why” behind user actions, potential confusion, and failures. For those more interested in gathering feedback from a particular audience, Ethnio is a site where clients can recruit target participants. Although the first 20 recruits are free, gathering 200 recruits costs $400. On the other end of the spectrum, Feng-GUI.com generates algorithms to simulate real user’s eye movements instead of recruiting live participants. If simple feedback is what you are looking for, Feedback Army is a basic usability test in which URLs are shared and viewers post comments and questions about the site. This service costs $10 for 10 reviews. However, without providing users with tasks, many issues may be left uncovered.  Once again, these sites are optimal for those interested in a specific aspect of usability testing.

If clients are seeking an inclusive analysis, there are several sites that provide online usability testing similar to facility testing. Loop 11 is one such site that has the advantage of providing data analysis tools to compare different sites systematically.  No codes or downloads are required, which makes the experience easier for both the client and participant. Testing is available for mobile devices, expanding testing possibilities. Loop 11 costs $350 per session which includes up to 1,000 participants, unlimited tasks and questions and real time results. Loop 11 would be a prime choice for those looking to test a large pool of users (even though a few is usually enough), while if only a few participants are needed, UserTesting.com provides a better bang for the buck.

OpenHallway is another remote usability testing site that is similar in function but has the major downfall that videos can only be up to 10 minutes in duration. Videos consist of the user’s screen and voice while they are completing the designated tasks. One has to wonder how many tasks can be completed in such a short time frame. UserTesting.com also mentioned that most of their videos are only 15 minutes in length but they have the ability to run longer. Although the set up with OpenHallway is regarded as very easy to use, technical issues seem to be a problem when recording and viewing video feed. A basic monthly plan costs $49, while the premium package that includes 30 hours of storage costs $199 per month. OpenHallway may be the best choice for those looking for a cheap and easy remote testing site that is basic in function and quality.

UserFly is a site that records on-screen user movement and mouse movement and clicks. However, it does not include auditory feedback from participants and the video recordings are stored on the UserFly servers for a maximum of 90 days. Prices range from free to $200 a month depending on the amount of captures and storage duration. This site would be useful for clients more interested in the design and navigation aspect of the site rather than user feedback based on their task experience. Those using tests in a professional setting may want their data permanently stored and in their possession.  UserVue, on the other hand, stresses the interaction between moderator and tester by allowing them to communicate via phone or chat while the test occurs. This site charges $149 for a one-month subscription.

With so many options, it is important to first assess your usability goals and identify what you are truly looking to get out of testing. If the website design and format is of primary concern, a simple mouse tracking or click heatmap might be sufficient. Those interested in user experience ought to purchase a more complete program that provides auditory user feedback as well as commentary to predetermined questions. The length of time that you will require usability services should also dictate whether you purchase a program for a single use, monthly or an annual plan. Luckily, there are sites and programs to cater to almost every need to fit your goals and budget.

It is always surprising to us here at Mediabarn to hear stories of moderators who conduct usability testing NOT in a professional research facility.

Q.      Why should I take 10%-20% of my client’s budget for a test to rent out a facility when I can just conduct the sessions cheaper in a hotel room, community center, Starbuck’s, or in a spare room in the client’s office? After all, those funds could be spent running additional sessions.

A.       Yes, those funds could be spent elsewhere, but the return will probably fall short of what you can gain from bringing in new clients. Research facilities provide a great experience to clients that they are sure to remember.  As a result, when these clients need future testing done, who do you think they will go to?  Renting a research facility is a quick and easy way to enhance the quality of the usability testing experience for all parties involved.

What’s in it for you?  How often is it that a group of people come together in hopes of learning about your research? Renting a research facility means you will have 8-10 clients in a room engaged in what you are doing. That’s 8-10 potential new clients, perhaps from a partner company, another department, or senior staff.  Chances are that in 18 months, someone from that group will leave their position and find new work at a new company, and will perhaps bring you in for their research. If you are able to provide a first-rate experience for observers, those observers will remember you. The bottom line is that it is a good investment; the more people see how valuable your research is, the more work (and money) you will receive in the long run.

How the client benefits: Research facilities allow your client to bring the team together, sit in a room (away from the distractions of the office), and watch real users interact with their product. The “war room” atmosphere is great for getting decision-makers together in one place to observe how users respond.  A professional facility creates an environment that fosters research while providing clients with an experience from which they are sure to derive benefit. We have hosted numerous client groups and have heard over and over again that the observers really enjoyed themselves. Usability sessions are often fun to watch, especially when provided with quality food and drinks, and the appearance of the beer and wine cart (for night sessions).

Q.      Why do clients need to be present at the facility when they can watch the recorded video sessions later on?

A.        Sure, the videos are recorded and posted on a secure server, but realistically we know almost no one takes the time to watch them.  Over the years we have posted videos for clients, and sadly we know they probably aren’t being watched.  More often than not, only one or two sessions are viewed, and for only about a fraction of the study length.  We have had teams of 20 people request access to the videos, with the good intention of getting folks to watch them, but most people never even sign in. At the facility, video footage is watched live and data can be analyzed right away in the viewing room.

How the Research Benefits: Unlike a Starbuck’s, the facility’s sole purpose is catering to the usability testing process. The features and amenities specific to research facilities are invaluable. The facility provides a quiet, professional setting without distractions and interruptions you would find at other locations. The materials that a usability site provides are going to be top of the line and are customized to deliver the best results possible.  The placement and set up of these tools have been perfected to result in optimization of performance. Having the necessary materials on site means no scrambling around beforehand and a shorter set up time. Don’t forget to estimate the cost of set up time for outside locations, the time it may take to re-establish the setup if there are malfunctions (which are inevitable), and the possibility of data being lost or damaged in transit. Most facilities have reliable backup systems that will securely protect your data.

Performing the sessions in a research facility allows analysts to control all aspects of the environment. Familiarity with your facility means less time fumbling with new equipment, optimal placement for recording materials and a room configuration designed to make users feel comfortable and relaxed.  In order to increase the reputability of the research conducted, all users should have a similar experience. Thus, controlling for consistency among users is of paramount importance.

Results from usability studies illustrate that simplicity is key. Research facilities make the usability testing experience as simple as possible for all those involved. At the end of the day, your client is paying for an off-site where they can gather top of the line research while having a good time watching you work. This benefits your business, the client’s goals, and ultimately (because everyone is engaged) it makes the product that much better for end users. Thus, everyone wins. In conducting usability studies, you are trying to improve the user experience, so why not let a research facility, such as Media Barn, enhance your experience!