How Market Research Experience Can Mislead You in Usability Studies

Diving into the emerging field of usability research after nearly a decade in traditional market research, I’ve learned that some concepts and processes are similar in both fields (e.g., surveys). There are, however, more differences than I expected. For example, when I first heard the term usability testing I translated that in my mind as in-depth interview. As I’ll discuss in a minute, there are significant differences even though the two concepts are related.

This list of three key differences will certainly change (and evolve) as I gain more UX experience. Here are my early thoughts about some of these differences. I am focusing on the differences that struck me as counter-intuitive from a market research perspective, rather than the biggest differences.

A focus group is not a usability test

In his book, Don’t Make Me Think!, Steve Krug says that in a focus group the participants react to designs or ideas. At the end of a focus group, one won’t be able to tell if people can use a website or product. In a usability test, however, there is no group.  A participant is shown a website or product prototype and asked to figure out what it is or to complete typical tasks. So the usability test isn’t really the same as an in-depth interview in market research.

Usability testing isn’t done in groups because participants have a hard enough time with not wanting to feel like they “failed” or admit “I don’t know” in front of the UX researcher (although these are often where our best insights come from).  And they certainly don’t want to admit these things in front of other strangers who are also testing a product.  My research participants aren’t competing with each other!

Also, I can’t control what some participants in a room will do/say and their words or actions may bias my other participants. For example, if I ask three participants siting (who are sitting in front of individual computers) to find the “donate” button on a website, one participant might reach over and help another one find it. S/he may think it is the nice thing to do, but now as a researcher I don’t know if that one participant would’ve been able to complete the task without help.

I do, however, use a lot of the same prompts and follow-up questions that I’d use in a focus group in a usability test, such as: Tell me more about X and anything else?

You only need to conduct usability research with 5 -10 participants

In market research, we’re used to looking for larger samples. We like to have focus groups with 8-10 participants (sometimes more).  And we like to conduct several focus groups about one topic. In usability research, the standard is 5. In market research we are trying to learn about unique needs/views of diverse groups and want to try to be representative even in qualitative work. By contrast, in usability, we are looking for needs/requirements that are common to all humans based on thinking that is common to most humans.

For example, the race or gender of a person who spots a missing home button on a website that should logically be there is less relevant than the race or gender of a respondent answering a customer satisfaction survey about their experiences in a store.

Jacob Nielsen’s article has some great addition thoughts on this point. You should be aware that this notion can be debated quite a lot, however. Some researchers test with 5 participants and some people think it best to test with 10 participants (and sometimes more). Laura Falkner, a UX researcher from UT Austin, published an article in 2003 that reached similar conclusions as Nielsen. Her results show that on average 5 users found 85.5% of usability problems and 10 users found 94.6% of usability problems. And the diminishing returns continue with an increasing number of users.

Ultimately, it depends on what you are testing. If you are testing medical devices, then you might need to uncover more than 94.6% of the usability problems and test with more than 10 participants.

You don’t have to conduct research with your users

If anyone could be a potential user of your product then y0u could conduct your research with anyone, including your mom. You don’t have to find people who fit the exact profile of your current users because it’s not as if only males between the ages of 18 and 25, who make more than $50,000 per year are the only people who will use your product. Nor are they the only people who should be able to use your product. You may not want to test with your mom because she’ll only say supportive, positive things about your product, but that’s a different issue.

An example: Let’s say your mom tests your shoe-buying website. You ask her to pretend she’s going hiking in the Grand Canyon and needs new hiking boots. She finds boots she likes in her size and then discovers that there is no “check out” button on the site. So she’s stuck on the hiking boots page and can’t make a purchase.

If you did this same test with a stranger who fits into some online shoe-buying persona, that stranger  would likely make the same discovery. You might, in this example, want to be sure your mom fits some basic screening criteria such as the ability to use a computer and the Internet, but those are pretty broad inclusion criteria compared to screeners for typical market research studies.

The exception to this is if only people with very specific knowledge will use your product. Some of my classmates tested a cloud storage platform and specifically sought out developers and people with knowledge of command line prompts.

Those are the three big differences that I’ve noticed so far. If you’re on a similar learning journey, do share any you’ve learned in the comments.