Top-10 Reasons for Unmoderated UX Research 🙌

What is unmoderated user testing?

Unmoderated user testing describes the process of gathering user feedback about a product, experience, or service without the interviewer (UX researcher) being present. The interviewee (user / participant) is guided in an automatic way through a series of prompts that were pre-written. The interviewer will be there in spirit, but not in real-time. That means no ability to course-correct, provide more detail, stop the user from ranting, etc.

In the past few years, a plethora of online tools have emerged to facilitate this unmoderated user research process. Tools like UserTesting, UserInterviews, dScout, Alpha, UserZoom, Optimal Workshop, and HotJar are to name a few.

Example of unmoderated with UserTesting from our work @ Meta. Standard process of user screening, task layout, video analysis, and insight creation.

These services enable unmoderated user research to happen by providing the online platforms to build, edit, launch, and share user feedback. Every platform has its own participant pool from which you can draw potential users. So, instead of making outreach to users to set up interviews (outbound model), the UX researcher will filter / screen-in participants from the pool of users to provide specific feedback (inbound model).

1. Unmoderated user research is much cheaper

Companies exist to make profits 💸. Therefore, the UX research discipline can perform its duty by mindfully managing costs. We've found that unmoderated user interviews is one way stretch the research budget further.

First of all, if your company has access to a platform like UserTesting, UserZoom, UserInterviews, etc. the business may have already paid for the access. Please leverage it to the fullest, and not leave it to rust (what the CEO is thinking).

Second, utilizing recruiting teams to obtain users for moderated interviews can be very expensive. If it's external, the agency will be charging for time, labor, outreach, and more. Internally, your company may have internal research operations doing the heavy lifting of locating users, screening, scheduling interviews, etc. Not much cheaper either, on the grand scheme. Plus you often need other teams involved, like data science (DS) to pull queries and take time to understand your requests. If there's a way to get the research done with less people involved, then that's a good choice. And unmoderated cuts out A LOT of the middle wo/men... Like all of them.

Third, incentives are much less for unmoderated participants. We're talking 10x less. On top of that, participants are only paid if they complete the full study. In our experience with moderated, we've paid lots participants who never even show up, or showed up for a half session and had to leave, or weren't the right fit, but showed up anyway so we had to pay them. In general, unmoderated participants are more than happy to provide 30-minutes of time for a fraction of the price. And they do all the work themselves, and on their own time schedule. Meanwhile the UX researcher doesn't even have to be there at the same time, or ever coordinate logistics.

Yay. Less money and capital spent. Please continue...

Return to overview ⤴

2. Recruiting is 3X faster

Unmoderated UX research is 3x faster ⚡️ than moderated. What takes 3 weeks in moderated, takes 1 in unmoderated.

Pulling queries, screening users, scheduling interviews, dealing with no-shows... forget about it. Unmoderated skips all of that, and more. There's considerable time needed to recruit for any moderated study. The process includes meetings with internal stakeholders, requesting user lists from data scientists, parsing list of users for best match, calendarizing upcoming 1:1 sessions, more meetings, and some more...

Meanwhile, unmoderated user testing allows you to establish your own screening criteria in a straightforward, then forget about the rest. Users will self-select into the your study. Often this can happen within minutes of launching your study. Yes, that fast 💨. If the user fits your required profile, then they quickly move on to interviewing (themselves!). The best part is you don't need any meetings to get the work done. It's just you and the testing platform. Better yet, the process easily allows you to set up multiple projects at the same time, with very few complications.

As an additional tip for unmoderated user testing , you can tailor your screening criteria to be fairly loose, strict, or somewhere in between. Often, we'll begin with very strict criteria to see what we get, and then loosen up as we go. Start slow and speed it up, or vice versa, and start fast to get wheels (participants) turning, then ratchet it down to hone it to perfection.

This is UXR at the speed of business .

Return to overview ⤴

3. The approach is more structured

On the one hand, the benefit of moderated UX research is you can do anything and everything in one study. In each interview, you can easily tailor which prototypes to show, modify which questions to ask, and alter the flow of the interview. This is great for certain circumstances, and absolutely necessary for others. The downside is this can lead to FOMO (fear of missing out), which results in packing too many things into one study. Also, this incentivizes the UX researcher to keep things loose 🤙 in their protocol to allow for changing-it-up on the fly.

Alternatively, unmoderated mandates that only so much can be accomplished. You can't do everything. For one, the participants aren't compensated as much, so they're not willing to spend more than 30 minutes in the study. That keeps things tight. Two, the same interview protocol is used for every participant, so things need to be buttoned up from the beginning. Three, you can't go too deep on too many subjects because you're not there to guide the participant if they get off balance.

All this means you need to be very structured in your approach to writing the interview protocol and Research Tasks. It can't be too long, it can't be too complicated, and it definitely can't be too loose. It's helpful to follow a standard arc in your Research Protocol. This gives you a common approach (user journey) to adhere across projects. It also allows internal stakeholders to understand the typical flow, and collaborate in the preparation process.

Keep it simple. Keep it standard. Keep it structured (the interview protocol that is).

Return to overview ⤴

4. Execution is increasingly automated

Besides using standard questions in your screening process, you can do the same for the interview tasks, to make things more streamlined. We recommend you create templates and re-use questions, tasks, and introduction statements for all your studies. All it takes it copy and paste thereafter to be asking the same foundational questions to a whole new set of participants. Over time, you'll create a great repository of questions / tasks from which to draw. Sure, you'll always have to create new questions specific to each study at hand, but a lot of the supporting elements will be saved in your repository, ready to go 🏁.

Better yet, you can be interviewing multiple participants at the same time! Unlike moderated, where you need to be present with every person to ask the questions, in unmoderated you don't. You can be sleeping, emailing, creating your next study, etc. Many times, several participants will be running through your study at the same time. You can conduct 10 interviews in 1 hour; not 10 interviews in 10 hours. Consider your mental relief alone!

Lastly, there's no camera work necessary. All sessions are recorded for you. No camera wo/man. No equipment to lug. No buttons to push. No shotgun mics to forget to turn on. Simply completed interview videos in your inbox.

Participants interviewing themselves, at the same time, with recordings, from an interview protocol you copied and pasted 30–40%... Yes to automation ⚙️.

Return to overview ⤴

5. Users may have more privacy

Have you ever been emailed a real list of customers with way too much information? Too many rows, too many columns, yikes! We have. And we don't like handling all the PII (personal identification information) or PHI (personal health information). You keep it 👉 moderated.

In unmoderated, you don't know more than you need. Participants sign up for your study willingly share what they're comfortable sharing. They know to what they've agreed. They're clear about what's being recorded, and familiar with the process as to why. They can easily stop if they feel uncomfortable, without social pressure. And importantly, you won't have a long list of non-participant PII to wipe off your drive.

Funny thing is, participants usually over share in all types of studies. They'll show you their screen when perhaps they shouldn't, or inform you of irrelevant, yet highly sensitive matter. So there's usually no issue in getting the good stuff from the participants you have. Best to keep privacy matters more strict when you can.

Because privacy matters, both to the user and to the company's bottom line.

Return to overview ⤴

6. Operations are easily scalable

Economies of scale rule in the business world. For long-term impact of UX research, we must find scalability in what we're doing. Unmoderated user testing provides an avenue to scale, unlike anything else we've seen in the field.

First and foremost, unmoderated platforms allow you to recruit participants from across the globe 🌎. Users from Brazil, France, Australia, U.S.A, Japan, etc. all in one study! Are you kidding me? Crazier yet, is it can all happen overnight. Participants from several countries will often respond to your user test. Now, not all product experiences you're working on will require this, or are scaled enough to support it. But some do, and this global scale is incredible.

In addition to scaling geographically, unmoderated studies can scale across time. We discussed earlier the notion of building template repositories for screeners, tasks, and questions. Again, we can't emphasize this enough for how this scales your time involvement in the now.

Considering the future, the video repository and highlight functionality allows UX researchers to save key insights for later. When a team needs to know something about the user, you can reach back into the videos vault easily and pull highlights that talk to those topics. Yes, you could have a video repository of your own, but some unmoderated platforms make this a very simple one-stop-shop for video editing and retrieval.

Scale across the globe, the present, and the future.

Return to overview ⤴

7. The experience can be higher quality

Getting users comfortable, set, and willing to speak their mind is 30% of the job in typical moderated interviews. Whereas in unmoderated, participants use their own tech devices, figure out their own problems, and do it all in their own physical space without your direct involvement. This makes the experience being tested very real. And when it comes to testing UX, real means higher quality. Observing, inquiring about, and analyzing real user behavior in their own natural context is our goal as UX researchers. Unmoderated makes this happen.

Furthermore, when users are in the comfort of their own home environment, without anyone present, they tend to dive deeper into topics / areas instead of shying away. This is extremely beneficial from a UX research standpoint. You're able to get to the core of problems and opportunities, faster. If the UX researcher was present, the whole experience can be unsettled. Social niceties, unconscious discrimination, and answering in ways "perceived" to be helpful are some problems encountered.

Better to leave the participant alone to share their heart out ❤️.

Return to overview ⤴

8. Tasks are truly consistent

Consistent user experiences makes things quantifiable. And quantitative (quant) data is a great partner to qualitative (qual) data. As UX researcher running moderated and unmoderated studies, we tend to lean more heavily on qual insights. So having supporting quant insights can be very powerful.

Since unmoderated tasks are truly consistent from one participant to the next, quant data is easily obtainable. Every participant is asked the same series of questions in the exact same manner. From this consistency in UX testing, you can quantify user behaviors, actions, and answers quite easily. This is something that's just not as possible in moderated sessions. There will be inevitable differences in the session between participants, even if subtle.

The consistent delivery of questions and tasks also benefit video analysis thereafter. Video clips 🎥 can be more easily snipped out, since unmoderated interviews follow a highly regular structure. Creating the highlight reels for sharing insights is all the more easy.

Easy peasy... insights that are quantifiable and trimmable.

Return to overview ⤴

9. Projects are highly iterative

The big bang UX research approach is flawed. Projects shouldn't tackle too much in one go. We see this problem over an over, especially when research doesn't happen enough. Inevitably, stakeholders try to pack too much in. Since moderated sessions take longer to plan / execute, they tend to become bloated 🎈.

Unmoderated user studies stay more lean. Quick moving projects allows for more iteration from one to the next. It's useful to communicate this notion to stakeholders as well. When asked to put one more series of questions in the study, you can easily retort, "Since we'll run another study next week, let's put that topic in the following study." This is helpful to keep a quick pace, and to simplify tasks for participants, which aids in their ability to focus.

UX research project [version_1], [version_2], [version_3]... only ever seen in unmoderated.

Return to overview ⤴

10. Unmoderated UX research is more fun 🎉

The future of UXR is asynchronous, unmoderated user feedback sessions. If you can get enough quality data about the user and their experience, without being there to interview every single one, that is huge. Yes, there's an additional nuance to talking to a real human, in real time, as there is with moderated sessions. This can be useful when first beginning a project, product, market, or industry. But over time — and in time — unmoderated sessions yield enough insight to make most businesses run at speed 🏎️.

It just so happens we find it fun and challenging to program computers to do our job for us. It's really interesting to imagine we're still running moderated sessions, only in an asynchronous manner via a computer interface. The most exciting part is to personify your interview style into the computer-driven tasks for the participants. Designing the user test takes real creativity, skill, and experience. You won't get it right on your 1st try. And maybe not on your 100th. But you can learn and evolve, and run hundreds of more. Unmoderated gives you that ability of speed and repetition to iterate as a UX research practitioner. In short order.

Time to be creative! Try new things as a UX researcher.

Return to overview ⤴

Challenges unmoderated UX research is perceived to have:

In every organization which we've worked, we've encountered some pushback against utilizing unmoderated UX research methods. Even though the organizations themselves pay for the service platforms (UserTesting, UserZoom, etc.), internal stakeholders often put up formidable barriers to entry 🙅. This is for various reasons. We'll begin by sharing typical comments a UX researchers encounter when proposing unmoderated as a method to generate insight for their product teams:

• "Will the right users even be in the participant pool?"
• "How will you ask them (users) to explain further, share more, and/or why?"
• "What if they get off course in the interview all by themselves?"
• "What if they can't get prototypes to work?"
• "Can't we get more info out of moderated sessions instead?"

Not to worry. We have answers for all of these hesitations. Contact us for tips and ideas if you're running up against comments like this yourself. In this article, we won't get into our best response them all, but generally one can counter with productive statements such as:

• "Let's begin by building the interview protocol, we can use it for moderated and unmoderated, whichever comes first (moderated will rarely be ready first)"
• "We can test the interview protocol in unmoderated, and if not sufficient insight, we'll still be able to run moderated as fast follow"
• "OK, we can run both simultaneously if you're worried (unmoderated will run 2-3 weeks ahead of moderated, by which time the team will have moved on for their own reasons)"
• "We can launch the unmoderated tonight/tomorrow/this week, and see what we get in the morning, just as an initial test (participants, responses, protocol, etc.)"
• Or, don't say anything, and just run unmoderated to hone your skills, protocol, practice, etc., and convince them next time 😁

As you can see, most of our responses emphasize action and 👉 forward progress over lengthy debate. This is how you can operate UXR at the speed of business. Don't get caught up in politics and jousting over methods. A wise person once said, "Don't let perfect be the enemy of good". Unmoderated UX research will prove itself when you give it a fighting chance. And you'll develop your own ways to counter the challenges that unmoderated UX research is perceived to have.

To be sure, there's a time and place for moderated user interviews. We won't act like there's not. Situations include: early-stage generative research, UX persona development, complicated UX testing, sensitive subjects, niche subjects, physical product experiences, and more. However, we see so much over-emphasis on the moderated testing front that we decided there's a need to showcase all the positives that unmoderated yields. And there's quite a few when you look closely 👀.

Back to top ⤴