The Magazine of the Usability Professionals' Association
By Tony Tulathumutte and Cyd Harrett
Remote user testing is primarily employed when speed, convenience, or international testing is called for. However, there are practical qualitative benefits to using remote methodologies, as well. In fact, the remote aspect of the process makes it suited to Wizard of Oz-style moderating techniques, in which a single trained moderator can hide a great deal of complex, behind-the-scenes setup, which can be invaluable for capturing user insights.
One new opportunity is the possibility of multiple people involved in the moderating process —in particular, clients or stakeholders who are invested in the outcome of the user research, but who are not themselves trained for moderating. No longer do clients have to stand behind a "two-way mirror," passively observing research sessions; remote methodologies make it possible for them to easily contribute to the process, providing richer results along with greater client buy-in and understanding of the project. This article addresses the benefits, tools, methodologies, and logistics of involving clients in research.
To get the most out of client participation, clients should be able to see and hear everything that the moderator can during the session. In its most basic form, this simply requires clients to sit alongside the moderator and listen in on the session, passing notes to the moderator whenever appropriate. However, there are many ways to use various remote research tools to streamline this process.
For remote web research, the tool choice is an important consideration. The UserVue screen-sharing tool is distinguished by its automated recording functionality, as well as its support for observer chat and video marking for later analysis. However, there are many screen sharing tools with basic built-in observer support, including GoToMeeting,NetMeeting, and Adobe Connect. The audio can be shared using any two-line phone with a conference call feature; the moderator can speak to the participant on one line, and clients can listen in on the other. It's important that their line remains muted, so participants don't hear them.
Since the moderator will clearly be unable to speak to anyone other than the participant, typically some form of online chat or instant messaging is used to communicate with clients.
This approach provides many advantages:
Participating clients should be properly briefed in advance of the research to make their contributions as useful and non-disruptive as possible.
To minimize questions to the moderator, provide clients with a document they can refer to should they have any questions about their participation or the course of the research. The document should include clear instructions about how to set up screen-sharing tools, how to chat or mark videos in UserVue, how to properly contact the moderator for requests via IM or chat, and so on. Generally, limiting the number of clients communicating directly with the moderator to one or two helps keep the requests manageable.
Clients should also be given expectations about what constraints the moderator will be working under during the session. Since the moderator will be taking notes and speaking to the user, it may not be possible to respond immediately to clients' questions. Style is another constraint: the moderator must keep the tasks natural and the language neutral, so client requests may not be acted upon right away, or precisely as requested. For example, a client may ask, "Ask the user what he thinks of the navigation bar," but the moderator may choose to get at the issue in a more natural and less leading manner with a series of questions: "Where would you go to look for laptops?" or "How would you get back to the main page?"
As mentioned above, clients may occasionally ask the moderator to address issues that lie far outside of the scope of the research or would be unduly complex to tackle within the allotted session time. Clients should be encouraged in advance to indicate important emerging topics to the moderator, but to generally adhere to the scope of the research. It may also be wise to politely request that clients defer to the moderator's judgment with regard to balancing emerging topics with established research goals.
Having clients actively engaged increases buy-in.
If clients are adequately prepared in the above manner, minimal management will be required during the study. However, the moderator should not assume that clients will follow the preparations to the letter and should be prepared to actively manage clients during the study.
A common issue is dealing with clients who make too many demands of the moderator to reasonably act upon within the allotted time. It is important to promptly acknowledge all client requests to reassure them that the moderator is listening, but it is not necessary to promise to act upon every request. In order to avoid appearing rushed or irritated with client requests, moderators should use polite, concise, and unambiguously friendly language to acknowledge them: "Understood. I'll see what I can do! " Moderators should also ask for clarification if the client request was unclear: "Not sure I quite understand, can you explain? Thanks!" Ultimately, if there are too many requests to handle, the moderator should have a short discussion with the clients at the end of the session, acknowledging it: "Sorry I couldn't get to all of your requests—it was important to leave time to achieve all of our study goals. I'll definitely try to handle as many of your requests as I can, so feel free to keep sending them along!" The aim is to make clients feel comfortable communicating with the moderator, but also that they understand if the moderator cannot comply with every request.
There may also be the problem of clients who don't seem to be paying enough attention to the session, or who do not ask any
questions at all. If clients are silent for a long time (ten minutes), it's good practice to ask clients if there's anything they're curious about or would like to have the moderator probe further on. If clients respond no, then the moderator can simply continue; if clients remain silent, then the moderator can follow up with a brief assessment of the session to encourage more client discussion: "This user seems to have a lot of trouble logging in. I'll try to find out more about that."
Finally, since moderators are responsible for gathering the feedback, they should also avoid soliciting advice or over-relying on clients for what to tell a participant; for example, "What should I ask the user now?" Client requests should be regarded as more of a tool for refining the research objectives, not a primary source of guidance for the moderator. The primary guidance, of course, should come from documents based on the research objectives, such as the moderator script.
It can be very productive to hold very short debriefing sessions after each session to let clients fully elaborate on things they may have said during the study. When all testing is complete, clients may participate in brainstorming sessions, contributing insights they may have gathered as outside observers. This is also a good opportunity for moderators to identify from clients’ interpretations any biases they may have about the research. For example, they may take a user’s comment that “the login process was okay” to mean that there are no problems with the login process, whereas a trained moderator may have identified numerous behavioral issues that contradict that perception. Moderators can then be prepared to address this bias during the drafting of the report. Finally, when the findings are presented, it’s nice to take time to acknowledge the clients’ contributions to the project, specifically calling out instances where they helped. Your ongoing efforts will benefit from having the assistance of clients who are active and interested and who have gained practical experience in user research
About the Authors:
Tony Tulathimutte works as a UX researcher for Bolt|Peters, a user experience firm specializing in remote research methodologies. Bolt|Peters is located in San Francisco, CA, USA.
Cyd Harrell has been the director of user experience at Bolt|Peters since July 2006, after several years at Charles Schwab. She received her formal training from Yale University, where she studied linguistics with a focus on socio-linguistics. She is a co-founder of San Francisco Women on the Web, an education and networking organization. She manages B|P’s consulting arm, handles sales, and researches as well; in the rest of her life she writes poetry, practices yoga, studies Thai cooking, and hangs out with her daughter.
Usability Professionals' Association
promoting usability concepts and techniques worldwide
User Experience Magazine is by and about usability professionals, featuring significant and unique articles dealing with the broad field of usability and the user experience.
This article was originally printed in User Experience Magazine, Volume 7, Issue 3, 2008.
© Usability Professionals' Association
Contact UPA at http://www.usabilityprofessionals.org/about_upa/contact_upa.html