Engaging Study Observers: An Overlooked Step in User Research

At AutoTrader.com, we have learned the value of actively involving observers when conducting usability studies. We have always invited stakeholders, such as product and project managers, visual designers, and interaction designers, as well as individuals with periphery interests such as marketing researchers and QA analysts to observe usability studies. However, our practice has evolved to improve our approach and provide a more valuable experience for observers and our usability team alike. The goal of this evolution has been to ensure that observing usability sessions is an effective use of time for our colleagues while also inviting them to provide valuable input for our findings reports. After reading this article, our hope is that you will not only gain insight into the advantages of incorporating observers into your practice, but that you will learn how to do so effectively.

Seeing is Believing

Attending usability study sessions provides observers with a unique opportunity to watch and listen to users as they interact with products. Prior to a study, stakeholders often find it difficult to imagine or articulate issues that may be encountered by end users, and they find it extremely valuable to witness issues unfold. It is during sessions that they get to see their concerns validated or eliminated and new issues and ideas uncovered.

Two (Or More) Heads Are Better Than One

Immediately after study sessions, our team conducts a debriefing where observers have the opportunity to learn from, and discuss findings with, the usability team and fellow observers. The atmosphere is one of energetic collaboration, with a diverse group represented in an open forum. In addition to seeing and hearing users’ perspectives, stakeholders are given the opportunity to see their project through the eyes of other observers, which provides valuable insight. They leave with enhanced understanding of issues and sometimes even with solutions. As the adage “two heads are better than one” suggests, valuable collective discoveries emerge beyond what a single observer might see.

The Magic of Mutualism

Observers are not the only ones to benefit from their inclusion. Researchers also benefit in gaining insights beyond what may have been captured in their session notes and recordings. Discussions with observers help researchers gauge if there is consensus on an issue or, conversely, when an issue may be contentious. Observers sometimes use interesting terminology and phrasing researchers might not be aware of, and key stakeholders oftentimes illuminate the history of a project, along with business considerations and challenges they face. As we describe findings and offer recommendations in the findings report, we’ve discovered that learnings from observer comments can help us craft our research debriefs in a more persuasive manner.

As a direct result of our observer management process, we have seen an increased interest in the work we do from both project stakeholders who are well-versed in our practice, and from observers who had not previously been exposed to our work. This has increased awareness in how we can help projects, giving us the opportunity to undertake wider variety of research activities.

Watching users interact with products is no longer framed as an “if” or “maybe” aspect of a project timeline—it is now framed as “when” and “how.”

Establishing Best Practices

As you can see, the value of inviting others to observe usability sessions has been firmly established in our practice for quite some time. However, the day-to-day logistics of exactly how to include observers effectively has seen significant evolution in our practice. We have learned a number of lessons along the way.

Before the Study: What Doesn’t Work

It used to be that when inviting observers to study sessions, a general email about the study was sent and observers were asked to reply with which sessions they wanted to attend. Our team tracked responses, but formal calendar invites weren’t sent. As you can imagine, observer no-shows were common.

Additionally, when preparing the observation room prior to a study, we did not place note-taking materials at each observer seat. Rather, note-taking materials were placed in a small stack between seats and the test plan and participant demographic sheet were placed at each seat. In most cases, note-taking materials went unnoticed, while observers noticed the test plans and participant demographic sheets immediately.

What Does Work

Our process for inviting observers to study sessions now involves the following steps:

  1. Outlook calendar invites are created for each session of the study.
  2. Key stakeholders are automatically invited to each session two weeks prior.
  3. An email describing the study and welcoming observers is sent out to others who may be interested in the sessions.
  4. Outlook calendar invites are sent to individuals who responded to the email.

We allow up to ten observers per session, with executives, upper management, and key stakeholders given highest priority. We then fill the remaining observer slots with members of our department and other research teams on a first-come, first-served basis. Observer no-shows have decreased significantly since we started sending calendar invites.

Additionally, from our first contact with potential observers, we now establish and define the observer role so that participation expectations are set. In the invitation email, we state that observers are responsible for recording at least two findings and sharing them with the group after each session. The observer role is emphasized again when the calendar invite is sent to each individual.

When setting up the observation room, we now strategically stack observer materials at each seat in the following order from top to bottom: large Post-It pad with a Sharpie, a cover sheet that lists brief observer “rules,” the study test plan, and the participant demographics list (see Figure 1). We also pre-label a whiteboard that can be used to organize the observer Post-Its after each session.

ign with instructions for observers

Figure 1. Materials that are provided to each observer.

During Each Study Session: What Doesn’t Work

Initially, to engage observers during study sessions, we tried allowing them to write down anything they found interesting on Post-Its using a Sharpie. We did not give them clear instructions or set expectations as to how the notes would be used. In the end, observers wrote very little.

We also invited observers to be the official note-taker during sessions, but most were not interested. When there was interest, the quality of the notes suffered due to difficulty determining what information was appropriate to record and in what format.

At one point, we asked nothing of observers during sessions and did not provide them with a means to take notes. Consequently, observer engagement suffered.

In general, due to the lack of organization and clear instruction with the above methods, observers were not able to channel their thoughts in a way that was useful. As a result, they often engaged in conversations with one another, creating a noisy environment and making it difficult for the note-taker and other observers to hear participants.

What Does Work

Our current approach is much more structured. In the five minutes before each session, a usability team member verbally reminds observers that their participation is important. Specifically, we emphasize that the Post-Its provide a means for us to consider their input in study reports, and we describe how each observer will be given the opportunity to share their notes.

We also emphasize the purpose of the study and provide examples of findings that are and are not appropriate to capture during a usability study, focusing on observations of user behavior rather than user opinions.

Observers have been notably quieter as they actively and enthusiastically record their observations on the Post-Its, knowing they will have an opportunity to discuss the issues after the session. We have found that giving observers a specific task, providing them with materials to complete the task, and explaining why their contribution is important works wonders at keeping the noise level down and involvement up.

After Each Study Session: What Doesn’t Work

Initially, our session debriefs were short, casual, and unstructured, with some rough notes captured informally on a whiteboard. Sometimes observers turned in handwritten notes, but we didn’t collect them in an organized fashion. Observers often left sessions without key takeaways, or with incorrect takeaways, and missed out on hearing insights from fellow observers because very few observers participated.

To increase collaboration and participation, we formalized our debriefs and recorded organized notes on a whiteboard. Originally, we organized notes by participant and only recorded findings that were unique to a given session. Observers were noticeably quiet during the debriefing or engaged in side conversations since some may not have attended previous sessions and did not know what was unique about the session they attended. Additionally, the notes were of limited use to our team when creating the findings report since we group our formal findings by screen or topic and not by participant.

We eventually began labeling the white board by topic and recorded notes with frequency information next to each finding. This change helped when transferring the notes to the findings report, but it did not increase observer engagement. We inadvertently created awkward, frequent lulls in conversation when turning to the whiteboard to handwrite findings, and ended up talking amongst ourselves more than we would have liked. Observers relied heavily on memory as discussion progressed, sometimes resulting in misremembering an event or discussing issues that were most memorable, while overlooking or downplaying other issues.

What Does Work

We have now evolved into a more structured method that sets expectations around the preferred level of observer involvement and provides clear guidelines for observer participation. We set the stage at the beginning of debriefs by reminding observers that they play an important role, describing the purpose of the debrief and how long it will take, and reminding them that everyone will get to share their notes.

Each observer then shares their Post-It notes with the entire room and helps categorize them on the whiteboard (see Figure 2). Involving the observers in the notes-mapping process assists in keeping the conversation flowing. For example, we might ask an observer, “Where do you think this goes on the board?” If time is running out, we focus on a rough mapping during the debriefing and correct as necessary after the debriefing.

Photo of room with notes covering two walls

Figure 2. A labeled whiteboard with categorized observer notes.

We designate a “parking lot” section on the whiteboard so we can politely acknowledge notes that are off topic. As a general rule of thumb, we set aside at least thirty minutes for debriefs after each session, and debriefing does not end until all observers have shared their notes. At the end of each debriefing, or earlier if needed, we caution our observers not to make generalizations too early and without having observed multiple sessions.

The levels of enthusiasm and energy remain consistent throughout the debriefing as observers take ownership of the documented findings. Lulls in conversation are no longer an issue since we quickly categorize the Post-Its, and observers are eager to empty their hands of Post-Its. Interestingly, as an added bonus, observer debriefing has almost become observer led, requiring less work of our team than any of the prior methods we tried.

Conclusion

We take great pride in having created an atmosphere of transparency and collaboration between our usability team and observers of usability studies. Building this atmosphere has not been without its challenges, but we have successfully found ways to balance observer inclusion with efficient study goal accomplishment.

By openly and directly communicating with observers prior to studies, providing them clear instructions and materials during studies, and giving them an organized way to express themselves after study sessions, your practice will only improve. As your practice improves, never forget to express your appreciation frequently. After all, a rising tide lifts all boats.

Fishel-Brown, S., Schroeder, E. (2013). Engaging Study Observers: An Overlooked Step in User Research. User Experience Magazine, 13(1).
Retrieved from http://www.uxpamagazine.org/engaging-study-observers/

Comments are closed.