UX Magazine logoThe Magazine of the Usability Professionals' Association

What Do You Mean? How to write good questions

By Jessica Enders

The work of a usability practitioner crosses many disciplines: from ergonomics to graphic design, from visual perception to web standards. The situation is no different when designing forms. This is why a background in survey methodology and social research can facilitate a transition into user-centered design.

The field of user interface design is relatively unaware of the well-established question design principles used in the fields of survey methodology and social research. Perhaps this is because usability has arisen largely from the study of ergonomics, human factors, and industrial design, which have considerable focus on the more physical and aesthetic attributes of design. Writing a good question, however, draws on the principles of linguistics, sociology, and psychology—areas that form the foundation of survey methodology and social research.
The problem is, no matter how well laid out the form is, or how streamlined the form filling and processing procedures are, the usability of a form will be significantly compromised if the questions on the form are poorly constructed. This article aims to address this potential gap by providing some models for good question design.

Why Worry About Questions And Answers?

When I tell people that I design forms and have a particular passion for writing good questions, they often respond with incredulity. “Surely writing questions isn’t that hard?” I can understand their perspective; we ask and answer questions hundreds of times in the course of a normal day. “What was your day like?” “Do you want salt and pepper on your sandwich?” “Can you get this finished by the end of the week?” “What time is the party on Saturday?”

We are comfortable with conversational questioning because the grammatical structure of this exchange is well understood by fluent speakers of the language being used. But asking an effective question relies on more than just an understanding of grammar. Consider the following example: “How long have you been using a walking stick?”

Suppose someone first started using a walking stick after she hurt her hip in 2000. She had surgery in 2002 and the surgery gave her two years of relief, during which time she did not use the stick. Unfortunately, the pain returned in 2006, so she started using the walking stick again.

If this question is being asked at the start of 2009, what answer would the person provide? An answer of “eight years” would be correct since it is the elapsed time since she first started using a walking stick. But an answer of “four years”—that is, of actual use—would also be reasonable answer.

If the question were being asked in conversation there would be many cues that would guide the person’s answer. For example, the person may know the individual asking the question and have a sense of why they are asking. Moreover, there’s a good chance that the person would ask for clarification before responding, or include explanations in her answer.

When someone is filling out a form, be it paper or electronic, they don’t have nearly as many cues and conversational techniques to draw on. In this situation, if the respondent doesn’t give up altogether in frustration, they will usually try to guess the designer’s meaning and answer accordingly. The respondent’s intentions are good, but unfortunately their actions—which are a direct response to the question—can result in inconsistency in the data. To make matters worse, looking at the responses for this form won’t raise any alarms, since they will all be valid.
An apparently straightforward question can result in inconsistent data, but that inconsistency will be essentially impossible to detect. This could have serious consequences if you were asking a more complex question about health or taxation.

A Model for Answering Questions

In order to write good questions, you need to know the different ways that a question can fail. In the example above, the problem is one of understanding the question (what does “how long” mean?) and judging the appropriate answer (do they want me to report elapsed or actual time?).

Comprehension and judgment are just two cognitive processes involved in answering a question. There are a number of models that survey methodologists and cognitive psychologists have proposed to describe all the different stages the respondent might go through. In their book “Forms That Work,” Jarrett and Gaffney describe a four stage model, which is, in turn, based on earlier work by Tourangeau, Rips, and Rasinski:

  1. Understand the question
  2. Find an answer
  3. Judge whether the answer fits the question
  4. Put the answer on the form

The form designer needs to be conscious of the possibility of error at each step. To see this in action, try answering the following question for yourself:“What did you have for lunch last Tuesday?”

The areas of difficulty that you might encounter include:

1. Understanding the question
Does “last Tuesday” mean the most recent one before today, or the one before that?

2. Finding an answer
Research suggests that to find an answer to this question, most people will:
Try and remember what they were doing last Tuesday and use that as a cue for what they would have been eating; or
Use a rule like, “I try to take leftovers from the previous night’s dinner to work for lunch each day, so what did I have for dinner the night before?” or “Tuesday is pay day so I always go out and have pasta.” Bad luck if last Tuesday was an exception to the rule!

3. Judging the answer
If last Tuesday was a special celebration at work and you ate something unusual, perhaps you considered whether you should say what you actually ate or what you would normally eat on a Tuesday. And just how sure are you that it was Tuesday, and not Monday or Wednesday, that you had that sushi?

4. Putting the answer on the form
Perhaps you’re all ready to write “rice noodles” when you find that you have to choose from a series of pre-determined response options. There’s a “pasta” option but no “noodles.”
What do you do now?

Writing Good Questions

We can use Jarrett and Gaffney’s model to prepare a kind of checklist of things that lead to a well-designed question.

1. Understand the question

2. Find an answer

There is an enormous amount of information and research on the workings of the human memory, too much to be summarized in a paragraph or two. However, there are key aspects that every question designer should be aware of. Be conscious that:

3. Judge whether the answer fits the question

4. Put the answer on the form

Rewriting Problem Questions

Two questions were used in this article to illustrate how difficult it can be to write a good question (“How long have you been using a walking stick?”) and the stages involved in the question answering process (“What did you have for lunch last Tuesday?”).
Applying the principles described in this article, some possible alternatives for asking these questions may be as follows:
For the walking stick question:

The lunch question is probably never going to yield a truly accurate response because it refers to a forgettable and unimportant event. However, if we did need to ask the question, we might choose to ask:

Which version is the best alternative depends heavily on exactly what data are needed and why.

It should also be apparent that some questions need to be made longer in order to be made more clear. This is one of the reasons that using fewer words, fewer questions, and fewer pages is not always appropriate in form design. The goal should always be greater clarity, whatever that requires

About the Author
Jessica Enders is principal of Formulate Information Design, a business specializing in the design of paper-based and electronic forms. Jessica has worked as a survey methodologist at the Australian Bureau of Statistics, a survey designer for Colmar Brunton Social Research, and as an interaction designer at The Hiser Group. Jessica has worked on forms for the public, private, and not-for-profit sectors and across many areas, from superannuation to measuring children’s progress.

UPA logo
Usability Professionals' Association

promoting usability concepts and techniques worldwide

User Experience Magazine is by and about usability professionals, featuring significant and unique articles dealing with the broad field of usability and the user experience.
http://www.usabilityprofessionals.org/upa_publications/user_experience/

This article was originally printed in User Experience Magazine, Volume 8, Issue 2, 2009.
http://www.usabilityprofessionals.org/upa_publications/past_issues/2009-2.html.

© Usability Professionals' Association
Contact UPA at http://www.usabilityprofessionals.org/about_upa/contact_upa.html