Conference Program Welcome! At a Glance Presentations Tutorials Workshops Tours Posters SIGs Other Conference Info Frequently Asked Questions Sponsors About the UPA Special Events Student Volunteers UPA home

Workshop 2 New Methods and Practices in Usability Testing Joseph Dumas, Ginny Redish, Jeff Rubin June 29, 1999, 8:30 a.m. - 5:30 p.m. Are you involved in usability testing? Are you doing testing now at different times or in different ways than you and others were doing five or more years ago? If you are, join Joe Dumas, Ginny Redish, and Jeff Rubin in a workshop at the UPA conference on New Methods and Practices in Usability Testing. We're interested in sharing what has changed about usability testing in the last five years. A UPA workshop (like a SIGCHI workshop) is a full-day discussion among specialists in the field (not a tutorial). You must apply for the workshop by writing a proposal (position statement) of no more than five pages. See the detailed description below for more on what we expect in your position statement. Participation is limited to 15. If you have any questions, write to one of us: Joe Dumas at jdumas@air-ne.org Ginny Redish at redish@ari.net Jeff Rubin at jrubin@jra4usability.com We are hoping that a new book, an anthology that will update the Dumas & Redish and the Rubin books, will come from this workshop. So, come with the idea of a chapter you might write for the book in mind. Detailed Description What general problem do we want to explore? Usability testing has become a standard tool in the methods that usability specialists use to evaluate the ease of learning and use of products or systems. Two book-length descriptions of usability testing practices were published in the early 1990s, Dumas and Redish (1993) and Rubin (1994). They both covered the methods and procedures that were common five or more years ago and amply covered the transition of testing from a formal, late-in-development method to a less formal, iterative approach. Since those books were published, however, the variety of usability testing methods, tools, and contexts has grown significantly. In addition, new software development approaches, such as participatory design, have incorporated testing but with modified methods. A third early book, Wiklund (1994), described the usability practices of 17 corporate usability lab groups, but many of the authors are no longer at those companies, and both the companies' and the authors' practices have changed. While usability specialists have continued to evolve and improve testing methods, the dissemination of information about new methods, tools, contexts, and issues has lagged behind. Certainly, articles have appeared in newsletters and journals from UPA's Common Ground to the special issue of Behaviour and Information Technology. There have been papers and panels at UPA and CHI, such as a workshop on remote testing. But people with experience with and data about new methods, tools, and contexts have not come together in one place to share that information or planned how to make it readily accessible to the larger usability community. At this workshop, participants will gather together and disseminate information about new usability testing methods, tools, and practices in specific contexts. The workshop leaders will be the authors of the two book-length descriptions of usability testing: Joe Dumas, Ginny Redish, and Jeff Rubin. If you think you would like to be a participant in the workshop, we encourage you to read this call for participation and submit a proposal. What are the goals of the workshop? We have three major goals for the workshop: To bring together usability specialists with experience with new usability testing methods, tools, and contexts, especially those not covered in our earlier books on testing so we all can share and discuss those methods and practices. To decide, as a group: Do new testing methods constitute a new vision for testing or an evolution from earlier practices? To discuss ways of disseminating information about usability testing issues to the wider community, perhaps planning an anthology that would serve as an update to the earlier books. What is the scope of the workshop? To limit the workshop to the time available, we plan to focus on the current situation in usability testing, not other usability evaluation methods, such as expert reviews or walkthroughs. To clarify what we mean by usability testing, we will adopt a slightly modified version of the definition proposed in Dumas and Redish (1993). That is, every usability test shares five characteristics: The primary focus is on the usability of the product. The participants represent typical users of the tested product. The participants perform tasks that probe the usability of the product. Tester(s) observe and record what participants do and say. Tester(s) analyze test data, diagnose usability problems, and recommend solutions. Comparisons of usability testing with other evaluation methods would be a secondary, not a primary, focus of the workshop. What specific issues will we address? As experienced workshop leaders and participants, we know that workshops are most successful when at least some of the issues come from the people who want to participate in them. Nonetheless, we have set down a tentative set of issues for you to consider, as shown below: What are the major differences between your testing methods and practices now and what you were doing five years ago? What specific contexts do you test in now that are different from what you were working on five years ago? How does testing now fit into current product design and development methods? How does that compare with what was happening five years ago? How visible are current testing programs in product development organizations? How does that compare with five years ago? Has the role of usability specialists, interface designers, technical communicators, developers, and others changed in relation to how usability testing is done? How is it different from five years ago? What are the limitations of current testing practices? How is testing likely to be different five years from now? Who should attend this workshop? Our expectation is that the workshop will have about 15 participants plus the three workshop organizers as workshop leaders. We would like you to be: Interested enough to prepare for the workshop and potentially to contribute to a book (anthology) describing new usability testing issues. An experienced and current tester who has thought deeply about the usability testing methods you use. An advocate for new ways of testing. Knowledgeable enough about the literature on usability testing to know whether the issues you bring to the workshop have already been described adequately in published sources. What should comprise your proposal? Proposals for participation in the workshop are limited to five pages and must include: A description of your experience conducting usability testing. An initial position on whether the way usability testing, as practiced in 1999, is an extension of practices begun in the early 1990s or needs a new definition and description. A description of the topic that you propose to discuss at the workshop and whether it is a new method, tool, or context. A justification for why the topic is important for usability testing practitioners to know about. We will select participants on the basis of the quality and uniqueness of their proposals. We will distribute the papers by all accepted participants to all other accepted participants. Everyone coming to the workshop will be expected to have read the papers before the workshop. We will spend workshop time in discussion of issues, not on presentations from participants on the contents of their proposals. How will we disseminate the information? In addition to reporting at the UPA conference, we will summarize the workshop for Common Ground. Our hope is that you will be ready to contribute a chapter to a book, inspired by the workshop, informed by the discussion at the workshop, and based on the specific methods, tools, and contexts that you bring to the workshop. However, proposals will be evaluated on the basis of their merit, not on your commitment to preparing material for a book after the workshop. Remember, the proposals are due on April 15th and are limited to five pages. Keep in mind that the closing date for the early registration fee for the UPA conference is May 14th. Registering for a Workshop Workshops are closed sessions; you may register for a workshop only with permission of the workshop organizer. To participate, send your position paper to the address below by May 27, 1999. Electronic applications may be submitted in plain text (.txt), rich text format (.rtf), or Microsoft Word for Windows. Although the workshop deadline is after the end of early registration, you can still qualify for the early conference registration discount if you register during the early registration period. Your registration should include the workshop you are requesting. Workshop registration (which is not discounted based on the registration date) will be processed when approved by the workshop coordinator. Joseph Dumas 470 Virginia Road Concord, MA 01742 +1 978 371 5885 jdumas@air-ne.org