Workshops

Usability practitioners may be frustrated when test-based recommendations aren't implemented. But they often aren't aware of constraints that influence recommendations' adoption . We bring together individuals from across the entire breadth of the usability community to identify commonly present obstacles to acceptance and to evolve strategies to counter such obstacles to change.

WORKSHOP DESCRIPTION

Usability evaluation techniques (including, among others, usability testing and heuristic review) are now widely incorporated into software design projects. However, the influence of evaluation-based recommendations on the ultimate (re-) design is often limited and sometimes tenuous. As a result, usability professionals are often frustrated and disappointed to find that data-driven recommendations designed to mitigate or solve seemingly obvious usability problems are not implemented after usability testing.

This workshop seeks to increase the adoption rate for usability recommendations by

• eliciting the technical constraints and organizational dynamics that undermine the impact of usability assessments, focusing on common or widespread conditions
• exploring how those barriers limit the implementation of
  (re-)design recommendations
• articulating interactions among the constraints and challenges that impede the adoption of usability recommendations
• innovating strategies to justify, position and prioritize recommendations to either accommodate or overcome these sources of resistance

Many factors influence the adoption of recommendations after usability evaluations are completed. One critical factor is the relationship between the evaluator(s) and the design team. Evaluators often take on a ‘consultative' role, emulating practices developed in domains such as organizational design or management consulting. On this model, consulting evaluators are engaged and briefed on the project requirements, conduct their evaluation, report their findings and move on to the next project. In many domains, this approach is quite successful. However, the iterative process of user-driven software design calls into question the efficacy and appropriateness of this one-time plug-in feedback approach.

In the (not uncommon) “worst case” scenario—an outside evaluator is engaged to perform one or more tests. In order to design each test, this team is given information about the specific software product’s performance requirements. However, evaluators rarely have the opportunity to gain insight into the contextual factors that continuously shape and constrain the development process. Usability professionals often know little or nothing about such important influences as the overall importance of the software project for the client organization, or the schedule of the design project, and how that schedule impacts the feasibility of implementing test-driven usability enhancements. In addition, they may not understand the availability of resources required to implement usability changes, or the skill levels of the developers available to implement the changes. They may use pre-test briefing information idiosyncratically to develop the evaluation protocol, and their recommendations may not be well justified in a Findings Report. Finally, little attention has been paid to developing strategies to position recommendations by invoking cost-effectiveness to support evidence-based arguments for enhancing software usability.

This workshop is intended to provide a forum and framework to begin the process of critical reflection on current practices in usability evaluation. By bringing together usability evaluators with software designers and other organizational staff who are their clients, we will focus on identifying practices that increase the positive impact of usability tests, in terms of achieving the highest proportion of usability-enhancing design improvements implemented within real-world constraints such as those listed above. By exploring approaches to usability evaluation that take practical constraints on implementation into account when prioritizing usability-enhancing recommendations, we can raise the perceived justification for, and thereby the probability of actual adoption. The resulting positive effects on product usability help all parties involved justify usability evaluation practices on the basis of cost-effectiveness—and THAT very powerfully “sells” management on usability as an essential product quality.

Why this workshop topic is appropriate at this time

Interest in usability as a strategic differentiator is high. Maintaining enthusiasm for the routine application of usability methods into the software design process depends critically on our ability to deliver practical recommendations for improvement. The organizers’ have broad experience with usability testing across a broad variety of organizational contexts, in both the public and private sector, as both evaluator and client. This diversity of experience has made them keenly aware of the many problems associated with contextually uninformed testing, including:

• variation in client organizations’ ability to assimilate and implement usability test recommendations
• misalignment of test objectives and redesign/reimplementation resources, resulting in impractical recommendations
• impractically positioned or politically impossible recommendations that may result in a client organization perception that a usability evaluation failed to provide benefits commensurate with its cost

However, usability testing is important and generally worthwhile, even if its impacts are limited. By exploring in this workshop both effective strategies for, and organizational challenges to the adoption of evaluation-driven usability recommendations, we can begin to identify and evolve “best practices.” The goal of following such practices is to enable evaluations to produce more recommendations that are positioned to favor acceptance and implementation by client organizations. Through such successful cases, the usability profession can increase the perceived usefulness of its members’ work and increase the actual cost-effectiveness of that work.

PARTICIPANT SELECTION CRITERIA

Prospective participants will be asked to submit a 3-5 page “case study” report outlining direct experience on a software/interface (re-) design project involving usability evaluations conducted by individuals not directly involved with that (re-) design project. For example, the study described might have involved an internal usability group or external consultants.

Using those reports as a filter, participants will be selected to maximize the diversity of perspectives within the workshop. It is our goal to include individuals from ‘all sides of the table,’ including representatives of groups such as:

• Usability evaluation consumers (e.g., Project Managers including budget oversight functions, strategic decision makers, other “concerned” stakeholders)
  
• ‘In-house’ usability practitioners
  
• Usability consultants (practitioners or managers)

• Implementers (developers, designers and graphics artists)
  
• Spokespersons for end user groups such as customer support staff members

Participants will be chosen to bring together a variety of experiences and backgrounds to provide a rich experiential mix that should lead to fertile joint exploration. This workshop is intended to be an opportunity to collaboratively construct new approaches to solving challenging problems. These problems arise from the need to enhance product usability effectively and affordably, so that the customer is satisfied and the value of usability engineering is reaffirmed.

It is our goal in this workshop to build better ways of communicating among the diverse “communities of practice” that usability engineering affects, by bringing together a group that represents those communities’ diverse perspectives toward user- and citizen-centric design activities and processes.

Click here to link to sample position paper

Applying to Participate in This Workshop

A workshop is a closed session. Admission to a workshop requires an approved “case study” report from you addressing the issues suggested by the coordinator(s). Please send your position paper (which should be roughly 1 to 3 pages) to John Bosley, bosley.john@bls.gov. Position papers received by March 24 will be accepted or rejected by March 31, in time for you to register before the early registration deadline on April 2. Position papers received by May 5 will be accepted or rejected by May 12, in time for the May 14 registration discount. Papers received after May 12 will be evaluated at the facilitator's discretion. If you want to register early for UPA and have not completed your position paper by these deadlines, you may register for the rest of the conference and add the workshop fee later.

PRE-WORKSHOP PARTICIPANT ACTIVITIES

Participants will be asked to develop a brief (3-5 page) case study report based on their direct experiences as a foundation for participation in the workshop.

The report should include

• Project description and purpose (e.g., Public Site/Intranet/Application design or re-design)
• Major objectives (to the extent those were articulated explicitly)
• Type and scope of usability evaluation, including timing relative to the development timeline of the project (e.g. conceptual design phase, early prototyping, design nearly complete, redesign)
• Significant customer roles and relationships, such as project manager, designers, evaluators and other decision makers
• Author’s role, in relation to customer roles, (e.g. main point(s) of contact with customer organization/design team or with usability evaluator)
• Outline major recommendations of the evaluations that were adopted (fully or partially)
• Outline major recommendations of the evaluations that were NOT adopted and outline why recommendations were not implemented (buttress with “objective” evidence where possible)
• Outline or discuss mitigating factors that, if known, might have increased recommendation adoption

In addition to this case-descriptive material, we request that participants attach the following:

• Questions or issues that the participant brings to the workshop for discussion and perspective-sharing
• Participant’s principal expectations about “take-away lessons” from the workshop
• A brief biography of the participant (less than 1 page)
  

PRE-WORKSHOP FACILITATION ACTIVITIES

• Review and select workshop participants to represent a broad range of usability-oriented communities upon which to build a collaboration
• Prepare and distribute an abridged case-study packet to participants prior to the conference
• Prepare and distribute a list of questions to spur discussion and additional questions for exploration during the workshop
• Prepare one “priming” case study based on moderators' shared experiences for presentation at the outset of the workshop. This case study will review
  • Project goals and management
  • Recommendations adopted and recommendations ignored…and why
  • Strategic lessons learned

This case study will provide a framework and foundation for discussion throughout the session.
Participants will receive this final case study at the time of the workshop. This draft may be revised and expanded prior to the workshop, to enhance its value to the workshop's pool of experiential knowledge.

• Prepare for discussion a possible framework for presenting the findings of the Workshop at a the UPA poster session

POST-CONFERENCE DISSEMINATION OF RESULTS

Immediately after the workshop the organizers will prepare a poster for presentation during the UPA poster session . This poster will provide as a springboard for further discussion and serve as the foundation for a User Experience article (to be written and submitted in the summer of 04.)

In addition, we will volunteer to moderate subsequent workshops (again, extrapolating from accumulated learnings) at appropriate sub-national venues such as the local chapter of the Usability Professionals' Association, local chapters of Society for Technical Communication, and the like.

It is our goal to present our findings (through more formalized presentations and articles) but also to continue to evolve those findings further by interacting with members of the concerned communities at future events (e.g., CHI & UPA as well as local events).

Finally, the workshop will provide the nucleus of what we hope will become an extended community of individuals involved in usability evaluations who are interested in communicating about their communities' interests and perspectives. It is our goal that this extended usability community will continue to share their successes and failures with one another long after the conference has concluded.

POST-CONFERENCE ACTIVITIES

Please see Post Conference Dissemination of Results

FACILITATORS

John J. Bosley, Ph.D.

Research Psychologist

Office of Survey Methods Research, US Bureau of Labor Statistics

Dr. John Bosley is a cognitive-experimental psychologist in the Office of Survey Methods Research (OSMR) at the Bureau of Labor Statistics. He is currently committed approximately 50% of his time to support of usability testing of BLS software applications, including on-line applications that are being developed for economic data collection via the Web as well as usability testing of web sites established by various BLS programs for public data dissemination. He serves as an expert resource and referral agent for the staff of other agencies who wish to evaluate the usability of their own statistical websites and/or obtain the services of outside usability consultants with whom he has worked in the past. He is also involved with a team of BLS usability professionals like himself and BLS mathematical statisticians that is working on development of innovative designs and analytic methods for use by usability professionals whose test datasets are small, non-random and otherwise ill-suited to traditional analytic techniques. His research interests include development of and validation through research of a construct he labels “statistical literacy for citizens,” and collaborative research with a number of academic teams that are developing and testing a variety of web-based aids to comprehension of statistical information such as index numbers or other complex “synthetic” statistical measures.

Kathleen A. Straub, Ph.D.

Chief Scientist / Executive Managing Director

Human Factors International