Half Day Tutorial Details

By Day - Selected
By Presenter
By Track
View My Agenda

Rapid Iterative Testing and Evaluation (RITE)

Experienced practitioners will learn how to conduct RITE sessions to obtain early feedback. Perfect for Agile projects.

The Rapid Iterative Testing and Evaluation (RITE) methodology is an effective way to gain user feedback on design concepts early in the development cycle. The method entails alternating testing days with iteration and requires quick analysis and design skills. RITE is an excellent way to integrate usability with Agile projects and to engage project teams in the process.

Half Day Tutorial by Richard Douglass (Nationwide), Carol Smith (Midwest Research, LLC )
Advanced Practitioners (AP)
6:30pm to 9:30pm on Monday, May 24, 2010

About the Half Day Tutorial

Tutorial: Rapid Iterative Testing and Evaluation (RITE)

Agenda: - Objectives - Overview: RITE

Activity: Planning - Case Study - Break - Overview: RITE Reporting Activity: Creating Final RITE Report - Q&A

RITE Objectives: Understand… - the difference between traditional usability testing and the RITE methodology. - how to set up a RITE usability test. - the strengths and limitations of the RITE approach. - how best to capture data and report out findings from RITE sessions.

RITE Testing Methodology - This methodology is primarily qualitative, exploratory, and conceptual. - It is most appropriate for the early requirements gathering and design phase of production where initial wireframes can be used to validate information architecture and gather feedback on proposed functionality. - The Rapid Iterative Testing and Evaluation (RITE) methodology is credited to Michael Medlock (2002) where he used this technique at Microsoft in association with game design.
- The Nationwide UE team has been experimenting with RITE for two years with a variety of financial services websites and applications. - Sessions tend to be much more conversational.

RITE Overview - A clickable prototype (typically PDF) is created by User Experience (UE) team. - Carefully recruited participants attempt representative tasks, and the UE team obtains qualitative user feedback (actions + comments). - The prototype is updated in one to two days based on feedback and the expertise of the UE team. The new version of the prototype is tested with the next day’s participants.

RITE Advantages - Easily modifiable prototypes - Faster feedback; more feedback - Prototype evolution

Activity: - Work in groups of three; select a “recorder” and a “presenter.”

Use the worksheet provided to create a sample planning document for a RITE session. Work as a group to determine the appropriate scope for the work, key objectives, schedule, and reporting components. - Report back to the group

Case Study - Background

Nationwide intends to provide a personalized customer experience for iPhone users in late 2009. The overall goal of the Nationwide Car Shopping app for iPhone is to provide car shopping assistance for users while away from a computer in a way that supports the Nationwide On Your Side brand promise. The application will accomplish this goal by providing:

Information – VIN decoding, vehicle safety ratings, pricing, total cost of ownership, used vehicle history, customer re-views. Tools - Compare and rate vehicles, estimate monthly payment, apply for an auto loan and get an auto quote.

  • RITE Sessions >> Typically each day of testing is followed by a full day of wireframe revisions. However, due to the accelerated schedule of this project, all three test days occurred in succession without a day in between. >> Conduct testing with clickable prototypes using a PC and a mouse to determine Information Architecture and to quickly vett the initial concept.

Day 1: Progression Example – Star Rating Method - Purpose: User is able to rank their impression of a vehicle (#1-5) as they are shopping to assist the user with ranking cars for their own use. - Only had one information or “I” icon to explain how to create “your own category.” - Challenges/Results:

Some of the categories were hard to rate (e.g. no neutral rating available). Star system did not resonate well (difference between a 3.5 vs. a 5.0?). - Proposed Changes: Move to a different rating system to address the “no rating” scenario. Look to make the “I” icon more readily available w/ the other categories.

Day 2: Progression Example – Star Rating Method, cont. - Features/What was Changed:

Thumb up or thumb down rating. “I” icon available on each category in order to explain each title via dialogue box. - Challenges/Results: Overall, performed better with the neutral rating option (thumb up, neutral, or thumbs down). The “thumb” rating was readily understood (quicker negative/positive rating). - Proposed Changes: Remove all information icons and allow users to rename all rating categories to allow higher flexibility.
What is important to a user varies from one to another (e.g. cuteness, number of seats, etc.). >> Need to provide a way to allow customization and input from others.

Day 3: Progression Example – Star Rating Method, cont. - Features/What was Changed:

Ability to rename all categories via “options” button. Share w/ friend feature added. Removed “I” icons since users now have the ability to rename if category not understood or relevant to user. - Challenges/Results: Overall, this design resonated well with users. The ability to rename all categories was received enthusiastically.

RITE: What Works Well - Best used early on in the project lifecycle. - Reduces risk of uncovering major issues in a latter stage validation study. - RITE does not reduce the value of conducting validation testing later in the project lifecycle. - Key stakeholder sign-off on wireframes must occur before RITE begins. - Full participation required – Ideally, project team members attend every session. At minimum, they should attend the end of day recap which typically lasts 30 minutes. - The final report often takes on a “Tadpole to Frog” type of look. Progressions of key sections are shown in wireframe format with explanation given for what changes were made and why.


RITE: Reporting -- "Tadpole to Frog" - Reporting can be very challenging due to:

Prototype changes multiple times over the course of the sessions. Findings build upon one another. Decisions are being made by the project team at the end of each RITE testing day.

Activity: - Continue to work in the same group of three as before. - Using the worksheet provided, work as a group to determine how best to capture one key evolution in the prototype as a result of your RITE work. - Report back to the group


Appendix: Practical Considerations 1) We found that the RITE methodology is best used early in the project lifecycle. It is most appropriate when dealing with early concepts that need to be vetted with users and can assist in quickly shaping designs. 2) A benefit of RITE is that it reduces the risk of running a later stage validation tests that identify the need for major changes. Risk is mitigated by early feedback obtained with low fidelity concepts. 3) RITE does not lessen the value of conducting validation testing later in the project lifecycle. Adding visual treatment to the wireframes and creating a much richer interactive prototype introduces elements that still need to be vetted with users. RITE alone is not sufficient for capturing all usability issues which may arise in the end experience. 4) Usability professionals must insist on project team involvement during RITE sessions. By its very nature, the methodology requires input from all stakeholders on the changes made to the wireframes between sessions. Without business input, the UX team may violate rules or other restrictions. Without technical input, the team can add functionality that cannot be delivered on the target platform. To help offset the risk of project team members not attending the sessions, schedule the testing as early as possible and send out invitations to key players. At minimum, the project team members should attend the end of day recap, which typically lasts 30 minutes. 5) The end of day recap sessions should occur as soon as possible after the last session of the day. It helps to have this meeting in a room with a whiteboard. If there are competing design approaches, low fidelity alternatives can be mocked up (A/B testing) for evaluation in subsequent sessions. 6) RITE methodology requires that the user centered design (UCD) professionals have a measure of experience working in the field. Since changes to the wireframes are being made with feedback from one to three users, it is incumbent upon the UCD members of the team to bring their experience to bear upon what is being learned. A junior or inexperienced UCD professional may jump to conclusions that are not valid.
7) Organizations or clients that require statistical significance may have a larger hurdle to cross in order to use the RITE methodology. If time on task, error rates, etc. is demanded, the qualitative nature of RITE may not fit the bill. 8) The UX Lead on RITE projects needs to be aware that this technique will require much more time than a typical usability test. As the person responsible for creating and updating the wireframes, they need to be available to attend all sessions, recap meetings, and quickly make recommended changes to the prototype. The result is that the UX Lead needs to be dedicated full-time to the project during the weeks of RITE sessions and needs to put in overtime hours as needed. 9) If it is anticipated that additional modules or functionality will be introduced during the course of testing, then the testing schedule should be extended.