A Strategic Approach to Metrics for User Experience Designers
Journal of Usability Studies, Volume 6, Issue 2, February 2011, pp. 52 - 59
Article Contents
Introduction
Good design produces customer value. It seems self evident to designers that usable applications and products that connect with customers' needs and wants are central to the value of their company. Unfortunately, user experience (UX) practitioners sometimes struggle with getting decision-makers in the organization to see the connection between user experience and customer value. Quantifying one’s financial return on the investment made in UX seems an impossible task.
Within most companies there is a significant constituency that places enormous weight on metrics and meeting objective goals. Designers should know this constituency and learn how to approach them. Think of this exercise as the user-centered design of your own services to your internal business partners. If you do, there are significant opportunities for usability professionals with a background in measurement to favorably influence managers’ decisions that affect their companies’ UX practice.
Are ROI Metrics for Usability Unusable?
In 2004 Dan Rosenberg wrote a provocative article entitled “The Myths of Usability ROI.” While praising the groundbreaking book on usability return on investment (ROI) by Bias and Mayhew (1994), Rosenberg pointed out a large number of shortcomings in the literature published since 1994 on usability ROI.
- The lack of empirical data that support ROI claims for usability.
- ROI studies of usability ignore other contributing factors to product improvement.
- Overly simple ROI calculations for usability don’t address executives’ concerns.
- Studies don’t weigh the ROI for usability activities against other investments.
According to Rosenberg, the “traditional ROI approach to defining and measuring the value of usability” doesn’t show the true value of UX activities (p. 23). In essence, typical UX metrics for ROI are unusable. As a corrective action, he proposed thinking strategically by tying one’s own UX activities to the Total Cost of Ownership (TCO) of a company’s products and services. This is certainly a strategic approach to showing ROI, but only if one’s company competes on TCO. However, the article raised a valid point: How do designers show the link between their own value and what matters to their company?
What Makes a Good Metric?
To answer that question, we must first make a digression. Most designers understand that some metrics are more valuable than others. Often, management will insist that everyone’s work “align with company goals,” usually expressed as metrics. However, not all metrics are worth aligning to. So, what makes a good metric? Price and Jaffe (2008) identify the following five qualities of a good metric.
- Strategic alignment. Alignment assumes that the company’s strategy is known by all, including management. Does the metric support the organization’s strategy?
- The metric drives action. Good metrics act as a target for employees to aim at. If the objectives stated in the metric aren’t met, then it’s understood by all that strong corrective action must be taken. Are data being reported that aren't used to drive action? Then the metric should be retired.
- The metric is important to stakeholders. Someone besides you needs to care about the metric, namely, people with power who can make trouble if the objectives for the metric isn’t being met: customers, customer service managers, heads of departments, and so on.
- The people being measured can change things. Suppose a design group is being measured on customer satisfaction with online applications. Does the design group have the authority and resources to change not only the interface design, but also integration with the mid-tier, database connectivity, etc.? In short, can the design group change everything that affects the performance of the applications? If not, then the metric may be valid, but it needs to be shared with another group.
- There’s a process in place to change things. Most usability professionals have been told, at one time or another, “go ahead and run your test and write your report on the product before it ships. We’ll make changes later,” only to realize that “later” never arrives. What’s missing here? A process that ensures timely changes are made based on the metric collected.
UX practitioners must learn to distinguish between good and bad metrics, and be willing to advocate for better metrics. In fact, relying on published studies for evidence of ROI in UX activities is unconvincing to executives because they were not conducted in the context of one’s own company. Different companies value different metrics. What UX practitioners need are not more published studies conducted at other companies, they need to learn how to collect the right UX data and derive metrics that demonstrate strategic value within the context of their own companies.
To Rosenberg’s point, a good business metric keeps a company focused on the right things, and helps executives make sound decisions. We now have a definition for the strategic use of metrics. Strategic thinking means (a) understanding UX’s value to the company, (b) identifying metrics that drive company—or department-wide decisions, and (c) drawing a clear and obvious connection between one’s measurable value and a company—or department-wide metric.