upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Strategic Approach to Metrics for User Experience Designers

Carl W. Turner

Journal of Usability Studies, Volume 6, Issue 2, February 2011, pp. 52 - 59

Article Contents


A Strategic Approach to Business Metrics: The Balanced Scorecard

Financials, of course, are a company's preferred and best-understood metric, and one to which all other metrics should tie to. Much of strategy, product selection, and service offerings are presented in dollars, as in the promise of future revenue. The known shortcoming of financial metrics is that they are backward looking measures, i.e., they don't necessarily predict future performance.

One well-known approach to developing strategy and effective metrics is the Balanced Scorecard (BSC; Kaplan & Norton, 1996). Its use is widespread among large companies; a survey of 1,430 executives globally revealed that 53% of companies use some form of the BSC (Rigby & Bilodeau, 2009). BSC addresses the shortcomings of financial measures by introducing three additional categories of measures: customer perspective, internal business perspective, and innovation and learning perspective. Strategy maps tie the four categories of measures together in the company’s theory of how each category of measures contributes to financial performance. All four perspectives provide UX designers with opportunities to contribute to strategic decisions.

Companies that adopt BSC recognize that return on investment doesn’t apply only to financial measures. They understand that investments can yield important returns in customer satisfaction, people, and process. This is invaluable information for usability practitioners who are trying to position their services within an organization, who must likewise realize that their own value may not tie directly to a company’s financials but to another category of measures.

If your company employs BSC, study the measures. If your company doesn’t employ BSC, then look at your company’s metrics for tracking performance. You will need to tie your performance to these measures, or introduce one of your own if possible. The following discussion refers to BSC, but can be used with other schemes, as long as the scheme contains metrics that are managed to—that is, the metrics satisfy the criteria for effective metrics.

Financial Perspective

Returns on investment are typically expressed in dollars. Dollar amounts are calculated using Net Present Value (NPV): the difference between an initial investment in a project and the cash flow the project generates, accounting for the time value of money. NPV is one of the main criteria used in selecting among new projects and services. NPV avoids the problems of the commonly-used payback period metric, which doesn't consider cash flows after the project has paid for itself.

The data that estimate NPV for a given project are usually generated by the marketing department or another business unit to justify a project initially. Obtain the estimates. Then estimate the expense needed to conduct analysis, design, and usability testing for the proposed project. These estimates are fairly straightforward if your company keeps historical records of project costs per task. Then estimate the percentage increase in sales per year (or savings per year if improving the use of customer self service) if the product is designed and usability tested properly. These data are harder to find.

To illustrate the use of NPV, assume that a project with a significant user interface is initially estimated to cost $90,000 and will produce $40,000 per year in sales for three years. Convert these data into NPV by calculating NPV using these data with the NPV formula, as shown in Table 1.

NPV = -C0 + C1 / (1 + r) + C2 / (1 + r)2 + ... + Cn / (1 + r)n

where

C0 = initial investment
C1 = cash flow in Year 1
C2 = cash flow in Year 2
r   = company's required rate of return on investment, or discount rate
Cn = number of years in the calculation

The term r, the discount rate, represents the percentage of profit the proposed investment must meet to be considered for funding. Proposals whose returns exceed the discount rate are then compared against others with the same level of risk for consideration. Companies that use NPV to compare projects determine their own discount rates.

Table 1. Net Present Value Without UX design

Table 1

To show the benefit of usability on NPV, add the estimated cost of doing usability on the project to the initial investment, assumed here to be $4,500. Add the increase in cash flow to each year's cash flow, assumed here to be 6%. Recalculate NPV, as shown in Table 2.

Table 2. Net Present Value With UX design

Table 2

The decision then is between a product with or without design and usability testing. For best results, focus on projects with a significant user interface that handles large numbers of transactions, such that a small percentage increase in success rates contributes a significant financial return to the company. If there is a positive difference then the decision to add usability to the project is nearly pre-ordained. If the difference in NPV is great enough, it could mean the difference between a project being selected or not. For more on the use of NPV in usability studies, see Karat (2005).

Note that using NPV correctly answers two of Rosenberg’s criticisms about usability ROI metrics. First, the discount rate is determined by the company, typically the finance department, and is an important criterion used by management to fund a project. Second, NPV allows managers to compare the proposed investment with other investments, in this instance, comparing returns with and without an investment in usability.

Customer Perspective

The BSC customer perspective dimension answers the question "How do customers see us?" This is an obvious place for user experience metrics. Customer satisfaction, customer retention/defection, and time to delivery are examples of metrics for customer perspective. These metrics are usually owned by customer service or marketing functions, but the usability function should own a share of responsibility for a significant metric in the customer perspective. The metric should drive decisions, as discussed in regards to the What Makes a Good Metric section. For example, a customer metric that can drive improvements to an existing application is self service usage. It could be a percentage change in usage, as measured directly by automated reports or indirectly by customer survey. The wide range of customer experience measures were surveyed by Tullis and Albert (2008). From these metrics, select a candidate measure and work with customer service or marketing to include an appropriate UX metric in the customer metrics. If it takes its measures seriously, the company will need to devote significant, appropriate resources to the hitting goal stated in the metric.

Internal Business Perspective

These measures drive process improvement projects by answering the question "What must we excel at?" Driving the design of service improvement is one of best opportunities for designers and usability practitioners to gain visibility. Price and Jaffe (2008) gave an excellent example of  how a good metric forced a great deal of process improvement at Amazon.com. Amazon was aware that it had a problem with the large number of calls to its call center despite its extensive web-based self service. It knew that the key to profitability was in persuading people to self serve. The company settled on cost per order (CPO) as a central metric. Obviously, calls handled by agents in the call center added to the cost of an order. Trying to reduce CPO pushed the company to discover the root cause for every call and to address each issue they found. It made a lot of decisions easier: improving the usability of its web site, simplifying the order process, and adding information on the site in the form of user-generated recommendations that were not available from its agents (Price & Jaffe, 2008). In short, the CPO metric drove decisions that required large investments be made to improve the customers’ user experience on the site.

Innovation and Learning Perspective

These metrics answer the question "What capabilities are needed to support the customer perspective and the internal business process?" That is, what does the company need to be able to do to meet its goals for customer satisfaction and process improvement? Aggressive customer satisfaction and process improvement goals nearly always require increased UX skills and capacity, as demonstrated in the Amazon example. The UX practitioner can act strategically by discussing with management the UX department’s needs for skill development, staffing increases, and increased visibility in the organization. If possible, put a “UX improvement” metric on the scorecard. The aim is to position UX as a valuable competency for meeting scorecard objectives. If the objective is on the scorecard, it will be tracked and decisions made based on outcomes.


Sidebar: How Effective Is Balanced Scorecard?

A large-scale study of companies that employ BSC showed some limited support for the effectiveness of BSC as a strategic tool (Malina & Selto, 2001). A primary finding was that it was difficult to isolate the contribution of a company’s approach to strategy from other factors such as the company’s ability to execute and the market it was competing in. Indeed, that is the very thing that makes it so hard to isolate the financial contribution of UX to an individual product or project: the entire team has to execute properly for the project to succeed.


Previous | Next