upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

The Effect of Culture on Usability: Comparing the Perceptions and Performance of Taiwanese and North American MP3 Player Users

Steve Wallace and Hsiao-Cheng Yu

Journal of Usability Studies, Volume 4, Issue 3, May 2009, pp. 136-146

Article Contents

Measuring User Performance

Subjects were then given a series of tasks to complete using the MP3 player. Using scenario-based analysis, a number of tasks were identified as being likely tasks that the target user would perform. These tasks included listening to the radio, playing a song, recording a voice, playing a game, and adjusting settings. The subjects were observed using the product as they carried out these tasks. As the subject was using the product she or he was observed to see how effective and efficient the product is to use.

Effectiveness was measured by recording whether or not a user could complete a task. Because time was a variable in this study no time limits were given. The user either completed the task or announced she or he was unable to complete the given task. The binomial (yes or no) result was then recorded and summated for all tasks attempted by the user.

Efficiency was measured using the time taken and the number of errors made. Errors were defined as an attempt to click on the screen or other hardware attachment that would not result in completing the task assigned. The resulting scores on the time taken and number of errors made however, were not combined into a total score for efficiency. As noted by Hornbaek (2006), this would result in two errors. Firstly, in order to combine the scores a weighting for each score would have to be identified, which is beyond the scope of this article. Secondly, combining scores may lead to the overlooking of important patterns in data representing each score. Finally, by separating the time taken and number of errors made we may better examine the usefulness of these variables as a measurement of efficiency.

User satisfaction was not measured by empirical means as the methods available (facial or verbal expressions) were considered too variable across cultures to be a culture-neutral method of measurement.

These measurements were then analyzed using a t-test (based on a small sample size and assumptions of a normal distribution) to identify whether the average result differed based on cultural background.

Measuring User Perceptions

Subjective measurements of effectiveness, efficiency, and user satisfaction were carried out using a Likert survey. The survey items were initially developed by Lund (2001) in the Usefulness, Satisfaction, Ease of Use (USE) survey on usefulness, ease of use, and user satisfaction, but were adapted in this survey to indicate efficiency, effectiveness, and user satisfaction. This survey was selected among others for a number of the following reasons:

The USE survey was found to be a highly reliable indicator of user perceptions as indicated by Cronbach's alpha. Similarly, Lund (2001) reports high levels of Cronbach's alpha when designing the survey. The following table shows the high level of internal consistency of this survey.

Table 2. Survey Reliability Estimates
Perceived Usability Factor Cronbach's Alpha
Efficiency 0.86
Effectiveness 0.92
User satisfaction 0.88
Total usability 0.96

The survey responses for each variable (efficiency, effectiveness, and user satisfaction) were summated and then analyzed for differences based on cultural background. Both the Mann-Whitney U test and the t-test were applied. The t-test is commonly used in studies such as this where the sample size is small and a normal distribution is assumed. However, one set of results, the number of tasks completed, was not found to be normally distributed, so the Mann-Whitney U test was also applied to the data. These tests gave similar results, so for convenience only the t-test results are given in this article.

In this section of the study one set of hypotheses is examined. The null hypothesis is that the mean or median values of all usability factors are the same for Taiwanese and North American users. The alternative hypothesis is that they differ.

Measuring Correlations

The correlation between variables was also estimated using Pearson's correlation co-efficient and the Spearman Rank Correlation. The latter measure was used for the reason mentioned earlier, that the number of completed tasks is not normally distributed. However, because results for both measures are similar, only measures of Pearson's correlation coefficient are used in this article.

Correlations are shown in two aspects. Firstly, the correlation between subjective and objective measures of usability and users' cultural background is examined. While focusing on the connection between culture and usability, other correlations were also estimated for the purpose of comparison. Secondly, a comparison is made of correlations between usability factors within each culture group-Taiwanese and North American users.

In this section of the study two sets of hypotheses are made. Firstly, correlations between culture and other usability factors are examined across both culture groups. The null hypothesis is that there is no significant correlation between culture and any usability factor. The alternative hypothesis is that there is a statistically significant correlation between culture and one or more usability factor. Secondly, correlations between usability factors in one culture group are examined to see whether they are significantly different from correlations in the other culture group. The null hypothesis is that there is no significant difference between any usability factor correlation in one culture group and its equivalent in the other culture group. The alternative hypothesis is that one or more usability factor correlations in a culture group differ significantly from the equivalent correlation(s) in the other culture group.

Previous | Next