upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

The Combined Walkthrough: Measuring Behavioral, Affective, and Cognitive Information in Usability Testing

Timo Partala and Riitta Kangaskorte

Journal of Usability Studies, Volume 5, Issue 1, Nov 2009, pp. 21 - 33

Article Contents


Recommendations

Based on this experiment, we suggest that combining expert evaluation and usability testing, and also the measurements of behavioral, affective, and cognitive aspects of interaction is a noteworthy and useful approach for usability testing. In the future, the evaluation of the users’ affective responses could also include gathering and analyzing qualitative interview data about the nature of their user experiences more systematically than in the current experiment in which the participants already gave some valuable comments. Further information about the users’ emotions could be also gathered, for example, using a discrete emotions framework (e.g., based on the model by Ekman (1992), consisting of anger, disgust, fear, joy, sadness, and surprise or the model by Frijda (1986), consisting of desire, happiness, interest, surprise, wonder, and sorrow). In the future, the third cognitive question (asking whether the participant notices that she or he has made progress towards the task goal) could be dropped, at least when the target of evaluation is similar to the interactive media software of this experiment. Finally, methods for systematically combining data from the different sources should be developed.

Conclusions

In all, the results of this experiment suggest that different kinds of information (behavioral, affective, and cognitive) can be successfully captured in practice in one usability testing session using the proposed combined walkthrough method. Especially the affective subjective ratings could be used successfully in capturing the variations in users’ affective states. The different measures were correlated, but a large coefficient of determination was found only between time and the number of usability problems. The effect sizes between the other variables ranged from small to medium. These results support the view that it is worth measuring user interaction from different aspects in order to gain a more multifaceted understanding of the interaction. This approach may act as a starting point for usability testing methods that aim at producing different types of information, but are nevertheless designed for cost-effectiveness.

Acknowledgements

The authors would like to thank all the voluntary test participants for their participation and the reviewers and the editor for their comments on the manuscript. This research was partly supported by European Structural Funds and State Provincial Office of Oulu. In addition, Timo Partala was funded by Tampere University of Technology when finalizing the research and by University of Tampere when developing the first version of the presented method.

 

Previous | Next