upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Intra- and Inter-Cultural Usability in Computer-Supported Collaboration

Ravi Vatrapu and Dan Suthers

Journal of Usability Studies, Volume 5, Issue 4, August 2010, pp. 172 - 197

Article Contents


Methodology

The formal definition of socio-technical affordance β =p/q/r in W = (T, S, O) has two important elements: technology T and individual actors S and O. The definition of appropriation of affordances has two important elements: affordances and intentions. Based on these two definitions, an experimental study was designed that introduced a variation in the cultural background of individuals (by selecting participants from a nation-state based ethnically stratified random sampling frame) but kept invariant the technological interface T and interactional setting. Briefly, the experimental study investigated how pairs of participants from similar and different cultures (American-American, American-Chinese, and Chinese-Chinese) appropriated affordances in a quasi-asynchronous computer-supported collaborative learning environment with external representations in order to collaboratively solve a public health science problem. Usability analysis reported in this paper was conducted on the empirical data generated by this experimental study.

In the next five subsections experimental design, materials, research hypotheses, sampling, and procedure are discussed briefly.

Experimental Design

The experimental study design consisted of three independent groups of dyads from similar and different cultures (American, Chinese) doing collaborative problem solving in a knowledge-mapping learning environment (described below). The three experimental conditions were the Chinese-Chinese intra-cultural condition, the American-American intra-cultural condition, and the Chinese-American inter-cultural condition.

In all three experimental conditions, the collaborative dyads were given the same experimental task. All the collaborative dyads interacted in the same computer-supported collaborative learning environment after reading the same instructions, software tutorial, and demonstration. The same instruments were administered to all participants. Internal validity and external validity were actively considered when designing and conducting the experiment. Construct validity was addressed by using existing instruments with high validity and reliability (Bhawuk & Brislin, 1992; House et al., 2004; Schwartz et al., 2001; Suinn, Ahuna, & Khoo, 1992). Brief descriptions of the experimental study’s software and topics are provided in the following sections.

Materials

The following sections discuss the software, protocol for workspace updates, and alternatives for action.

Software

The computer-supported collaborative learning environment used in this experimental study has an Information Viewer on the left of the screen (see Figure 1) in which materials relevant to the problem are displayed. This Information Viewer functions as a simple web browser, but the presentation of materials is constrained as discussed in the next section. The environment has a shared workspace or Information Organizer section on the right side of the screen in which participants can share and organize information they gather from the problem materials as well as their own interpretations and other ideas. The Discussion tool below the Information Viewer on the left enables participants to discuss their ideas in a threaded discussion format.

The Information Organizer workspace includes tools derived from Belvedere (Suthers et al., 2008) for constructing knowledge objects under a simple typology relevant to the task of identifying the cause of a phenomenon (e.g., a disease), including data (green rectangles, for empirical information) and hypotheses (pink rectangles, for postulated causes or other ideas). There are also linking tools for constructing consistency (For) and inconsistency (Against) relations between other objects, visualized as green links labeled + and red links labeled - respectively. Unspecified objects and Unknown links are also provided for flexibility. Finally, an embedded note object supports a simple linear (unthreaded) discussion that appears similar to a chat tool, except that a note is interactionally asynchronous and one can embed multiple notes in the knowledge-map and link them like any other object. In the Threaded Discussion section of the environment (see bottom left of Figure 1) participants can embed references to knowledge-map objects in the threaded discussion messages by selecting the relevant one or more graph object while composing the message. The references show up as small icons in the message. When the reader selects the icon, the corresponding object in the knowledge-map is highlighted, indicating the intended referent. Figure 1 displays a captioned screenshot of the environment used in the experimental study.

Figure 1

Figure 1. Screenshot from I3P1's session

Mutual awareness of participants’ artifacts is supported in the software environment as follows: all knowledge-map nodes and threaded discussion messages carry the name of the participant who first created it. The mutual awareness features of artifacts and of activity are shown in Figure 1, a screenshot taken from I3P1 (I stands for American-Chinese inter- cultural session, 3 stands for the number of experimental session in this condition, P1 stands for Participant 1). In Figure 1, the I3P1’s screen name of “Teri” (screen name selected by participant) appears on the title bar of the application window and on knowledge-map nodes and message created by I3P1. Similarly, I3P2’s screen name of “Sue” appears on artifacts created by her. Artifacts marked with a solid red triangle in the top-right corner are from I3P2 and are yet to be opened by I3P1. The yellow circle on the threaded discussion message of I3P1 in the lower-left region of Figure 1 indicates artifacts created by “Teri” (I3P1) but not yet read by the study partner, “Sue” (I3P2). Thus each participant is potentially aware of the new artifacts from the study partner as well as the artifacts not yet read by their study partner.

Protocol for workspace updates

To simulate asynchronous online interactions that are typical of many computer-supported collaboration settings in learning and work, the actions of each participant in the shared workspace were not displayed immediately in the other participant’s workspace. As a person worked, the actions of that person were sent to the other participant’s client application, but were queued rather than displayed. Participants were given a new report after playing the game of Tetris™. Tetris™ was chosen as it presents a different sensory-motor perceptual task than the primary experimental task of collaborative knowledge-map co-construction.  The game simulates taking a break from the task in real-world asynchronous settings (Suthers et al., 2008) . After the game of Tetris™, all of the queued actions on that client were displayed. Conflicts that might arise when both participants edited the same object were resolved through operational transformations (Sun, Jia, Zhang, Yang, & Chen, 1998). The delayed updating protocol simulates one aspect of the experience of asynchronous collaboration: a participant sees what one’s partner has done upon returning to a workspace after a period of time. It excludes the possibility of synchronous conversation in which one participant posts a message in the workspace and receives an immediate reply. The Refresh feature of the software enables one to get all updates to that point in time.

Alternates for action

The software environment provides multiple alternatives for appropriation of affordances and multiple ways to relate to the social other (the study partner). For example, participants can discuss with each other using the threaded discussion tool or the embedded notes tool. Participants can also use the knowledge-map objects to discuss the task at hand or any other topic of interest. Participants can refer to artifacts by deictic referencing (this, that, etc.) or use the cross-referencing feature of the threaded discussion. Participants can externalize the perceived relations between their concepts by creating external evidential relations between objects in the knowledge-map, by spatial arrangement, or by mentioning them in discussion. Participants have multiple ways of sharing the information presented to them (threaded discussion, embedded notes, and knowledge-map).

The research strategy was to provide participants with a feature rich collaborative environment with multiple alternates for action. By incorporating systematic variation in the assignment of participants to the collaborative dyad based on their cultural background and gender, the experimental design measured and observed systemic differences in how participants used the tools and resources of the technology (research question 1, appropriation of affordances) and related to each other during and after their interaction (research question 2, technological intersubjectivity). This in turn helped us analyze the empirical data from a cultural usability perspective.

Task

The study presented participants with a “science challenge” problem that requires participants to identify the cause of a disease known as Amyotrophic Lateral Sclerosis-Parkinsonism/Dementia (ALS-PD). This disease is prevalent on the island of Guam and has been under investigation for over 60 years, in part because it shares symptoms with Alzheimer’s and Parkinson’s diseases. Only recently have investigators converged on both a plausible disease agent (a neurotoxic amino acid in the seed of the Cycad tree) and the vector for introduction of that agent into people (native Guamians’ consumption of fruit bats that eat the seed). Over the years numerous diverse hypotheses have been proposed and an even greater diversity of evidence of varying types and quality explored. These facts along with the relative obscurity, multiple plausible hypotheses, contradicting information, ambiguous data, and high interpretation make this a good experimental study task for measuring cultural effects on appropriation of affordances and on technological intersubjectivity. Cognition, social behavior, and communication are all involved in this particular task and software. For example, instead of having a socially awkward and culturally inappropriate (for social harmony maintenance) verbal disagreement with the other-actor about a data node contradicting a hypotheses node, the self-actor could choose to create an evidential “Against” link between the two nodes. For other rationale, see the hypotheses listed in (Vatrapu, 2008).

All experimental study materials were in English. All participants began with a mission statement that provided the problem description and task information. Four mission statements corresponding to the four participant assignment configurations (Chinese vs. American x P1 vs. P2) were administered (http://lilt.ics.hawaii.edu/culturalreps/materials/). Due to the distribution of conflicting evidence, sharing of information across participants and study sessions was needed to expose the weakness of the genetics explanation as well as to construct the more complex explanation involving bats and cycad seeds. Given the nature of the information distribution between the two collaborating participants, working out that the consumption of bats as an optimal hypothesis as a cause for contracting ALS-PD involved making these cross-report collaborative connections and also considering and rejecting other probable factors. The study task and task materials were designed to highlight the social division of cognitive labor between the collaborating dyad. The experimental study encouraged participants to interact with each other by including the following reinforcing task instruction on each report (set of four articles): “Please share and discuss this information with you colleague...”

Participants

Participants were recruited from the graduate student community at the University of Hawai‘i at Mānoa. Each participant was offered a payment of $75 (U.S.) for participating in the study. Participant selection and treatment assignment are discussed in the following sections.

Sampling

There is a tendency in cross-cultural, computer-mediated communication research to use cultural models bounded by modern nation-states. Nationality based stratified sampling frames remain a methodologically convenient way to select participants provided that cultural homogeneity of the participants is not to be assumed but empirically measured. We used the Portrait Value Questionnaire (PVQ) individual values survey (Schwartz et al., 2001) and the Global Leadership and Organizational Behavior Effectiveness (GLOBE)  instrument (House et al., 2004) to empirically assess differences in the two participant groups at the individual and group levels respectively.

Selection

Participants were selected based on the sampling frame consisting of two randomly stratified subject pools of graduate students either from mainland USA or from the People’s Republic of China and the Republic of China (Taiwan). That is, all the American participants in the study self-reported growing up in mainland USA and similarly, all the Chinese participants self-reported growing up in China or Taiwan. We used the stratified random sampling frame to control for the context effects of Hawaii. For detailed rationale, please see (Vatrapu, 2007, pp. 129-130).

Assignment

Participants were randomly assigned to either the intra- or the inter-cultural profiles and the same or different gender profiles. Excluding six pilot studies, a total of 33 experimental sessions involving 66 pairs of participants were conducted. Data from three experimental sessions was discarded due to issues of a missing screen recording, a software crash, and a disqualification. As a result, there were 10 pairs of participants for each of the three treatment groups: Chinese-Chinese intra-cultural, American-American intra-cultural, and American-Chinese inter-cultural. All the three conditions were gender-balanced because gender can substantially influence social interaction (Tannen, 1996). Each treatment group included three female-female, three male-male, and four female-male dyads.

Instruments

The following sections discuss the demographic questionnaire, the self-perception PVQ, the GLOBE cultural dimensions instrument, the individual essays, the peer-perception PVQ, the acculturation SL-ASIA questionnaire, intercultural sensitivity, and the user satisfaction QUIS questionnaire.

Demographic questionnaire

A demographic questionnaire (Vatrapu, 2007, pp. 275-276) was administered to collect participants’ familiarity with each other, with online learning environments, with usability evaluation studies as well as data about age, gender, ethnic background, duration of stay in the USA, and duration of stay in the state of Hawai‘i. All participants were requested to make a self-report of their Cumulative Grade Point Average (CGPA) and also assign a release form for obtaining official records of their CGPA, Graduate Record Examination (GRE) scores, and the Test of English as a Foreign Language (TOEFL; Chinese participants only).

Self-perception: Portrait value questionnaire (PVQ)

The 40 item version of the PVQ instrument (Vatrapu, 2007, pp. 277-279) recommended for intercultural contexts (S.H. Schwartz, personal communication, 2006) was used in the study. The PVQ scale measured cultural values at the individual level. Cronbach’s “alpha measures of internal consistency range from .37 (tradition) to .79 (hedonism) for the PVQ (median, .55)” (Schwartz et al., 2001, p.532). Gender specific versions of the self perception PVQ scale were administered.

GLOBE cultural dimensions instrument

The GLOBE instrument (House et al., 2004) was used to measure cultural values at the group level (Vatrapu, 2007, pp. 280-293). Section 1 (“The way things are in your society”) and Section 3 (“The way things generally should be in your society”) of the original GLOBE instrument were used in this study. Section 1 of the GLOBE instrument measures a responder’s perceptions of their society. Section 3 of the GLOBE instrument measures a responder’s preferences for their society. According to the Guidelines for the Use of GLOBE Culture and Leadership Scales2 (2004) “the construct validity of the culture scales was confirmed by examining the correlations between the GLOBE scales with independent sources (e.g., Hofstede’s culture dimensions, Schwartz’s value scales, World Values Survey, and unobtrusive measures)” (p.5). Phrasing of “this country” has been changed to “my home society” to remove possible ambiguity for Chinese graduate students who might rate Hawai‘i, USA instead of the society they grew up in.

Individual essays

At the end of the collaborative science problem-solving session, each participant wrote an essay. This constituted the immediate post-test. Identical essay writing instructions were provided to all participants. The instructions asked the participants to (a) state the hypotheses they considered, (b) whether and how their hypotheses differed from those of their study partners, and (c) their final conclusion.

Peer-perception: PVQ

Technological intersubjectivity after interaction was measured by the second immediate post-investigative test. This was the administration of the PVQ (Schwartz et al., 2001) instrument with a reversal of the direction of assessment (Vatrapu, 2007, pp. 304-306). This time instead of assessing themselves, participants assessed their collaborative partners. Based on their collaborative interactions, each participant rated his/her impressions of the study partner on the PVQ.

Acculturation: Suinn-Lew Asian Self Identity Acculturation (SL-ASIA) questionnaire

Acculturation is a process that occurs when members of two or more cultures interact together. This becomes an external variable in cross-cultural research conducted with participants from an immigrant culture in a host culture (in our case, Chinese participants in Hawai‘i, USA). This external variable can be controlled by measuring the acculturation level of the participants belonging to the minority immigrant culture (Triandis, Kashima, Shimada, & Villareal, 1986). Participants with a high level of acculturation can be best used as members of the majority host culture or not included in the study (Triandis et al., 1986). This research project used the SL- ASIA scale (Suinn et al., 1992) to measure the acculturation levels of the Chinese participants (Vatrapu, 2007, pp. 307-311). This scale was chosen as it is specifically designed for Asians. Suinn et al. (Suinn et al., 1992) reported an internal-consistency estimate of .91 for the SL-ASIA instrument.

Intercultural sensitivity: Intercultural sensitivity instrument

Intercultural sensitivity is a vital skill for intercultural collaborations (Bhawuk & Brislin, 1992). The SL-ASIA scale provided a measure of Chinese participants assimilation to the USA. The intercultural sensitivity instrument (ICSI(Bhawuk & Brislin, 1992) was used to measure American participants self-assessment of intercultural sensitivity (Vatrapu, 2007, pp. 312-315). Bhawuk and Brislin (1992) report that “the ICSI was validated in conjunction with intercultural experts at the East-West Center with an international sample (n=93)” (p. 423). The word “Japan” in the original ICSI scale was changed to “China” to fit the context of Chinese-American collaboration setting of the experiment. Part three of the original ICSI instrument was not used, as pilot studies indicated that it was irrelevant to the purposes of this experimental study.

User Satisfaction: Questionnaire for User Interaction Satisfaction (QUIS)

The QUIS 7.0 questionnaire (Harper, Slaughter, & Norman, 2006) was administered to collect the participants subjective perceptions and preferences of the learning. The QUIS has high reliability, Cronbach’s alpha = 0.95 and high construct validity (alpha = 0.86), (Harper et al., 2006).

Procedure

Two students participated in each session. Experimental sessions lasted about 3.5 hours on average. Informed consent was obtained from all participants for both the pilot studies and the experimental studies. After signing the informed consent forms, participants completed a demographic survey. They were then given a CGPA/GRE/TOEFL score release form, a self-perception PVQ (Schwartz et al., 2001), and the GLOBE instrument (House et al., 2004). After completing these three forms, participants were brought into a common room. Participants were then introduced to the software and the structure of the experimental study through an identical set of instructions and demonstrations across all three conditions.

After the software demonstration, the two participants were led back to their respective workstations in two different rooms. They were then instructed to begin work on the study task. Participants had up to 90 minutes to work on the information available for this problem. The update protocol described in (Suthers et al., 2008) was used to synchronize the workspaces of the two participants. At the conclusion of the investigative session, each participant was given up to 30 minutes to write an individual essay. The CSIC environment remained available to each participant during the essay writing, but the participants were requested not to engage in any further communication. After each participant had finished writing the individual essay, the other-perception PVQ instrument (Schwartz et al., 2001) and the QUIS instrument (Harper et al., 2006) were administered. This concluded the experimental session. Participants then completed the payment forms and were debriefed.

2http://www.thunderbird.edu/wwwfiles/sites/globe/pdf/GLOBE_Culture_and_Lead ership_Scales_Guidelines.pdf

Previous | Next