upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

When Links Change: How Additions and Deletions of Single Navigation Links Affect User Performance

Lauren F.V. Scharff and Philip Kortum

Journal of Usability Studies, Volume 5, Issue 1, Nov 2009, pp. 8 - 20

Article Contents


Methods

The following sections discuss the experiment design, the participants, the task, and the materials used.

Experiment Design

This study had users look for target information on a site over the course of two visits. The second visit occurred either immediately after the first visit, one week later, or three weeks later. A shortcut link to the information was either consistently present, consistently absent, or varied between user visits. Users’ performance was measured on the number of pages viewed, the time it took to locate the target information (or give up trying), and the accuracy with which they found the target information.

The study used a 2 (search task: first or second) x 2 (shortcut link presence consistency: yes or no) x 2 (presence of shortcut link on second search task: yes or no) x 3 (time delay: no delay, one-week delay, or three-week delay) between-participants design. When the shortcut link was present, it was always in the same location in the left-side navigation. Participants completed two search tasks, with the time delay between the two searches varying across participants in a blocked manner across different semesters. All of the participants were assigned to one of the four consistency x link presence conditions in a random fashion.

Participants

Thirty-eight undergraduate student participants from a mid-sized state university in east Texas completed the no delay study. Sixty-nine undergraduate student participants completed the one-week delay study; thirty-nine of the participants were students from a private university in southeastern Texas, and thirty of the participants were from the mid-sized state university. Fifty-five undergraduate students from the state university completed the three-week delay study. The participants tested in this experiment had moderately high self-reported Web literacy skills. On a scale of 1 to 5, where 1 indicates higher comfort, participants rated their general comfort in finding information on other sites at 1.84 (SD = .66). They further rated their average success in navigating Web sites at 1.98 (SD = .66) on a 5point scale where 1 indicates higher success. Regardless from which institution or in which of the three studies the students participated, all received course credit or extra credit for participating. Some of the students participated individually while others participated in larger groups as part of a class or seminar. In all cases, consistent instructions were given and students had access to a dedicated task computer and did not interact with any other participants. Total testing time for each participant was less than 30 minutes for the no delay experiment, and less than 15 minutes for each part of the one-week delay and the three-week delay experiments.

Task

When beginning the search task, participants started at the front page of a test Web site and were then allowed to freely explore the entire site until they found the answer to the question or chose to give up the search. When the shortcut link was available, the target information was only one click away. Even though the search task question and the remainder of the content on the Web pages were the same for both the first and second visits, the answer to the task question was changed to prevent users from using memory from their first visit to complete the task on the second visit. Once the user found the target information, they could click an Answer Question button, which was always present in a frame at the bottom of all of the pages (see Figure 1). If the user believed that the target information was not located on the site, the user could also chose an Information Not Available button at any time.

For every task, we recorded search times (in milliseconds). Start time was recorded once a user clicked the Begin Task button and the front page of the site had begun to load. Time recording stopped when the user clicked the Answer Question button or the Information Not Available button. A page log was also created that counted and identified the pages the user visited. This page count included the front page of the site and all pages that were viewed up to and including the page that was displayed when the Answer Question or Information Not Available button was clicked. User input into the task answer box was recorded so that we could measure the accuracy of the answers given to the task question.

Following the first search task, participants were asked to rate how comfortable they were finding information on Web sites in general and how successful they believed they were at navigating through Web sites in general. They responded using a 5-point Likert scale (always, usually, occasionally, rarely, never). They were also asked if they had previously explored the site before. Following those questions, participants were then given the 10-item System Usability Scale (SUS) (Brooke, 1996) to measure the usability of the portions of the site that were used by a participant. Following the second search task, participants were asked if the site changed from the first time they had used it and, if yes, how so. They were also given an additional opportunity to give comments about the experimental tasks or the Web site.

Materials

The experiment used a modified version of the Stephen F. Austin Teaching Excellence Center Web site. If the shortcut link was not present in the left-hand navigation of the site, then the target information was at minimum three clicks from the front page. Several of the navigation links that led outside the Teaching Excellence Center pages were disabled in order to retain full control over the data collection. There were two versions of the site. In one of the versions, a shortcut link was present in the left-hand navigation (fourth from the bottom in a list of ten links). In the other version there were only nine links in the left-hand navigation, with the shortcut link being removed. Other than the presence or absence of the link and the altered answer to the task question, these two test sites were identical. Figure 1 shows the front page of the site. This is the link-present version of the site. The shortcut link, while highlighted with the red arrow in this figure (the red arrow was not present during the study), was consistent with the other navigational links in the menu during the actual experiment.

Figure 1

Figure 1. The site presented to users in the experiment. This is the version of the site with the shortcut link present, as indicated by the red arrow. The red arrow did not appear to the participants.

The participants were first presented with a task screen that asked them the following question: When the Teaching Excellence Center opened, Dr. Steve Davis spoke at the opening ceremonies. What item did Dr. Davis donate at this event? On the site with the shortcut, the link was labeled Opening Speaker Event.

A bottom frame (1-inch height x screen width) contained both the search question (in order to reduce the memory load of the user) and the two response buttons (Answer Question and Information Not Available); the bottom frame was always present on every page during the study. When a user clicked the Answer Question button, a response box was shown in the same space that allowed the user to type in an answer. The page that the user was viewing when he or she clicked the Answer Question button remained displayed so that any errors in the response would not likely be due to the user forgetting the answer. If the user clicked the Information Not Available button, the user was prompted to continue to the debriefing questions. The answer to the first task was Handbook of the Teaching of Psychology and for the second task the answer was Guide for Successful Lectures.

 

Previous | Next