upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

When Links Change: How Additions and Deletions of Single Navigation Links Affect User Performance

Lauren F.V. Scharff and Philip Kortum

Journal of Usability Studies, Volume 5, Issue 1, Nov 2009, pp. 8 - 20

Article Contents


Results

For each participant for each search, we recorded accuracy, search time, link use, navigation path, and page count. Ratings of comfort and success did not significantly correlate with accuracy, search time, or page count; thus, these ratings were not included as covariates in the subsequent analyses. SUS scores were within the acceptability ranges outlined by Bangor, Kortum, and Miller (2008), so this variable was also dropped from further consideration.

Prior to any multivariate analyses, we performed a simple comparison of the effect of link presence versus link absence. For the dependent variable of search time, users were significantly faster when the link was present (M=90.8 sec) than when it was not present (M=158.6 sec), t(226) = 7.98, p<.001. For the dependent variable of page count, users viewed fewer pages when the link was present (M=3.7) than when the link was not present (M=10.1), t(194) = 12.1, p<.001.

To better understand the interaction between variables, for both search times and page counts a 2 (search task) x 2 (consistency) x 2 (link presence on second search) x 3 (delay) mixed ANOVA was performed. Search task (first or second) was the within variable and the remaining were between variables. Search times were measured with millisecond accuracy but are reported in seconds. Figures 2 and 3 show means for all conditions collapsed across the three delays.

For search times, there were several significant main effects. However, these were modified by significant interactions. On average, the first search was slower than the second, F(1,152) = 74.61, p<.01; consistent searches were faster than inconsistent searches, F(1,152) = 15.70, p<.01; the three-week delay led to the slowest searches, F(2, 152) = 9.49, p<.01 (M = 118.24, 108.28, 146.11 secs for immediate, one-week delay, and three-week delay, respectively); there was no effect for link presence on the second search. Search task, consistency, and delay interacted significantly so that all groups, except the one-week delay with inconsistent link presence group, showed a decrease in search times on the second search, F(2, 152) = 3.46, p = .034. Finally, search task, consistency, and link presence on the second search interacted significantly, F(1, 152) = 46.32, p<.01. Users showed the largest improvement (decrease in search times) in the second search when a link was added (NY condition, shortcut link was present [Y] or not [N]), but they were also faster on the second search with the consistent conditions (NN and YY). Those users who first had a link and then had it removed showed an increase in search times for the second search. See Figure 2 for a plot of this interaction. Overall, delay did not change the pattern of responses, and users with the condition where the link was consistently present (YY) had the fastest search times (M=84.5 sec).

Figure 2

Figure 2. Search times (seconds) and standard errors for each condition collapsed across delay. Users showed improvements for all conditions except when the shortcut link was removed (YN condition). Letter pairs along the axis indicate whether the shortcut link was present (Y) or not (N) with the first letter representing the first search and second letter the second search.

For page counts, there were two significant main effects. Consistent searches led to fewer page counts than inconsistent searches, F(1,152) = 6.97, p<.01, and having a shortcut link on the second search led to significantly fewer page counts, F(1, 152) = 20.46, p<.01. There was no effect for search (first or second) or for delay (M = 6.64, 6.59, 7.31 pages for immediate, one-week delay, and three-week delay, respectively). However, these were modified by significant interactions. Similar to the results with search time, there was a significant 3-way interaction between search, consistency, and link presence on the second search, F(1, 152) = 49.04, p<.01. Specifically, there was a significant increase in the number of pages users viewed when the link was removed on the second search (YN) and a significant decrease in the number of pages viewed when the link was added (NY). Users in both the NN and YY conditions showed a non-significant decrease in the number of pages on the second search. Users in the YY condition explored the fewest pages overall (M=3.3). See Figure 3 for a plot of this interaction. Again, delay did not change the pattern of responses.

Figure 3

Figure 3. Page counts and standard errors for each condition collapsed across delay. Removal of the link led to a significant increase in the number of pages and addition of the link led to a significant decrease in the number of pages. Letter pairs along the axis indicate whether the shortcut link was present (Y) or not (N) with the first letter representing the first search and second letter the second search.

Page count statistics alone do not tell the entire story. While there is clearly a significant difference in the page counts for the inconsistent conditions, the pattern of page visits also appears to vary to an extent noticeable by participants. Indeed, the participants in the inconsistent conditions were more likely to report that the site had changed in some way than the participants in the consistent conditions. Using a color chart showing page viewing hot spots for the site, Figures 4 through 7 show the different viewing patterns for the consistent and inconsistent conditions. These figures reflect the averages of page views for all users in the three-week delay condition. The results are similar for the no delay and the one-week delay conditions. The top row of boxes in each figure shows the links available in the left-hand navigation, with the far left box representing the home page link and the second-to-left box representing a link to Services and Workshops. The layout of the other boxes indicates the organization of the deeper pages that are accessed when clicking on links. In all figures, the star indicates the page where the target information was located.

Figure 4 shows the viewing pattern from the consistent NN condition. As can be seen, viewing patterns remain relatively consistent between the first and second viewings. Figure 5 shows the patterns from the consistent YY condition. As we expect, the patterns are different than the NN condition, with most of the page views confined to the top level pages. Within the YY condition, however, the viewing patterns are again consistent between the first and second viewings. In contrast, Figures 6 and 7 show the different patterns of page views in the inconsistent conditions. In Figure 6 the pattern is shown for the NY condition, where users did not have the link in the first visit and then did in their second visit. The pattern of viewing is tighter and more localized in the second condition, as users found the shortcut link and utilized it to navigate directly to the target material. Figure 7 shows that the opposite is true in the YN condition. Here, users had the link in their first visit, but the link was removed on the return visit. As can be seen, the pattern of page views (especially the use of multiple left-navigation links) spreads out as users look for the target pages or the links leading to it. Also note that in all N conditions, regardless of whether they were the first or second task, the left-navigation link Speakers and Workshops is red. In contrast, in all Y conditions, it is either blue or green because many of the users avoided the path by jumping directly to the target page using the shortcut link.

Figure 4

Figure 4. Color temperature chart of pages visited for the NN consistent condition. Colors indicate the average number of times a page was viewed in the given condition. The top diagram is the first visit (link not present) and the bottom diagram is the second visit (link not present) three weeks later. The no delay and one-week delay conditions have similar patterns. The star shows the location of the target information.

Figure 5

Figure 5. Color temperature chart of pages visited for the YY consistent condition. Colors indicate the average number of times a page was viewed in the given condition. The top diagram is the first visit (link present) and the bottom diagram is the second visit (link present) three weeks later. The no delay and one-week delay conditions have similar patterns. The star shows the location of the target information.

Figure 6

Figure 6. Color temperature chart of pages visited for the NY inconsistent condition. Colors indicate the average number of times a page was viewed in the given condition. The top diagram is the first visit (link not present) and the bottom diagram is the second visit (link present). This figure shows the three-week delay, but the no delay and one-week delay conditions are similar. The star shows the location of the target information.

Figure 7

Figure 7. Color temperature chart of pages visited for the YN inconsistent condition. Colors indicate the average number of times a page was viewed in the given condition. The top diagram is the first visit (link present) and the bottom diagram is the second visit (link not present). This figure shows the three-week delay, but the no delay and one-week delay conditions are similar. The star shows the location of the target information.

For the dependent variable of accuracy, users were significantly more accurate overall when a link was present (M=94%) than when it was not (M=63%), t (223) = -6.84, p<.001. More specifically, users were significantly more accurate when a link was added for the second search (NY), t (39) = -4.15, p<.001, and significantly less accurate when the link was removed (YN), t (50) = 4.14, p<.001. Importantly, however, even if a link was removed after the first search (YN), users were no less accurate than if the link had never been present, t (106) = 1.33, p>.05. See Figure 8 for a plot of this interaction. As with search times and page counts, delay did not affect the pattern of results.

Figure 8

Figure 8. Accuracy (in percent) for each condition collapsed across delay. Letter pairs along the axis indicate whether the shortcut link was present (Y) or not (N) with the first letter representing the first search and second letter the second search.

An investigation of link use showed that approximately half of the users used the link in the second search if it was not present during the first search (NY condition). However, regardless of link use, users showed improvement across the two searches. For search times, both those who did and those who did not use the added link showed a significant decrease in their search times, t (28)=, p<.001 and t(15)=3.53, p<.002, respectively. Similarly, for page counts, both those who did and those who did not use the added link showed a significant decrease in the number of pages viewed, t(28)=5.54, p<.001 and t(15)=3.53, p<.002, respectively.

 

Previous | Next