1. Web Usability Research at Microsoft Corporation


    Amy Kanerva, Kevin Keeker, Kirsten Risden, Eric Schuh, & Mary Czerwinski

    Microsoft Corporation





    The Interactive Media Usability Group at Microsoft Corporation covers a broad range of Microsoft divisions and product categories. Our group covers Web content, reference products, games, desktop finance products, children's products, and input devices—well over 100 different products in all. Although we have a fairly large group of researchers (10 full-time and 11 part-time employees), covering so many different product areas challenges even the most knowledgeable usability engineer. Sharing empirically based principles and design guidelines across team members is critical as development cycles shorten.

    As usability engineers at Microsoft, we support the design process with traditional lab and field research techniques, as well as with other, more "streamlined" usability activities (such as participatory design, heuristic evaluations of preliminary specifications, and testing of low-fidelity paper prototypes). Specific usability goals, identified at the beginning of the product cycle, must be met before the design is frozen. To ensure meeting these aims, usability goals, milestones, and research activities are built directly into the development schedule.

    This chapter reviews the development of guidelines and principles for three Web research areas that we have found to be critical: organization of the content, navigation, and engagement. Each section of the chapter touches on important questions and varied research methodologies used to address one of these research areas.

    Early in the design cycle, we perform research to understand the optimal organization of the Web site. The first section of this chapter describes how we collect card-sort data from users and analyze the data into an organized, conceptual Web site model by using hierarchical cluster analysis.

    After the optimal organization has been designed from a user’s perspective, our usability engineers working on Web products turn their attention to navigation issues. Identifying the structural design properties that support effortless navigation through the site can be worked on very early in the design cycle, often with flow charts or paper and pencil prototypes. Because of the vast number of ways to research this aspect of a Web site, the second section focuses most heavily on the principles and guidelines that have emerged from this work.

    Simply having a good organization and structured design, however, is not enough to ensure a successful Web site. Once the foundational design of a Web site is intact, a usability engineer needs to ensure that the content is appealing. The chapter closes with a discussion of our research efforts for an engaging experience on the World Wide Web (WWW).

    Each of these three areas of Web design is discussed in the order in which it occurs in the design process at Microsoft. For each section, we discuss the motivation behind the usability work, the general methodology developed for this topic area, and a selected set of Web design guidelines. Even though this chapter does not focus on all of the standard methods used by our group throughout the development of Web products (e.g., paper prototyping, heuristic evaluation, iterative lab testing), these methods are used heavily by usability researchers at Microsoft and have been critical to the guidelines effort.


    1. Organization
      1. Motivation
      2. Taxonomies are ubiquitous in everyday life—the division of movies into drama, comedy, and horror at the video store; the intricate Dewey decimal and Library of Congress classifications at the library; the placement of foods in the grocery store. A well-constructed taxonomy is a boon to end users, and enables them to quickly and accurately locate desired information. Conversely, a poorly built taxonomy can lead to a considerable expenditure of time and effort that fails to produce the desired information.

        There are several approaches to building a taxonomy. The organic approach can rely on a combination of naive intuition and incorporation of existing structures. On the face of it, the Microsoft Network (MSN) version 1.3 (http://www.msn.com) and America Online™ (AOL) version 3.0 (http://www.aol.com) online services both seem to be examples of this organic approach. The expert approach relies on professional indexers to build and code a taxonomy. Yahoo’s (http://www.yahoo.com) categories are the clearest example of this expert approach. The user-centered approach incorporates user feedback in both the construction and validation of the hierarchy. Because the taxonomy is built—and tested on—end users, the user-centered approach has a higher probability of mapping with common mental representations than does either of the other approaches used in isolation. The result is a hierarchy that is easier to use.

        The first step in building a user-centered hierarchy is developing an understanding of how people organize information space. This understanding can be gained by using hierarchical cluster analysis to analyze data collected via card-sorting tasks. The resulting tree structures can form the foundation for organizing the site. Moreover, it is possible to identify meaningful patterns in the resulting hierarchy which are indicative of general underlying cognitive processes. These patterns can be abstracted to general principles and guidelines for organizing Web content.


      3. General Approach
      4. Early in the development cycle, long before any content has been created or any interface has been designed, it is possible to do research that dramatically impacts the overall usability of the site. Card-sorting studies can be conducted by using little more than the names of the site elements. From these humble origins, a strong, user-centered organizational structure can be built.

        Typical card-sorting studies involve 40 to 50 participants. Researchers may use different populations that suit specific research questions. For example, children can help to investigate the organization of kids’ Web sites, and the hierarchies generated from novices and experts can be compared for underlying similarities and differences to aid design.

        Participants in these studies are typically given a number of index cards, each printed with a description of a particular part of a Web site. When testing the organization of existing Web pages, it is possible to use a printout of the pages instead of a written description. Participants are asked to sort the index cards into meaningful or related piles and are often asked to provide a label for each pile. Comparing labels to the clusters that emerge from analysis can facilitate interpretation.

        The results of the sorting task are then entered on a spreadsheet. Items that occur together in a group are given an identical category number. These sort values are used to build a frequency of co-occurrence matrix across all the subjects on which various cluster analysis techniques can be applied. Each entry (i.e., each item described on a card) starts out as a single unit or cluster. Additional entries or clusters are iteratively joined to it on the basis of a similarity criterion. The end result of this process is a tree-diagram that provides a view of the number of clusters and the content of each cluster.

        Another variant of this approach can be used to test the ease of use of a proposed organizational scheme. In this version, participants are given a printout of the suggested hierarchy, as well as cards containing descriptions of the content. Participants are then asked to replace the cards into the hierarchy as they see fit. Participants may also be asked to rate their certainty about placing the card in the "right" place. Various "goodness of fit" metrics can be calculated (e.g., percentage of people who categorize the item the same way the originator of the hierarchy did, average certainty ratings), and these can give a sense of how good the proposed hierarchy is. It is also possible to develop a better understanding of how people represent the information space, especially when there are common "misclassifications."


      5. Guidelines

      Several cluster studies have been conducted to help organize information available on MSN, for version 2.0 of the product. These studies focused on the organization of Web content, proprietary content, and children’s content. Although many results are too specific to be of broad interest, several recurring principles emerged:


      1. People pay attention to the source of information, at least as much as they pay attention to the content itself. As a result, the University of Washington Psychology page (http://weber.u.washington.edu/~psychol) tends to be classified with other academic pages (even those from completely different departments) rather than with other psychology pages.When tagging Web content for search mechanisms, designers should make sure to tag the source information. Designers of search mechanisms might want to group search results by source.


      2. People make the most classification "errors" at the deep levels of the hierarchy, when they are asked to make fine distinctions they may not completely understand. Designers should reduce the number of low-level categories and make sure they are readily discriminable.

      3. People have difficulties classifying objects that exist at different levels of abstraction. For example, people classify the New Yorkers for Ralph Nader Web page (http://www.vais.net/~nader96/ny.html) as applying to politics, U.S. politics, presidential campaigns, local politics, politicians, and celebrities. Items that exist at multiple levels of abstraction should be classified at each level of abstraction.

      4. A certain percentage of the population simply does not have the requisite world- knowledge to locate a site. For example, 15% of people in one study believed that Madagascar was in Europe. Designers should consider incorporating common misconceptions into a hierarchy even if they are not "correct."


      After the basic organization has been developed by taking into account user feedback, usability attention turns to navigation.


    2. Navigation

      1. Motivation
      2. Easy navigation is a critical component of a Web site’s success. Even if the basic organization is clear, a poorly designed navigation structure causes users difficulty. Inconsistent navigation may also affect perceptions about a site’s content. Several studies have shown that engagement is positively correlated with ease of navigation. Thus, building clear navigation is a key factor to immersion and engagement.

        Designers of Web navigation schemes often fail to provide distinctive landmarks that indicate where users are located and where they can go. Real-world navigation often involves following clear paths or trails, but Web sites frequently do not provide maps or trails to guide movement through Web environments. At an extreme, the only way users can tell where they are in some Web pages is to remember where they came from. This process can be like trying to remember, in order, a long list of random numbers.

        To help users to develop a sense of a Web site’s space, the navigation design needs to systematically add cues and provide structural elements that serve as spatial landmarks and navigational devices. Landmarks and navigational devices are particularly important in designing Web sites for children as they are less likely to have general knowledge about spatial navigation. At the same time, when strong landmarks are absent, adults tend to revert to less developmentally advanced ways of reasoning about a space. Our goal in a series of recent studies was to understand how to engineer fluid navigation into Web sites.


      3. General Approach
      4. Usability activities focusing on navigation tend to occur relatively early in the design of our Web products, although competitive analysis can support this process at any stage of design. One goal of our research was to extend our existing navigation design guidelines that we derived from spatial cognition and design literature. The extensions to these guidelines would be based on our own studies with Web users, by conducting usability research that helped us establish a basic understanding of which cues seem to be most relevant for Web site design. The scope of this research relied on successive usability studies with a combination of low-fidelity paper prototypes, higher fidelity online prototypes, and existing online service providers such as AOLÔ (http://www.aol.com) and PathfinderÔ (http://www.pathfinder.com). Across multiple studies, data about behaviors during use and data about postuse memory of the Web sites were collected. Different studies included various dependent measures (e.g., user errors categorized according to a specific coding system [see Table 14.1], search times, and users’ memory of their traversal paths, analyzed for accuracy and completeness).



        Table 14.1. Error Classification Scheme Used During Web Usability Studies



        Type of Error

        Behavioral Indice Observed

        Where am I?

        Behavior indicates users do not know where they are in the structure of the Web site.

        How can I get to other places?

        Behavior indicates users do not know how they can navigate to another place from the Web site.

        Where can I go?

        Behavior indicates users do not know where they can go in the Web site.



      5. Guidelines

      Web site design should facilitate navigation by using visually salient landmarks in the form of labels, buttons, headings, and icons. These visual cues should indicate the current location and potential pathways to clearly described ultimate goals. The following guidelines provide an overview of design techniques and elements that can be used to support spatial learning and navigation. All these guidelines have been empirically validated in the laboratory with target end users.


      1. Simplify the virtual "landscape" for users so that spatial information can be easily learned: Designers should not overload pages with choices; as a general rule, five choices per page are sufficient, unless they can be grouped as a single chunk of information. Designers should make all choices in an image map fit in one screen. Directories and table of contents pages must have intuitive and recognizable ways of grouping and organizing entries. If a navigation bar contains both links to places and access to tools, the design must highlight the difference between these button groups. Designers should avoid nested scroll bars, and should especially avoid placing a nested scroll bar next to a major scroll bar. When using scrolling, designers should signal that there is additional content below or to the side.


      2. Make users aware of where they are at all times: Metaphors can be helpful in communicating the organization of a space. When using a metaphor, carry it throughout the navigation design. Otherwise, the lack of consistency actually impedes its effective use. Navigation bars should be very noticeable and should communicate which category users are in. Every time a site is mentioned, it should be given the same name, and each page should be labeled with its topic. Users should recognize changes as they navigate from one page to the next.

      3. Make users aware of where and how to travel in the space: Create a persistent navigation structure to guide users to a home base at the major, intermediate, and minor levels of the Web site organization. Never assume that users can use the Back button to get out of pages; at the same time, always assume that some of them will use the Back button instead of the button you have created to get them out, and plan for this outcome. Never assume that all readers arrive at a page from the same place. Avoid navigational dead ends and links to incomplete pages. Place icons and buttons in a location that takes into consideration where attention is likely to be drawn.

      Make sure to distinguish graphic content images from navigational icons and buttons. Avoid intrapage links. If intrapage links must be used, ensure that users can see a link back to the top of the page. Make sure that navigation buttons and icons are meaningful for target users. Make sure icons that are clickable in one location are clickable throughout the site, unless they are grayed out. Make sure that a particular icon always takes users to the same place wherever it is encountered. Avoid ambiguous terms, if possible. Use labels that indicate specifically where a navigation element takes a user. Use tool tips to reinforce and clarify functions and destinations of buttons and icons. Build preferred paths into designs (rather than using a tour or guide). Preferred paths may be important for engagement with some Web sites (e.g., narratives, productivity).



      Once designers know the basic organization of the Web site and have successfully designed in the ways users will navigate through it, they look at the content. We have been creating content geared toward entertaining home users. The following section focuses on how we approached this research.


    4. Researching Appealing Web Content

    6. Motivation
    7. At the end of 1996, the Microsoft Network discontinued its use of proprietary protocols, moved to the Web, and delivered all its content via standardized Web protocols, common scripting languages, and ActiveX controls. MSN version 2.0 also kicked off a movement to broaden the appeal of online services by bringing more entertaining content to the Web. Like television, theater, and movie entertainment, these new content offerings were called shows.

      Web shows create a special challenge for usability researchers. Although shows must be usable, typical usability goals like simplicity and consistency may not always be appropriate in an entertainment context. People want their games to be challenging, their books to be frustratingly suspenseful, and their tearjerkers to make them cry. Usability researchers must try to keep interface problems from lessening the impact of the content while realizing that content and user interface (UI) are inextricably tied.

      Shows also present a measurement problem for researchers. The ultimate success of entertainment software is difficult to measure in tangible, reliable behaviors like task time or number of errors. In addition to traditional measures, researchers have to make strong use of natural observation and subjective questionnaires. The ultimate measure of success for a show is whether people enjoy it, become engaged in the experience, and return to the show again and again.


      1. General Approach
      2. One part of Microsoft’s approach to show research has involved creating a reliable, subjective questionnaire to measure appeal. In doing so, we have tried to identify a set of factors that contribute to the appeal of Web content.

        Our observations have been drawn from many types of entertainment venues. Users look for particular characteristics in certain types of content. For example, electronic magazines need to be timely, while networked Web games need to accurately target a range of skill levels. We assume, however, that the lessons learned in one domain may be, in some way, applicable to another domain. For example, we expect soap operas to have strong, identifiable characters, but we do not expect reference Web sites to have strong, identifiable characters. Interestingly, movie reviewers play both a reference and an entertainment role. A reference Web site with a strong editorial viewpoint, in addition to an unbiased perspective, could be very appealing.

        Factors relating to appeal were iteratively derived via a combination of theoretical and empirical means. Theoretical insights came from researching the Human–Computer Interface (HCI) and psychological literatures and from frequent discussions about entertaining media. Empirical research focused on kids’ entertainment titles and age nonspecific entertainment content for the WWW.

        In an ongoing Microsoft research project, children rated their favorite Web sites or computer products on Likert-style questionnaires (younger children marked their ratings on a graphic response bar with iconic positive and negative poles). Usability engineers derived items from Lepper’s (1988) research on enhancing intrinsic motivation, Whalen’s and Cziksentmihalyi’s (1991) research on flow states, and UI research by Hix and Hartson (1993). A factor analysis of the questionnaire verified Lepper’s (1988) suggestion that challenge, control, contextualization, and curiosity are important underlying dimensions of intrinsic motivation.

        A literature review of HCI research on games suggested additional items to assess: ease of use and attractive use of graphics and sound (Carroll & Thomas, 1988; Igbaria, Schiffman, & Wieckowski, 1994; Malone, 1981; Neal, 1990; Pausch, Gold, Skelly, & Thiel, 1994). Concurrent discussions about entertaining media focused on emotional reactions to stories told via television, movies, and books. A range of primary emotional reactions was pulled from the psychological literature on emotion and added to the questionnaire. User comments and market research suggested that users returned to those Web sites that had the correct amount of quality information on a topic interesting to a particular user.

        At each stage, item sets were analyzed empirically. Weak questionnaire items were removed or revised to leave a reliable core item set. The resulting item sets were used to research MSN shows. Systematic analysis of data collected during iterative usability testing, site visits, and broad distribution surveys over a year has recently shown that most of the attributes listed in the following Guidelines section predict how much people like a show and how many hits a show gets once it is released. To further validate the appeal factors, we expect to compare appeal questionnaire ratings to measures of actual MSN show usage rates.


      3. Guidelines

      A structured series of guidelines for enhancing appeal was derived over the course of extended research and refinement of the questionnaire. Each guideline may not be applicable to every show. Some guidelines are more important for certain types of entertaining experience than for others.


      1. Ease of use—clear goals: The goal of the show is immediately clear, and the steps to achieve the goal are clear.

      2. Ease of use—clear structure: The design clearly communicates the show's core activities and sets appropriate expectations. The terms (especially the title and the sections of the show) adequately communicate the consequences of selection or action. The core activities require few actions to locate. People always have more than one course of action to choose from. People are able to get out of sequences that they do not want to be in.


      3. Ease of use—clear feedback: Page headings help users to understand where they are located. Design elements (e.g., headings, scores, visual metaphors) indicate what is happening.


      4. Content quality—valuable: The topic matter is interesting to the core audience, and people have an opportunity to learn.

      5. Content quality—timely information: The show features the latest information available on the topic.

      6. Content quality—appropriate depth and breadth: The show has the right amount of variety for the topic area. The articles are the right lengths, and the show goes into the appropriate level of depth in its topic areas.

      7. Content quality—attractive use of media: The graphics and audio are appealing to the core audience.

      8. Content quality—safe novelty: Users experience a mix of the known and unknown, in a novel twist on the familiar.The metaphors, language, graphics, and sounds set appropriate expectations for the show.

      9. Content quality—challenging: The level of challenge in the show's content is appropriate for the core audience. People find new challenges as they gain experience.

      10. Content quality—evokes emotion: Users feel emotion while experiencing the show.


    8. Summary

      Designing the organization, navigational structure, and appeal of a Web site from the perspective of users can be a difficult process because of the large number of Web sites and the rapid development cycles with which usability engineers typically work. It has been our observation that pulling guidelines and principles from the empirical data we collect during Web interaction can have positive impact on Web site design. This chapter has presented three of the many important areas of Web design and has shown how the Interactive Media Usability Group at Microsoft is involved during these design stages. In the end, the usability group needs to work to ensure that users come back to our Web sites again and again. Our research efforts continue to focus on Web designs that evoke emotion, interest, and pleasure for our target audiences.



      1. References


Carroll, J. M., & Thomas, J. C. (1988). Fun. SIGCHI Bulletin, 19(11), 21–24.

Hix, D., & Hartson, R. (1993). Developing user interfaces. New York: Wiley.

Igbaria, M., Schiffman, S. J., & Wieckowski, T. J. (1994). The respective roles of perceived usefulness and perceived fun in the acceptance of microcomputer technology. Behaviour and Information Technology, 13(6), 349–361.

Lepper, M. R. (1988). Motivational considerations in the study of instruction. Cognition and Instruction, 5, 289–309.

Malone, T. W. (1981). Heuristics for designing enjoyable user interfaces: Lessons from computer games. In Proceedings of the Conference on Human Factors in Computer Systems (pp. 63–68). Gaithersburg, MD:ACM.

Neal, L. (1990). Implications of computer games for system design. In Diaper, D. (Ed.), Human–computer interaction—Interact ’90 (pp.93–99). New York:Elsevier.

Pausch, R., Gold, R., Skelly, T., & Thiel, D. (1994). What HCI designers can learn from video game designers. In Proceedings of computer human interaction (pp. 177–178). Boston:ACM.

Whalen, S. & Csikszentmihalyi, M. (1991). Putting flow theory into educational practice. (Report to The Benton Center for Curriculum and Instruction). Chicago: University of Chicago Press.