Chapter 9 Conclusions

9.1 Approach

The WIMP (Windows, Icons, Menus, and Pointer) interface metaphor was designed to match the computing capabilities available in the late 1970's and early 1980's. Computers are no longer limited to small black-and-white displays and an impoverished input bandwidth. Continued vigorous growth in processor power and memory capacity, in addition to modern innovations such as large flat-screen color displays, real-time 3D graphics on the PC platform, and high degree-of-freedom input devices make the limitations of the WIMP interface paradigm increasingly apparent.

A host of devices, displays, and interaction techniques have been proposed as candidates for the post-WIMP interface. A key challenge is to discover interaction techniques that do not necessarily behave like the real world, yet nonetheless seem natural. Directly mimicking reality will not necessarily produce the desired results. A demonstration of a new interaction technique that does not directly mimic reality may not be enough to fully take advantage of the technique. To understand why interaction techniques do or do not work, and to design new interaction techniques which meet these criteria, the interface designer needs to understand the human. In essence, my approach focuses not on a particular technology, but rather on the capabilities of the human participant.

9.2 Contributions

My work contributes a system which has undergone extensive informal usability testing in the context of real domain experts doing real work, but it also presents the results of experimental evaluations which illustrate general behavioral principles. Some of the informal design issues which my user observations have suggested are now supported by formal experimental evaluations. And the selection of experimental evaluations has been tempered by design experience, so there is some assurance that the experiments apply to design in a meaningful way. Together, these approaches make a decisive statement that using both hands for virtual manipulation can result in improved user productivity.

The primary contributions of my work are:

Revisiting passive haptic issues: Facile virtual manipulation requires studying the feel of the interface, and not just the look of the interface.
Application to neurosurgical visualization: I contribute the design, implementa-tion, and usability testing of a three-dimensional volume visualization application which has been well accepted by neurosurgeons.
Two-handed virtual manipulation techniques: Using the nonpreferred hand as a dynamic frame of reference is an important design principle for virtual manipulation. The asymmetrical hierarchical organization of two-handed manipulation is not generally known by interface designers, nor will it be obvious how to apply this knowledge when they design two-handed interfaces without benefit of the knowledge contributed by this dissertation.
Basic knowledge about two hands: I contribute experimental data which shows that performance is best when the preferred hand operates relative to the frame-of-reference of the nonpreferred hand.
Two hands are not just faster than one hand: Two hands together provide the user with information which one hand alone cannot. Using both hands can influence how users think about a task.
Research methodology: As a whole, the dissertation provides a case study for an interdisciplinary approach to the design and evaluation of new human-computer interaction techniques.

9.3 High level conclusions

This work has demonstrated the inherently interdisciplinary nature of human-computer interaction. The collaboration of neurosurgeons, computer scientists, and psychologists has enabled a decisive contribution to the virtual manipulation and two-handed interaction research fields. Without any one part of this collaboration, the work would not be as convincing.

This work also illustrates the synergy between behavioral principles, usually proposed by the psychology community, and design principles, sought after by the human-computer interaction community. Behavioral science has direct application to design: human-computer interaction can profit from increased collaboration with perceptual psychologists, cognitive psychologists, and motor behavior researchers.

9.4 Future Work

Multimodal input has been an underlying theme of this thesis. An exploration of multimodal input which integrates both hands, the voice (voice recognition), and possibly other effectors such as the head, legs, and feet is a natural extension of this work. It seems quite likely that Guiard's kinematic chain model can be profitably applied to more than just the two hands working together. For example, an organist uses both hands on the organ keyboard, uses both feet to control pedals, and sometimes sings as well; during microsurgery, the surgeon holds an implement in each hand, issues verbal orders to the surgical team, and sometimes even uses the feet to steer the surgical microscope and to control the zoom factor. Can computer interfaces take advantage of these skills as well?

Much previous work has discussed all of these capabilities as if they were independent input channels, but as Guiard's work suggests they are not independent at all, but rather form a hierarchy of interdependent effectors. And there are certainly other constraints which interface designers have only begun to understand: for example, Karl [96] has found that issuing voice commands can interfere with short term memory under some conditions. As a second example, this time from the psychology literature, Cremer [44] has reported that during a finger tapping task, concurrent speech activity can disrupt right-hand performance but not left-hand performance, while a concurrent visuospatial task can disrupt left hand performance. Clearly, there are some constraints on multimodal human performance which are not yet understood at the level of interface design principles.

Interfaces which combine both active force feedback and two-handed interaction, for applications such as telemanipulation [163] or exploration of computer-generated virtual spaces, seem like a promising research area. This might involve using an armature such as the Phantom [147] in one hand with a virtual object held in the other hand, or using a pair of armatures, one for each hand. For example, one possible application the user interface group [173] has envisioned is to add active force feedback to the Worlds-in-Miniature application (fig. 2.12 on page 28) [164], so that the user can feel objects held in a miniature world in the nonpreferred hand. Such a configuration might also allow one to explore behavioral issues with two hands and active haptic feedback. For example, in a prototype of this configuration, George Williams (working with the user interface group [173]) found that when the preferred hand holds a force reflecting stylus up to a virtual object held by the passive nonpreferred hand, there is a compelling "ghost sensation" that the nonpreferred hand is also receiving force feedback.

Large flat-panel display technology is rapidly coming to market. As scale increases, the need arises for tasks with both macrometric and micrometric movement scales, which would seem to make such a configuration ideal for two-handed interaction. Large, flat displays can be simulated now using back-projection techniques (for example, the ActiveDesk (fig. 7.4 on page 139), the Immersadesk (fig. 2.14 on page 30) [49], or the Reactive Workbench [52]) and researchers have already started to explore two-handed interaction techniques for these systems.

In medical imaging in general, there are still tremendous opportunities to improve physician user interfaces. A current trend is the "filmless hospital" which replaces the traditional physical film media with purely digital patient images, and has led to the introduction of so-called PACS systems (Picture Archiving and Communications Systems). Working with digital images raises a host of interface, security, archiving, and telecommunication issues which never arose with film-based imaging, and as a result, physicians find many of the commercially available PACS systems very difficult to use.

In terms of the neurosurgery application, there is still much to be learned by turning the props-based interface into a clinical tool. Multimedia Medical Systems [122] is currently exploring the issues of turning my interface design into a commercial application, for neurosurgery as well as other medical applications. By deploying a commercial version of the system at a significant number of hospitals, there is the potential for this effort to provide a real success story for spatial interaction technologies.

9.5 Epilogue

For me, perhaps the most striking element of all of this work is how much a resourceful design can get away with, in terms of helping the user to see things that aren't really there, while using some very simple technology. My interface design is perhaps more notable for the technologies it does not use than for those it does use: it does not use head tracking; it does not use stereoscopic projection; it does not use gloves; it does not use an immersive head-mounted display. Nonetheless, it allows the user to see and explore the interior of a virtual object in interesting new ways. The point I am trying to make is that most applications of interactive 3D graphics do not need a full scale virtual reality implementation, but often some rather simple technology in the desk top setting can do the trick.

The technology trap is an easy one to fall into. Fred Brooks has recounted the story of the world's first molecular visualization system in virtual reality [22], which he and his colleagues designed to allow a molecular chemist to walk around a room-filling molecule and inspect it from different standpoints. What is one of the first things Brooks's molecular chemist collaborator asked for? A chair!

Indeed, my chief neurosurgeon collaborator had initially talked to me about doing an immersive walkthrough of the human brain. It probably would have failed for many of the same reasons that Brooks's room-filling molecule was not completely satisfactory for molecular visualization. This teaches another valuable lesson: domain experts are expert primarily at the work they do, and not at the technology that might best help them to do that work better. The interface I designed uses six degree-of-freedom input devices and two-handed manipulation not because these are interaction techniques that I or my neurosurgeon collaborators had a priori chosen to investigate, but rather because they are well suited to some of the problems that neurosurgeons are trying to solve.

[Top] [Prev] [Next] [Bottom]