Contribution

Researchers exploring 3D visualizations currently choose between two approaches: create a 3D environment with all new applications, or host 3D elements inside 2D applications. Neither is ideal: with the first a developer is faced with the impossible task of duplicating the functionality of a tremendous number of useful common application. While the second, keeping 3D visualizations inside a window, limits the range of designs that can be pursued and the degree of integration. 

We have developed a novel approach to bring existing, unmodified Windows applications into a running 3D virtual environment. The result is a working platform for experimentation in 3D user interfaces, in which the user retains all familiar productivity tools. This also allows for a smooth transition between traditional 2D interfaces and our new 3D territory.

Implementation Issues

The current base implementation runs on Windows PCs with a 400 MHz Intel Pentium II processor, 128 MB of RAM, and AGP support. We initially used the NVIDIA Riva TNT2 graphics accelerator with 32MB of memory and AGP texturing support. With this setup we achieved frame rates above 20 frames per second (fps) without substantial optimization of the code. The newest emerging graphics accelerator cards will certainly provide "real-time" updates of well over 30 fps and with with even less texture-mapping bottlenecks.

While we intend the TaskGallery to become usable as a primary visual shell, at this early stage of development it runs as a full-screen application on top of the standard Windows 2000 shell. Standard Windows applications started by the TaskGallery automatically appear within its 3D environment.

All TaskGallery code was implemented in C++, using the Win32 and Direct3D APIs. The software architecture is divided into five major components as follows:

1.  objects to simplify and abstract Win32 platform functionality like window creation, devices and events

2. management of redirected windows

3. a simple 3D renderer that abstracts the Direct3D interface and performs render state maintenance and texture management

4. a library for scene graph management, animation, and basic graphical object types

5. the high-level objects that implement behavior and visual representations of Rooms, Tasks, and Windows.

Redirection

The key technical challenge in building a 3D window manager like the TaskGallery is to get existing applications to work in the 3D environment without changing, rewriting, or recompiling them.  This requires both output and input redirection facilities in the operating system. 

Output redirection causes applications, without the application "knowing it," to render to off-screen bitmaps instead of the screen, provides access to those bitmaps so they can be used as textures in the 3D environment, and sends notification whenever an application has updated its visual display in any way.  

Input redirection causes mouse and keyboard events to be received by an application, such as Word, rather than the TaskGallery's main window, but with mouse coordinates translated from 3D to 2D.

Texture Management

Another key problem is managing textures. Keeping a texture for each window and each task can quickly fill any texture memory available. In addition, because each application is always running, the texture for each application has to update often -- thus requiring high texture download bandwidth. AGP graphics cards make it possible to texture from main memory with little performance penalty. This helps enormously, but careful management of textures is still required.

For example, in our current implementation of the TaskGallery, only the windows in the current task have their textures updated dynamically. And these changes only happen when an element changes within the window. So the texture bandwidth requirement for a fairly static application, such as Word, is much lower than that of a dynamic window, such as a web page with a Flash animation.

Audio

A brief description of the audio components of the TaskGallery is here.

Generality

The low-level components of this system were implemented by the USER and GDI groups of the Windows 2000 team (primarily Vadim Gorokhovsky). Some components use standard features of the operating system, such as the window hide/show messages. Others were added to a private build of Windows 2000 for this work. These new features work without recompiling existing applications because their implementation consists of changes to libraries which all applications load dynamically at runtime as well as the Windows 2000 driver that contains the Window Manager and the Graphics subsystem.

The details of this solution are specific to the Windows 2000 operating system. However, the components needed will be similar on other operating systems. We believe that similar changes are possible for any OS that uses the X Window System, as long as the X server runs on the same machine as the client applications and the window manager so that bitmap sharing is efficient.