Contribution
Researchers exploring 3D visualizations currently choose between
two approaches: create a 3D environment with all new applications, or host 3D
elements inside 2D applications. Neither is ideal: with the first a developer is
faced with the impossible task of duplicating the functionality of a tremendous
number of useful common applications. While the second, keeping 3D visualizations
inside a window, limits the range of designs that can be pursued and the degree
of integration.
We have developed a novel approach to bring existing, unmodified
Windows applications into a running 3D virtual environment. The result is a
working platform for
experimentation in 3D user interfaces, in which the user retains all familiar
productivity tools. This also allows for a smooth transition between traditional
2D interfaces and our new 3D territory.
Implementation Issues
The current base
implementation runs on Windows PCs with a 400 MHz Intel Pentium II processor,
128 MB of RAM, and AGP support. We initially used the NVIDIA Riva TNT2 graphics
accelerator with 32MB of memory and AGP texturing support. With this setup we
achieved frame rates above 20 frames per second (fps)
without substantial optimization of the code. The newest emerging graphics
accelerator cards will certainly provide "real-time" updates of well
over 30 fps and with with even fewer texture-mapping bottlenecks.
While we intend the
TaskGallery to become usable as a primary visual shell, at this early stage of
development it runs as a full-screen application on top of the standard Windows
2000 shell. Standard Windows applications started by the TaskGallery
automatically appear within its 3D environment.
All TaskGallery code
was implemented in C++, using the Win32 and Direct3D APIs. The software
architecture is divided into five major components as follows:
-
objects
to simplify and abstract Win32 platform functionality like window creation,
devices and events
-
management of
redirected windows
-
a simple 3D renderer
that abstracts the Direct3D interface and performs render state maintenance
and texture management
-
a library for scene
graph management, animation, and basic graphical object types
-
the high-level
objects that implement behavior and visual representations of Rooms, Tasks,
and Windows.
Redirection
The key technical
challenge in building a 3D window manager like the TaskGallery is to get
existing applications to work in the 3D environment without changing, rewriting,
or recompiling them. This requires
both output and input redirection facilities in the operating system.
Output redirection
causes applications, without the application "knowing it," to render
to off-screen bitmaps instead of the screen, provides access to those bitmaps so
they can be used as textures in the 3D environment, and sends notification
whenever an application has updated its visual display in any way.
|
|
|
Here a standard application (Internet Explorer)
has been redirected into the 3D TaskGalllery environment. The
application has not in any way been modified and in fact does not
"know" that it is being displayed in 3D. |
Input redirection
causes mouse and keyboard events to be received by an application, such as Word,
rather than the TaskGallery's main window, but with mouse coordinates translated
from 3D to 2D.
|
|
|
Because input to the application is filtered
through the same 3D transform that has been used to display the
application, the mouse clicks are correctly interpreted. Here a user is
activating a typical hierarchical menu from an unmodified application
(Internet Explorer). |
Texture Management
Another key problem is
managing textures. Keeping a texture for each window and each task can quickly
fill any texture memory available. In addition, because each application is
always running, the texture for each application has to update often -- thus
requiring high texture download bandwidth. AGP graphics cards make it possible
to texture from main memory with little performance penalty. This helps
enormously, but careful management of textures is still required.
For example, in our current implementation of the TaskGallery, only the windows
in the current task have their textures updated dynamically. And these changes
only happen when an element changes within the window. So the texture bandwidth
requirement for a fairly static application, such as Word, is much lower than
that of a dynamic window, such as a web page with a Flash animation.
Audio
A brief description of the audio components of the TaskGallery is here.
Generality
The low-level
components of this system were implemented by the USER and GDI groups of the
Windows 2000 team (primarily Vadim
Gorokhovsky). Some components use standard features of the operating system,
such as the window hide/show messages. Others were added to a private build of
Windows 2000 for this work. These new features work without recompiling existing
applications because their implementation consists of changes to libraries which
all applications load dynamically at runtime as well as the Windows 2000 driver
that contains the Window Manager and the Graphics subsystem.
The details of this
solution are specific to the Windows 2000 operating system. However, the
components needed will be similar on other operating systems. We believe that
similar changes are possible for any OS that uses the X Window System, as long
as the X server runs on the same machine as the client applications and the
window manager, so that bitmap sharing is efficient.
|
