The Microsoft Interactive TV System:
An Experience Report
Michael B. Jones
Research, Microsoft Corporation
One Microsoft Way
Redmond, WA 98052
This paper describes the Microsoft Interactive Television (MITV) system. An overview of the different components of the system is presented. The development process is described. Finally, experiences and lessons learned from the system are presented.
The system is designed to be able to provide traditional television service, video-on-demand service, and custom interactive applications, such as interactive shopping services, to subscriber bases scaling from a few hundred subscribers up through major metropolitan areas. An early version of the system was deployed in a trial of 100 subscribers in the Redmond, Washington area with TCI and Viacom from April, 1995 to early 1997 using PCs to emulate set-top boxes; a trial of 298 subscribers was deployed from March 1996 to April 1997 by NTT and NEC in Yokosuka, Japan using client-premises set-top boxes.
From initial conception in early 1993 thorough the trial deployments, an eventual staff of over 200 people evolved the design and implementation of the system to meet changing requirements under tight deadlines. The team delivered a working system on time (a rare occurrence in the software industry!). This paper will describe both the eventual architecture of the MITV system and some of the compromises made and lessons learned in building and deploying this substantial commercial software system.
By early 1993 the combination of low-cost broadband networking, low-cost powerful microprocessors, and advanced digital compression techniques had enabled the prospect of eventual widespread deployment of interactive television systems. Both cable TV operators and telephone companies envisioned the possibility of providing a wide new range of compelling content, increasing their revenue bases and allowing them to create and enter new businesses, particularly once widely anticipated telecommunications deregulation occurred.
Microsoft, as did others such as Oracle [Laursen et al. 94] and Silicon Graphics [Nelson et al. 95], saw interactive television (ITV) as a new kind of software platform, potentially outnumbering the PC if successful. Thus, we set out to develop a compelling ITV system.
Key properties that we wanted the resulting system to have were:
The remainder of this paper presents details of the system developed, describes the development process, presenting some of the "war stories" and lessons learned, and analyzes the degree to which these technical goals were met. Finally, the paper presents a discussion of why ITV was a business failure despite being a technology success, particularly in light of the concurrent emergence of the Internet as a popular communication medium.
2. System Overview
This section provides a brief overview of the entire system. Each of the elements mentioned here is discussed in greater detail in sections 3 to 10.
Figure 2-1 shows an overview of the Microsoft Interactive TV system as deployed by Nippon Telephone & Telegraph (NTT) and Nippon Electric Corporation (NEC) in Yokosuka, Japan in March, 1996.
Figure 2-1: Overview of MITV system
From March, 1996 to April, 1997 this system served 298 paying subscribers, providing the following applications to users:
Applications written and deployed by NTT include:
While NTT's consumer interactive television trial concluded in April, 1997, they retain a working demonstration installation of the system in Yokosuka.
2.1 Network Client
The set-top box is a custom-designed diskless computer using a Pentium processor, a custom video and graphics subsystem including an MPEG-2 [Mitchell et al. 97] audio/video decoder and sprite-based graphics compositing, an Asynchronous Transfer Mode (ATM) [CCITT 93] network interface card, and various peripheral devices.
The set-top box software uses a small, portable real-time kernel upon which subsets of the Win32, DirectDraw, and DirectSound APIs have been layered. Subsets of the standard Microsoft C/C++ run-time library and Microsoft Foundation Classes (MFC) are provided. A DCE RPC (a remote procedure call standard from the Open Software Foundation's Distributed Computing Environment) implementation is present for communicating with remote services, and is used for implementing the DCOM distributed object facility and a distributed namespace. (COM[Microsoft 94], the Component Object Model, is the object model underlying OLE2. DCOM is a distributed version of COM allowing cross-machine object references and invocations.) DCOM is used to implement many application services, including program invocation, remote database services (used for program and movie guide data), per-viewer and per-set-top-box persistent state (such as profile data and last channel watched), financial transaction services, and logging. Finally, on top of these lower layers, the actual user-visible applications, such as the video-on-demand player, the digital broadcast player, the electronic program guide, and the system navigator (the channel surfing application), are built.
Approximately 12 megabytes of general-purpose set-top box memory are in use when running typical applications, such as video-on-demand. This includes 3MB of Kanji font memory and 3MB of video buffer memory. In addition to the general-purpose memory, 1.7 megabytes of special video memory and 2 megabytes of dedicated MPEG-2 decode memory are in use.
2.2 Head-End Servers
The server machines in the ITV head-end are all NEC SV98 PCs with 90MHz Pentium processors and 80MB of memory. They are running Japanese Windows NT Server 3.51, plus various services specific to interactive TV. Fifteen server machines are used in this trial.
The primary head-end services are the Tiger distributed video server[Bolosky et al. 96, Bolosky et al. 97], an extended DHCP server (Dynamic Host Control Protocol, RFC-1531 [IETF 93], an Internet standard for dynamically assigning host IP addresses), the DCOM implementation, the Class Store (which also includes a network boot server), the distributed directory service, OleDB (remote database services), Microsoft SQL Server (a database server), and several services supporting financial transactions.
Not all ITV head-end services run on all head-end machines. For instance the Tiger controller server runs on a dedicated machine, and the Tiger cubs (the actual server machines from which striped video is delivered) occupy ten dedicated server machines, leaving four other general head-end server machines in this configuration.
The NTT trial in Yokosuka uses an ATM network over optical fiber at 155Mbits/second (OC-3) to connect head-end servers and subscriber's homes. All data, other than digital broadcast streams, is sent using UDP datagrams [IETF 80]. (Broadcast streams for this trial are MPEG-2 encoded in real-time by a separate NTT facility and transmitted directly into the ATM network over a 622Mbit/second OC-12 line using MPEG transport stream packets.)
While this trial happens to use an "ATM-to-the-home" network, it should be noted that the only attributes of this network that higher layers of software depend upon are the ability to provide guaranteed bandwidth communication and the availability of IP. Other physical networks, such as hybrid fiber/coax, ADSL (Asynchronous Digital Subscriber Line, a method of transmitting high bandwidth data over standard telephone wires), switched Ethernet, etc., could have equally well been used.
DHCP is used to assign client machine IP addresses and provide other boot-time information. TFTP (the Internet Trivial File Transfer Protocol, RFC 1350 [IETF 92]) is used by the ROM boot code to load a dynamic boot file. Clients of UDP include Video Transport, DCE RPC, the network time service, and SNMP. TCP is also used.
2.4 Other System Attributes
All textual data in the ITV system is represented using 16-bit Unicode[Unicode 91] characters, both on the head-end servers and on the client machines. This was done to allow the same code to be used for both European and Asian language deployments, with only changes of resource and configuration files, eliminating the need for "internationalized" code to handle multi-byte character encodings for non-European languages.
All textual data is displayed on the set-top box via a TrueType font renderer and a small set of scalable TrueType fonts that were specially designed for good readability on television screens.
3. Set-Top Box Hardware
The set-top box used in this trial was designed by Microsoft and manufactured by NEC. It uses a standard PC backplane, a 90 MHz Pentium processor (but clocked at 75 MHz), and a PCI bus. An NEC PCI ATM card is being used in the trial. Unlike a PC, the set-top box contains no disk, keyboard, mouse, or BIOS.
Custom video and audio hardware for the set-top box contains a MPEG-2 decoder, NTSC (the U.S. and Japanese analog television encoding standard) encoders & decoders, a tuner, and an audio mixer. A custom graphics chip called the Burma is capable of dynamically scaling and alpha blending (semi-transparently overlaying) multiple video and computer-generated graphics surfaces using different pixel representations into a single, flicker-filtered output image in real time. (Flicker filtering reduces the flicker associated with NTSC's interlaced display and slow refresh rate.)
The set-top box also has a bi-directional infrared port for communicating with the hand controller, a smart card interface, a serial port (used for debugging), a microphone input, auxiliary audio & video inputs, and separate audio and video outputs for TV and VCR.
The processor has an 8Ki/8Kd on-chip cache. There is no second level cache. The system was designed for 8MB of RAM, although it was typically used with 16MB and was eventually deployed with 24MB (more about this in section 12.4). The graphics system uses 2MB of RAMBUS memory, plus the MPEG-2 decoder contains 2MB of RAM. The system has 1/2 MB of boot ROM.
3.1 Burma Graphics Chip
The Burma graphics chip is central to Microsoft's interactive TV set-top box. In particular, it provides us with the capability of combining real-time video with dynamic computer-generated graphics under programmatic control.
The primary job of the Burma is to dynamically composite sets of video and computer-generated images into an output image. Images are represented as lists of spans, where a span is a horizontal line of pixels. Since spans can be of different lengths and have different origins, Burma images need not be rectangular.
Pixels can be represented in these data formats: 8-bit palletized color, 8-bit palletized color plus 8-bit alpha value, 16-bit Red-Green-Blue (RGB) (5R:6G:5B), 32-bit Luminance-Chrominance (YCrCb) pixel pairs (8-bit Y0, 8-bit Y1, 8-bit shared Cr, 8-bit shared Cb), 24-bit RGB, and 24-bit RGB plus 8-bit alpha. Those formats without per-pixel alpha values have per-span alpha values. Flicker filtering can be controlled on a per-span basis.
Major functional blocks within the Burma include two video capture engines, a color space converter, a static composition engine (a blitter) with source transparent color checking, a dynamic composition engine (performs alpha blending), a flicker filter, video output, a PCI interface with pre-fetcher and a RAMBUS interface.
The Burma was designed to be clocked at 62.5MHz and is actually clocked at 50MHz. The 8-bit RAMBUS channel was designed to be clocked at 250MHz, and was actually clocked at 200MHz (4 × 50 MHz). RAMBUS transfers occur on both up and down transitions, giving an ultimate achievable speed of 500Mbytes/second or a BLT rate of 250Mbytes/second. In practice, about 50% of that is achievable with the Burma.
The Burma chip is implemented as a custom ASIC fabricated using a .35 micron process. The chip size is 8.68 × 8.68 mm. It contains 252K total gates, or roughly 1 million transistors. The logic is 140K gates, plus 56Kbits of RAM.
3.2 Infrared Hand Controller
A remote control device was custom designed and built for the MITV system. Along with the usual number keys and up/down channel and volume buttons and the standard VCR control Play, Stop, Pause, Fast-Forward, and Rewind buttons, it also has some special features designed for interactive television. One of these is a circular disk that can be depressed in any direction to indicate movement; for instance, mouse-style pointing can be implemented using this control. The Menu and Action buttons are for making and confirming choices. A Help button is provided. The A and B buttons can be used in application-specific ways. Finally, an ID button is used during the set-top box login sequence.
Figure 3-1: Hand Controller
4. The MMOSA Real-Time Kernel
The MMOSA real-time kernel was developed as a cooperative effort between members of the operating systems research group within Microsoft Research and a team of development staff committed to building a small, embedded-systems kernel meeting the needs of interactive television. MMOSA is the internal name for the ITV kernel: it stands for MultiMedia Operating System Architecture.
Besides meeting the development team's needs, the kernel has also been used for investigating research issues in consumer real-time computing systems. As a research vehicle, the kernel is called Rialto (see [Jones et al. 96], [Draves et al. 97], [Draves & Cutshall 97], and [Jones et al. 97]). In practice, MMOSA and Rialto share a common code base.
MMOSA is portable and is designed for both space and time efficiency. Versions run on Mips boards, 486 and Pentium PCs, and our Pentium-based set-top box.
MMOSA provides processes with protected address spaces and multiple threads per process. The kernel is itself a distinguished process and much of its code is mapped into other processes, resulting in significant code savings due to the ability to use kernel routines in user processes; in effect, the kernel serves as a shared library for all other processes. All functions available to user processes are also available in the kernel, allowing code to be developed in user processes and later run in the kernel completely unmodified, if so desired.
Synchronization between threads is provided via standard Mutex and Condition Variable objects [ISO 96]. They are implemented in such as way as to provide a generalization of priority inheritance [Sha et al. 90], preventing priority inversions from occurring.
Adaptive deadline-based real-time scheduling is accomplished using Time Constraints. See [Jones et al. 96] and [Jones et al. 97] for a description.
Most services in MMOSA are invoked via object invocation. The kernel directly supports optimized cross-process object invocation, with a null RPC round trip taking 51 microseconds on a 90 MHz Pentium workstation. Server threads are created in the destination process by the kernel on an as-needed basis to run RPC requests, and are cached and reused upon return.
MMOSA employs a novel virtual memory architecture that is both space efficient and highly preemptible. It supports address translation, sparse virtual address spaces, memory mapping, pages that are created upon first use in a controlled fashion, and growable kernel thread stacks. See [Draves et al. 97] for further discussion of the MMOSA VM architecture.
MMOSA makes extensive use of dynamic loading, and in fact, the linkage to MMOSA system calls appears to client programs to be the same as to any other dynamic linked library (although it is handled differently by the kernel). MMOSA supports the sharing of DLLs between kernel and user space.
Other features of MMOSA include a flexible local namespace in which services are registered, and simple stream and random-access I/O interfaces, used among other things, for program loading and device support.
Native MMOSA device drivers are written using the same system calls and interfaces as normal code. The one additional piece of functionality available to drivers is a special form of the Condition_Signal() synchronization primitive that may be called from an interrupt context. As a rule, MMOSA drivers are expected to return from interrupts promptly and to do any additional work needed from within a thread context so as to minimize the effect on scheduling latency.
MMOSA supports a serial-line based debugging protocol permitting remote source-level debugging. Finally, it implements a DLL called KernView that is knowledgeable about many kernel data structures, and may be dynamically loaded to examine them. The data structure values are exported via the namespace in a fashion allowing various state parameters of individual processes and threads to be easily examined.
5. Set-Top Box Device Support
As described above, driver code for native MMOSA drivers is written using the same system calls and interfaces as other code. Many of them can be run as user processes, although typically they are loaded into the kernel.
Native drivers are deployed on the set-top box for the Burma, the MPEG-2 decoder card, the sound chip, the hand controller port, the serial lines, the digital joystick port, and the front panel.
Other native drivers exist for PC devices, including the VGA, the keyboard, serial and bus-based mice, the floppy controller, hard disks, several ISA and PCI Ethernet cards, several sound cards, and the ET4000 MPEG-1 video card.
In addition to native MMOSA drivers, a wrapper layer has also been written using MMOSA primitives to allow binary Windows NT NDIS network miniport device drivers to be used under MMOSA without modification.
Miniport drivers used include a DEC PCI Ethernet card, the Intel EtherExpress Pro, the Intel EtherExpress 16, a Fore Systems PCI ATM card, and an NEC PCI ATM card.
6. Win32e and Other Middleware
In early 1995 a decision was made to implement a small subset of the Win32 interfaces on top of MMOSA and to write set-top box applications to that interface. This subset is called Win32e (where the "e" stands for "embedded"). This was done for several reasons:
This subset contains approximately 18% of the 1300 Win32 APIs. It includes only Unicode interfaces (no single-byte character interfaces). The set chosen was based on profiles of the actual APIs used by existing multimedia titles and games, with redundant calls being eliminated. The user interface and graphics calls were explicitly tailored to a television environment and not to a general-purpose workstation.
Win32 kernel abstractions implemented included processes, threads, synchronization objects (critical sections, events, semaphores), resource loading, time and locale functions, and memory management. Notably absent are filesystem calls.
The window I/O functions implemented are compatible with Win32, but optimized for an ITV input model. The hand controller replaces both the mouse and the keyboard. Elements such as window decorations, icons, the clipboard, keyboard support, and double clicking have been dropped. Windows, window messages, Z-order management, clipping, activation, and a small set of rendering operations are provided. BLT'ing, TrueType font rendering, and region filling are supported; StretchBlt, alpha blending, diagonal lines, pens, curves, paths, polygons, and bitmap fonts are not.
6.2 DirectX Implementations
High performance computer-generated graphics and sound are supported via an implementation of the DirectDraw and DirectSound interfaces. These implementations are compatible with those deployed with Windows 95. DirectX clients and servers communicate via COM object interfaces using MMOSA lightweight RPC.
DirectDraw consists of a core, device-independent layer and a device-dependent hardware abstraction layer (HAL). MMOSA HALs were implemented for the VGA, ET4000 (an MPEG-1 decoder chip), and Burma display devices. The Win32e graphics functions are implemented on and integrated with the DirectDraw system.
DirectSound provides support for a variety of software audio formats including Pulse Code Modulation (PCM) (in which sound is represented as a series of discrete numbers at regular intervals) and both Microsoft and Interactive Multimedia Association (IMA) Adaptive Differential Pulse Code Modulation (ADPCM) formats (in which each sample is encoded as a difference from the previous sample). It allows efficient, low latency (25 ms) software mixing of up to 10 PCM streams and analog mixing of PCM, ADPCM, and MPEG audio sources. It implements per-buffer, per-channel, and master volume controls. Both static and streaming buffers are supported.
6.3 ITV User Interface Components
User interface via a television screen and a remote control is different in a number of ways than traditional computer interfaces. The display resolution is reasonably poor, no keyboard exists, pointing is difficult, controls need to be visually attractive, and input focus must be apparent even in the presence of live video. A substantial user interface effort went into designing and implementing on-screen visual controls tailored to this environment.
The result of this effort was a set of scalable tiled user interface components requiring only limited memory that are realizable using textured bitmaps. A tabbed input model is used, with animated focus being provided around the active control. Unlike traditional Windows controls, ITV user interface controls can be placed on a DirectDraw surface and alpha blended. In fact, several of the applications superimpose translucent controls over live video.
Figure 6-1: Scalable ITV UI button using tiled texture bitmaps and luminance masks
Programmatically, all controls look like their desktop brethren with some additional properties and messages. Visual customization is accomplished using tiled textures and compressed luminance masks, supporting reuse. The bitmaps for controls and dialogs reside on the Class Store and are downloaded as needed.
Controls implemented include buttons, spin dials, list boxes, a grid control, a progress indicator, edit boxes, and an audio/video control (in which video is displayed).
6.4 COM and Distributed Systems Support
By mid 1993 the decision had been made to use a Distributed Component Object Model (DCOM) [Microsoft 94] framework as a basis for building a reliable, scalable distributed interactive television system. This decision allows for location-transparent object invocation, as is fitting for an environment with both diskless and stateful computing nodes.
At the core of our object strategy is a distributed implementation of the COM object system and a number of the core OLE2 system objects such as IStream, IStorage, IPropertyStorage, and IMoniker. This supports transparent object creation and invocation for all of in-process objects, cross-process objects on the same machine, and remote objects. Any needed dynamic link libraries (DLLs) are automatically loaded and parameters for remote calls are transparently marshalled. Remote object invocation calls are sent using an extended DCE RPC implementation.
Implementations of remote objects are obtained by the COM implementation using a Class Store service. Objects are retrieved using numeric 128-bit class-ID names, although string names are also supported for non-object resources such as programs, font files, and bitmaps.
Also hosted on DCE RPC is an extensible distributed namespace. The namespace is used by client code for locating and connecting to a number of potentially replicated services at boot time and application start time. Data and services stored in the namespace include per-customer and per-set-top box data, the electronic program guide service, the customer service application service, logging services, and financial transaction services.
An OLE Database (OleDB) client is also present, allowing applications to perform queries and other operations on remote databases. OleDB is implemented using remoted COM interfaces. Remote databases include the broadcast program guide information and the video-on-demand movie database.
6.5 Purchase Mediator
An important client-side service called the Purchase Mediator allows applications to have customers securely pay for services rendered using a standard payment user interface. One important function of the purchase mediator is to isolate the merchant from the customer's wallet. It constructs the intersection of the set of payment methods accepted by the merchant and available to the customer, lets the customer choose a payment method, and then authorizes the transaction.
Purchases are made using journalled, signed digital certificates, using an STT/SET-like protocol. (STT and SET [SETCo 97] are industry-standard electronic payment protocols.) This provides for non-repudiation of confirmed transactions. Financial transactions are discussed more in section 8.5.
Clockwork is the modular multimedia stream control architecture used for processing and synchronizing continuous media streams. Streams are processed using active special-purpose "filters". Stream data flows are transformed by and transmitted between filters in the filter graph. Tight media synchronization can be maintained between and within media streams. Clockwork supports server-push streaming, which is the model used by Tiger.
Filters initially used in the NTT trial include:
Other filters implemented include those for MPEG-1, AVI video (Audio/Video Interleaved, a set of Microsoft multimedia formats), including RLE (Run-Length Encoding), Video1, and Indeo (an Intel video format), and both PCM and ADPCM audio filters.
Clockwork supports both software- and hardware-based rendering filters.
7. ITV Applications
The set-top box applications provide the end-user experience. This section describes the key applications.
The Navigator application serves a function roughly analogous to a shell or a window manager in traditional character-based or window-based systems. It is always present on the set-top box, although is usually not visible. It implements most cross-application user interface functionality such as channel changing (which may result in new application object invocations), master volume control, audio track selection, and status inquiries.
The Navigator is responsible for set-top box management functions such as user login and soft power down / warm boot. It also implements a simple reminder service.
Figure 7-1: Navigator screen shot (U.S. version)
7.2 Electronic Program Guide
The Electronic Program Guide (EPG) application provides an interactive TV guide. It displays program lineup and episode information for current and future broadcast programming choices, displays the current channel contents in a window, and allows you to select a broadcast channel. It accesses the program guide database from head-end servers using OleDB.
Figure 7-2: EPG screen shot (U.S. version)
7.3 Movies On Demand
The Movies On Demand (MOD) application is used to choose, order, and play stored video content on demand. Video data is sourced from Tiger[Bolosky et al. 96, Bolosky et al. 97] video servers. VCR-like controls, including pause, fast-forward, and rewind are provided. The time between confirming a video purchase and video play is on the order of two seconds.
Figure 7-3: MOD screen shot (U.S. version)
In the trial, the MOD application is actually used for four different "channels": a movie-on-demand channel, a sports-on-demand channel, a cooking-on-demand channel, and an animation-on-demand (cartoon) channel. NTT charged customers between ¥300 (~$3.00) and ¥600 (~$6.00) per movie, and also charged for the sports and cartoon channels. The cooking channel was free.
7.4 Other ITV Applications
An additional application deployed in the trial is the customer service application, which was written by NTT. It provides for subscriber logon, channel lockout, billing queries, and other administrative functions.
A Karaoke-on-demand application was written by NTT. This is essentially the MOD application with the addition of audio mixing from the microphone input. NTT also wrote an on-line shopping application.
A version of the Internet Explorer web browser was also ported by Microsoft to the set-top box but was not deployed in the trial since NTT did not perceive significant demand for a web browser at that time.
8. Head-End Services
The fifteen server machines in the NTT ITV head-end are all NEC SV98 PCs with 90MHz Pentium processors and 80MB of memory. They are running Japanese Windows NT Server 3.51, plus various services specific to interactive TV, which are described in this section.
8.1 Tiger Video Fileserver
Tiger is a distributed, fault-tolerant real-time fileserver. It provides data streams at a constant, guaranteed rate to a large number of clients, in addition to supporting more traditional filesystem operations. It forms the basis for multimedia (video on demand) fileservers, but may also be used in other applications needing constant rate data delivery. Tiger efficiently balances user load against limited disk, network and I/O bus resources. It accomplishes this balancing by striping file data across all disks and all computers in the distributed Tiger system, and then allocating data streams in a schedule that rotates across the disks. For a more thorough description of Tiger see [Bolosky et al. 96] and [Bolosky et al. 97].
The Tiger system deployed in the NTT trial uses one dedicated Tiger controller server and ten dedicated Tiger Cub servers (the machines across which the data are striped). Each cub has a single NEC OC-3 (155Mbit/s) ATM adapter and 12 Seagate Elite 9GB wide disks (ST410800W), formatted with 2K sectors. Each of these disks has 9400MB of storage, for a system total of 1.08TB. With half this capacity used for mirroring and 6Mbit/s MPEG-2 video, that gives 208.9 hours of video storage. Each disk is theoretically capable of supplying 3.7MB/s, of which 3.3MB/s is primary, with the remainder reserved for covering for other, failed disks. Practically, the bandwidth limit on the system is actually imposed by the ATM cards -- not the disks. The system as configured uses a rotating cycle of 127 one-second schedule slots, supporting a theoretical limit of 127 simultaneous viewers, and a practical limit of about 100 simultaneous viewers that can play videos with a reasonable (few second) start latency.
8.2 Class Store
The Class Store service provides a general repository both for objects needed by both set-top boxes and COM-based services on head-end machines. It provides EXE files and DLLs when COM objects are activated. COM object implementations and proxy/stub DLLs are stored in the Class Store and downloaded at run time by COM. In the remote case, these objects are sent using DCE RPC over UDP. It supports multiple versions, dependencies, different client hardware and software architectures, and multiple locales.
The Class Store server also provides several other services. It provides boot images for set-top boxes via an integrated TFTP server. It allows font files, bitmaps, and binaries to be downloaded by name. It also implements a network time server.
8.3 Directory Services
Another core component of the MITV distributed computing environment is a distributed, scalable, extensible directory service. The namespace is fully distributed, with no central database; its contents are maintained by a collection of cooperating services running on each participating node.
The directory service allows multiple namespaces to be mounted under junction points, providing a basis for locating fault-tolerant replicated servers. (Junction points are similar to the "union directories" used in the 3D Filesystem [Korn & Krell 90] and the Translucent Filesystem [Hendricks 90].) Namespace replication was used in a similar way in the Time-Warner Orlando trial [Nelson et al. 95]. General COM objects can be registered at any point in the namespace.
Clients of the directory service include the DCOM implementation, Integrated Subscriber Services, Financial Transaction Services, the Electronic Program Guide, the NTT Customer Service Application, and logging services.
8.4 Interactive Subscriber Services
The Interactive Subscriber Services (ISS) service serves as a repository for context-specific data needed by remote applications. It resides within the general directory services namespace. Within the its namespace, type-specific objects are registered supporting property types, property values, and data type enforcement. This is implemented using a general schema engine. The runtime schema is compiled from schema information stored in an Access database.
Data stored within ISS includes subscriber, viewer, set-top box, and merchant specific data. For instance, the last channel watched and ratings lockouts are stored on a per-viewer basis. Subscriber location data (address, etc.) is stored here. Set-top box specific data used at boot time such as an identifier for the initial viewer is also kept here.
8.5 Financial Transaction Services
The Financial Transaction Services (FTS) provide support for subscriber billing, both for regular service and for explicit purchases made by viewers. An important (if unglamorous) part of this job is providing a standard way of interfacing to credit card providers and Billing and Subscriber Management Systems (BSMSs) -- legacy billing systems used by cable companies. This is done by encapsulating the BSMS functionality behind a standard DCOM object running on the head-end servers. This object translates between the native data formats and access methods used by the BSMS and the data formats expected by the remote Purchase Mediator interface and data presented to applications by ISS.
It is worth pointing out that the master copy of much of the subscriber data actually resides in the BSMS. For instance, the subscriber name and billing address almost certainly resides there, in which case the FTS and ISS servers must ensure whatever consistency is necessary between the BSMS data and that in ISS.
FTS is designed to support a variety of secure on-line payment methods using digitally signed, journalled payment certificates. It uses an STT-like network data representation, although it is not real STT since when we had to commit the code in November 1995 STT was not sufficiently complete.
Finally, unlike real STT, the network data is not actually encrypted for the NTT trial. We had originally planned to use public/private key pairs stored on set-top box smart cards. When plans to use smart cards for the trial were shelved late in the development cycle due to software development time constraints, a fallback plan was put into place where the network data representation has the same information content as an STT certificate, but is not encrypted, allowing real encryption to be easily added after the trial. This was deemed an appropriate tradeoff since the network for the NTT trial is closed.
8.6 OleDB and SQL Server Database
The head-end servers contain several SQL server databases. The two primary databases are the electronic program guide data and the video-on-demand title data. Set-top boxes make remote queries to these databases as object invocations to an OLE database (OleDB) server.
The ability to perform arbitrary queries allows for a much richer content browsing experience than would otherwise be possible. For instance, it allows one to look for movies by title, actors, director, genre, and other diverse criteria.
9. Network Support
Several interesting pieces of network development work were done to support interactive television. This section discusses them.
9.1 MITV Network Requirements
The MITV system is designed to be network transport independent. All communication between ITV applications is done using IP datagrams. In addition to the ability to host IP, the other property that MITV depends upon is the ability to provision dedicated bandwidth between communicating parties. These requirements can be met over a number of link and physical layers including ATM, ADSL (Asynchronous Digital Subscriber Line, a high-speed data transmission technology using standard telephone wires), HDSL (High-speed Digital Subscriber Line (another similar technology using standard phone lines), hybrid fiber/coax (systems running fiber to neighborhoods and coaxial cable the final distance into the home), and even dedicated switched Ethernet or FDDI. In our lab we have built and run Ethernet, FDDI, and ATM-based systems.
9.2 NTT Trial Network
The particular network deployed by NTT is an ATM-over-fiber network, with OC-3 (155Mbit/s) links to set-top boxes and MITV server machines, and an OC-12 (622Mbit/s) link to a separate digital broadcast server providing real-time MPEG-2 encoded video for all regular cable broadcast channels. The digital broadcast system sends groups of 26 188-byte MPEG transport stream packets in raw ATM AAL-5 [CCITT 93] packets. This is the only non-IP-based network component of the NTT system, and is specific to this deployment.
9.3 Connection-Oriented NDIS
Windows NT network card drivers have typically been built using the Network Device Interface Standard (NDIS) port/miniport model. This model allows the hardware vendor to write only the hardware-specific portions of the device driver (the miniport), with the common components being shared (the port).
Quality-of-service (QoS) -based and connection-oriented network technologies (such as ATM) introduce a new level of complexity. These require signaling protocols (such as ATM UNI 3.1) and adaptation protocols (such as IP over ATM or LAN emulation). Traditionally, network interface card hardware vendors have been faced with the daunting task of including software to support these protocols within the hardware specific miniport driver.
As part of the interactive television work, and in cooperation with the Windows NT networking group, we developed NDIS extensions for networks that are connection oriented and/or offer quality-of-service (QoS) guarantees (such as ATM). These extensions allow the modularity of the miniport model to be extended to the more complex networks required in the ITV environment. These interface extensions allow both signaling modules (Call Managers in NDIS 4.1 terms), such as UNI 3.1 (the ATM standard signalling protocol) or SPANS (an earlier proprietary ATM signalling protocol by Fore Systems), and protocol layers, such as IP-over-ATM (IP/ATM, a superset of RFC-1577 [IETF 94]) or LAN Emulation, to be provided that are independent of the miniport driver. This frees the network interface card vendor to focus on developing hardware, and allows us to refine our signaling and IP/ATM protocol modules without being constrained to specific hardware.
Common binary miniport drivers for both Ethernet and ATM devices are used on both Windows NT and MMOSA. Co-NDIS shipped with Windows NT 4.0.
9.4 IP/ATM and UNI 3.1
Two kernel software modules provided with the operating system are the IP/ATM layer and the UNI 3.1 Call Manager. These operate together to manage the ATM Virtual Circuits (VCs) underlying ATM-based MITV implementations. IP/ATM operates at a layer in the protocol stack equivalent to the ARP layer. Our implementation is a superset of RFC-1577. IP/ATM uses the services of the Call Manager to provide virtual circuits between various endpoints at different fixed bandwidths and priorities depending upon the type of traffic to be carried between those endpoints. Unlike traditional IP/ATM (which does not exploit the QoS guarantees of ATM networks), our IP/ATM uses heuristics to channel IP packets to appropriate VCs.
For instance, in the NTT system, IP/ATM uses 10Mbit/second VCs for large data downloads from head-end servers to client machines. The return path is 1Mbit/second, which is sufficient for the lower volume control traffic sent from client machines towards the head-end servers. IP/ATM classifies submitted packets according to priority and implied bandwidth requirements, and may use more than one VC between the same two physical endpoints, to carry different classes of traffic.
In addition to IP/ATM, the Video Transport uses the Call Manager to obtain VCs appropriate for carrying the constant bit rate video streams. These are provisioned at 7Mbits/second (sufficient to carry continuous 6Mbit/second traffic, plus a bit of extra to leave gaps between blocks transmitted by different cubs).
All traffic in the NTT trial is carried as Constant Bit Rate (CBR) or Unspecified Bit Rate (UBR) traffic. We did not consider Variable Bit Rate (VBR) to offer significant benefits in bandwidth efficiency or reliability. Available Bit Rate (ABR) would have been ideal for carrying the bursty IP traffic, however, true ABR support in hardware is only just beginning to be viable.
Video traffic and certain control traffic (with strict latency requirements) is carried on CBR VCs. All other traffic is carried on UBR VCs.
In the NTT environment, certain elements internal to the ATM network do not provide Switched Virtual Circuit (SVC) service. As a result, Permanent Virtual Circuits (PVCs) must be provisioned in the network, in advance. These are described to the Call Manager on each host by computer-generated PVC files. The Call Manager doles these VCs out to IP/ATM and the Video Transport as required.
In environments in which SVC service is available, the Call Manager sets up and tears down VCs dynamically as required by IP/ATM and the Video Transport. Note that software layers above the Call Manager, including IP/ATM, the Video Transport and all higher protocol layers, are isolated from the underlying SVC or PVC nature of the network.
9.5 ATM Funnel
Tiger data for the NTT trial is sent over special multipoint-to-point ATM virtual circuits supported by a number of switch vendors. Each Tiger cub successively sends data over different virtual circuits to the switch, with the data leaving the switch all being merged onto a single virtual circuit connected to the client.
This eliminates the need to establish separate virtual circuits between each cub/client pair for every video stream, reducing the VC count in this case by a factor of 10. It is the responsibility of the senders to prevent interleaved sends.
On non-ATM transports, on which point-to-multipoint connections are not possible, Tiger cubs instead send to intermediate "redirector" machines that combine inputs from multiple cubs into one output stream.
9.6 Transport-Independent Network Protocols
The primary transport-independent network protocol used by MITV is UDP [IETF 80]. Distributed services, including remote object invocations, are built using DCE RPC over UDP. Tiger uses custom UDP-based protocols for transmitting video data and for communicating among the Tiger cubs.
Set-top box network addresses and other boot-time parameters are obtained via DHCP [IETF 93]. SNMP (Simple Network Management Protocol, RFC 1157 [IETF 90], an Internet host management standard) is used for a few set-top box management functions such as remote rebooting and logging control.
TCP [IETF 81] is used by Tiger control flows and for set-top box debugging via Telnet shell windows.
10. Set-Top Box Boot Sequence
Set-top boxes boot as follows. First, a small boot loader in EPROM is run. This decompresses a boot image also in the ROM and jumps to its entry point. The ROM boot image contains the MMOSA kernel, the network protocol stack and drivers, and a network boot program. The network boot program sends a DHCP request. It waits for a DHCP reply containing not only the usual IP address, but also some MITV-specific extended results such as sets of IP addresses of Class Store servers and a boot filename to use. It then loads a new boot image from a Class Store via TFTP and jumps to it.
Like the ROM boot image, the dynamic boot image contains MMOSA and networking code, but also contains a larger set of services and data that will always be resident, such as the Win32e, DirectDraw, and Navigator implementations, and the TrueType font files. This image also sends a DHCP request to establish its IP address, and waits for a reply containing extended results. In particular, the second boot uses results telling the set-top box what directory server to use and what its machine name is. It then connects with the directory server, looks up other data using the machine name, and is soon fully initialized, ready for user input in about twenty seconds.
Like many consumer devices, the set-top-box and software supports a soft power down / warm boot feature using the Power button on the remote control. Soft power down turns off the display and frees some resources, but leaves software running to respond to a soft power-on request. Soft power-on takes about five seconds -- about the same time required for the TV screen to warm up.
11. System Integration
Before a complex hardware and software system with many separately developed components can be successfully deployed, the components must be assembled and tested as a whole. Many cross-component problems or mismatches will likely only likely be discovered during system integration. Understanding its potential complexity and pitfalls, we aggressively pursued early and realistic system integration.
To do system integration for the NTT trial we constructed a lab at Microsoft containing a 1/4 scale replica of the actual system to be deployed in Yokosuka. Special power equipment provided Japanese power (100V, 50Hz) to the lab. Everything from server machines and ATM switches to real-time MPEG-2 encoders and OS software was precisely duplicated.
Having the lab was critical for developing the NTT-specific portions of the system. For instance, broadcast streams were sent using a different transport protocol, and requiring very different code paths, than stored videos. Only during system integration, did we learn that decoding broadcast streams would occupy so much of the CPU that other activities would respond sluggishly, necessitating some last-minute tuning work.
Other components requiring substantial integration work were the financial transaction and billing systems. In this case, NTT needed to test their code to interface their legacy billing system to the MITV financial code.
12. Project Scope and Measurements
From its initial conception in early 1993 through the NTT deployment in March, 1996 the project grew enormously. What started as the whiteboard sketches of a core team of about a dozen individuals from the Advanced Technology and Research groups eventually grew to an effort with a dedicated staff of approximately 207 people, including:
86 developers (including 6 hardware engineers)
35 program managers
23 user education (documentation) people
8 product support people
4 marketing people specifically for the NTT trial
3 administrative assistants
2 systems support engineers
plus a multitude of others, such as marketing staff, who played key roles in the ITV effort, but who also had other concurrent responsibilities elsewhere in the company. Conservatively, the Microsoft ITV effort took at least 400 man-years on Microsoft's part through March, 1996.
12.2 Source Code and Tools
Another measure of the complexity of the project is the amount of data involved. For instance, as of the December 1995 developer kit release, the source tree contained 28,705 files, with 862 megabytes of source code, documentation, and support tools. The source code alone occupies 135 megabytes, and contains 4.7 million lines of code, almost all of it C or C++.
12.3 Deployed Binaries
The dynamically loaded set-top box boot file used for the NTT deployment is 4.6 megabytes. The largest contributors to this (in kilobytes) are as follows:
|Kanji TrueType font||
|DCE RPC implementation||
|ITV user interface controls||
|Win32 graphics output server code||
|Win32 user input server code||
|IP protocol stack||
|Distributed COM (DCOM) code||
|NDIS driver support code||
|Directory services code||
|Win32 user input client code||
|Visual C++ runtime code||
|Win32 kernel client code||
|Roman TrueType font||
|MPEG port driver||
|File lock manager||
|Network boot code||
|Burma device driver||
|MPEG API support code||
|NT kernel emulation for NDIS||
|ITV help services||
|Win32 kernel server code||
|Command line shell||
Altogether, the boot file contained 8 executable images, 46 dynamically linked libraries (DLLs), 5 drivers (which are actually also DLLs), 2 font files, and 5 data files.
After boot, libraries, applications, and fonts available for dynamic loading to the client contain an additional 5.8 megabytes (some of which is duplicated in the bootfile, and so consequently never downloaded). Some highlights of this set that are used, with sizes in kilobytes, are:
|Sixty-one bitmaps (31K and 1K sizes)||
|Tiger network client code||
|Tiger stream synchronization filter||
|Microsoft foundation classes (MFC)||
|Movie play application||
Finally, on top of Japanese Windows NT 3.51 on the server machines, an additional 9.5 megabytes of ITV-specific server files were deployed on the servers. Highlights of this include:
|Tiger controller and cub files||
|COM objects running on servers||
|Twenty-seven SQL database files||
12.4 Set-Top Box Memory Usage
The following table gives typical memory usage figures in megabytes for a set-top box playing a movie. The static memory figures represent memory that will be in use for all application scenarios. Dynamic memory is allocated on a demand basis and may be freed upon application exit. Finally, the Burma memory figure represents the component's use of special Burma chip RAMBUS video memory.
Size in MBytes
|Kernel, network, shell||0.40||0.80||0|
|Win32e kernel, COM, Class Store, C runtime||0.54||0.31||0|
|Win32e graphics I/O, DirectX, hand controller||0.65||0.18||0.29|
|ITV interface controls||0.14||0.29||0|
|Typical OS plus video player application||4.88||7.13||1.65|
Thus, for video-on-demand, roughly 12 megabytes out of 24 megabytes of system memory are in use, along with 1.65 out of 2 megabytes of the video memory.
By far, the two largest consumers of set-top-box memory are the Kanji font and video buffers. Both of these warrant further discussion.
The Kanji font requires approximately 3 megabytes of memory due to the large number of complex characters that can be represented in Japanese. Ideally this would have been stored in ROM. Unfortunately, when the set-top box was designed, only enough ROM address lines were provided for 1/2 megabyte of ROM. While useful for the compressed initial boot file, this was insufficient for any later purposes. This problem would likely have been rectified in a follow-on set-top box design.
As deployed, MITV plays MPEG-2 videos encoded at 6 megabits per second. Our Tiger servers are configured to deliver video in one second buffers. Thus, each buffer requires 0.75 megabytes. The MPEG playback system as implemented keeps four such buffers, or 3MB of buffering for playback of stored video alone.
The ITV developers understand how they could reduce this buffering by a substantial factor, and in fact, had built systems using far less buffering in the lab. Nonetheless, it was decided to deploy the NTT trial with the older, memory-intensive playback code in order to eliminate one possible risk factor in meeting our schedule.
One other significant user of memory is the buffering for playback of broadcast video streams received via MPEG transport. This is sent completely differently than stored video, and uses different code paths. The Navigator, which implements broadcast stream playing, allocates four 64Kbyte buffers for demultiplexing interleaved audio and video tracks into elemental streams, using another 256Kbytes of dynamic memory.
The Microsoft set-top box was designed with an initial memory target of 8MB. Due to several factors, most of them related to our ship schedule, it became clear that something between 12 and 16 megabytes would be actually be required, and so developers were given set-top boxes with 16 megabytes. All development and most testing were done with this configuration. For the trial, NTT actually deployed each set-top box with 24 megabytes each. Even though this much memory was not necessary for the initial applications deployed, this gave them the flexibility to deploy larger applications in the future without doing hardware upgrades in their customer's homes.
13. Project Time Lines
|Overall ITV Project|
|Early 93||ITV project conception|
|Jun 94||Tiger announcement|
|Apr 95||Redmond ITV trial begins|
|Jun 95||First ITV software developer's kit released|
|Dec 95||Second developer's kit released|
|Mar 96||Yokosuka trial begins|
|Mar 97||Microsoft ITV project cancelled|
|Apr 97||Yokosuka trial concludes|
|Mar 93||Tiger conception|
|Jul 93||Disk scheduling demo for Bill Gates|
|Sep 93||MPEG-1 video|
|Nov 93||AVI video over Ethernet demo for Bill Gates|
|May 94||NCTA: two dozen clients, ATM, 5 cubs|
|May 95||NCTA: 25 clients off of single machine on 100MB Ethernet|
|Early 93||Initial design decisions|
|Dec 93||Fast COM RPC in modified Windows NT|
|Jan 94||Real-time scheduling modifications to NT|
|Feb 94||Booted native MMOSA on Mips|
|Jun 94||Booted native MMOSA on PC|
|Oct 94||Modified Windows NT running MMOSA binaries|
|May 95||Booted from disk on set-top box prototype|
|Jul 95||Network boot on PC|
|Aug 95||Network boot on set-top box prototype|
|Summer 93||Began set-top box design|
|Spring 94||Burma design|
|Fall 94||Choice of NEC as manufacturing partner|
|Nov 94||Decision to use x86 processor|
|Feb 95||Burma tape-out|
|May 95||First set-top box prototypes arrive|
|Jun 95||Burma chips arrive|
|Jul 95||Burma fully exercised, declared alive|
|Other Notable Dates|
|Jul 94||DCOM implementation begun|
|Feb 95||Decision made to build Win32e|
|Apr 95||Initial Win32e running on MMOSA|
|Jul 95||DCOM running on Win32E|
|Aug 95||ATM on MMOSA|
|Sep 95||1/4 scale version of NTT hardware in lab|
|Sep 95||MPEG-1 video from MMOSA PC disk|
|Oct 95||Navigator running on MMOSA|
|Nov 95||MPEG-2 video from ATM on set-top box|
|Feb 95||DirectSound working|
|Feb 95||Digital broadcast stream working in lab|
|Jan 97||EPG date calculation bug occurred|
14. Key Decisions
This section discusses some of the decisions that were pivotal to the shape and outcome of MITV.
15. War Stories
This section relates a number of experiences we had along the way...
16. Technical and Business Perspective
ITV was a technical success, both in Microsoft's and our competitor's trials, but it was a business failure.
The Microsoft/NTT ITV trial in Yokosuka, Japan was successfully operated between March, 1996 and April, 1997 for 298 paying subscribers. Customers used applications written both by Microsoft and NTT. During the deployment only one user-visible bug occurred and it caused no unscheduled downtime. Following the successful conclusion of the trial, NTT discontinued providing ITV services for customer use but kept a smaller demonstration version of the system running.
Since early 1993, the landscape for interactive TV has changed. While then widespread deployment looked likely, even by early 1996 when the trial began it was becoming clear that access providers would not be able to justify the costs of installing high-bandwidth data lines into people's homes based on the revenues expected from ITV. (This was referred to as the "last mile" problem.) Furthermore, many of the applications, such as home shopping, originally envisioned for ITV had begun to appear on the Internet. No, the online catalogs could not show video clips of models modelling the clothing for sale, as envisioned for ITV, but web pages with text and static images were nonetheless proving "good enough" to gain consumer acceptance. Popular momentum had clearly shifted from ITV to the World Wide Web.
Based on this business assessment, having completed the work needed for the Yokosuka trial, Microsoft cancelled its Interactive TV project in March, 1996, retargetting many pieces of it to related projects such as our web server, web browser, web audio and video, electronic commerce, and hand-held devices efforts.
Despite the apparent death of Interactive TV, we nonetheless believe that nearly all the capabilities originally planned for ITV will be available to consumers within the next few years -- although they will be called The Internet. While the high-bandwidth digital pipes into people's homes that will enable true video-on-demand are not yet there (the "TV" part of Interactive TV), market forces are clearly working towards providing them in response to demand for faster Internet connectivity. And the Internet is already a (very) high-volume open authoring and content delivery platform (the "Interactive" part of ITV), making this goal a reality far more quickly than the most optimistic schedules in anyone's ITV plans.
17. Conclusions and Evaluation
The Microsoft Interactive TV team met stringent deadlines, shipping a working, fault-tolerant product that provides a compelling user experience and for which it is easy to develop new applications and content. Of the many goals outlined for the project, only that of hitting a cost-effective price point for widespread deployment failed to be met. The reasons for this are plain.
The competitive fervor of the early ITV efforts put tremendous perceived time-to-market pressures on all parties involved. When decisions had to be made between building a working system on schedule but over ideal cost targets, and keeping unit costs small but being late, the choice was clear, particularly for small initial trials with only hundreds of customers.
If development were to have continued on the system, the next job would have been to revisit the places that compromises were made that raised per-subscriber costs, making it a leaner, more cost-effective system. We understand how to reduce many of the costs, such as video buffer memory, quite quickly and dramatically.
Like that of any large system, the actual development path for MITV contained both significant successes and many false starts. Remaining flexible and being willing to change courses in mid-stream in several areas was essential to our successes. Like a free market, real software development on a large scale can be somewhat chaotic. But in practice, this "chaos" seems to be the essential alternative to centrally planned, rigid software architectures.
Some specific lessons and observations from the MITV system:
It is my hope in writing this description of the architecture and development of the Microsoft Interactive TV system that readers will gain two things: an understanding of the particular technical choices and designs employed by Microsoft to implement an interactive television system, and a better appreciation for the scope and complexity (and the fun!) of developing real commercial software systems.
Yoram Bernet, Bill Bolosky, Scott Cutshall, Mike Daly, Kirt Debique, Steve DeBroux, Craig Dowell, Rich Draves, Bob Fitzgerald, Alessandro Forin, Bob Fries, Angus Gray, Johannes Helander, Julian Jiggins, Ted Kummert, Debi Mishra, Robert Nelson, Gilad Odinak, Steve Proteau, Rick Rashid, David Robinson, Anthony Short, Jim Stuart, James Stulz, and Brian Zill all provided valuable input for this paper and were some of the key people involved in shipping a working ITV system on schedule.
[Bolosky et al. 96] William J. Bolosky, Joseph S. Barrera, III, Richard P. Draves, Robert P. Fitzgerald, Garth A. Gibson, Michael B. Jones, Steven P. Levi, Nathan P. Myhrvold, Richard F. Rashid. The Tiger Video Fileserver. In Proceedings of the Sixth International Workshop on Network and Operating System Support for Digital Audio and Video, Zushi, Japan. IEEE Computer Society, April, 1996.
[Bolosky et al. 97] William J. Bolosky, Robert P. Fitzgerald, and John R. Douceur. Distributed Schedule Management in the Tiger Video Fileserver. In Proceedings of the 16th ACM Symposium on Operating Systems Principles, Saint-Malo, France, October, 1997.
[CCITT 93] CCITT, Draft Recommendation I.363. CCITT Study Group XVIII, Geneva, 19-29 January 1993.
[Draves et al. 97] Richard P. Draves, Gilad Odinak, Scott M. Cutshall. The Rialto Virtual Memory System. Technical Report MSR-TR-97-04, Microsoft Research, February, 1997.
[Draves & Cutshall 97] Richard P. Draves and Scott M. Cutshall. Unifying the User and Kernel Environments. Technical Report MSR-TR-97-10, Microsoft Research, March, 1997.
[IETF 80] J. Postel. User Datagram Protocol (UDP). Internet Engineering Task Force Request for Comments (RFC) 768, August 1980. http://www.ietf.org/rfc/rfc768.txt.
[IETF 81] J. Postel. Transmission Control Protocol (TCP). Internet Engineering Task Force Request for Comments (RFC) 793, September, 1981. http://www.ietf.org/rfc/rfc793.txt.
[IETF 90] J.D. Case, M. Fedor, M.L. Schoffstall, C. Davin. A Simple Network Management Protocol (SNMP). Internet Engineering Task Force Request for Comments (RFC) 1157, May, 1990. http://www.ietf.org/rfc/rfc1157.txt.
[IETF 92] K. Sollins. Trivial File Transfer Protocol (TFTP). Internet Engineering Task Force Request for Comments (RFC) 1350, July, 1992. http://www.ietf.org/rfc/rfc1350.txt.
[IETF 93] R. Droms. Dynamic Host Configuration Protocol (DHCP). Internet
Engineering Task Force Request for Comments (RFC) 1531, October, 1993.
[IETF 94] M. Laubach. Classical IP and ARP over ATM. Internet Engineering Task Force Request for Comments (RFC) 1577, January, 1994. http://www.ietf.org/rfc/rfc1577.txt.
[ISO 96] ISO/IEC 9945-1: Information technology -- Portable Operating System Interface (POSIX) -- Part 1: System Application Program Interface (API) [C Language], IEEE, July, 1996.
[Jones et al. 96] Michael B. Jones, Joseph S. Barrera III, Alessandro Forin, Paul J. Leach, Daniela Rosu, Marcel-Catalin Rosu. An Overview of the Rialto Real-Time Architecture. In Proceedings of the Seventh ACM SIGOPS European Workshop, Connemara, Ireland, pp. 249-256, September, 1996.
[Jones et al. 97] Michael B. Jones, Daniela Rosu, Marcel-Catalin Rosu. CPU Reservations and Time Constraints: Efficient, Predictable Scheduling of Independent Activities. In Proceedings of the 16th ACM Symposium on Operating Systems Principles. Saint-Malo, France, October, 1997.
[Korn & Krell 90] David G. Korn and Eduardo Krell. A New Dimension for the Unix File System. Software -- Practice and Experience. 20(S1):19-34, June, 1990.
[Hendricks 90] David Hendricks. A Filesystem For Software Development. In Summer USENIX Conference Proceedings, pp. 333-340. June, 1990.
[Laursen et al. 94] Andrew Laursen, Jeffrey Olkin, Mark Porter. Oracle Media Server: Providing Consumer Based Interactive Access to Multimedia Data. In Proceedings of the 1994 ACM SIGMOD International Conference on Management of Data, Minneapolis, MN, pp. 470-477. May, 1994.
[SETCo 97] Secure Electronic Transactions Specification. SET Secure Electronic Transaction LLC, 1997.
[Microsoft 94] OLE2 Programmer's Reference, Volume One. Microsoft Press, 1994.
[Mitchell et al. 97] Joan L. Mitchell, William B. Pennebaker, Chad E. Fogg, Didier J. LeGall. MPEG Video Compression Standard. Digital Multimedia Standards Series, J. L. Mitchell and W. B. Pennebaker, Editors. Chapman and Hall, New York, 1997.
[Nelson et al. 95] Michael N. Nelson, Mark Linton, Susan Owicki. A Highly Available, Scalable ITV System. In Proceedings of the 15th ACM Symposium on Operating Systems Principles, Copper Mountain, CO, pp. 54-67. December, 1995.
[Sha et al. 90] L. Sha, R. Rajkumar, and John Lehoczky. Priority Inheritance Protocols: An Approach to Real-time Synchronization. In IEEE Transactions on Computers, vol. 39, no. 3, pp. 1175-1198, 1990.
[Unicode 91] The Unicode Standard: Worldwide Character Encoding. Unicode Consortium, Reading, MA, 1991.