River System

Preliminary version of 12/00

Tobias Mayr and Jim Gray

 

During the Fall 2000 we developed the basis of a river system that allows the parallelization of existing application based on streaming exchange of record data.

 

The preliminary river system was built in two steps:

  1. First, we implemented the basic one-to-one byte stream communication mechanisms, forwarding fixed-size buffers using asynchronous IO between nodes across the network and to and from disk. We did a thorough performance study on this and documented the effort in “Performance of the On-to-One Data Pump”.
  2. In a second step, we implemented the mechanisms for moving records across the byte streams and for merging and partitioning such record streams. This, together with a conceptual overview of the whole effort is documented in “River Design”

 

The completion of the launch mechanism and the XML specifications for rivers remain future work. The document above has sections on these issues and how we intend to solve them.

 

To get access to the existing code please contact Jim or Tobias.

 

Back to Jim Gray’s home page.