River System
Preliminary
version of 12/00
Tobias Mayr and Jim Gray
During the Fall 2000 we developed the basis of a river
system that allows the parallelization of existing application based on
streaming exchange of record data.
The preliminary river system was built in two steps:
- First,
we implemented the basic one-to-one byte stream communication mechanisms,
forwarding fixed-size buffers using asynchronous IO between nodes across
the network and to and from disk. We did a thorough performance study on
this and documented the effort in “Performance
of the On-to-One Data Pump”.
- In a
second step, we implemented the mechanisms for moving records across the
byte streams and for merging and partitioning such record streams. This,
together with a conceptual overview of the whole effort is documented in “River Design”
The completion of the launch
mechanism and the XML specifications for rivers remain future work. The
document above has sections on these issues and how we intend to solve them.
To get access to the existing
code please contact Jim
or Tobias.
Back to Jim Gray’s home page.