Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Seaweed: Scalable Distributed Querying

Traditional solutions to unavailability rely on replicating data, which fundamentally limits scalability. In Seaweed we take a different approach: delay-aware querying. Data is queried in-situ as and when it becomes available and incremental results are provided to the user, thus trading delay for result completeness without sacrificing scalability. Seaweed also generates completeness predictors which tell the user what percentage of relevant data is on unavailable machines, and when results from that data will become available, thus providing with the user with a "progress bar" for each query.

Research Areas

Seaweed: Scalable Distributed Querying

The aim of this project is to build a scalable infrastructure for querying large and highly distributed data sets, spread over thousands to millions of nodes. For example, querying performance and failure statistics in a data center; endsystem-based network monitoring in a large enterprise network; and collection of crash and failure statistics from Windows machines worldwide: all these require a way to query the data stored on a large number of endsystems while meeting the twin challenges of scalability and endsystem unavailability.

Traditional solutions to unavailability rely on replicating data, which fundamentally limits scalability. In Seaweed we take a different approach: delay-aware querying. Data is queried in-situ as and when it becomes available and incremental results are provided to the user, thus trading delay for result completeness without sacrificing scalability. Seaweed also generates completeness predictors which tell the user what percentage of relevant data is on unavailable machines, and when results from that data will become available, thus providing with the user with a "progress bar" for each query.

Current Status

We have a working prototype of Seaweed, and have tested its scalability and the accuracy of completeness prediction in simulation.

Project Members

Publications