Dryad is a high-performance general-purpose distributed computing engine that is designed to simplify the task of implementing distributed applications on clusters of Windows-based computers. DryadLINQ allows developers to implement Dryad applications in managed code by using an extended version of the LINQ programming model and API.
Academic Release Now Available
The Academic Release includes everything you need to start developing DryadLINQ applications on a Windows HPC Server 2008 cluster, including programming guides and code samples. Send your feedback by visiting the DryadLINQ community site on Microsoft Connect.
Try it today: Download the Academic Release of Dryad and DryadLINQ
Learn More
- Dryad and DryadLINQ in Depth
- About LINQ (Language-Integrated Query)
Background: Dryad and DryadLINQ
The scientist's challenge
You have all the data for an important scientific problem. You just need to analyze it. With terabytes of data, you need a powerful data processing application in order to have results to talk about at the upcoming national meeting. The analysis might be straightforward in principle, but actually doing it is going to be tough.
For many important scientific investigations, efficiently analyzing large data sets is a major challenge. For example, astronomers use the Sloan Digital Sky Survey to investigate problems such as the distribution of “dark matter” around distant galaxies. The current data set—SDSS Data Release 7—covers more than a quarter of the sky and contains more than 50 TB of data representing 357 million unique objects.
Your best bet is to create a distributed application that runs on a cluster of relatively inexpensive networked PCs. However, implementing such an application is a non-trivial task: distributed applications must manage numerous threads, allocate resources across numerous individual multicore computers, handle hardware failures, and so on. Writing the code could take months, and you are a scientist, not a programming expert.
As an alternative to writing all the code yourself…
Microsoft Dryad is a high-performance, general-purpose distributed computing engine that handles some of the most difficult aspects of cluster-based distributed computing. It's powerful: Microsoft routinely uses Dryad applications to analyze petabytes of data on clusters of thousands of computers.
But Dryad applications are not easy to implement. To further simplify things, Microsoft has developed DryadLINQ, which allows developers to use an extended version of the LINQ programming model and API to implement Dryad applications. DryadLINQ code is similar to what you'll see in a conventional LINQ application, and the application core is often only a few lines of code. Behind the scenes though, a DryadLINQ provider automatically converts the LINQ query into a Dryad job and executes the query as a distributed application on a cluster.
With DryadLINQ and a Dryad cluster…
Even a novice at parallel processing or cluster-based computing can implement a high-performance distributed application to efficiently analyze terabytes of data. As an example, consider one time-consuming problem: Q18 of the Sloan Digital Sky Survey, which searches the data set for possible gravitational lenses:
Q18: Find all objects within 1' of one another that have similar colors, where the color ratios u-g, g-r, r-I are less than 0.05m. Magnitudes are logarithms, so these differences correspond to ratios.
To address this problem, Microsoft researchers used DryadLINQ to run the query on a 40-node Dryad cluster consisting of 40 off-the-shelf networked Windows-based computers. Dryad took about an hour to install.
- The query itself is a three-way join over two input tables, one with 11 GB of data and the other with 41.8 GB.
- To perform the query, the team used Microsoft Visual Studio and a standard Windows-based workstation to implement aDryadLINQ application that consists of approximately 100 lines of Microsoft Visual C# code.
- The team manually distributed the data across the cluster and ran the application from the workstation. The DryadLINQ provider set up the Dryad job and ran the query on the cluster.
The results came back in under two minutes—not even enough time for a quick cup of coffee.
Tools and Services for Researchers
- Tools and Services Home
- Tools for Computational Science
- Database, HPC, and Cloud Services for Scientists
- Managing Scientific Workflows
- Publication and Conference Management for Scientists
- Software Tools for Computational Science
- Tools for Web Search Research
DryadLINQ Documentation
- An Introduction to Dryad and DryadLINQ
- Dryad and DryadLINQ Installation Guide
- DryadLINQ Programming Guide
Papers, Presentations, and Articles
- Dryad/DryadLINQ: Composable Large-Scale Applications
- Dryad/DryadLINQ Overview: Data Deluge in Scientific Research
- DryadLINQ: Programming Model for Sorting, Filtering, Preprocessing
- Cluster Computing with Dryad
- Machine Learning in DryadLINQ
- DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language
Contact Us
To ask technical questions, to report bugs or to make suggestions, please visit the DryadLINQ community site on Microsoft Connect.
For general inquiries, please contact us at dryadrel@microsoft.com.
