The computation core of many data-intensive applications can be best expressed as matrix computations. The MadLINQ project addresses the following two important research problems: the need for a highly scalable, efficient and fault-tolerant matrix computation system that is also easy to program, and the seamless integration of such specialized execution engines in a general purpose data-parallel computing system.

MadLINQ exposes a unified programming model to both matrix algorithm and application developers. Matrix algorithms are expressed as sequential programs operating on tiles (i.e., sub-matrices). For application developers, MadLINQ provides a distributed matrix computation library for .NET languages. Via the LINQ technology, MadLINQ also seamlessly integrates with DryadLINQ, a data-parallel computing system focusing on relational algebra.

The system automatically handles the parallelization and distributed execution of programs on a large cluster. It outperforms current state-of-the-art systems by employing two key techniques, both of which are enabled by the matrix abstraction: exploiting extra parallelism using fine-grained pipelining and efficient on-demand failure recovery using a distributed fault-tolerant execution engine. We describe the design and implementation of MadLINQ and evaluate system performance using several real-world applications.

}, author = {Zhengping Qian and Xiuwei Chen and Nanxi Kang and Mingcheng Chen and Yuan Yu and Thomas Moscibroda and Zheng Zhang}, booktitle = {EuroSys 2012}, month = {April}, publisher = {ACM}, title = {MadLINQ: Large-Scale Distributed Matrix Computation for the Cloud}, url = {http://research.microsoft.com/apps/pubs/default.aspx?id=158914}, year = {2012}, }