Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Parallelizing large-scale data processing applications with data skew: a case study in product-offer matching

Ekaterina Gonina, Anitha Kannan, John Shafer, and Mihai Budiu

Abstract

The last decade has seen a surge of interest in large-scale data-parallel processing engines. While these engines share many features in common with parallel databases, they make a set of different trade-offs. In consequence many of the lessons learned for programming parallel databases have to be re-learned in the new environment. In this paper we show a case study of parallelizing an example large-scale application (offer matching, a core part of online shopping) on an example MapReduce-based distributed computation engine (DryadLINQ). We focus on the challenges raised by the nature of large data sets and data skew and show how they can be addressed effectively within this computation framework by optimizing the computation to adapt to the nature of the data. In particular we describe three different strategies for performing distributed joins and show how the platform language allows us to implement optimization strategies at the application level, without system support. We show that this exibility in the programming model allows for a highly effective system, providing a measured speedup of more than 100 on 64 machines (256 cores), and an estimated speedup of 200 on 1280 machines (5120 cores) of matching 4 million offers.

Details

Publication typeInproceedings
Published inInternational Workshop on MapReduce and its Applications (MapReduce) 2011
> Publications > Parallelizing large-scale data processing applications with data skew: a case study in product-offer matching