Reservation-based Scheduling: If You’re Late Don’t Blame Us!

  • ,
  • Djellel E Difallah ,
  • Chris Douglas ,
  • Subru Krishnan ,
  • Rahgu Ramakrishnan ,
  • Sriram Rao

MSR-TR-2013-108 |

Published in SoCC 2014

Publication | Publication

The continuous shift towards data-driven approaches to business, and a growing attention to improving return on investments (ROI) for cluster infrastructures is generating new challenges for big-data frameworks. Systems, originally designed for big batch jobs, now handle an increasingly complex mix of computations. Moreover, they are expected to guarantee stringent SLAs for production jobs and minimize latency for best-effort jobs.

​In this paper, we introduce reservation-based scheduling, a new approach to this problem. We develop our solution around four key contributions: 1) we propose a reservation definition language (RDL) that allows users to declaratively reserve access to cluster resources, 2) we formalize planning of current and future cluster resources as a Mixed-Integer Linear Programming (MILP) problem, and propose scalable heuristics, 3) we adaptively distribute resources between production jobs and best-effort jobs, and 4) we integrate all of this in a scalable system named Rayon, that builds upon Hadoop / YARN.

We evaluate Rayon on a 256-node cluster, against workloads derived from Microsoft, Yahoo!, Facebook, and Cloudera’s clusters. To enable practical use of Rayon, we hardened our system and open-sourced it as part of Apache Hadoop.