Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Sriram Rao

PARTNER SCIENTIST MANAGER
.

I lead the Cloud and Information Services Lab at Microsoft.  The lab comprises of two sub-groups: Systems and Machine Learning.  As a group, we do innovative research by building production quality systems, publishing papers in top conferences, and contributing code to open source projects.  We work closely with Microsoft's Big Data teams.  The systems sub-group in particular focuses on systems infrastructure aspects in the "Big data" space.

On a personal side, I am broadly interested in building storage and compute infrastructure for datacenter settings.    I enjoy building and deploying systems in practice as well as releasing them as open source.  In building these systems, my work leverages upon technology trends in datacenter computing.

My recent work has been in the area of resource management for Big data clusters.  We have focused on a building a scale-out resource management substrate for big-data workloads.  While the ideas are general, we have implemented our ideas on top of Apache Hadoop YARN:

  • Mercury (Hybrid Centralized/Distributed Scheduling; also, see YARN-2877)
  • Rayon (Rayon ships as part of Apache Hadoop 2.6; see YARN-1051)
  • Tetris (Packing tasks of Big data jobs to improve cluster efficiency)
  • Corral (Network-aware scheduling of Big data jobs)

Some of the previous systems I have built and released as open source projects are:

  • Kosmos distributed filesystem: I have designed/implemented/deployed  (KFS) to manage PB's of storage. 
  • Sailfish: I have also designed/implemented Sailfish, a compute infrastructure which improves handling of intermediate data (i.e., "shuffle" phase in a Map-Reduce computation).  Our results show that Sailfish can improve job completion times at scale by 20% to 5x.  

At CISL, I am working on building Hadoop related services on Windows Azure. I also collaborate extensively with colleagues in MSR-Redmond.

A full list of my publications is here.

Publications