Practical Performance Models for Complex, Popular Applications

Perhaps surprisingly, no practical performance models exist for popular (and complex) client applications such as Adobe’s Creative Suite, Microsoft’s Office and Visual Studio, Mozilla, Halo 3, etc. There is currently no tool that automatically answers program developers’, IT administrators’ and end-users’ simple what-if questions like “what happens to the performance of my favorite application X if I upgrade from Windows Vista to Windows 7?”. This paper describes our approach towards constructing practical, versatile performance models to address this problem. The goal is to have these models be useful for application developers to help expand application testing coverage and for IT administrators to assist with understanding the performance consequences of a software, hardware or configuration change. This paper’s main contributions are in system building and performance modeling. We believe we have built applications that are easier to model because we have proactively instrumented them to export their state and associated metrics. This application-specific monitoring is always on and interesting data is collected from real, "in-the-wild" deployments. The models we are experimenting with are based on statistical techniques. They require no modifications to the OS or applications beyond the above instrumentation, and no explicit a priori model on how an OS or application should behave. We are in the process of learning from models we have constructed for several Microsoft products, including the Office suite, Visual Studio and Media Player. This paper presents preliminary findings from a large user deployment (several hundred thousand user sessions) of these applications that show the coverage and limitations of such models. These findings pushed us to move beyond averages/means and go into some depth into why client application performance has an inherently large variance.

PDF file

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SIGMETRICS’10, June 14–18, 2010, New York, New York, USA. Copyright 2010 ACM 978-1-4503-0038-4/10/06 ...$10.00.


> Publications > Practical Performance Models for Complex, Popular Applications