Towards Pragmatic Library-based Replay

Deterministic replay is a powerful approach for debugging multithreaded and distributed applications that involve complex nondeterminism. Recently, several replay tools based on user-space libraries were proposed. Such library-based tools are lightweight and efficient, and focus on the application being debugged. However, several significant problems remain, such as in-address-space tool interference, absence of support for advanced features in modern operating systems (e.g. asynchronous I/O), and error-prone engineering efforts for wrapping a large set of API functions to enable logging and replaying. We propose three mechanisms to cope with these problems. First, we resolve the tool interference problem by space separation, using a virtual execution layer composed of intercepted functions. The virtual execution layer isolates both memory and execution of the application from supporting libraries and in-address-space replay tool, hence systematically removes the interference. Second, based on an event-view of the execution, we are able to handle advanced operating features such as asynchronous I/O and exceptional control flow (e.g. signal). Third, instead of manually coding the wrappers, we use an annotation-aware code generator so that the process of producing the wrappers is more robust and efficient. We have handled more than 750 API functions and the code generator is responsible for more than 37,000 lines of wrapper code. To the best of our knowledge, these three mechanisms, space separation, event-based replay, and annotation-aware code generation are novel for library-based deterministic replay. We have successfully replayed many large and complex software packages with low overhead, including both server and distributed system, including those developed by ourselves and by the community. MySQL and Apache, for instance, experienced around 10% slowdown.