Distributed systems are becoming key engines of IT companies for data centric computing. However, it is very hard to diagnose and improve the performance of a distributed system. Through analyzing the performance of a distributed system step by step, we demonstrate a series of technologies, including inspecting a running distributed system, non-intrusively aligning machine time, creating system snapshot, calculating running cost, digging process relationship and application topology, and detecting bottleneck.