Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Mining Dependency in Distributed Systems through unstructured log analysis

Jian-Guang LOU, Qiang FU, Yi WANG, and Jiang LI

Abstract

Dependencies among system components are crucial to locating root errors in a distributed system. In this paper, we propose an approach to mine inter-component dependencies from unstructured logs. The technique requires neither additional system instrumentation nor any application specific knowledge. In the approach, we first parse each log message into its log key and parameters. Then, we find dependent log key pairs belong to different components by leveraging co-occurrence analysis and parameter correspondence. After that, we use Bayesian decision theory to estimate the dependency direction of each dependent log key pair. We further apply time delay consistency to remove false positive detections. Case studies on Hadoop show that the technique successfully identifies the dependencies among the distributed system components.

Details

Publication typeProceedings
Published inthe 2nd Workshop on Analysis of System Logs (selected as BEST PAPER and published in SIGOPS OS Review)
PublisherAssociation for Computing Machinery, Inc.
> Publications > Mining Dependency in Distributed Systems through unstructured log analysis