Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Thread detection in dynamic text message streams

Dou Shen, Qiang Yang, Jian-Tao Sun, and Zheng Chen

Abstract

Text message stream is a newly emerging type of Web data which is produced in enormous quantities with the popularity of Instant Messaging and Internet Relay Chat. It is beneficial for detecting the threads contained in the text stream for various applications, including information retrieval, expert recognition and even crime prevention. Despite its importance, not much research has been conducted so far on this problem due to the characteristics of the data in which the messages are usually very short and incomplete. In this paper, we present a stringent definition of the thread detection task and our preliminary solution to it. We propose three variations of a single-pass clustering algorithm for exploiting the temporal information in the streams. An algorithm based on linguistic features is also put forward to exploit the discourse structure information. We conducted several experiments to compare our approaches with some existing algorithms on a real dataset. The results show that all three variations of the single-pass algorithm outperform the basic single-pass algorithm. Our proposed algorithm based on linguistic features improves the performance relatively by 69.5% and 9.7% when compared with the basic single-pass algorithm and the best variation algorithm in terms of F1 respectively.

Details

Publication typeInproceedings
Published inSIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
URLhttp://doi.acm.org/10.1145/1148170.1148180
Pages35–42
ISBN1-59593-369-7
AddressNew York, NY, USA
PublisherACM
> Publications > Thread detection in dynamic text message streams