Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Lexical Co-occurrence, Statistical Significance, and Word Association

Dipak L. Chaudhari, Om P. Damani, and Srivatsan Laxman

Abstract

Lexical co-occurrence is an important cue for detecting word associations. We propose a new measure of word association based on a new notion of statistical significance for lexical co-occurrences. Existing measures typically rely on global unigram frequencies to determine expected co-occurrence counts. Instead, we focus only on documents that contain both terms (of a candidate word-pair) and ask if the distribution of the observed spans of the word-pair resembles that under a random null model. This would imply that the words in the pair are not related strongly enough for one word to influence placement of the other. However, if the words are found to occur closer together than explainable by the null model, then we hypothesize a more direct association between the words. Through extensive empirical evaluation on all publicly available benchmark data sets, we show the advantages of our measure over existing co-occurrence measures.

Details

Publication typeInproceedings
Published inProceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP 2011), Edinburgh, UK
PublisherACL/SIGPARSE
> Publications > Lexical Co-occurrence, Statistical Significance, and Word Association