Language ID for a Thousand Languages

Fei Xia, Carrie Lewis, and William Lewis

Abstract

ODIN, the Online Database of INterlinear text, is a resource built over language data harvested from linguistic documents (Lewis, 2006). It currently holds approximately 190,000 instances of Interlinear Glossed Text (IGT) from over 1100 languages, automatically extracted from nearly 3000 documents crawled from theWeb. A crucial

step in building ODIN is identifying the languages of extracted IGT, a challenging task due to the large number of languages and the lack of training data. We demonstrate that a coreference approach to the language ID task significantly outperforms existing algorithms as it provides an elegant solution to the unseen language problem. We also discuss several issues that make automated Language ID and the maintenance of ODIN very difficult.

Details

Publication typeInproceedings
Published ineLanguage, LSA Annual Meeting Extended Abstracts
PublisherLinguistics Society of America
> Publications > Language ID for a Thousand Languages