Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Language ID in the Context of Harvesting Language Data off the Web

Fei Xia, William Lewis, and Hoifung Poon

Abstract

As the arm of NLP technologies extends beyond a small core of languages, techniques for working with instances of language data across hundreds to thousands of languages may require revisiting and recalibrating the tried and true methods that are used. Of the NLP techniques that has been treated as “solved” is language identification (language ID) of written text. However, we argue that language ID is far from solved when one considers input spanning not dozens of languages, but rather hundreds to thousands, a number that one approaches when harvesting language data found on the Web. We formulate language ID as a coreference resolution problem and apply it to a Web harvesting task for a specific linguistic data type and achieve a much higher accuracy than long accepted language ID approaches.

Details

Publication typeInproceedings
Published inProceedings of the 12th Conference of the European Chapter of the ACL (EACL-2009)
PublisherAssociation for Computational Linguistics
> Publications > Language ID in the Context of Harvesting Language Data off the Web