Language ID in the Context of Harvesting Language Data off the Web

As the arm of NLP technologies extends beyond a small core of languages, techniques for working with instances of language data across hundreds to thousands of languages may require revisiting and recalibrating the tried and true methods that are used. Of the NLP techniques that has been treated as “solved” is language identification (language ID) of written text. However, we argue that language ID is far from solved when one considers input spanning not dozens of languages, but rather hundreds to thousands, a number that one approaches when harvesting language data found on the Web. We formulate language ID as a coreference resolution problem and apply it to a Web harvesting task for a specific linguistic data type and achieve a much higher accuracy than long accepted language ID approaches.

final.pdf
PDF file

In  Proceedings of the 12th Conference of the European Chapter of the ACL (EACL-2009)

Publisher  Association for Computational Linguistics
Association for Computational Linguistics

Details

TypeInproceedings
> Publications > Language ID in the Context of Harvesting Language Data off the Web