Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Extracting Parallel Sentences from Comparable Corpora using Document Level Alignment

Jason R. Smith, Chris Quirk, and Kristina Toutanova

Abstract

The quality of a statistical machine translation (SMT) system is heavily dependent upon the amount of parallel sentences used in training. In recent years, there have been several approaches developed for obtaining parallel sentences from non-parallel, or comparable data, such as news articles published within the same time period (Munteanu and Marcu, 2005), or web pages with a similar structure (Resnik and Smith, 2003). One resource not yet thoroughly explored is Wikipedia, an online encyclopedia containing linked articles in many languages. We advance the state of the art in parallel sentence extraction by modeling the document level alignment, motivated by the observation that parallel sentence pairs are often found in close proximity. We also include features which make use of the additional annotation given byWikipedia, and features using an automatically induced lexicon model. Results for both accuracy in sentence extraction and downstream improvement in an SMT system are presented.

Details

Publication typeInproceedings
Published inHuman Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL
PublisherAssociation for Computational Linguistics
> Publications > Extracting Parallel Sentences from Comparable Corpora using Document Level Alignment