A Comparative Study of Parameter Estimation Methods for Statistical Natural Language Processing

Jianfeng Gao, Galen Andrew, Mark Johnson, and Kristina Toutanova

Abstract

This paper presents a comparative study of five parameter estimation algorithms on four NLP tasks. Three of the five algorithms are well-known in the computational linguistics community: Maximum Entropy (ME) estimation with L2 regularization, the Averaged Perceptron (AP), and Boosting. We also investigate ME estimation with L1 regularization using a novel optimization algorithm, and BLasso, which is a version of Boosting with Lasso (L1) regularization. We first investigate all of our estimators on two re-ranking tasks: a parse selection task and a language model (LM) adaptation task. Then we apply the best of these estimators to two additional tasks involving conditional se-quence models: a Conditional Markov Model (CMM) for part of speech tagging and a Conditional Random Field (CRF) for Chinese word segmentation. Our experiments show that across tasks, three of the estimators — ME estimation with L1 or L2 regularization, and the Averaged Perceptron — are in a near statistical tie for first place.

Details

Publication typeInproceedings
Published inAnnual Meeting of the Association for Computational Linguistics (ACL)
PublisherAssociation for Computational Linguistics
> Publications > A Comparative Study of Parameter Estimation Methods for Statistical Natural Language Processing