A Comparative Study of Parameter Estimation Methods for Statistical Natural Language Processing

This paper presents a comparative study of five parameter estimation algorithms on four NLP tasks. Three of the five algorithms are well-known in the computational linguistics community: Maximum Entropy (ME) estimation with L2 regularization, the Averaged Perceptron (AP), and Boosting. We also investigate ME estimation with L1 regularization using a novel optimization algorithm, and BLasso, which is a version of Boosting with Lasso (L1) regularization. We first investigate all of our estimators on two re-ranking tasks: a parse selection task and a language model (LM) adaptation task. Then we apply the best of these estimators to two additional tasks involving conditional se-quence models: a Conditional Markov Model (CMM) for part of speech tagging and a Conditional Random Field (CRF) for Chinese word segmentation. Our experiments show that across tasks, three of the estimators — ME estimation with L1 or L2 regularization, and the Averaged Perceptron — are in a near statistical tie for first place.

gao07comparative.pdf
PDF file

In  Annual Meeting of the Association for Computational Linguistics (ACL)

Publisher  Association for Computational Linguistics
All copyrights reserved by ACL 2007

Details

TypeInproceedings
> Publications > A Comparative Study of Parameter Estimation Methods for Statistical Natural Language Processing