Using the Penn Treebank to Evaluate Non-Treebank Parsers

This paper describes a method for conducting evaluations of Treebank and non-Treebank parsers alike against the English language U. Penn Treebank (Marcus et al., 1993) using a metric that focuses on the accuracy of relatively non-controversial aspects of parse structure. Our conjecture is that if we focus on maximal projections of heads (MPH), we are likely to find much broader agreement than if we try to evaluate based on order of attachment. We hope that this method may find wider acceptance and be useful in establishing a generally applicable framework for evaluation in natural language parsing. We employ this method in an evaluation of NLPWin (Heidorn, 2000), a parser developed at Microsoft Research without reference to the Penn Treebank, and, for comparison, the well-known statistical Treebank parser of Charniak (2000).

PDF file
PostScript file

Publisher  European Language Resources Association
Printed / Distributed with the permission of ELRA. This paper was published within the proceedings of the LREC'2004 Conference. © 2004 ELRA - European Language Resources Association. All rights reserved.


> Publications > Using the Penn Treebank to Evaluate Non-Treebank Parsers