Multilingual Summarization Evaluation
http://research.microsoft.com/~lucyv/MSE2006.htm
The 2nd Multilingual Summarization Evaluation will be held in conjunction with the COLING-ACL 2006 Workshop "Task-focused Summarization and Question Answering", July 23, 2006, and the results of the evaluation will be reported during the COLING-ACL Workshop. This evaluation repeats the first Multilingual Summarization Evaluation held in 2005 as part of the ACL workshop "Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization", and is similar to Task 4 in DUC 2004 with a few changes.
Important dates:
April 17th - May 10th Apply for participation
May 17th Test data available from LDC
June 2nd Submissions due to lucyv@microsoft.com for evaluation
June 21st Evaluation results returned to participants
July 5th Papers due (to be posted on this website)
Instructions for joining:
Interested participants should send an email message to Lucy Vanderwende at lucy.vanderwende@microsoft.com by May 10th. Your email must include:
contact name
site (organization) name
contact email address
contact phone number
shipping address
Upon receipt of your message, we will send you the LDC user agreement, which you should complete and fax to LDC. LDC will then send download instructions in 1-2 days.
After May 10th, a discussion list will be set up for all participants. If you do not want to be on the discussion list, please include that information in your email message to lucy.vanderwende@microsoft.com.
Task Description:
Given a cluster of documents on the same event, some in English, some translated from Arabic (Arabic source is also available), generate a 100-word summary of the event. Clusters contain on average 10 documents per cluster. The distribution between Arabic and English varies between clusters.
Evaluation:
We will use ROUGE exclusively, with the settings recommended by DUC 2006.
Sites will be allowed up to 3 submissions.
Data:
25 clusters from the Multilingual Summarziation Evaluation 2005 are available for training, as well as the DUC2004 data for Task 4, available at http://duc.nist.gov.
25 clusters will be used for testing. These clusters were created by running a clustering algorithm developed by Columbia over the the TDT4 corpus, which contains 41,728 Arabic documents and 23,602 English documents. ISI's MT system was used to translate the Arabic data. Both source and translation are available in the cluster. Human annotators at the LDC sorted through the automatically created clusters, to select 50 (25 clusters for last year, 25 clusters for this year) that were good to use, editing the clusters as needed. Four humans wrote a 100-word summary for each cluster. Thus, there are 4 model summaries per cluster.
Organizers:
Jade Goldstein, U.S. Department of Defense
Lucy Vanderwende, Microsoft Research
Liang Zhou, USC/ISI