The Workshop Program is now available! (http://research.microsoft.com/~lucyv/WS7_program.txt)
Task-Focused Summarization and Question Answering, a COLING/ACL 2006 workshop
http://research.microsoft.com/~lucyv/WS7.htm
Sydney, Australia
July 23, 2006
Multilingual Summarization Evaluation
http://research.microsoft.com/~lucyv/MSE2006.htm
Workshop Description
This one-day workshop will focus on the challenges that the Summarization and
QA communities face in developing useful systems and in developing evaluation
measures. Our aim is to bring these two communities together to discuss the
current challenges and to learn from each other's approaches, following the
success of a similar workshop held at ACL-05, which brought together the
Machine Translation and Summarization communities.
A previous summarization workshop (Text Summarization Branches Out, ACL-04)
targeted the exploration of different scenarios for summarization, such as
small mobile devices, legal texts, speech, dialog, email and other genres.
We encourage a deeper analysis of these, and other, user scenarios, focusing
on the utility of summarization and question answering for such scenarios and
genres, including cross-lingual ones.
By focusing on the measurable benefits that summarization and question
answering has for users, we hope one of the outcomes of this workshop will be
to better motivate research and focus areas for summarization and question
answering, and to establish task-appropriate evaluation methods. Given a user
scenario, it would ideally be possible to demonstrate that a given evaluation
method predicts greater/lesser utility for users. We especially encourage
papers describing intrinsic and extrinsic evaluation metrics in the context of
these user scenarios.
Both summarization and QA have a long history of evaluations: Summarization
since 1998 (SUMMAC) and QA since 1999 (TREC). The importance of summarization
evaluation is evidenced by the many DUC workshops; in DUC-05, extensive
discussions were held regarding the use of ROUGE, ROUGE-BE, and the pyramid
method, a semantic-unit based approach, for evaluating summarization systems.
The QA community has related evaluation issues for answers to complex
questions such as the TREC definition questions. Some common considerations
in both communities include what constitutes a good answer/response to an
information request, and how does one determine whether a "complex" answer is
sufficient? In both communities, as well as in the distillation component of
the 2005 DARPA program GALE, researchers are exploring how to capture semantic
equivalence among components of different answers (nuggets, factoids or SCUs).
There also have been efforts to design new automatic scoring measures, such as
ROUGE-BE and POURPRE. We encourage papers discussing these and other metrics
that report on how well the metric correlates with human judgments and/or
predicts effectiveness in task-focused scenarios for summarization and QA.
This workshop is a continuation of ACL 2005 for the summarization community,
In which those interested in evaluation measures participated in a joint
Workshop on evaluation for summarization and MT. As a sequel to the ACL 2005
workshop, in which the results of the first Multilingual multi-document
summarization evaluation (MSE) were presented
(http://www.isi.edu/~cyl/MTSE2005/MLSummEval.html),
we plan to report and discuss the results of the 2006 MSE evaluation.
In summary, we solicit papers on any or all of the following three topics:
- Task-based user scenarios requiring question answering
(beyond factoids/lists) and/or summarization, across genres and languages
- Extrinsic and intrinsic evaluations, correlating extrinsic measures with
outcome of task completion and/or intrinsic measures with human judgments
previously obtained.
- The 2006 Multilingual Multi-document Summarization Evaluation
Anyone with an interest in summarization, QA and/or evaluation is encouraged
to participate in the workshop. We are looking for research papers in the
aforementioned topics, as well as position papers that identify limitations in
current approaches and describe promising future research directions.
SUMMARIZATION TASK: Multilingual Summarization Evaluation
Details for MSE 2006 are available at
http://research.microsoft.com/~lucyv/MSE2006.htm.
For description and results of last year’s MSE task, please see:
http://www.isi.edu/~cyl/MTSE2005.
Send email to lucy.vanderwende@microsoft.com to be added to MSE mailing
list.
PAPER FORMAT:
Papers should be no more than 8 pages, formatted following the guidelines that
will be made available on the conference Web site. The reviewing process will
be blind, so authors' names, affiliations, and all self-references should not
be included in the paper. Authors who cannot submit a PDF file electronically
should contact the organizers at least one week prior to the May 1st deadline.
Proceedings will be published in conjunction with the main HLT/NAACL
proceedings.
Please submit your papers at:
http://www.softconf.com/acl/W7-COLINGACL2006/submit.html
IMPORTANT DATES:
Task-focused Summarization and Question Answering Workshop
Submission Due: May 1st
Notification of Acceptance: May 22nd
Camera-ready papers due: June 1st
Workshop date: July 23, 2006
Multilingual Summarization Evaluation:
Dates to be announced. Send email to lucy.vanderwende@microsoft.com
to be added to email distribution list.
WORKSHOP ORGANIZERS
Tat-Seng Chua, National University of Singapore; chuats@comp.nus.edu.sg
Jade Goldstein, U.S. Department of Defense; jgstewa@afterlife.ncsc.mil
Simone Teufel, Cambridge University; simone.teufel@cl.cam.ac.uk
Lucy Vanderwende, Microsoft Research; lucy.vanderwende@microsoft.com
PROGRAM COMMITTEE
Regina Barzilay (MIT)
Sabine Bergler (Concordia University, Canada)
Silviu Cucerzan (Microsoft Research)
Hang Cui (National University of Singapore)
Krzysztof Czuba (Google)
Hal Daume III (USC/ISI)
Hans van Halteren (Radboud University Nijmegen, Netherlands)
Sanda Harabagiu (University of Texas, Dallas)
Chiori Hori (CMU)
Eduard Hovy (USC/ISI)
Hongyan Jing (IBM Research)
Guy Lapalme (University of Montreal)
Geunbae (Gary) Lee (Postech Univ, Korea)
Chin-Yew Lin (USC/ISI)
Inderjeet Mani (MITRE)
Marie-France Moens (Katholieke Universiteit Leuven, Belgium)
Ani Nenkova (Columbia University)
Manabu Okumura (Tokyo Institute of Technology)
John Prager (IBM Research)
Horacio Saggion (University of Sheffield, UK)
Judith Schlesinger (IDA/CCS)
Karen Sparck Jones (University of Cambridge)
Nicola Stokes (University of Melbourne)
Beth Sundheim (SPAWAR Systems Center)
Tomek Strzalkowski (University at Albany)
Ralph Weischedel (BBN)