<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-1013">
  <Title>ROUGE: A Package for Automatic Evaluation of Summaries</Title>
  <Section position="8" start_page="0" end_page="0" type="concl">
    <SectionTitle>
7 Conclusions
</SectionTitle>
    <Paragraph position="0"> In this paper, we introduced ROUGE, an automatic evaluation package for summarization, and conducted comprehensive evaluations of the automatic measures included in the ROUGE package using three years of DUC data. To check the significance of the results, we estimated confidence intervals of correlations using bootstrap resampling. We found that (1) ROUGE-2, ROUGE-L, ROUGE-W, and ROUGE-S worked well in single document summarization tasks, (2) ROUGE-1, ROUGE-L, ROUGE-W, ROUGE-SU4, and ROUGE-SU9 performed great in evaluating very short summaries (or headline-like summaries), (3) correlation of high 90% was hard to achieve for multi-document summarization tasks but ROUGE-1, ROUGE-2, ROUGE-S4, ROUGE-S9, ROUGE-SU4, and ROUGE-SU9 worked reasonably well when stopwords were excluded from matching, (4) exclusion of stopwords usually improved correlation, and (5) correlations to human judgments were increased by using multiple references.</Paragraph>
    <Paragraph position="1"> In summary, we showed that the ROUGE package could be used effectively in automatic evaluation of summaries. In a separate study (Lin and Och, 2004),  the DUC 2001, 2002, and 2003 multi-document summarization tasks ROUGE-L, W, and S were also shown to be very effective in automatic evaluation of machine translation. The stability and reliability of ROUGE at different sample sizes was reported by the author in (Lin , 2004). However, how to achieve high correlation with human judgments in multi-document summarization tasks as ROUGE already did in single document summarization tasks is still an open research topic.</Paragraph>
  </Section>
class="xml-element"></Paper>