<?xml version="1.0" standalone="yes"?>
<Paper uid="C04-1094">
  <Title>Using Syntactic Information to Extract Relevant Terms for Multi-Document Summarization</Title>
  <Section position="3" start_page="0" end_page="0" type="intro">
    <SectionTitle>
2 Evaluation of automatically extracted
</SectionTitle>
    <Paragraph position="0"> key concepts It is necessary, in the context of an interactive summarization system, to measure the quality of the terms suggested by the system, i.e., to what extent they are related to the key topics of the document set.</Paragraph>
    <Paragraph position="1"> (Lin and Hovy, 1997) compared different strategies to generate lists of relevant terms for summarization using Topic Signatures. The evaluation was extrinsic, comparing the quality of the summaries generated by a system using different term lists as input. The results, however, cannot be directly extrapolated to interactive summarization systems, because the evaluation does not consider how informative terms are for a user.</Paragraph>
    <Paragraph position="2"> From an interactive point of view, the evaluation of term extraction approaches can be done, at least, in two ways: Evaluating the summaries produced in the interactive summarization process. This option is difficult to implement (how do we evaluate a human produced summary? What is the reference gold standard?) and, in any case, it is too costly: every alternative approach would require at least a few additional subjects performing the summarization task.</Paragraph>
    <Paragraph position="3"> Comparing automatically generated term lists with manually generated lists of key concepts.</Paragraph>
    <Paragraph position="4"> For instance, (Jones et al., 2002) describes a process of supervised learning of key concepts from a training corpus of manually generated lists of phrases associated to a single document. null We will, therefore, use the second approach, evaluating the quality of automatically generated term lists by comparing them to lists of key concepts which are generated by human subjects after a multi-document summarization process.</Paragraph>
  </Section>
class="xml-element"></Paper>