<?xml version="1.0" standalone="yes"?>
<Paper uid="W00-0407">
  <Title>Evaluation of Phrase-Representation Summarization based on Information Retrieval Task</Title>
  <Section position="3" start_page="59" end_page="60" type="intro">
    <SectionTitle>
2 Evaluation Method
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="59" end_page="60" type="sub_section">
      <SectionTitle>
2.1 Summarization Methods to be
Compared
</SectionTitle>
      <Paragraph position="0"> In this experiment, we compare the effectiveness of phrase-represented summaries to summaries created by other commonly used summarization methods. From the viewpoint of the phrase-represented summary, we focus the comparison of the units that constitute summaries. The units to be compared with phrases are sentences (created by the sentence extraction method) and words (by the keyword enumeration method).</Paragraph>
      <Paragraph position="1"> We also compare &amp;quot;leading fixed-length characters,&amp;quot; which are often used as substitutes for summaries by WWW search engines. The generation method for each summary is described as follows.</Paragraph>
      <Paragraph position="2"> (A) Leading fixed-length characters: extract the first 80 characters of the document body.</Paragraph>
      <Paragraph position="3"> (B) Sentence extraction summarization: select important sentences from a document.</Paragraph>
      <Paragraph position="4"> The importance score of each sentence is calculated from the simple sum of the importance scores of the words in a sentence (Zechner 1996).</Paragraph>
      <Paragraph position="5"> (C) Phrase-representation summarization: described in Chapter 1.</Paragraph>
      <Paragraph position="6"> (D) Keyword enumeration summarization: list up important words or compound nouns.</Paragraph>
      <Paragraph position="7"> http://www, fujixerox.co.jp/headlineJ2000/0308__nton e,r_biz,hlml (in English) s This phrase lacks the subject because the original sentence lacks it. Cases are usually omitted in Japanese if they can be easily inferred.</Paragraph>
      <Paragraph position="9"> I In (B), (C), and (D), the same method of calcuiating the importance scores of words is 18ilm~'-,~t~tmm\]m ~~~.~..~ used in common, and lengths of summaries are Ill ..I kept to be 60 to80 characters. ~~ .,,,~~e~ As you can see each summary is generic, i.e. ./ not created for any specific queries. Because the ~.~ I phrase-representation summarization method is applied to Japanese, we examine the effective- Relevanti ) ness of these four methods in Japanese. =__.- .1 Relevant i~ 2 I 2.2 Previous Work The best-known example of task-based Irrelevant--- .... 3 -~ ! I~ I evaluation on information retrieval is the ad hoc ~ ~~ I task in the TIPSTER Text Summarization Evaluation Conference (SUMMAC) (Mani, et al.</Paragraph>
      <Paragraph position="10"> ' ') Accuracy 1998). Hand (1997) details the proposed task-based evaluation under TIPSTER. Jing, et al.</Paragraph>
      <Paragraph position="11"> (1998) describe how various parameters affect the evaluation result through a relatively large task-based experiment. Evaluation conferences like SUMMAC are not yet held for Japanese summarization systems 4. Mochizuki and Okumura (1999) applied the SUMMAC methodology to Japanese summarization methods for the first time. Most previous experiments are concerned with SUMMAC, accordingly the methods resemble each other.</Paragraph>
    </Section>
    <Section position="2" start_page="60" end_page="60" type="sub_section">
      <SectionTitle>
2.3 Framework of Evaluation
</SectionTitle>
      <Paragraph position="0"> ~Fhe framework of task-based evaluation on information retrieval is shown in Fig. 2.</Paragraph>
      <Paragraph position="1"> Task-based evaluation in general consists of the following three steps: (l) Data preparation: Assume an information need, create a query for the information need, and prepare simulated search results with different types of summaries.</Paragraph>
      <Paragraph position="2"> (2) Relevance assessment: Using the summades, human subjects assess the relevance of the search results to the assumed information needs.</Paragraph>
      <Paragraph position="3"> (3) Measuring performance: Measure the accuracy of the subjects' assessment by comparing the subjects' judgement with the correct relevance. The assessment process is also timed.</Paragraph>
      <Paragraph position="4">  We designed our evaluation method through detailed examination of previous work. The consideration points are compared to the SUMMAC ad hoc task (Table l). A section number will be found in the &amp;quot;*&amp;quot; column if we made an improvement. Details will be discussed in the section indicated by the number in the next chapter.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>