<?xml version="1.0" standalone="yes"?>
<Paper uid="W00-1420">
  <Title>Multilingual Summary Generation in a Speech-To-Speech Translation System for Multilingual Dialogues*</Title>
  <Section position="8" start_page="153" end_page="153" type="evalu">
    <SectionTitle>
6 Evaluation
</SectionTitle>
    <Paragraph position="0"> We have performed a small evaluation of the overall system as described in this paper. Basis for the evaluation were the transcripts of four German-English negotiation dialogues. For each dialogue the resulting features of the negotiation (maximally 47, e.g., location, date for a meeting, speakers name and title, book agent) were annotated by a lmman, and then compared with the result of running the dialogues through the system and generating the summaries.</Paragraph>
    <Paragraph position="1"> The features in the summary were compared using the following classifications: * Corr The feature approximately corresponds to the human annotation. This means that the feature is either (1) a 100% match; (2) it was not sufficiently specified or (2) too specific. An example of (2) is when the correct date included a time, which was not captured. An example of (3) is when a date with time was annotated but the feature contained just a (late.</Paragraph>
    <Paragraph position="2">  o Miss A feature is not included in the summary.</Paragraph>
    <Paragraph position="3"> o False A feature was erroneously iimluded in the  sumlnary, meaning that the feature was not part of the dialogue or it received a wrong value.</Paragraph>
    <Paragraph position="4"> Obviously, our approach tries to be on the safe side; the summary contains only those features that the system thinks both partners agreed on. The main reasons for not getting higher numbers is twofold. The recognition of dialogue acts, and thus the recognition of the intension behind the utterances reaches a 70% recall (Reithinger and Klesen, 1997). We also still make errors during the content extraction.</Paragraph>
  </Section>
class="xml-element"></Paper>