<?xml version="1.0" standalone="yes"?>
<Paper uid="W02-0712">
  <Title>Automatic Interpretation System Integrating Free-style Sentence Translation and Parallel Text Based Translation</Title>
  <Section position="5" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
4 Evaluation
</SectionTitle>
    <Paragraph position="0"> In this section, we evaluate the prototype system with respect to the extent that typical traveler utterances are covered by the registered sentences and whether the user can easily find a registered sentence that matches a natural utterance.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.1 Coverage provided by the Registered
Sentences
</SectionTitle>
      <Paragraph position="0"> The system can provide correct translation if the input sentence can be matched to a registered sentence; otherwise, it can only provide a translation through machine translation whose quality is uncertain. To determine the proportion of commonly used sentences for which the system would provide a correct translation, we evaluated the coverage provided by the registered sentences for sentences randomly extracted from travel conversation corpora.</Paragraph>
      <Paragraph position="1"> Table 4 displays the evaluation results. In the table, &amp;quot;closed set&amp;quot; denotes the result for sentences extracted from the corpora we referred to when developing the registered sentences and &amp;quot;open set&amp;quot; de- null notes the result for sentences extracted from corpora not referred to when developing the registered sentences. The registered sentences covered 72.6% of the sentences in the closed set and 52.9% of the sentences in the open set.</Paragraph>
      <Paragraph position="2"> The coverage of almost 73% for the closed set suggests that roughly 27% of the sentences in the corpora are not used for typical travel conversation.</Paragraph>
      <Paragraph position="3"> If the open set includes an equal proportion of atypical sentences, the registered sentences cover about 72% of the typical sentences used for travel in the open set.</Paragraph>
      <Paragraph position="4"> We therefore believe that our prototype system can provide reliable translation for a minimum set of utterances necessary for overseas travel.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.2 Basic Performance of the Registered
Sentence Retrieval
</SectionTitle>
      <Paragraph position="0"> Since registered sentences are retrieved mainly by using sentences inputted with natural language, a poorly performing retrieval system may prevent the user finding an appropriate sentence that is registered in the system. To determine whether the system can reliably retrieve an appropriate sentence from the utterance the user first thinks of, we randomly picked 116 subtasks for which registered sentences had been developed, had experimental subjects compose a natural sentence that could be used to accomplish each subtask, and evaluated whether the retrieval result when using a composed sentence as a key included the registered sentence.</Paragraph>
      <Paragraph position="1"> Table 5 shows the evaluation results. The user could find a registered sentence corresponding to the natural utterance with our retrieval system for 63.8% of the traveler's subtasks when a sentence for the subtask was registered in the system.</Paragraph>
      <Paragraph position="2"> Although this retrieval system performance is not sufficient when we take into consideration the decreased performance caused by recognition errors in the input sentence, we expect to improve the retrieval performance by adding expressions synonymous with those in each sentence template to the index used for registered sentence retrieval.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>