<?xml version="1.0" standalone="yes"?>
<Paper uid="P97-1022">
  <Title>Fertility Models for Statistical Natural Language Understanding</Title>
  <Section position="10" start_page="171" end_page="171" type="concl">
    <SectionTitle>
6 Results
</SectionTitle>
    <Paragraph position="0"> The translation models were trained with 5627 context-independent ATIS sentences and smoothed with 600 sentences. In addition, 3567 training sentences were manually aligned and included in a separate training experiment. This allows comparison between an unannotated corpus and a partially annotated one.</Paragraph>
    <Paragraph position="1"> We employ a trivial decoder and language model since our emphasis is on evaluating the performance of different translation models. Our decoder is a simple pattern matcher. That is, we accumulate the different formal language patterns seen in the training set, and score each of them on the test set. The language model is just the unsmoothed unigram probability distribution of the patterns. This LM has a 10% chance of not including a test pattern and its use leads to pessimistic performance estimates. A more general language model for ATIS is presented in (Koppelman et al., 1995). Answers are generated by an SQL program which is a deterministically constructed from the formal language of our system.</Paragraph>
    <Paragraph position="2"> The accuracy of these database answers is measured using ARPA's Common Answer Specification (CAS) metric.</Paragraph>
    <Paragraph position="3"> The results are presented in table 3 for ARPA's December 1993 blind test set. The column headed DEC93 reports results on unsupervised training data, while the column entitled DEC93a contains the results from using models trained on the partially annotated corpus. The rows correspond to various translation models. Model 1 is the word-pair translation model used in simple machine translation and understanding models (Brown et al., 1993; Epstein et al., 1996). The models labeled &amp;quot;Clump&amp;quot; use a basic clumped model without fertility. The models labeled &amp;quot;Poisson&amp;quot; and &amp;quot;General&amp;quot; use the Poisson and general fertility models presented in this paper.</Paragraph>
    <Paragraph position="4"> The &amp;quot;HW&amp;quot; and &amp;quot;BG&amp;quot; suffixes indicate the results when p(e\[f) is computed with a headword or bigram model.</Paragraph>
    <Paragraph position="5"> The partially annotated corpus provides an increase in performance of about 2-3% for most models. For General-LM, results increased by 8-10%.</Paragraph>
    <Paragraph position="6"> The Poisson and general fertility models show a 25% gain in performance over the basic clump model when using the partially annotated corpus. This is a reduction of the error rate by 10-20%. The unannotated corpus also shows a comparable gain.</Paragraph>
    <Paragraph position="7"> Acknowledgement: This work was sponsored in part by ARPA and monitored by Fort Huachuca HJ1500-4309-0513. The views and conclusions contained in this document should not be interpreted as representing the official policies of the U.S. Government. null</Paragraph>
  </Section>
class="xml-element"></Paper>