<?xml version="1.0" standalone="yes"?>
<Paper uid="P97-1037">
  <Title>Transformation Step Original CorPora + Categorization + 'por2favor ' + Word Splitting Translation Errors \[~.\]</Title>
  <Section position="6" start_page="292" end_page="294" type="evalu">
    <SectionTitle>
4 Experimental Results
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="292" end_page="293" type="sub_section">
      <SectionTitle>
4.1 The Task and the Corpus
</SectionTitle>
      <Paragraph position="0"> The search algorithln proposed in this paper was tested on a subtask of the &amp;quot;'Traveler Task&amp;quot; (Vidal, 1997). The general domain of the task comprises typical situations a visitor to a foreign country is faced with. The chosen subtask corresponds to a scenario of the hulnan-to-human communication situations at the registration desk in a hotel (see Table 4).</Paragraph>
      <Paragraph position="1"> The corpus was generated in a semi-automatic way. On the basis of examples from traveller booklets, a prol)abilistic gralmnar for different language pairs has been constructed from which a large corpus of sentence pairs was generated. The vocabulary consisted of 692 Spanish and 518 English words (ineluding punctuatioll marks). For the experiments, a trailfing corpus of 80,000 sentence pairs with 628,117 Spanish and 684.777 English words was used. In addition, a test corpus with 2.730 sentence pairs different froln the training sentence pairs was constructed. This test corpus contained 28.642 Spanish a.nd 24.927 English words. For the English sentences, we used a bigram language model whose perplexity on the test corpus varied between 4.7 for the original text. and 3.5 when all transformation steps as described below had been applied.</Paragraph>
    </Section>
    <Section position="2" start_page="293" end_page="293" type="sub_section">
      <SectionTitle>
4.2 Text Tl-ansformations
</SectionTitle>
      <Paragraph position="0"> The purpose of the text transformations is to make the two languages resenable each other as closely as possible with respect, to sentence length and word order. In addition, the size of both vocabularies is reduced by exploiting evident regularities; e.g. proper names and numbers are replaced by category markers. We used different, preprocessing steps which were applied consecutively:  * Original Corpus: Punctuation marks are treated like regular words.</Paragraph>
      <Paragraph position="1"> * Categorization: Some particular words or  word groups are replaced by word categories. Seven non-overlapping categories are used: three categories for names (surnames, name and female names), two categories for numbers (regular numbers and room numbers) and two categories for date and time of day.</Paragraph>
      <Paragraph position="2">  * 'D_'eatment of 'pot :favor': The word 'pot :favor' is always moved to the end of the sentence and replaced by the one-word token ' pot_favor '.</Paragraph>
      <Paragraph position="3"> * Word Splitting: In Spanish, the personal pronouns (in subject case and in object, case) can be part of the inflected verb form. To counteract this phenomenon, we split the verb into a verb part and pronoun part, such as 'darnos&amp;quot; -- &amp;quot;dar _nos' and &amp;quot;pienso&amp;quot; -- '_yo pienso'. * Word Joining: Phrases in the English language such as &amp;quot;Would yogi mind doing ...' and '1 would like you to do ...&amp;quot; are difficult to handle by our alignment model. Therefore, we apply some word joining, such as 'would yo~t mi71d&amp;quot; -- 'wo~dd_yo',_mind&amp;quot; and ~would like ' -&amp;quot;wotdd_like '.</Paragraph>
      <Paragraph position="4"> * Word Reordering: This step is applied to the Spanish text to take into account, cases like  the position of the adjective in noun-adjective phrases and the position of object, pronouns. E.g. &amp;quot;habitacidT~ dobh'-- 'doble habitaci6~'. By this reordering, our assumption about the monotony of the alignment model is more often satisfied.</Paragraph>
      <Paragraph position="5"> The effect of these transformation steps on the sizes of both vocabularies is shown in Table 2. In addition to all preprocessing steps, we removed the punct.uation marks before translation and resubstituted t.hena by rule into the target sentence.</Paragraph>
    </Section>
    <Section position="3" start_page="293" end_page="294" type="sub_section">
      <SectionTitle>
4.3 Translation Results
</SectionTitle>
      <Paragraph position="0"> For each of the transformation steps described above, all probability models were trained anew, i.e, the lexicon probabilities p(fle), the alignment probabilities p(ili - 6) and the bigram language probabilities p(ele'). To produce the translated sentence in normal language, the transformation steps in the target language were inverted.</Paragraph>
      <Paragraph position="1"> The translation results are summarized in Table 3. As an aut.omatic and easy-to-use measure of the translation errors, the Levenshtein distance between the automatic translation and the reference translation was calculated. Errors are reported at the word level and at. the sentence level: * word leveh insertions (INS). deletions (DEL), and total lmmber of word errors (\VER).</Paragraph>
      <Paragraph position="2"> * sentence level: a sentence is counted as correct only if it is identical to the reference sentence.</Paragraph>
      <Paragraph position="3"> Admittedly, this is not a perfect measure. In particular, the effect of word ordering is not taken into account appropriately. Actually, the figures for sentence error rate are overly pessimistic. Many sentences are acceptable and semantically correct translations (see the example translations in Table 4),  As can be seen in Table 3. the translation errors can be reduced systen~at.ically by applying all transformation steps. The word error rate is reduced from 21.2{,} t.o 5.1{2~: the sentence error rate is reduced from 85.55~, to 30.1%. The two most inaportant transformation steps are categorization and word joining. What is striking, is the large fi'action of deletion errors. These deletion errors are often caused by the omission of word groups like 'for me please &amp;quot;and &amp;quot;could you &amp;quot;. Table 4 shows some example translations (for the best translation results). It can be seen that the semantic meaning of the sentence in the source language may be preserved even if there are three word errors according t.o our performance criterion. To study the dependence on the amount of training data, we also performed a training wit.la only 5 000 sentences out of the training corpus. For this training condition, the word error rate went up only slightly, namely from 5.15}. (for 80,000 training sentences) to 5.3% (for 5 000 training sentences).</Paragraph>
      <Paragraph position="4"> To study the effect of the language model, we tested a zerogram, a unigram and a bigram language model using the standard set of 80 000 training sentences. The results are shown in Table 5. The  t.ranslatiol~.</Paragraph>
      <Paragraph position="5"> O: He hecho la reserva de una habitacidn con televisidn y t.el~fono a hombre del sefior Morales. R: I have made a reservation for a room with TV and telephone for Mr. Morales. A: I have made a reservation for a room with TV and telephone for Mr. Morales. O: Sfibanme las maletas a mi habitacidn, pot favor.</Paragraph>
      <Paragraph position="6"> R: Send up my suitcases to my room, please.</Paragraph>
      <Paragraph position="7"> A: Send up my suitcases to my room, please.</Paragraph>
      <Paragraph position="8"> O: Pot favor, querr{a qua nos diese las llaves de la habitacidn. R: I would like you to give us the keys to the room, please. A: I would like you to give us the keys to the room, please. O: Pot favor, me pide mi taxi para la habitacidn tres veintidds? R: Could you ask for nay taxi for room number three two two for me. please'? A: Could you ask for my taxi for room number three two two. please? O: Por favor, reservamos dos habitaciones dobles con euarto de bafio. R: We booked two double rooms with a bathroom.</Paragraph>
      <Paragraph position="9"> A: We booked two double rooms with a bathroom, please.</Paragraph>
      <Paragraph position="10"> O: Quisiera qua nos despertaran mafiana a las dos y cuarto, pot favor. R: l would like you to wake us up tomorrow at. a quarter past two. please. A: I want you to wake us up tomorrow at a quarter past two. please. O: Rep/seme la cuenta de la l~abitacidn ochocientos veintiuno. R: Could .you check the bill for room number eight two one for me, please'? A: Check the bill for room lmmber eight two one.</Paragraph>
      <Paragraph position="11"> WER decreases from 31.1c/c for the zerogram model to 5.1% for the bigram model.</Paragraph>
      <Paragraph position="12"> The results presented here can be compared with the results obtained by the finite-state transducer approach described in (Vidal, 1996: Vidal, 1997), where the same training and test conditions were used. However the only preprocessing step was categorization. In that work. a WER of 7.1c)~. was obtained as opposed to 5.1(7c presented in this paper.</Paragraph>
      <Paragraph position="13"> For smaller amounts of training data (say 5 000 sentence pairs), the DP based search seems to be even lnore superior.</Paragraph>
      <Paragraph position="14"> Table 5: Language model perplexity (PP), word error rates (INS/DEL. WER) and sentence error rates (SER) for different language models.</Paragraph>
    </Section>
    <Section position="4" start_page="294" end_page="294" type="sub_section">
      <SectionTitle>
4.4 Effect of the Word Reordering
</SectionTitle>
      <Paragraph position="0"> In more general cases and applications, there will ahvays be sentence pairs with word alignments for which the monotony constraint is \]lot satisfied. However even then, the nlonotouy constraint is satisfied locally for the lion's share of all word alignments in such sentences. Therefore. we expect t.o extend the approach presented by the following methods: * more systelnatic approaches to local and global word reorderiugs that try to produce the same word order in both languages.</Paragraph>
      <Paragraph position="1"> * a multli-level approach that allows a small (say 4) number of large forward and backward transitions. Within each level, the monotone alignment model can still be applied, and only when moving from one level to the next, we have to handle the problem of different word orders.</Paragraph>
      <Paragraph position="2"> To show the usefulness of global word reordering. we changed the word order of some sentences by hand. Table 6 shows the effect of the global re-ordering for two sentences. In the first example, we changed the order of two groups of consecutive words and placed an a.dditional copy of the Spanish word &amp;quot;euest, a'&amp;quot; into the source sentence. In the second example, the personal pronoun &amp;quot;'me&amp;quot; was placed at the end of the source sentence. In both cases, we obtained a correct translation.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>