<?xml version="1.0" standalone="yes"?>
<Paper uid="P03-1011">
  <Title>Loosely Tree-Based Alignment for Machine Translation</Title>
  <Section position="6" start_page="4" end_page="4" type="evalu">
    <SectionTitle>
5 Experiments
</SectionTitle>
    <Paragraph position="0"> We evaluate our translation models both in terms agreement with human-annotated word-level alignments between the sentence pairs. For scoring the viterbi alignments of each system against gold-standard annotated alignments, we use the alignment error rate (AER) of Och and Ney (2000), which measures agreement at the level of pairs of words:  where A is the set of word pairs aligned by the automatic system, and G the set aligned in the gold standard. We provide a comparison of the tree-based models with the sequence of successively more complex models of Brown et al. (1993). Results are shown in Table 2.</Paragraph>
    <Paragraph position="1"> The error rates shown in Table 2 represent the minimum over training iterations; training was stopped for each model when error began to increase. IBM Models 1, 2, and 3 refer to Brown et al. (1993). &amp;quot;Tree-to-String&amp;quot; is the model of Yamada and Knight (2001), and &amp;quot;Tree-to-String, Clone&amp;quot; allows the node cloning operation of Section 2.1. &amp;quot;Tree-to-Tree&amp;quot; indicates the model of Section 3, while &amp;quot;Tree-to-Tree, Clone&amp;quot; adds the node cloning operation of Section 3.1. Model 2 is initialized from the parameters of Model 1, and Model 3 is initialized from Model 2. The lexical translation probabilities</Paragraph>
    <Paragraph position="3"> (fje) for each of our tree-based models are initialized from Model 1, and the node re-ordering probabilities are initialized uniformly. Figure 1 shows the viterbi alignment produced by the &amp;quot;Tree-to-String, Clone&amp;quot; system on one sentence from our test set.</Paragraph>
    <Paragraph position="4"> We found better agreement with the human alignments when fixing P ins (left) in the Tree-to-String model to a constant rather than letting it be determined through the EM training. While the model learned by EM tends to overestimate the total number of aligned word pairs, fixing a higher probability for insertions results in fewer total aligned pairs and therefore a better trade-off between precision and recall. As seen for other tasks (Carroll and Charniak, 1992; Merialdo, 1994), the likelihood criterion used in EM training may not be optimal when evaluating a system against human labeling. The approach of optimizing a small number of metaparameters has been applied to machine translation by Och and Ney (2002). It is likely that the IBM models could similarly be optimized to minimize alignment error - an open question is whether the optimization with respect to alignment error will correspond to optimization for translation accuracy.</Paragraph>
    <Paragraph position="5"> Within the strict EM framework, we found roughly equivalent performance between the IBM models and the two tree-based models when making use of the cloning operation. For both the tree-to-string and tree-to-tree models, the cloning operation improved results, indicating that adding the flexibility to handle structural divergence is important when using syntax-based models. The improvement was particularly significant for the tree-to-tree model, because using syntactic trees on both sides of the translation pair, while desirable as an additional source of information, severely constrains possible alignments unless the cloning operation is allowed.</Paragraph>
    <Paragraph position="6"> The tree-to-tree model has better theoretical complexity than the tree-to-string model, being quadratic rather than quartic in sentence length, and we found this to be a significant advantage in practice. This improvement in speed allows longer sentences and more data to be used in training syntax-based models. We found that when training on sentences of up 60 words, the tree-to-tree alignment was 20 times faster than tree-to-string alignment. For reasons of speed, Yamada and Knight (2002) limited training to sentences of length 30, and were able to use only one fifth of the available Chinese-English parallel corpus.</Paragraph>
  </Section>
class="xml-element"></Paper>