<?xml version="1.0" standalone="yes"?>
<Paper uid="W03-0303">
  <Title>Word Alignment Based on Bilingual Bracketing</Title>
  <Section position="6" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
5 Experiments
</SectionTitle>
    <Paragraph position="0"> All the settings described so far are based on our previous experiments on Chinese-English (CE) alignment.</Paragraph>
    <Paragraph position="1"> These settings are then used directly without any adjustment of the parameters for the French-English (FE) and Romanian-English (RE) word alignment tasks. In this section, we will first describe our experiments on Chinese-English alignment, and then the results for the shared task on French-English and Romanian-English.</Paragraph>
    <Paragraph position="2"> For Chinese-English alignment, 365 sentence-pairs are randomly sampled from the Chinese Tree-bank provided by the Linguistic Data Consortium. Three persons manually aligned the word-pairs independently, and the consistent alignments from all of them were used as the reference alignments. There are totally 4094 word-pairs in the reference set. Our way of alignment is very similar to the &amp;quot;SURE&amp;quot; (S) alignment defined in the shared task. The training data we used is 16K parallel sentence-pairs from Hong-Kong news data. The English POS tagger we used is Brill's POS tagger [Brill 1994]. The base noun detector is [Ramshaw 1995]. The alignment is evaluated in terms of precision, recall, F-measure and alignment error rate (AER) defined in the shared task. The results are  Table-1 shows the effectiveness of using each setting on this small size training data. Here the boosted model gives a noticeable improvement over the baseline. However, our observations on the trial/test data showed very similar results for boosted and non-boosted models, so we present only the non-boosted results(standard Model1) for the shared task of EF and RE word alignment.</Paragraph>
    <Paragraph position="3"> Adding POS further improved the performance significantly. The AER drops from 44.04 to 41.29. Adding additional base noun phrase boundaries did not give as much improvement as we hoped. There is only slight improvement in terms of AER and F-measure. One reason is that noun phrase boundaries is more directly related to phrase alignment than word-alignment. A close examination showed that with wrong phrase-alignment, word-alignment can still be correct. Another reason is that using the noun phrase boundaries this way may not be powerful enough to leverage the English structure information in Bilingual Bracketing. More suitable ways could be bilingual chunk parsing, and refining the bracketing grammar as described in [Wu 1997].</Paragraph>
    <Paragraph position="4"> In the shared task experiments, we restricted the training data to sentences upto 60 words. The statistics for the training sets are shown in Table-2. (French/Romanian are source and English is target language).</Paragraph>
    <Paragraph position="5">  There are 447 test sentence pairs for English-French and 248 test sentence pairs for Romanian-English. After the bilingual bracketing, we extracted only the explicit word alignment from lexical rules: A ! e=f, where neither e nor f is the null(empty) word. These explicit word alignments are more directly related to the translation quality in our SMT system than the null-word alignments.</Paragraph>
    <Paragraph position="6"> Also the explicit word alignments is in accordance with the &amp;quot;SURE&amp;quot; (S) alignment defined in the shared tasks. However the Bilingual Bracketing system is not adapted to the &amp;quot;PROBABLE&amp;quot; (P) alignment because of the inherent one-to-one mapping. All the AERs in the following tables are calculated based solely on S alignment without any null alignments collected from the bracketing results.</Paragraph>
    <Paragraph position="7">  For the limited resource task, we trained Model-1 lexicons in both directions: from source to target denoted as p(fje) and from target to source denoted as p(ejf). These two lexicons are then plugged into the Bilingual Bracketing algorithm separately to get two sets of bilingual bracketing word alignments. The intersection of these two sets of word alignments is then collected. The resulting AERs are shown in Table-3 and Table-5 respectively. For the unlimited resource task, we again tagged the English sentences and base noun phrase boundaries as mentioned before. Then corresponding Model-1 lexicon was trained and Bilingual Bracketing carried out. Using the same strategies as in the limited resource task, we got the results shown in Table-4 and Table-6.</Paragraph>
    <Paragraph position="8"> The table above show that adding English POS and base noun detection gave a consistent improvement for all conditions in the French-to-English alignment. The intersection of the two alignments greatly improves the precision, paired with a reduction in recall, still resulting in an overall improvement in F-measure and AER.</Paragraph>
    <Paragraph position="9"> For the Romanian-English alignment the POS tagging and noun phrase boundaries did not help. On the small corpus the increase in vocabulary resulted in addition unknown words in the test sentences which introduces additional alignment errors.</Paragraph>
    <Paragraph position="10"> Comparing the results of the French-English and Romanian-English alignment tasks we see a striking difference in precision and recall. Whereas the French-English alignment has a low precision and a high recall its the opposite for the Romanian-English alignment. The cause lays in different styles for the manual alignments.</Paragraph>
    <Paragraph position="11"> The French-English reference set contains both S and P alignments, whereas the Romanian-English reference set was annotated with only S alignments. As a result, there are on average only 0.5 S alignments per word in the FE reference set, but 1.5 S alignments per word in the RE test set.</Paragraph>
  </Section>
class="xml-element"></Paper>