<?xml version="1.0" standalone="yes"?>
<Paper uid="W93-0301">
  <Title>Robust Bilingual Word Alignment for Machine Aided Translation</Title>
  <Section position="4" start_page="5" end_page="5" type="evalu">
    <SectionTitle>
3 Evaluation
</SectionTitle>
    <Paragraph position="0"> Word_align was first evaluated on a representative sample of Canadian Hansards (160,000 words in English and French). The sample was kindly provided by Simard et al., along with alignments of sentence boundaries as determined by their panel of 8 judges (Simard et al., 1992).</Paragraph>
    <Paragraph position="1"> Ten iterations of the EM algorithm were computed to estimate the parameters of the model.</Paragraph>
    <Paragraph position="2"> The window size was set to 20 words in each direction, and the minimal threshold for t(fJe) was set to 0.005. We considered connections whose source and target words had frequencies between 3 and 1700 (1700 is the highest frequency of a content word in the corpus. We thus excluded as many 4As mentioned earlier, we do not estimate directly translation probabilities for the null English word.</Paragraph>
    <Paragraph position="3"> function words as possible, but no content words).</Paragraph>
    <Paragraph position="4"> In this experiment, we used French as the source language and English as the target language.</Paragraph>
    <Paragraph position="5"> Figure 3 presents the alignment error rate of word_align. It is compared with the error rate of word_align's input, i.e. the initial rough alignment which is produced by char_align. The errors are sampled at sentence boundaries, and are measured as the relative distance between the output of the alignment program and the &amp;quot;true&amp;quot; alignment, as defined by the human judges 5. The histograms present errors in the range of-20-20, which covers about 95% of the data s. It can be seen that word_align decreases the error rate significantly (notice the different scales of the vertical axes). In 55% of the cases, there is no error in word_align's output (distance of 0), in 73% the distance from the correct alignment is at most i, and in 84% the distance is at most 3.</Paragraph>
    <Paragraph position="6"> A second evaluation of word_align was performed on noisy technical documents, of the type typically available for AT&amp;T Language Line Services. We used the English and French versions of a manual of monitoring equipment (about 65,000 words), both scanned by an OCR device. We sampled the English vocabulary with frequency between three and 450 occurrences, the same vocabulary that was used for alignment. We sampled 100 types from the top fifth by frequency of the vocabulary (quintile), 80 types from the second quintile, 60 from the third, 40 from the fourth, and 20 from the bottom quintile. We used this stratified sampling because we wanted to make more accurate statements about our error rate by tokens than we would have obtained from random sampling, or even from equal weighting of the quintiles. After choosing the 300 types from the vocabulary list, one token for each type was chosen at random from the corpus.</Paragraph>
    <Paragraph position="7"> By hand, the best corresponding position in the French version was chosen, to be compared with word_align ' s output.</Paragraph>
    <Paragraph position="8"> Table 2 summarizes the results of the second experiment. The figures indicate the expected relative frequency of each offset from the correct alignment. This relative frequency was computed according to the word frequencies in the stratified sample. As shown in the table, for 60.5% of the tokens the alignment is accurate, and in 84% the offset from the correct alingment is at most 3. These figures demonstrate the usefulness of word_align for constructing bilingual lexicons, and its impact on 5As explained eaxlier, word_align produces a partial Mignment. For the purpose of the evaluation, we used linear interpolation to get Mignments for all the positions in the sample.</Paragraph>
    <Paragraph position="9"> 6Recall that the window size we used is 20 words in each direction, which means that word_align cannot recover from larger errors in char_align.</Paragraph>
    <Paragraph position="10">  square error) by a factor of 5 over char_align alone (notice the vertical scales).</Paragraph>
    <Paragraph position="11"> the quality of bilingual concordances (as in Figure 1). Indeed, using bilingual concordances which are based on word_align's output, the translators at AT&amp;T Language Line Services are now producing bilingual terminology lexicons at a rate of 60-100 terms per hour! This is compared with the previous rate of about 30 terms per hour using char_align's output, and an extremely lower rate before alignment tools were available.</Paragraph>
  </Section>
class="xml-element"></Paper>