<?xml version="1.0" standalone="yes"?>
<Paper uid="P97-1017">
  <Title>Machine Transliteration</Title>
  <Section position="3" start_page="128" end_page="129" type="metho">
    <SectionTitle>
2 A Modular Learning Approach
</SectionTitle>
    <Paragraph position="0"> Bilingual glossaries contain many entries mapping katakana phrases onto English phrases, e.g.: (aircraft carrier --, ~ T ~ ~ 7 I. ~ ~ ~3 7&amp;quot; ). It is possible to automatically analyze such pairs to gain enough knowledge to accurately map new katakana phrases that come along, and learning approach travels well to other languages pairs. However, a naive approach to finding direct correspondences between English letters and katakana symbols suffers from a number of problems. One can easily wind up with a system that proposes iskrym as a back-transliteration of aisukuriimu. Taking letter frequencies into account improves this to a more plausible-looking isclim.</Paragraph>
    <Paragraph position="1"> Moving to real words may give is crime: the i corresponds to ai, the s corresponds to su, etc. Unfortunately, the correct answer here is ice cream. After initial experiments along these lines, we decided to step back and build a generative model of the transliteration process, which goes like this:  1. An English phrase is written.</Paragraph>
    <Paragraph position="2"> 2. A translator pronounces it in English.</Paragraph>
    <Paragraph position="3"> 3. The pronunciation is modified to fit the Japanese sound inventory.</Paragraph>
    <Paragraph position="4"> 4. The sounds are converted into katakana.</Paragraph>
    <Paragraph position="5"> 5. Katakana is written.</Paragraph>
    <Paragraph position="6">  This divides our problem into five sub-problems.</Paragraph>
    <Paragraph position="7"> Fortunately, there are techniques for coordinating solutions to such sub-problems, and for using generative models in the reverse direction. These techniques rely on probabilities and Bayes' Rule. Suppose we build an English phrase generator that produces word sequences according to some probability distribution P(w). And suppose we build an English pronouncer that takes a word sequence and assigns it a set of pronunciations, again probabilistically, according to some P(plw). Given a pronunciation p, we may want to search for the word sequence w that maximizes P(wtp ). Bayes&amp;quot; Rule lets us equivalently maximize P(w). P(plw). exactly the two distributions we have modeled.</Paragraph>
    <Paragraph position="8"> Extending this notion, we settled down to build five probability distributions:  1. P(w) -- generates written English word sequences. null 2. P(elw) -- pronounces English word sequences.</Paragraph>
    <Paragraph position="9"> 3. P(jle) -- converts English sounds into Japanese sounds.</Paragraph>
    <Paragraph position="10">  4. P(k\[j) ~ converts Japanese sounds to katakana writing.</Paragraph>
    <Paragraph position="11"> 5. P(o{k) ~ introduces misspellings caused by optical character recognition (OCR).</Paragraph>
    <Paragraph position="12">  Given a katakana string o observed by OCR, we want to find the English word sequence w that maximizes the sum, over all e, j, and k, of</Paragraph>
    <Paragraph position="14"> Following (Pereira et al., 1994; Pereira and Riley, I996), we implement P(w) in a weighted finite-state aceeptor (WFSA) and we implement the other distributions in weighted finite-state transducers (WF-STs). A WFSA is an state/transition diagram with weights and symbols on the transitions, making some output sequences more likely than others. A WFST is a WFSA with a pair of symbols on each transition, one input, and one output. Inputs and outputs may include the empty symbol e. Also following (Pereira and Riley, 1996), we have implemented a general composition algorithm for constructing an integrated model P(zlz) from models P(~IY) and P(ylz), treating WFSAs as WFSTs with identical inputs and outputs. We use this to combine an observed katakana string with each of the models in turn. The result is a large WFSA containing all possible English translations. We use Dijkstra's shortest-path algorithm {Dijkstra, 1959) to extract the most probable one.</Paragraph>
    <Paragraph position="15"> The approach is modular. We can test each engine independently and be confident that their results are combined correctly. We do no pruning, so the final WFSA contains every solution, however unlikely. The only approximation is the Viterbi one, which searches for the best path through a WFSA instead of the best sequence (i.e., the same sequence does not receive bonus points for appearing more than once).</Paragraph>
  </Section>
  <Section position="4" start_page="129" end_page="132" type="metho">
    <SectionTitle>
3 Probabilistic Models
</SectionTitle>
    <Paragraph position="0"> This section describes how we desigued and built each of our five models. For consistency, we continue to print written English word sequences in italics  (golf ball), English sound sequences in all capitals (G AA L F B A0 L). Japanese sound sequences in lower case (g o r u h u b o o r u)and katakana sequences naturally ( =':t. 7 .~- ~).</Paragraph>
    <Section position="1" start_page="129" end_page="129" type="sub_section">
      <SectionTitle>
3.1 Word Sequences
</SectionTitle>
      <Paragraph position="0"> The first model generates scored word sequences, the idea being that ice cream should score higher than ice creme, which should score higher than nice kreem. We adopted a simple unigram scoring method that multiplies the scores of the known words and phrases in a sequence. Our 262,000-entry frequency list draws its words and phrases from the Wall Street Journal corpus, an online English name list, and an online gazeteer of place names.&amp;quot; A portion of the WFSA looks like this: los / 0.000087 federal / O.O013~ angele s~ ~ month 10.000992 An ideal word sequence model would look a bit different. It would prefer exactly those strings which are actually grist for Japanese transliteratots. For example, people rarely transliterate auxiliary verbs, but surnames are often transliterated. We have approximated such a model by removing high-frequency words like has, an, are, am, were, their, and does, plus unlikely words corresponding to Japanese sound bites, like coup and oh.</Paragraph>
      <Paragraph position="1"> We also built a separate word sequence model containing only English first and last names. If we know (from context) that the transliterated phrase is a personal name, this model is more precise.</Paragraph>
    </Section>
    <Section position="2" start_page="129" end_page="130" type="sub_section">
      <SectionTitle>
3.2 Words to English Sounds
</SectionTitle>
      <Paragraph position="0"> The next WFST converts English word sequences into English sound sequences. We use the English phoneme inventory from the online CMU Pronunciation Dictionary, 3 minus the stress marks. This gives a total of 40 sounds, including 14 vowel sounds (e.g., AA, AE, UW), 25 consonant sounds (e.g., K, 1tlt, It), plus our special symbol (PAUSE). The dictionary has pronunciations for 110,000 words, and we organized a phoneme-tree based WFST from it:</Paragraph>
      <Paragraph position="2"> Note that we insert an optional PAUSE between word pronunciations. Due to memory limitations, we only used the 50,000 most frequent words.</Paragraph>
      <Paragraph position="3"> We originally thought to build a general letter-to-sound WFST, on the theory that while wrong (overgeneralized) pronunciations might occasionally be generated, Japanese transliterators also mispronounce words. However, our letter-to-sound WFST did not match the performance of Japanese translit- null erators, and it turns out that mispronunciations are modeled adequately in the next stage of the cascade.</Paragraph>
    </Section>
    <Section position="3" start_page="130" end_page="132" type="sub_section">
      <SectionTitle>
3.3 English Sounds to Japanese Sounds
</SectionTitle>
      <Paragraph position="0"> Next, we map English sound sequences onto Japanese sound sequences. This is an inherently information-losing process, as English R and L sounds collapse onto Japanese r, the 14 English vowel sounds collapse onto the 5 Japanese vowel sounds, etc. We face two immediate problems:  1. What is the target Japanese sound inventory? 2. How can we build a WFST to perform the se- null quence mapping? An obvious target inventory is the Japanese syllabary itself, written down in katakana (e.g., &amp;quot;) or a roman equivalent (e.g., hi). With this approach, the English sound K corresponds to one of 2 (ka), -'Y (ki), ~' (ku), ~ (ke), or = (ko), depending on its context. Unfortunately, because katakana is a syllabary, we would be unable to express an obvious and useful generalization, namely that English g usually corresponds to Japanese k, independent of context. Moreover, the correspondence of Japanese katakana writing to Japanese sound sequences is not perfectly one-to-one (see next section), so an independent sound inventory is well-motivated in any case. Our Japanese sound inventory includes 39 symbols: 5 vowel sounds, 33 consonant sounds (including doubled consonants like kk), and one special symbol (pause). An English sound sequence like (P R OW PAUSE S AA K ER) might map onto a Japanese sound sequence like (p u r o pause s a kk a a). Note that long Japanese vowel sounds are written with two symbols (a a) instead of just one (an). This scheme is attractive because Japanese sequences are almost always longer than English sequences. null Our WFST is learned automatically from 8,000 pairs of English/Japanese sound sequences, e.g., ( (s AA K ER) --* (s a kk a a)). We were able to produce'these pairs by manipulating a small Englishkatakana glossary. For each glossary entry, we converted English words into English sounds using the previous section's model, and we converted katakana words into Japanese sounds using the next section's model. We then applied the estimation-maximization (EM) algorithm (Baum, 1972) to generate symbol-mapping probabilities, shown in Figure 1. Our EM training goes like this: 1. For each English/Japanese sequence pair, compute all possible alignments between their elements. In our case. an alignment is a drawing . that connects each English sound with one or more Japanese sounds, such that all Japanese sounds are covered and no lines cross. For example, there are two ways to align the pair ((L</Paragraph>
      <Paragraph position="2"> 2. For each pair, assign an equal weight to each of its alignments, such that those weights sum to 1. In the case above, each alignment gets a weight of 0.5.</Paragraph>
      <Paragraph position="3"> 3. For each of the 40 English sounds, count up instances of its different mappings, as observed in all alignments of all pairs. Each alignment contributes counts in proportion to its own weight. 4. For each of the 40 English sounds, normalize the scores of the Japanese sequences it maps to, so that the scores sum to 1. These are the symbol-mapping probabilities shown in Figure 1.</Paragraph>
      <Paragraph position="4"> 5. Recompute the alignment scores. Each alignment is scored with the product of the scores of the symbol mappings it contains.</Paragraph>
      <Paragraph position="5"> 6. Normalize the alignment scores. Scores for each pair's alignments should sum to 1.</Paragraph>
      <Paragraph position="6"> 7. Repeat 3-6 until the symbol-mapping probabil null ities converge.</Paragraph>
      <Paragraph position="7"> We then build a WFST directly from the symbol-mapping probabilities:</Paragraph>
      <Paragraph position="9"> Our WFST has 99 states and 283 arcs.</Paragraph>
      <Paragraph position="10"> We have also built models that allow individual English sounds to be &amp;quot;swallowed&amp;quot; (i.e., produce zero Japanese sounds). However, these models are expensive to compute (many more alignments) and lead to a vast number of hypotheses during WFST composition. Furthermore, in disallowing &amp;quot;swallowing,&amp;quot; we were able to automatically remove hundreds of potentially harmful pairs from our training set, e.g., ((B AA R B ER SH AA P) -- (b a a b a a)). Because no alignments are possible, such pairs are skipped by the learning algorithm; cases like these must be solved by dictionary lookup anyway. Only two pairs failed to align when we wished they had--both involved turning English Y UW into Japanese u, as in ((Y UW K AH L EY L IY) ~ (u kurere)).</Paragraph>
      <Paragraph position="11"> Note also that our model translates each English sound without regard to context. We have built also context-based models, using decision trees receded as WFSTs. For example, at the end of a word, English T is likely to come out as (= o) rather than (1;). However, context-based models proved unnecessary  case), as learned by estimation-maximization. Only mappings with conditional probabilities greater than 1% are shown, so tile figures may not sum to 1.</Paragraph>
      <Paragraph position="12">  for back-transliteration. 4 They are more useful for English-to-Japanese forward transliteration.</Paragraph>
    </Section>
    <Section position="4" start_page="132" end_page="132" type="sub_section">
      <SectionTitle>
3.4 Japanese sounds to Katakana
</SectionTitle>
      <Paragraph position="0"> To map Japanese sound sequences like (m o o 1: a a) onto katakana sequences like (~--$t--), we manually constructed two WFSTs. Composed together, they yield an integrated WFST with 53 states and 303 arcs. The first WFST simply merges long Japanese vowel sounds into new symbols aa, ii, uu, ee, and oo. The second WFST maps Japanese sounds onto katakana symbols. The basic idea is to consume a whole syllable worth of sounds before producing any katakana, e.g.:  This fragment shows one kind of spelling variation in Japanese: long vowel sounds (oo) are usually written with a long vowel mark (~-) but are sometimes written with repeated katakana (~).</Paragraph>
      <Paragraph position="1"> We combined corpus analysis with guidelines from a Japanese textbook (Jorden and Chaplin, 1976) to turn up many spelling variations and unusual katakana symbols: * the sound sequence (j +-) is usually written ~, but occasionally C/:.</Paragraph>
      <Paragraph position="2">  ~-~- when it occurs.</Paragraph>
      <Paragraph position="3"> and so on.</Paragraph>
      <Paragraph position="4"> Spelling variation is clearest in cases where an English word like swiIeh shows up transliterated variously (:~ ~&amp;quot; :, C/-, :~4 ~, C/-, x ~, 4 ~, 4-) in different dictionaries. Treating these variations as an equivalence class enables us to learn general sound mappings even if our bilingual glossary adheres to a single narrow spelling convention. We do not, however, 4And harmfully restrictive in their unsmoothed incarnations.</Paragraph>
      <Paragraph position="5"> generate all katakana sequences with this model; for example, we do not output strings that begin with a subscripted vowel katakana. So this model also serves to filter out some ill-formed katakana sequences, possibly proposed by optical character recognition.</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="132" end_page="132" type="metho">
    <SectionTitle>
3.5 Katakana to OCR
</SectionTitle>
    <Paragraph position="0"> Perhaps uncharitably, we can view optical character recognition (OCR) as a device that garbles perfectly good katakana sequences. Typical confusions made by our commercial OCR system include ~ for ~-', C/-for -)', T for 7, and 7 for 7&amp;quot;. To generate pre-OCR text, we collected 19,500 characters worth of katakana words, stored them in a file, and printed them out. To generate post-OCR text, we OCR'd the printouts. We then ran the EM Mgorithm to determine symbol-mapping (&amp;quot;garbling&amp;quot;) probabilities. Here is part of that table:  This model outputs a superset of the 81 katakana symbols, including spurious quote marks, alphabetic symbols, and the numeral 7.</Paragraph>
  </Section>
  <Section position="6" start_page="132" end_page="133" type="metho">
    <SectionTitle>
4 Example
</SectionTitle>
    <Paragraph position="0"> We can now use the models to do a sample backtransliteration. We start with a katakana phrase as observed by OCR. We then serially compose it with the models, in reverse order. Each intermediate stage is a WFSA that encodes many possibilities. The final stage contains all back-transliterations suggested by the models, and we finally extract the best one.</Paragraph>
    <Paragraph position="1"> We start with the masutaazutoonamento problem from Section 1. Our OCR observes: ~ x ~,--;~deg 1. ---/- j :/ 1.</Paragraph>
    <Paragraph position="2"> This string has two recognition errors: ~' (ku) for $ (ta), and C/-(chPS) for &amp;quot;3-(na). We turn the string into a chained 12-state/ll-arc WFSA and compose it with the P(k\[o) model. This yields a fatter 12-state/15-arc WFSA, which accepts the correct spelling at a lower probability. Next comes the P(jlk) model, which produces a 28-state/31-arc WFSA whose highest-scoring sequence is: mas ut aazut o o ch im ent o Next comes P(elj ), yielding a 62-state/241-arc WFSA whose best sequence is:</Paragraph>
    <Paragraph position="4"> Next to last comes P(wle), which results in a 2982state/4601-arc WFSA whose best sequence (out of myriads) is: masters tone am ent awe This English string is closest phonetically to the Japanese, but we are willing to trade phonetic proximity for more sensical English; we restore this WFSA by composing it with P(w) and extract the best translation: masters tournament (Other Section 1 examples are translated correctly as earth day and robert scan leonard.)</Paragraph>
  </Section>
class="xml-element"></Paper>