<?xml version="1.0" standalone="yes"?>
<Paper uid="C04-1022">
  <Title>Automatic Learning of Language Model Structure</Title>
  <Section position="6" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
5 Experiments and Results
</SectionTitle>
    <Paragraph position="0"> In our application of GAs to language model structure search, the perplexity of models with respect to the development data was used as an optimization criterion. The perplexity of the best models found by the GA were compared to the best models identi ed by a lengthy manual search procedure using linguistic knowledge about dependencies between the word factors involved, and to a random search procedure which evaluated the same number of strings as the GA. The following GA options gave good results: population size 30-50, crossover probability 0.9, mutation probability 0.01, Stochastic Universal Sampling as the selection operator, 2point crossover. We also experimented with reinitializing the GA search with the best model found in previous runs. This method consistently improved the performance of normal GA search and we used it as the basis for the results reported below. Due to the large number of fac- null = n-gram order, Word = word-based models, Hand = manual search, Rand = random search, GA = genetic search.</Paragraph>
    <Paragraph position="1"> tors in the Turkish word representation, models were only optimized for conditioning variables and backo paths, but not for smoothing options. Table 1 compares the best perplexity results for standard word-based models and for FLMs obtained using manual search (Hand), random search (Rand), and GA search (GA).</Paragraph>
    <Paragraph position="2"> The last column shows the relative change in perplexity for the GA compared to the better of the manual or random search models. For tests on both the development set and evaluation set, GA search gave the lowest perplexity. In the case of Arabic, the GA search was  performed over conditioning factors, the back-o graph, and smoothing options. The results in Table 2 were obtained by training and testing without consideration of out-of-vocabulary (OOV) words. Our ultimate goal is to use these language models in a speech recognizer with a xed vocabulary, which cannot recognize OOV words but requires a low perplexity for other  (with unknown words).</Paragraph>
    <Paragraph position="3"> word combinations. In a second experiment, we trained the same FLMs from Table 2 with OOV words included as the unknown word token. Table 3 shows the results. Again, we see that the GA outperforms other search methods.</Paragraph>
    <Paragraph position="4"> The best language models all used parallel back-o and di erent smoothing options at di erent backo graph nodes. The Arabic models made use of all conditioning variables (Word, Stem, Root, Pattern, and Morph) whereas the Turkish models used only the W, P, C, and R variables (see above Section 4).</Paragraph>
  </Section>
class="xml-element"></Paper>