<?xml version="1.0" standalone="yes"?>
<Paper uid="P01-1068">
  <Title>Multi-Class Composite N-gram Language Model for Spoken Language Processing Using Multiple Word Clusters</Title>
  <Section position="10" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
5 Evaluation Experiments
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
5.1 Evaluation of Multi-Class N-grams
</SectionTitle>
      <Paragraph position="0"> We have evaluated Multi-Class N-grams in perplexity as the next equations.</Paragraph>
      <Paragraph position="1">  The Good-Turing discount is used for smoothing. The perplexity is compared with those of word 2-grams and word 3-grams. The evaluation data set is the ATR Spoken Language Database (Takezawa et al., 1998). The total number of words in the training set is 1,387,300, the vocabulary size is 16,531, and 5,880 words in 42 conversations which are not included in the training set are used for the evaluation.</Paragraph>
      <Paragraph position="2"> Figure1 shows the perplexity of Multi-Class 2-grams for each number of classes. In the Multi-Class, the numbers of following and preceding classes are fixed to the same value just for comparison. As shown in the figure, the Multi-Class 2-gram with 1,200 classes gives the lowest perplexity of 22.70, and it is smaller than the 23.93 in the conventional word 2-gram.</Paragraph>
      <Paragraph position="3"> Figure 2 shows the perplexity of Multi-Class 3-grams for each number of classes. The number of following and preceding classes is 1,200 (which gives the lowest perplexity in Multi-Class 2-grams). The number of pre-preceding classes is  changed from 100 to 1,500. As shown in this figure, Multi-Class 3-grams result in lower perplexity than the conventional word 3-gram, indicating the reasonability of word clustering based on the distance-2 2-gram.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
5.2 Evaluation of Multi-Class Composite
N-grams
</SectionTitle>
      <Paragraph position="0"> We have also evaluated Multi-Class Composite N-grams in perplexity under the same conditions as the Multi-Class N-grams stated in the previous section. The Multi-Class 2-gram is used for the initial condition of the Multi-Class Composite 2-gram. The threshold of frequency for introducing word successions is set to 10 based on a preliminary experiment. The same word succession set as that of the Multi-Class Composite 2-gram is used for the Multi-Class Composite 3gram. The evaluation results are shown in Table 1. Table 1 shows that the Multi-Class Composite 3-gram results in 9.5% lower perplexity with a 40% smaller parameter size than the conventional word 3-gram, and that it is in fact a compact and high-performance model.</Paragraph>
    </Section>
    <Section position="3" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
5.3 Evaluation in Continuous Speech
Recognition
</SectionTitle>
      <Paragraph position="0"> Though perplexity is a good measure for the performance of language models, it does not always have a direct bearing on performance in language processing. We have evaluated the proposed model in continuous speech recognition.</Paragraph>
      <Paragraph position="1"> The experimental conditions are as follows:  Table 2 shows the evaluation results. As in the perplexity results, the Multi-Class Composite 3-gram shows the highest performance of all models, and its error reduction from the conventional word 3-gram is 16%.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>