<?xml version="1.0" standalone="yes"?>
<Paper uid="W99-0615">
  <Title>I HMM Specialization with Selective Lexicalization*</Title>
  <Section position="2" start_page="0" end_page="121" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> Hidden Markov 'Models are widely used for statistical language modelling in various fields, e.g., part-of-speech tagging or speech recognition (Rabiner and Juang, 1986). The models are based on Markov assumptions, which make it possible to view the language prediction as a Markov process. 'In general, we make the first-order Markov ass'umptions that the current tag is only dependant on the previous tag and that the current word is only dependant on the current tag. These are very 'strong' assumptions, so that the first-order Hidden Markov Models have the advantage of drastically reducing the number of its parameters. On the other hand, the assumptions restrict the model from utilizing enough constraints provided by the local context and the resultant model consults only a single category 'as the contex.</Paragraph>
    <Paragraph position="1"> A lot of effort has been devoted in the past to make up for the insufficient contextual information of the first-order probabilistic model.</Paragraph>
    <Paragraph position="2"> The second order Hidden Markov Models with &amp;quot; The research underlying this paper was supported t) 3&amp;quot; research grants fl'om Korea Science and Engineering Foundation.</Paragraph>
    <Paragraph position="3"> appropriate smoothing techniques show better performance than the first order models and is considered a state-of-the-art technique (Merialdo, 1994; Brants, 1996). The complexity of the model is however relatively very high considering the small improvement of the performance. null Garside describes IDIOMTAG (Garside et al., 1987) which is a component of a part-of-speech tagging system named CLAWS. IDIOMTAG serves as a front-end to the tagger and modifies some initially assigned tags in order to reduce the amount of ambiguity to be dealt with by the tagger. IDIOMTAG can look at any combination of words and tags, with or without intervening words. By using the IDIOMTAG, CLAWS system improved tagging accuracy from 94% to 96-97%. However, the manual-intensive process of producing idiom tags is very expensive although IDIOMTAG proved fruitful.</Paragraph>
    <Paragraph position="4"> Kupiec (Kupiec, 1992) describes a technique of augmenting the Hidden Markov Models for part-of-speech tagging by the use of networks.</Paragraph>
    <Paragraph position="5"> Besides the original states representing each part-of-speech, the network contains additional states to reduce the noun/adjective confusion, and to extend the context for predicting past participles from preceding auxiliary verbs when they are separated by adverbs. By using these additional states, the tagging system improved the accuracy from 95.7% to 96.0%. However, the additional context is chosen by analyzing the tagging errors manually.</Paragraph>
    <Paragraph position="6"> An automatic refining technique for Hidden Markov Models has been proposed by Brants (Brants, 1996). It starts with some initial first order Markov Model. Some states of the model are selected to be split or merged to take into account their predecessors. As a result, each of  new states represents a extended context. With this technique, Brants reported a performance cquivalent to the second order Hidden Markov Models.</Paragraph>
    <Paragraph position="7"> In this paper, we present an automatic refining technique for statistical language models. First, we examine the distribution of transitions of lexicalized categories. Next, we break out the uncommon ones from their categories and make new states for them. All processes are automated and the user has only to determine the extent of the breaking-out.</Paragraph>
  </Section>
class="xml-element"></Paper>