<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-0505">
  <Title>Efficient Hierarchical Entity Classifier Using Conditional Random Fields</Title>
  <Section position="9" start_page="37" end_page="38" type="evalu">
    <SectionTitle>
7 Results
</SectionTitle>
    <Paragraph position="0"> The major goal of this paper was to build a classifier that could learn all the WordNet synsets in a reasonable amount of time. We will first discuss the improvement in time needed for training and labeling and then discuss accuracy.</Paragraph>
    <Paragraph position="1"> We want to test the influence of the number of labels on the time needed for training. Therefor, we created different training sets, all of which had the same input (246 sentences tagged with POS labels), but a different number of labels. The first training set only had 5 labels (&amp;quot;ADJ&amp;quot;, &amp;quot;ADV&amp;quot;, &amp;quot;VERB&amp;quot;, &amp;quot;entity&amp;quot; and &amp;quot;NONE&amp;quot;). The second had the same labels except we replaced the label &amp;quot;entity&amp;quot; with either &amp;quot;physical entity&amp;quot;, &amp;quot;abstract entity&amp;quot; or &amp;quot;thing&amp;quot;. We continued this procedure, replacing parent nouns labels with their children (i.e.</Paragraph>
    <Paragraph position="2"> hyponyms) for subsequent training sets. We then trained both a CRF using a hierarchical feature selection and a standard CRF on these training sets.</Paragraph>
    <Paragraph position="3"> Fig. 5 shows the time needed for one iteration of training with different numbers of labels. We can see how the time needed for training slowly increases for the CRF using hierarchical feature selection but increases fast when using a standard CRF. This is conform to eq. 7.</Paragraph>
    <Paragraph position="4"> Fig. 6 shows the average time needed for labeling a sentence. Here again the time increases slowly for a CRF using hierarchical feature selection, but increases fast for a standard CRF, conform to eq. 6.</Paragraph>
    <Paragraph position="5"> Finally, fig 7 shows the error rate (on the training data) after each training cycle. We see that a standard CRF and a CRF using hierarchical feature selection perform comparable. Note that fig 7 gives the error rate on the training data but this  can differ considerable from the error rate on unseen data.</Paragraph>
    <Paragraph position="6"> After these tests on a small section of the Semcor corpus, we trained a CRF using hierarchical feature selection on 7/8 of the full corpus. We trained for 23 iterations, which took approximately 102 hours. Testing the model on the remaining 1/8 of the corpus resulted in an accuracy of 77.82%. As reported in (McCarthy et al., 2004), a baseline approach that ignors context but simply assigns the most likely sense to a given word obtains a accuracy of 67%. We did not have the possibility to compare the accuracy of this model with a standard CRF, since as already stated, training such a CRF takes impractically long, but we can compare our systems with existing WSD-systems.</Paragraph>
    <Paragraph position="7"> Mihalcea and Moldovan (Mihalcea and Moldovan, 1999) use the semantic density between words to determine the word sense. They achieve an accuracy of 86.5% (testing on the first two tagged files of the Semcor corpus). Wilks and Stevenson (Wilks and Stevenson, 1998) use a combination of knowledge sources and achieve an accuracy of 92%3. Note that both these methods use additional knowledge apart from the WordNet hierarchy.</Paragraph>
    <Paragraph position="8"> The sentences in the training and testing sets were already (perfectly) POS-tagged and noun chunked, and that in a real-life situation additional preprocessing by a POS-tagger (such as the LT-POS-tagger4) and noun chunker (such as described in (Ramshaw and Marcus, 1995)) which will introduce additional errors.</Paragraph>
  </Section>
class="xml-element"></Paper>