<?xml version="1.0" standalone="yes"?>
<Paper uid="N03-1002">
  <Title>Japanese Named Entity Extraction with Redundant Morphological Analysis</Title>
  <Section position="6" start_page="0" end_page="0" type="intro">
    <SectionTitle>
4 Evaluation
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.1 Data
</SectionTitle>
      <Paragraph position="0"> We use CRL NE data (IREX Committee, editor, 1999) for evaluation of our method. CRL NE data includes 1,174 newspaper articles and 19,262 NEs. We perform five-fold cross-validation on several settings to investigate the length of contextual feature, the size of redundant morphological analysis, feature selection and the degree of polynomial Kernel functions. For the chunk tag scheme we use IOB2 model since it gave the best result in a pilot study. F-Measure (a11 a2 a50 ) is used for evaluation.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.2 The length of contextual feature
</SectionTitle>
      <Paragraph position="0"> Firstly, we compare the extraction accuracies of the models by changing the length of contextual features and the direction of chunking. Table 4 shows the result in accuracy for each of NEs as well as the total accuracy of all NEs. For example, &amp;quot;L2R2&amp;quot; denotes the model that uses the features of two preceding and two succeeding characters. &amp;quot;For&amp;quot; and &amp;quot;Back&amp;quot; mean the chunking direction: &amp;quot;For&amp;quot; specifies the chunking direction from left to right, and &amp;quot;Back&amp;quot; specifies that from right to left.</Paragraph>
      <Paragraph position="1"> Concerning NE types except for &amp;quot;TIME&amp;quot;, &amp;quot;Back&amp;quot; direction gives better accuracy for all NE types than &amp;quot;For&amp;quot; direction. It is because suffixes are crucial feature for NE extraction. &amp;quot;For&amp;quot; direction gives better accuracy for &amp;quot;TIME&amp;quot;, since &amp;quot;TIME&amp;quot; often contains prefixes such as &amp;quot; a12a14a13 &amp;quot;(a.m.) and &amp;quot;a12a14a15 &amp;quot;(p.m.).</Paragraph>
      <Paragraph position="2"> &amp;quot;L2R2&amp;quot; gives the best accurary for most of NE types.</Paragraph>
      <Paragraph position="3"> For &amp;quot;ORGANIZATION&amp;quot;, the model needs longer contextual length of features. The reason will be that the key prefixes and suffixes are longer in this NE type such as &amp;quot; a16a18a17a18a19a21a20 &amp;quot;(company limited) and &amp;quot; a22a24a23a21a25 &amp;quot;(research institute). null</Paragraph>
    </Section>
    <Section position="3" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.3 The depth of redundant morphological analysis
</SectionTitle>
      <Paragraph position="0"> Table 5 shows the results when we change the depth (the value n of the n-best answers) of redundant morphological analysis.</Paragraph>
      <Paragraph position="1"> Redundant outputs of morphological analysis slightly improve the accuracy of NE extraction except for numeral expressions. The best answer seems enough to extract numeral experssions except for &amp;quot;MONEY&amp;quot;. It is because numeral expressions do not cause much errors in morphological analysis. To extract &amp;quot;MONEY&amp;quot;, the model needs more redundant output of morphological analysis. A typical occurs at &amp;quot;a26a28a27a30a29a32a31a34a33 &amp;quot; (Canadian dollars = MONEY) which is not including training data and is analyzed as &amp;quot;a26a35a27a14a29 &amp;quot; (Canada = LOCATION). The similar error occurs at &amp;quot;a36a21a37a38a31a10a33 &amp;quot; (Hong Kong dollars) and so on.</Paragraph>
    </Section>
    <Section position="4" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.4 Feature selection
</SectionTitle>
      <Paragraph position="0"> We use POS tags, characters, character types and NE tags as features for chunking. To evaluate how they are  3-best answers of redundant morphological analysis, Feature(POS, Character, Character Type and NE tag), Polynomial kernel of degree 2.</Paragraph>
      <Paragraph position="1"> effective we test four settings, that is, &amp;quot;using all features (ALL)&amp;quot;, &amp;quot;excluding characters (a51 Char.)&amp;quot;, &amp;quot;excluding character types (a51 Char. Type)&amp;quot; and &amp;quot;excluding subcategory of POS tags (a51 POS subcat.)&amp;quot;. Table 6 shows the results for these settings.</Paragraph>
      <Paragraph position="2"> &amp;quot;Excluding Characters&amp;quot; gives the worst accuracy, implying that characters are indispensable for NE extraction. &amp;quot;Excluding POS subcat.&amp;quot; results in worse accuracy. Some subcategories of POS include semantic information for proper nouns such that name, organization and location, and they are useful for NE extraction.</Paragraph>
      <Paragraph position="3"> For numeral expressions, &amp;quot;excluding Char Type&amp;quot; gives better accuracy. The reason is that numbers in Kanji are not defined in our character type definition.</Paragraph>
    </Section>
    <Section position="5" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.5 The degree of polynomial Kernel functions
</SectionTitle>
      <Paragraph position="0"> We alter degrees of kernel functions and check how the combination of features affects the results. As shown in Table 7, degree 2 gives the best accuracy for most of NE types. The result shows that the combination of two features is effective for extract NE extraction. However, the tendency is not so significant in numeral expressions.</Paragraph>
    </Section>
    <Section position="6" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.6 The effect of thesaurus
</SectionTitle>
      <Paragraph position="0"> Character Types and NE tags.</Paragraph>
      <Paragraph position="1"> In the experimentation above, we follow the features used in the preceding work (Yamada et al., 2002). Isozaki  feature set. Table 8 shows the result when the class names in the thesaurus is used as features. Note that we introduced the leaf node tag for each morpheme. The thesaurus information is effective for NEs except for &amp;quot;ARTIFACT&amp;quot; and &amp;quot;TIME&amp;quot;. Since &amp;quot;ARTIFACT&amp;quot; includes many unseen expressions, even if we introduce the information of the thesaurus, we cannot improve this model. Concerning &amp;quot;TIME&amp;quot;, the words and characters in this NE type are limited. The information of thesaurus may not be necessary for &amp;quot;TIME&amp;quot; expression extraction. In this paper, we did not encode the tree structure of the thesaurus. Introducing hierarchical relationships in the thesaurus is one of our future works.</Paragraph>
    </Section>
    <Section position="7" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.7 Discussion
</SectionTitle>
      <Paragraph position="0"/>
    </Section>
  </Section>
class="xml-element"></Paper>