<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-0705">
  <Title>Applying Coreference to Improve Name Recognition</Title>
  <Section position="10" start_page="99" end_page="99" type="evalu">
    <SectionTitle>
8 Experiments
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="99" end_page="99" type="sub_section">
      <SectionTitle>
8.1 Training and Test Data
</SectionTitle>
      <Paragraph position="0"> For our experiments, we used the Beijing University Insititute of Computational Linguistics corpus - 2978 documents from the People's Daily in 1998, one million words with name tags - and the training corpus for the 2003 ACE evaluation, 223 documents. 153 of our ACE documents were used as our test set.</Paragraph>
      <Paragraph position="1">  The 153 documents contained 1614 names. Of the system-tagged names, 959 were considered 'obscure': were not on a name list and had a margin below the threshold. These were the names to which the rules and classifier were applied. We ran all the following experiments using the MUC scorer.  The test set was divided into two parts, of 95 documents and 58 documents. We trained two name tagger and classifier models, each time using one part of the test set along with all the other documents, and evaluated on the other part of the test set. The results reported here are the combined results for the entire test set.</Paragraph>
    </Section>
    <Section position="2" start_page="99" end_page="99" type="sub_section">
      <SectionTitle>
8.2 Overall Performance Comparison
</SectionTitle>
      <Paragraph position="0"> Table 4 shows the performance of the baseline system; Table 5 the system with rule-based corrections; and Table 6 the system with both rules and the SVM classifier.</Paragraph>
      <Paragraph position="1">  Table 6 Results for Single Document System The gains we observed from coreference within single documents suggested that further improvement might be possible by gathering evidence from several related documents.  We did this in two stages. First, we clustered the 153 documents in the test set into 38 topical clusters. Most (29) of the clusters had only two documents; the largest had 28 documents. We then applied the same procedures, treating the entire cluster as a single document. This yielded another 1.0% improvement in overall F score (Table 7).</Paragraph>
      <Paragraph position="2"> The improvement in F score was consistent for the larger clusters (3 or more documents): the F score improved for 8 of those clusters and remained the same for the 9 th . To heighten the multi-document benefit, we took 11 of the small  Borthwick (1999) did use some cross-document information across the entire test corpus, maintaining in effect a name cache for the corpus, in addition to one for the document. No attempt was made to select or cluster documents.</Paragraph>
      <Paragraph position="3"> (2 document clusters) and enlarged them by retrieving related documents from sina.com.cn. In total, we added 52 texts to these 11 clusters. The net result was a further improvement of 0.3% in F score (Table 8).</Paragraph>
    </Section>
    <Section position="3" start_page="99" end_page="99" type="sub_section">
      <SectionTitle>
8.3 Contribution of Coreference Features
</SectionTitle>
      <Paragraph position="0"> Since feature selection is crucial to SVMs, we did experiments to determine how precision increased as each feature was added. The results are shown in Figure 3. We can see that each feature in the SVM helps to select correct names from the output of the baseline name tagger, although some (like FirstMention) are more crucial than others.</Paragraph>
      <Paragraph position="1">  Scores are still computed on the 153 test documents ; the retrieved documents are excluded from the scoring.</Paragraph>
    </Section>
    <Section position="4" start_page="99" end_page="99" type="sub_section">
      <SectionTitle>
8.4 Comparison to Cache Model
</SectionTitle>
      <Paragraph position="0"> Some named entity systems use a name cache, in which tokens or complete names which have been previously assigned a tag are available as features in tagging the remainder of a document.</Paragraph>
      <Paragraph position="1"> Other systems have made a second tagging pass which uses information on token sequences tagged in the first pass (Borthwick 1999), or have used as features information about features assigned to other instances of the same token (Chieu and Ng 2002). Our system, while more complex, makes use of a richer set of global features, involving the detailed structure of individual mentions, and in particular makes use of both name - name and name - nominal relations.</Paragraph>
      <Paragraph position="2"> We have compared the performance of our method (applied to single documents) with a voted cache model, which takes into account the number of times a particular name has been previously assigned each type of tag:  Compared to a simple voted cache model, our model provides a greater improvement in name recognition F score; in particular, it can substantially increase the precision of name recognition. The voted cache model can recover some missed names, but at some loss in precision.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>