<?xml version="1.0" standalone="yes"?>
<Paper uid="C00-2136">
  <Title>Automatic Acquisition of Domain Knowledge for Information Extraction</Title>
  <Section position="6" start_page="942" end_page="944" type="evalu">
    <SectionTitle>
4 Results
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="942" end_page="943" type="sub_section">
      <SectionTitle>
4.1 Event Extraction
</SectionTitle>
      <Paragraph position="0"> 'l'he, most nal;m'a.l measm'e of efl'ecl;iveness of our discovery procedure is the performmme of ml extraction systmn using the, discovered t)~tterns.</Paragraph>
      <Paragraph position="1"> However, il; is not 1)ossil)le to apply this reel;rio direei;ly because the discovered t)al;terns lack some of the information required tbr entries ill :{\V('. did not el)serve a significam; difl'erencc in 1)crfi)rlIiHl\[CO, bet, ween the two tormulas 4 alt(t 5 in o111&amp;quot; experiin(mrs; the results whit:h tbllow use 5.</Paragraph>
      <Paragraph position="2">  the pattern base: information about the event type (predicate) associated with the pattern, and the mapping from pattern elements to predicate arguments. We have evaluated ExDIsco by manually incorporating the discovered patterns into the Proteus knowledge bases and running a full MUC-style evaluation.</Paragraph>
      <Paragraph position="3"> We started with our extraction system, Protens, which was used in MUC-6 in 1995, and has undergone continual improvements since the MUC evaluation. We removed all the scenario-specific clause and nominalization patterns. 4 We then reviewed all the patterns which were generated by the ExDIsco, deleting those which were not relewmt to the task, or which did not correspond directly to a predicate already implemented tbr this task) The remaining pat;terns were augmented with intbnnation about the corresponding predicate, and the relation between the pattern and the predicate al'guments, a The resulting variants of Proteus were applied to the formal training corpus and the (hidden) formal test corpus for MUC-6, and the output evaluated with the MUC scorer.</Paragraph>
      <Paragraph position="4"> The results on the training corpus are:  give rise to scenario events. For example, &amp;quot;Mr Smith, former president of IBM&amp;quot;, may produce an event record where l%ed Smith left IBM. These patterns were left in Proteus for all the runs, and they make some contribution to the relatively high baseline scores obtained using just the seed event patterns.</Paragraph>
      <Paragraph position="5"> ~ExD~sco found patterns which were relevant to the task lint could not be easily aceomodated in Proteus. For instance &amp;quot;X remained as president&amp;quot; could be relevant, particularly in the case of a merger creating a new corporate entity, but Proteus was not equipped to trundle such iIfformation, and has not yet been extended to incorporate such patterns.</Paragraph>
      <Paragraph position="6">  The tables show the recall and precision measures for the patterns, with F-measure being the harmonic mean of the two. The Seed pattern base consists of just the initial pattern set, given in the table on the previous page. ~ib this we added the patterns which the system discovered automatically after about 100 iterations, producing the pattern set called ExDIsco. For comparison, M anual-MUC is the pattern base lnanually develot)ed on the MUC-6 training corpus-1)repared over the course of 1 month of full-time work by at least one computational linguist (during which the 100-document training corpus was studied in detail). The last row, Manual-now, shows the current pertbrmance of the Proteus system. The base called Ultiolt contains the union of ExDIScO and Manual-No'w.</Paragraph>
      <Paragraph position="7"> We find these results very encouraging: Proteus performs better with the patterns discovered by ExI)IscO than it did after one month of manual tinting and development; in fact, this perfi)rmance is close to current levels, which are the result of substantial additional developmeut. These results umst be interpreted, however, with several caveats. First, Proteus performance depends on many fimtors besides the event patterns, such as the quality of name re, cognition, syntactic mmlysis, anaphora reso~ lution, inferencing, etc. Several of these were improved since the MUC formal evaluation, so some of the gain over the MUC formal evaluation score is attritmtable to these factors. How~ ever, all of the other scores are comparable in these regards. Second, as we noted above, the patterns were reviewed and augmented manually, so the overall procedure is not entirely automatic. However, the review and augmentation process took little time, as compared to the manual corpus analysis and development of the pattern base.</Paragraph>
    </Section>
    <Section position="2" start_page="943" end_page="944" type="sub_section">
      <SectionTitle>
4.2 Text filtering
</SectionTitle>
      <Paragraph position="0"> We can obtain a second measure of pertbrmance by noting that, in addition to growing the tmttern set, ExDIsco also grows the rele- null vance rankings of documents. The latter cnn be evahlated directly, wil;hollt human intervention. We tested Exl)IsC, o ~tgainst two cor\])orn: th(; 100 documents from MUC-6 tbrmal training, a:nd the 100 documents from the MUC-6 formal test (both are contained anlong the 10,000 ExDIsoO training set) r. Figure 1 shows recall t)\]otted against precision on the two corpora, over 100 iterations, starting with the seed patte, nls in section 3.d. This view on the discovery procedure is closely related to the MUC %exttill;ering&amp;quot; task, in which the systems are jlulged at the \]evel of doc,wm, e,'nt.s rather thmt event slots. It; is interesting to (:omt)m:e Exl)IsCO's results with how other MUC-6 part\]tit)ants performed on the MUC-b' test cortms , shown anonymously.</Paragraph>
      <Paragraph position="1"> ExDIscO attains values within the range of the MUC participald;S, all of which were either heavily-supervised or m~mually coded systems.</Paragraph>
      <Paragraph position="2"> II; is important to bear in mind that ExI)Isco had no benefit of training material, or any intbrmation beyond the seed pattern set.</Paragraph>
      <Paragraph position="3"> Figure 2 shows the 1)ertbrmance, of text filtering on the Acquisition task, again, given the seed in section 3.4. ExDisco was trained on |;lie same WSJ eorlms, and tested against a set of 200 documents. We retrieved this set using keyword-based IR, search, and judged their relevance by halId.</Paragraph>
      <Paragraph position="4"> rThesc judgements constituted the truth which was used only for evaluation, not visible to ExDISCO</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>