<?xml version="1.0" standalone="yes"?>
<Paper uid="M93-1020">
  <Title>TRW: DESCRIPTION OF THE DEFT SYSTEM AS USED FOR MUC- 5</Title>
  <Section position="4" start_page="244" end_page="248" type="evalu">
    <SectionTitle>
RESULTS
</SectionTitle>
    <Paragraph position="0"> The results of the final MUC evaluation were strongly influenced by th e unavailability of the parser, which was an essential component of the DEFT approac h to MUC-5. The resulting scores indicate the magnitude of the problems inherent in a simple pattern-matching strategy which is not informed with even a crude semanti c grammar. It should be noted that a decision Was made to focus only on a subset o f templates and slots required for the preliminary run . These were the documen t template, tie-up-relationship, and entity . The F-measures for the final evaluatio n  Not surprisingly, these were the lowest scores for any system in the evaluation . A detailed analysis of the run is of little utility, however there are some points o f interest seen in the walk-through sample document .</Paragraph>
    <Paragraph position="1"> Walkthrough Documen t The identifying data (document number, source, and date) were correctly extracted . Some simple atomic patterns were defined in a l)I :I &amp;quot;I' lexicon for tie-up relations . These were to be factored into a semantic grammar ; as noted, the parser was not available at the time of the run . &amp;quot;Therefore, the patterns were run as a simple search . It correctly identified the presence of a joint venture in the sample document , incorrectly instantiating two tie-up templates (one for each of two out of thre e references to the venture) and entering their ll)s in the content slot of the document template . DEFT currently does not determine that multiple references hav e a common object unless the frames overlap .</Paragraph>
    <Paragraph position="2"> A single entity was mis-identified, &amp;quot;Jiji Press Ltd .,&amp;quot; which is actually the documen t source. This entity was incorrectly associated with the first tie-up . The foregoin g explanation of the DEFT scoping mechanism makes it clear why this false associatio n  took place. The name of the &amp;quot;BRiDGIS'TONE SPORTS CO .&amp;quot; was correctly reconstructed from the corporate designator (&amp;quot;CO .&amp;quot;) and assigned to the first tie-up . The name of the joint venture, &amp;quot;BRIDGESTONE SPORTS TAIWAN CO .,&amp;quot; was also constructed and associate d with the second tie-up instance. No other features were correctly identified .</Paragraph>
    <Paragraph position="3"> Among the other corporate names, the algorithm used by DEFT would not hav e identified &amp;quot;UNION PRECISON CAS'T'ING CO .,&amp;quot; but did identify &amp;quot;&amp;quot;I'AGA CO .&amp;quot; However, this entity was considered out of scope of the tie-up templates and was (incorrectly) no t attached to one . DEFT had no facility for recognizing &amp;quot;BRIDGESTONE SPORTS&amp;quot; nor for tracking the reference to &amp;quot;TI IL NEW COMPANY .&amp;quot;</Paragraph>
    <Section position="1" start_page="245" end_page="245" type="sub_section">
      <SectionTitle>
What Worke d
</SectionTitle>
      <Paragraph position="0"> DEFT was effective at recognizing literal strings and 'patterns contained in it s lexicons. DEFT frequently generated correct entity names that were not in th e corporate name lexicon using a set of heuristics that reasoned backwards from a designator. For example, &amp;quot;BRIDGESTONE SPORTS CO .&amp;quot; was constructed . DEFT of course had little problem with the tagged items for the document template . These are precisely the kinds of elemental functions that DEFT is expected to perform well .</Paragraph>
      <Paragraph position="1"> DEFT recognized the occurrence of sonic of the joint ventures, based on a very limite d set of patterns that were originally defined for use in connection with a semantic grammar. This set could have been extended to produce improved recall had w e known the parser would not he available . &amp;quot;These few successes indicate that even a simple pattern-based approach can recognize concepts of this type in restricte d cases .</Paragraph>
    </Section>
    <Section position="2" start_page="245" end_page="246" type="sub_section">
      <SectionTitle>
What Failed
</SectionTitle>
      <Paragraph position="0"> The lexicons and extraction phases that were rapidly developed for MtJC-5 containe d some hugs that were not observed during training ; some corporate names were missed, for example, that should have been constructed . The chief failings were inadequate lexicons for identifying joint ventures and inadequate scoping . These two problems combined to suppress the instantiation of the many valid entities tha t DEFT found, but could not associate with a tie-up relation and therefore did no t report . In general, the system was configured to reduce the anticipate d overgeneration, with the expectation that tie-ups and entity relations would b e identified and scoped by the semantic grammar ; in the absence of the parser , undcrgeneration became severe .</Paragraph>
      <Paragraph position="1"> System Training and Resources Expended The effort expended on MUC-5 testing and documentation was approximately tw o person-weeks. System development activities undertaken independently of MUC- 5 were exploited for the Mt1C-5 evaluation run . These included:  I'he total level of effort for all actin ities impacting M1,1C-5 c\&amp;quot;as therefore roughly 5 . 5 person-months .</Paragraph>
      <Paragraph position="2">  As we have noted, key system components were ultimately unavailable for the MUC- 5 evaluation . Although we won't know &amp;quot;how we would have done&amp;quot; until th e components are completed and our internal tests against the MUG data are repeated, i t is our expectation that significant improvement will he obtained with a littl e additional effort-- although performance is neither expected nor required t o approach that of true NM systems, given our view of DEFT as an integration environment.</Paragraph>
      <Paragraph position="3"> Most of the effort in creating a new DEFT application usually centers on lexicon development. For MUC-5, most lexicons were batch loaded from the data supplied via the Consortium for Lexical Research . A few lexicons for joint venture identificatio n and scoping were developed manually . These were quite simple and their actua l creation required minimal time .</Paragraph>
      <Paragraph position="4"> Much of the time on MUG-5 was occupied with writing C-code for extraction routines, particularly for corporate names . The need to write so much code for a new application is a current weakness in DEFT which will he remedied to a degree whe n the parser becomes available .</Paragraph>
      <Paragraph position="5"> Of course, a key activity was the analysis of the test corpus and development of a semantic grammar appropriate to the IiJV problem . The results of this analysis wer e manifested in the tie-up relation lexicon and the BNF grammar for the parser . Only the former was ready in time for the evaluation . Analysis was a cyclical, iterativ e process; refinement continued during system training .</Paragraph>
      <Paragraph position="6"> DEFT system training consisted of a series of runs against samples of the trainin g corpus, utilizing the frame review tool to examine the results . Lexicons were manually refined as a result of missed objects and false hits . Early runs resulted i n changes to the hatch loading sequence for some of the lexicons (e.g. the corporat e designators) . Feedback into the grammar would also have been derived from thi s process, had the parser been available and time permitted . As it was, time was insufficient even for lexicon refinement ; for example, a few key errors in th e corporate designator lexicon resulting from a hug in the program that prepared th e file provided through the Consortium for hatch uploading were noted only after th e final evaluation run was analyzed . This was partially responsible for some of th e undergeneration .</Paragraph>
    </Section>
    <Section position="3" start_page="246" end_page="248" type="sub_section">
      <SectionTitle>
What We Learned
</SectionTitle>
      <Paragraph position="0"> It came as no surprise that simple patterns are inadequate to extract the complex ideas expressed in the IiJV documents . We view the results as validating the concep t that DEFT, operating as a standalone system, is best qualified to perform on problem s involving well-defined, constrained sets of text objects to be extracted, even with the addition of a &amp;quot;meta-pattern&amp;quot; or grammatical capability . DEFT should excel on such problems when throughput is a major consideration .</Paragraph>
      <Paragraph position="1"> The selection (and on-going implementation) of a mechanism for expressing meta-patterns that is compatible with all of the goals discussed earlier is a major outcom e of our MUC work, even though it was not available in time . We believe that thi s approach will significantly empower DEFT and broaden the range of applications fo r which it is a suitable tool, while increasing the flexibility with which it can be integrated with other text analysis tools . This will prove highly valuable to ou r  current government customers, as well as future DL F1' users in the government o r commercial sector .</Paragraph>
      <Paragraph position="2"> DEFT's potential as an integration environment was underscored by the fact that w e successfully ran documents through : * A complex set of extraction phase s * With extremely large lexicon s that are beyond the scope of anything that has been tried in existing DEF T applications . The robustness of the architecture and efficiency of the patter n searches were our major consolation in the MUG-5 evaluation . We therefore look .for opportunities to combine DEFT's system engineering and search capabilities with th e sophisticated analytical power of NI .U-based solutions when real-world problems ar e encountered which are out of scope of DEFI&amp;quot;s simple extraction mechanisms .</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>