<?xml version="1.0" standalone="yes"?>
<Paper uid="M91-1012">
  <Title>MCDONNELL DOUGLAS ELECTRONIC SYSTEMS COMPANY : MUC-3 Test Results and Analysi s</Title>
  <Section position="4" start_page="92" end_page="92" type="metho">
    <SectionTitle>
LIMITING FACTOR S
</SectionTitle>
    <Paragraph position="0"> Our main problems resulted from working with a new and incomplete system . Too ofte n we had to devote our time to fixing bugs or making improvements in the skimmer, in th e graphic representation tools, and in the knowledge addition tools . Starting with a smal l vocabulary and little linguistic and domain knowledge was disadvantageous . Adding a lot of knowledge to the system over a short period of time caused many problems to surface (e .g., initializing the system became a time-waster, system tables overflowed several times) . Lack of an internal representation for information extracted from the text was yet another limitation o n development.</Paragraph>
  </Section>
  <Section position="5" start_page="92" end_page="92" type="metho">
    <SectionTitle>
TRAININ G
</SectionTitle>
    <Paragraph position="0"> We used the entire development corpus, including the key templates, for gatherin g domain information . For example, we used the key templates to get fairly complete lists o f perpetrators, targets, instruments, and so on . Similarly, we searched the corpus for keywords , temporal, locative, and other patterns . Many of our domain-specific grammar rules wer e crafted using the results of such searches.</Paragraph>
    <Paragraph position="1"> The first 100 messages of the development set served as a primary development an d testing vehicle. TST1 messages were run occasionally in order to gauge progress on unseen message sets.</Paragraph>
    <Paragraph position="2"> In order to shake out bugs in the system, we processed half the development set in batches of 100 messages several days before the testing deadline .</Paragraph>
  </Section>
  <Section position="6" start_page="92" end_page="93" type="metho">
    <SectionTitle>
STRENGTHS AND WEAKNESSE S
</SectionTitle>
    <Paragraph position="0"> In general, skimming worked much better than expected . Based merely on our initial results for MUC3, we conclude that skimming is a powerful adjunct to deeper processing of text .</Paragraph>
    <Paragraph position="1"> We feel that with several months' work, continued development of skimming techniques , combined with knowledge base and vocabulary development, would substantially raise our MUC 3 score.</Paragraph>
    <Paragraph position="2"> Skimming provides extremely fast, simple, and robust text processing . While keyword and pattern-based methods for NLP have usually met with scorn, we feel a review of thes e methods is called for.</Paragraph>
    <Paragraph position="3"> On the other hand, we are aware of the limitations of any approach that doesn't analyz e text as deeply as possible . In order to segment incidents with great accuracy, linguistic contex t  as well as script-level understanding of the text are required . Many reference resolutio n problems also require such knowledge .</Paragraph>
    <Paragraph position="4"> In the near future, we will merge our skimming capability with a bottom-up syntacti c analysis mechanism, and also incorporate script-based understanding mechanisms . The INLET customization tools have proved their worth by supporting hierarchy , grammar rule, and vocabulary addition. Even our qualified success would have been impossibl e without the effectiveness of the knowledge addition framework .</Paragraph>
  </Section>
  <Section position="7" start_page="93" end_page="93" type="metho">
    <SectionTitle>
HITS AND MISSES
</SectionTitle>
    <Paragraph position="0"> Our system is fairly good at determining the incident type, using a hierarchy of key words and patterns. With just a few specialized rules, the system is able to process appositive s to find perpetrators, perpetrator organizations, physical targets, and human targets . An extensive temporal grammar was developed, though not much correlation of multiple tempora l references has been implemented . A similar situation holds for locative phrases .</Paragraph>
    <Paragraph position="1"> Simple gaps in knowledge and vocabulary caused many misses on the TST2 messages .</Paragraph>
    <Paragraph position="2"> Missing vocabulary (e .g., &amp;quot;killings&amp;quot;), missing domain rules (e.g., &amp;quot;explosion caused damage&amp;quot;) , missing generic rules (e.g., &amp;quot;actor participated in action on object&amp;quot;), and missing mechanism s of various kinds led to substantially lower performance than we would expect of a more matur e INLET system . Missing mechanisms include lack of threat handling, lack of any inferencin g capability, lack of spelling correction, and lack of rejection of incidents for even simple reasons (e.g., an abstract object such as the &amp;quot;economy&amp;quot; is attacked) .</Paragraph>
  </Section>
  <Section position="8" start_page="93" end_page="93" type="metho">
    <SectionTitle>
PORTABILIT Y
</SectionTitle>
    <Paragraph position="0"> Because INLET is a customization shell, portability of the specific knowledge added to th e system is not a major concern . In 2 man-months, we were able to achieve a 25% recall and 35% precision score with a relatively immature INLET system . When the system is completed , we expect similar customization time to result in a better system for the particular domain an d task.</Paragraph>
    <Paragraph position="1"> The skimmer framework and knowledge addition framework are generic, as is the core knowledge base and vocabulary . On top of this layer is a substantial body of domain-specifi c code and knowledge, which would necessarily have to be replaced for a new domain and task .</Paragraph>
  </Section>
  <Section position="9" start_page="93" end_page="94" type="metho">
    <SectionTitle>
LESSONS LEARNED
</SectionTitle>
    <Paragraph position="0"> We have demonstrated that the INLET knowledge addition framework and skimmer ca n quickly support customization to a new domain and task . We have found that the graphic interface for knowledge addition has speeded up customization over a system like VOX . Finally, we have found that skimming is a critical adjunct to deeper NLP .</Paragraph>
    <Paragraph position="1"> Our participation in MUC3 has shown us the high value of formal testing and compariso n with other NLP efforts . We intend to continue using the MUC3 corpus and testing system for ou r system development, test, and evaluation .</Paragraph>
  </Section>
class="xml-element"></Paper>