<?xml version="1.0" standalone="yes"?>
<Paper uid="M91-1016">
  <Title>SYNCHRONETICS : MUC-3 TEST RESULTS AND ANALYSI S</Title>
  <Section position="3" start_page="108" end_page="108" type="metho">
    <SectionTitle>
ALLOCATION OF TIM E
</SectionTitle>
    <Paragraph position="0"> The most time-consuming of our activities were the development of the semantic net software, and th e development of the phrase and sentence interpreters . Next came the development of the grammars for the two parsers, and the template generation software . Development of the dictionary was quite rapid, thank s to our automatic acquisition software . The activity we spent the least amount of time on was the encodin g of world knowledge into the knowledge base .</Paragraph>
  </Section>
  <Section position="4" start_page="108" end_page="109" type="metho">
    <SectionTitle>
LIMITING FACTORS
</SectionTitle>
    <Paragraph position="0"> Our primary limiting factor was the tenuous nature of the lines of communication between our team members .</Paragraph>
    <Paragraph position="1"> With personnel spread across six different sites, we were forced to rely on weekly meetings to resolve problems that would ordinarily be cleared up on a daily basis if everyone were working at the same site .</Paragraph>
    <Paragraph position="2">  The second limiting factor for our system was the amount of time we had available to us . Most of the system was developed from scratch (only the NL-Builder software was written prior to the commencemen t of our project). We had only a few weeks between the time we were first able to process 100 texts and th e time that the final test was due. Thus, we were unable to be as careful as we would have liked to be in th e development of the final system configuration .</Paragraph>
    <Paragraph position="3"> The third limiting factor for our system was the lack of a detailed and well thought out world model . We did most of our development using a very small world model that had fewer than 50 concepts . Just before running the final test, we quickly developed and switched to a world model containing almost 90 0 concepts. However, we did not have time to examine it closely before running the test . We believe that we could considerably increase our system's performance for the slots we are currently filling by improving th e world model .</Paragraph>
  </Section>
  <Section position="5" start_page="109" end_page="109" type="metho">
    <SectionTitle>
SUCCESSES AND FAILURES
</SectionTitle>
    <Paragraph position="0"> Our biggest successes were the development of the dictionary, and the speed of the parsers. Our automati c acquisition software allowed us to obtain a dictionary of 10000 words quite painlessly . Together, both parsers took less than one hour to process every word of all 100 texts, running on a DecStation 3100 .</Paragraph>
    <Paragraph position="1"> Our biggest failures were lack of development of the knowledge base and the speed of the semantic net I/ O routines . Our knowledge base was a last-minute effort, which significantly degraded system performance . The semantic net I/O routines were slow enough to be the main time drain on the three non-parser components .</Paragraph>
    <Paragraph position="2"> For these reasons the knowledge base and the semantic net I/O routines are our prime candidates to b e rewritten .</Paragraph>
  </Section>
  <Section position="6" start_page="109" end_page="109" type="metho">
    <SectionTitle>
REUSABILITY
</SectionTitle>
    <Paragraph position="0"> We expect to be able to reuse all system components except for the template generator in other projects .</Paragraph>
    <Paragraph position="1"> We are currently working on a project to automatically convert linear text to hypertext . We plan to use ou r MUC system as the front end to the conversion system . This will require only the development of software to generate hypertext links based on the semantic net built by the MUC system, and the development of a new knowledge base for the target domain .</Paragraph>
  </Section>
  <Section position="7" start_page="109" end_page="111" type="metho">
    <SectionTitle>
LESSONS LEARNED
</SectionTitle>
    <Paragraph position="0"> Participation in MUC-3 has led us to the following conclusions : * Our software engineering paradigm (which is thrust upon us by virtue of the fact that our personne l are spread out across several sites) is a poor one, but it is not fatal .</Paragraph>
    <Paragraph position="1"> * Several person-years of work is needed to build a parser-based system that has the poieniial to do well at the MUC task . Even then, a weakness in any component can easily reduce the system's abilities t o those of a stupid keyword-matching system.</Paragraph>
    <Paragraph position="2">  * Evaluation of natural language processing systems through a MUC-like competition is significantl y complicated by the fact that it is hard to know what is being measured . Nonetheless, we believe that our architecture will be excellent for evaluation of the various components of a natural languag e processing system, because we will be able to mix and match the components that go into our system . We will have this flexibility because each of our components is a stand-alone program, and because al l of our programs communicate with each other via the same semantic net representation language . For example, if we develop both a script-processing component and an anaphora component, we will be abl e to put them together in either order, or omit either or both of them . By comparing the results of eac h of these configurations, we will gain insight into the relative merits of these two forms of processing .</Paragraph>
  </Section>
class="xml-element"></Paper>