<?xml version="1.0" standalone="yes"?>
<Paper uid="M93-1019">
  <Title>SRI: Description of the JV-FASTUS System Used for MUC-5</Title>
  <Section position="2" start_page="221" end_page="222" type="intro">
    <SectionTitle>
RESULTS OF THE EVALUATIO N
</SectionTitle>
    <Paragraph position="0"> The EJV-FASTUS system achieved an error rate per slot fill of 0 .70 and richness-normalized error of 0 .82 .</Paragraph>
    <Paragraph position="1"> The error rate was the third-best result reported, and was exceeded only by systems that had significantly longer domain-specific development time . The richness-normalized error was the second best of all systems reported . This error rate corresponds to a recall of 34%, a precision of 56%, and an equally weighted F-metri c of 42.67 .</Paragraph>
    <Paragraph position="2"> The JJV-FASTUS system achieved an error rate per slot fill of 0 .70 and richness-normalized error of 0.79. This was also the third-best error rate of all sites reported, and was exceeded only by systems that ha d significantly longer development time. This error rate corresponds to a recall of 34%, a precision of 62% , and an equally weighted F-meteric of 44 .21.</Paragraph>
    <Paragraph position="3"> After the MUC-4 conference, SRI embarked on an effort to rationalize our MUC-4 system, refine it s overall architecture, and add a user interface to facilitate defining and maintaining the finite-state transducer s comprising the system . With the exception of a very skeletal system for English joint ventures based on a corpus of about 10 articles extracted by hand from current issues of the Wall Street Journal, which was used a s the basis of a demonstration at the ARPA Human Language Technology (HLT) meeting, no domain-specifi c development was undertaken until the beginning of April 1993 . The first end-to-end test of EJV-FASTU S on 100 texts was conducted on April 30, with the result of an unimpressive error rate of 0 .93 (F-measure  A month later, at the end of May, we ran the first end-to-end test of JJV-FASTUS on 100 texts, with an equally unimpressive error rate of 0 .93 (F-measure 12 .69). Although we had developed a version of FASTU S called MIMI (a Japanese word for &amp;quot;ears&amp;quot;) for information extraction from Japanese spoken dialogues [3] , M I M I's base system is really the same as the English system, since it operates on a Romaji encoding of speec h rather than on Kanji characters . It was also in the domain of conference room scheduling. JJV-FASTUS was thus developed from scratch on an entirely new character basis and a new domain .</Paragraph>
    <Paragraph position="4"> We succeeded in raising the system performance from this baseline to our reported results in 3 months  of work. We feel that this experience confirms the adequacy of the tools provided by JV-FASTUS for the rapid development of information extraction systems in new domains and in new languages . We stopped development of the English system at approximately 3 :00 p .m. on August 1 . At that time, our improvemen t curve was still extremely steep . Work during the morning of August 1 resulted in a 0 .5 point improvement in F-measure . We had not even attempted to produce revenue objects, and our treatment of times an d facilities was extremely sketchy . The improvement curve of the Japanese system was even steeper . When we stopped development at about 7 :00 p.m., the F-measure had climbed 2 .8 points in one day of work on fixing the postprocessing routine .</Paragraph>
  </Section>
class="xml-element"></Paper>