<?xml version="1.0" standalone="yes"?>
<Paper uid="M92-1001">
  <Title>OVERVIEW OF THE FOURTH MESSAGE UNDERSTANDIN G EVALUATION AND CONFERENC E</Title>
  <Section position="2" start_page="0" end_page="3" type="intro">
    <SectionTitle>
INTRODUCTIO N
</SectionTitle>
    <Paragraph position="0"> The Fourth Message Understanding Conference (MUC-4) is the latest in a serie s of conferences that concern the evaluation of natural language processing (NLP ) systems. These conferences have reported on progress being made both in th e development of systems capable of analyzing relatively short English texts and i n the definition of a rigorous performance evaluation methodology . MUC-4 was preceded by a period of intensive system development by each of the participating organizations and blind testing using materials prepared by NRaD and SAIC tha t are described in this paper, other papers in this volume, and the MUC- 3 proceedings [1] .</Paragraph>
    <Paragraph position="1"> The overall objective of the evaluations is to advance our understanding of th e merits of current text analysis techniques, as applied to the performance of a realistic information extraction task . As a task, information extraction require s &amp;quot;understanding&amp;quot; of the texts, but it presents a more limited challenge than would a task requiring production of an in-depth representation of the contents o f complete texts . The inputs to the analysis/extraction process consist of naturally occurring newswire texts that were obtained in electronic form. The outputs of th e process are a set of templates or semantic frames resembling the contents of a partially formatted database .</Paragraph>
    <Paragraph position="2"> MUC-3 and MUC-4 offer benchmarks for the field of NLP in general and fo r information extraction technology in particular . One of the fundamental ways i n which MUC-3 and MUC-4 are distinct from earlier efforts is in their choice of texts : MUC-3 and MUC-4 made use of news articles on the subject of Latin America n terrorism, whereas the previous conferences had made use of naval tactica l message narratives [2] . The MUC-4 evaluation and conference featured a n enhanced evaluation methodology, greater participation, and significantly mor e conclusive results than those recorded in the MUC-3 proceedings .</Paragraph>
    <Paragraph position="3"> Evaluating end-to-end systems in the context of a common task helps in severa l ways to bridge the gap between research and technology. First, it makes it easier for both technology producers and technology consumers to understand an d appreciate the value of the methods that are being explored and applied .</Paragraph>
    <Paragraph position="4"> It also formerly the Naval Ocean Systems Center (NOSC) .</Paragraph>
    <Paragraph position="5">  serves as an example of achievable results and inspires ideas for real-life applications such as statistical analysis or trends analysis of world events , routing/retrieval of texts of personal interest, and feeding of data to expert system s and database management systems . Finally, it encourages the development of large, experimental testbed systems in 'which to conduct research, and th e evaluation results can provide insight into research bottlenecks that are impedin g the development of full-scale, usable systems .</Paragraph>
    <Paragraph position="6"> This paper presents an overall view of the MUC-4 evaluation and, to a larg e extent, reflects the content of introductory presentations made at the conference. This paper is also an overview of the conference proceedings, which include s papers contributed by the organizations that participated in the evaluation (Part s II and III) and by individuals who were involved in designing aspects of th e evaluation (Part I) . The ordering of papers does not necessarily correspond to th e order in which the presentations were made during the conference . The proceedings also includes a number of appendices (Part IV) containing material s pertinent to the evaluation .</Paragraph>
  </Section>
class="xml-element"></Paper>