<?xml version="1.0" standalone="yes"?>
<Paper uid="M95-1002">
  <Title>OVERVIEW OF RESULTS OF THE MUC-6 EVALUATION</Title>
  <Section position="3" start_page="0" end_page="13" type="intro">
    <SectionTitle>
EVALUATION TASKS
</SectionTitle>
    <Paragraph position="0"> Documentation of the four evaluation tasks is contained in appendices C-F to this volume . A basic characterization of the challenge presented by each task is as follows :  * Named Entity (NE) -- Insert SGML tags into the text to mark each string that represents a person ,organization, or location name, or a date or time stamp, or a currency or percentage figure . * Coreference (CO) -- Insert SGML tags into the text to link strings that represent coreferring nou n phrases.</Paragraph>
    <Paragraph position="1"> * Template Element (TE) -- Extract basic information related to organization and person entities , drawing evidence from anywhere in the text.</Paragraph>
    <Paragraph position="2"> * Scenario Template (ST) -- Drawing evidence from anywhere in the text, extract prespecified eventinformation, and relate the event information to the particular organization and person entities involve d  in the event.</Paragraph>
    <Paragraph position="3"> The two SGML-based tasks required innovations to tie system-internal data structures to the original text s o that the annotations could be inserted by the system without altering the original text in any other way . This capability has other useful applications as well, e .g., it enables text highlighting in a browser . It also facilitates information extraction, since some of the information in the extraction templates is in the form of literal tex t strings, which some systems have in the past had difficulty reproducing in their output .</Paragraph>
    <Paragraph position="4"> The inclusion of four different tasks in the evaluation implicitly encouraged sites to design general-purpos e architectures that allow the production of a variety of types of output from a single internal representation in orde r  to allow use of the full range of analysis techniques for all tasks . Even the simplest of the tasks, Named Entity , occasionally requires in-depth processing, e .g., to determine whether &amp;quot;60 pounds&amp;quot; is an expression of weight or of monetary value . Nearly half the sites chose to participate in all four tasks, and all but one site participated in a t least one SGML task and one extraction task .</Paragraph>
    <Paragraph position="5"> The variety of tasks designed for MUC-6 reflects the interests of both participants and sponsors in assessin g and furthering research that can satisfy some urgent text processing needs in the very near term and can lead t o solutions to more challenging text understanding problems in the longer term . Identification of certain common types of names, which constitutes a large portion of the Named Entity task and a critical portion of the Templat e Element task, has proven to be largely a solved problem . Recognition of alternative ways of identifying an entit y constitutes a large portion of the Coreference task and another critical portion of the Template Element task an d has been shown to represent only a modest challenge when the referents are names or pronouns . The mix of challenges that the Scenario Template task represents has been shown to yield levels of performance that ar e smilar to those achieved in previous MUCs, but this time with a much shorter time required for porting.</Paragraph>
    <Paragraph position="6"> Summary scores for all systems evaluated are contained in appendix B . Note that for each task, sites were assigned different &amp;quot;code names&amp;quot; that were used in lieu of the site names to identify systems up to the time of th e conference . Some of the site reports in the proceedings may refer to other sites by these code names whe n discussing cross-system performance figures .</Paragraph>
  </Section>
class="xml-element"></Paper>