<?xml version="1.0" standalone="yes"?>
<Paper uid="W97-1311">
  <Title>Event Coreference for Information Extraction</Title>
  <Section position="7" start_page="79" end_page="79" type="evalu">
    <SectionTitle>
6 Evaluation
</SectionTitle>
    <Paragraph position="0"> We have not been able to carry out direct evaluation of our approach to event coreference. To do so would require manually annotating coreferential events in a corpus of significant size, and we have not had the resources to do so. However, we have attempted to gain some indirect measure of the successfulness of the approach by toggling event coreference on and off and observing the effect on the ability of the system to fill MUC-6 management succession templates correctly. The hypothesis here is that effective event coreference will lead to higher scores in the template filling task for at least two reasons. First, role players in events (which become slot fillers in the scored templates, e.g. persons and organisations) should become available due to event coreference. Second, spurious succession events should be eliminated due to proper event coreference.</Paragraph>
    <Paragraph position="1"> The MUC-6 management succession scenario task involved filling an object-oriented template consisting of five objects, each with associated slots (twenty slots in total). The top level object was a template object and contained one or more succession_event objects which in turn contained an organization object and one or more in_and_out objects, themselves containing organization and person objects (a precise definition of the template and the task can be found in DARPA (1995)).</Paragraph>
    <Paragraph position="2"> Table 1 shows the gross results of running the system against the 100 articles in the MUC-6 scenario task test corpus. Our system is easily reconfigured to run with or without attempting event coreference. The two rows in the table show the effects without and with event coreference. The 'Overall' column show the effects on the overall scenario template filling task, i.e., on recall and precision scores for all objects and slots in the templates. The 'Succession Events' column shows the effect just for the succession_event objects in the templates, and is therefore a more direct measure of template filling performance where we might expect event coreference to have an effect.</Paragraph>
    <Paragraph position="3"> As can be seen from the table, the effect over-all is not particularly significant. However, the effect on succession events alone is more substantial, with precision going up five percentage points and recall dropping only one, when event coreference is switched on. Closer examination revealed that the event coreference mechanism successfully avoided the proposal of 11 spurious succession events in the evaluation corpus, which included 196 possible events.</Paragraph>
    <Paragraph position="4"> We stress that this is a crude measure of our event coreference algorithm - really just an indication of its utility in the information extraction task. However, even as such, it shows that the algorithm is performing correctly, on balance, and that event coreference is worth addressing in an IE system.</Paragraph>
  </Section>
class="xml-element"></Paper>