<?xml version="1.0" standalone="yes"?>
<Paper uid="P98-1011">
  <Title>Evaluating a Focus-Based Approach to Anaphora Resolution*</Title>
  <Section position="7" start_page="76" end_page="77" type="evalu">
    <SectionTitle>
6 Results
</SectionTitle>
    <Paragraph position="0"> The MUC scorer does not distinguish between different classes of anaphora (pronouns, definite noun phrases, bare nouns, and proper nouns), but baseline figures can be established by running the LaSIE system with no attempt made to resolve any pronouns:  Training: 42.47. 73.67. 52.67.</Paragraph>
    <Paragraph position="1"> Evaluation: 44.77. 73.97. 55.77.</Paragraph>
    <Paragraph position="2"> LaSIE with the simple pronoun resolution heuristics of the non-focus-based mechanism achieves the following:  Training: 58.27. 71.37. 64.17.</Paragraph>
    <Paragraph position="3"> Evaluation : 56.07. 70.27. 62.37.</Paragraph>
    <Paragraph position="4"> showing that more than three quarters of the estimated 20% of pronoun coreferences in the corpora are correctly resolved with only a minor loss of precision.</Paragraph>
    <Paragraph position="5"> LaSIE with the focus-based algorithm achieves the following: ~Recall is a measure of how many correct (i.e. manually annotated) coreferences a system found, and precision is a measure of how many coreferences that the system proposed were actually correct. For example, with 100 manually annotated coreference relations in a corpus and a system that proposes 75, of which 50 are correct, recall is then 50/100 or 50% and precision is 50/75 or 66.7%.</Paragraph>
    <Paragraph position="6">  Training: 55.47. 70.37. 61.97.</Paragraph>
    <Paragraph position="7"> Evaluation: 53.37. 69.77. 60.47.</Paragraph>
    <Paragraph position="8"> which, while demonstrating that the focus-based algorithm is applicable to real-world text, does question whether the more complex algorithm has any real advantage over LaSIE's original simple approach.</Paragraph>
    <Paragraph position="9"> The lower performance of the focus-based algorithm is mainly due to an increased reliance on the accuracy and completeness of the grammatical structure identified by the parser. For example, the resolution of a pronoun will be skipped altogether if its role as a verb argument is missed by the parser. Partial parses will also affect the identification of EE boundaries, on which the focus update rules depend. For example, if the parser fails to attach a prepositional phrase containing an antecedent, it will then be missed from the focus registers and so the IRs (see (Azzam, 1995)). The simple LaSIE approach, however, will be unaffected in this case.</Paragraph>
    <Paragraph position="10"> Recall is also lost due to the more restricted proposal of candidate antecedents in the focus-based approach. The simple LaSIE approach proposes antecedents from each preceding paragraph until one is accepted, while the focus-based approach suggests a single fixed set. From a theoretical point of view, many interesting issues appear with a large set of examples, discussed here only briefly because of lack of space. Firstly, the fundamental assumption of the focus-based approach, that the focus is favoured as an antecedent, does not always apply. For example: In June, a few weeks before the crash of TWA Flight 800, leaders of several Middle Eastern terrorist organizations met in Teheran to plan terrorist acts. Among them was the PFL of Palestine, an organization that has been linked to airplane bombings in the past. Here, the pronoun them corefers with organizations rather than the focus leaders. Additional information will be required to override the fundamental assumption.</Paragraph>
    <Paragraph position="11"> Another significant question is when sentence focus changes. In our algorithm, focus changes when there is no reference (pronominal or otherwise) to the current focus in the current  EE. In the example used in section 4.1, this causes the focus at the end of the first sentence to be that of the last EE in that sentence, thus allowing the pronoun it in the subsequent sentence to be correctly resolved with the plane. However in the example below, the focus of the first EE (the writ) is the antecedent of the pronoun it in the subsequent sentence, rather than the focus from the last EE (the ...flight): The writ is for &amp;quot;damages&amp;quot; of seven passengers who died when the Airbus A310 flight crashed. It claims the deaths were caused by negligence.</Paragraph>
    <Paragraph position="12"> Updating focus after the complete sentence, rather than each EE, would propose the correct antecedent in this case. However neither strategy has a significant overall advantage in our evaluations on the MUC corpora.</Paragraph>
    <Paragraph position="13"> Another important factor is the priorities of the Interpretation Rules. For example, when a personal pronoun can corefer with both CF and AF, IRs select the CF first in our algorithm. However, this priority is not fixed, being based only on the corpora used so far, which raises the possibility of automatically acquiring IR priorities through training on other corpora.</Paragraph>
  </Section>
class="xml-element"></Paper>