<?xml version="1.0" standalone="yes"?>
<Paper uid="P00-1022">
  <Title>A Computational Approach to Zero-pronouns in Spanish</Title>
  <Section position="6" start_page="0" end_page="0" type="concl">
    <SectionTitle>
4 Evaluation
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.1 Experiments accomplished
</SectionTitle>
      <Paragraph position="0"> Our computational system (SUPAR) has been trained with a handmade corpus8 with 106 zero7 For example in: Peter es un genio (Peter is a genius), the tagger does not indicate that the object does not have both masculine and feminine linguistic forms. Therefore, a feminine subject would use the same form: Jane es un genio (Jane is a genius).</Paragraph>
      <Paragraph position="1"> Consequently, although the tagger says that the verb, es (is), is copulative, and the object, un genio (a genius) is masculine, this gender could not be used as a restriction for the zero-pronoun in the following sentence: [?] Es un genio.</Paragraph>
      <Paragraph position="2"> 8 This corpus has been provided by our colleagues in pronouns. This training has mainly supposed the improvement of the set of preferences, i.e. the optimum order of preferences in order to obtain the best results. After that, we have carried out a blind evaluation on unrestricted texts.</Paragraph>
      <Paragraph position="3"> Specifically, SUPAR has been run on two different Spanish corpora: a) a part of the Spanish version of The Blue Book corpus, which contains the handbook of the International Telecommunications Union CCITT, published in English, French and Spanish, and automatically tagged by the Xerox tagger, and b) a part of the Lexesp corpus, which contains Spanish texts from different genres and authors.</Paragraph>
      <Paragraph position="4"> These texts are taken mainly from newspapers, and are automatically tagged by a different tagger than that of The Blue Book. The part of the Lexesp corpus that we processed contains ten different stories related by a sole narrator, although they were written by different authors.</Paragraph>
      <Paragraph position="5"> Having worked with different genres and disparate authors, we feel that the applicability of our proposal to other sorts of texts is assured.</Paragraph>
      <Paragraph position="6"> In Figure 2, a brief description of these corpora is given. In these corpora, partial parsing of the text with no semantic information has been used.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.2 Evaluating the detection of zero-
</SectionTitle>
      <Paragraph position="0"> pronouns To achieve this sort of evaluation, several different tasks may be considered. Each verb must first be detected. This task is easily the University of Alicante, which were required to propose sentences with zero-pronouns.</Paragraph>
      <Paragraph position="1"> accomplished since both corpora have been previously tagged and manually reviewed. No errors are therefore expected on verb detection. Therefore, a recall9 rate of 100% is accomplished. The second task is to classify the verbs into two categories: a) verbs whose subjects have been omitted, and b) verbs whose subjects have not. The overall results on this sort of detection are presented in Figure 3 (success10 rate of 88% on 1,599 classified verbs, with no significant differences seen between the corpora). We should also remark that a success rate of 98% has been obtained in the detection of verbs whose subjects were omitted, whereas only 80% was achieved for verbs whose subjects were not. This lower success rate is justified, however, for several reasons. One important reason is the non-detection of impersonal verbs by the POS tagger. This problem has been partly resolved by heuristics such as a set of impersonal verbs (e.g. llover (to rain)), but it has failed in some impersonal uses of some verbs.</Paragraph>
      <Paragraph position="2"> For example, in sentence (3), the verb es (to be) is not usually impersonal, but it is in the following sentence, in which SUPAR would fail: (3) [?] Es hora de desayunar ([It][?] is time to have breakfast) Two other reasons for the low success rate achieved with verbs whose subjects were not omitted are the lack of semantic information and the inaccuracy of the grammar used. The second reason is the ambiguity and the unavoidable incompleteness of the grammars, which also affects the process of clause splitting.</Paragraph>
      <Paragraph position="3"> In Figure 3, an interesting fact can be observed: 46% of the verbs in these corpora have their subjects omitted. It shows quite clearly the importance of this phenomenon in Spanish. Furthermore, it is even more important in narrative texts, as this figure shows: 61% with the Lexesp corpus, compared to 26% with the technical manual. We should also observe that The Blue Book has no verbs in either the first or the second person. This may be explained by the style of the technical manual, which usually 9 By &amp;quot;recall rate&amp;quot;, we mean the number of verbs classified, divided by the total number of verbs in the text.</Paragraph>
      <Paragraph position="4"> 10 By &amp;quot;success rate&amp;quot;, we mean the number of verbs successfully classified, divided by the total number of verbs in the text.</Paragraph>
      <Paragraph position="5">  consists of a series of isolated definitions, (i.e. many paragraphs that are not related to one another). This explanation is confirmed by the relatively small number of anaphors that are found in that corpus, as compared to the Lexesp corpus.</Paragraph>
      <Paragraph position="6"> We have not considered comparing our results with those of other published works, since, (as we have already explained in the Background section), ours is the first study that has been done specifically for Spanish texts, and the designing of the detection stage depends mainly on the structure of the language in question. Any comparisons that might be made concerning other languages, therefore, would prove to be rather insignificant.</Paragraph>
    </Section>
    <Section position="3" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.3 Evaluating anaphora resolution
</SectionTitle>
      <Paragraph position="0"> As we have already shown in the previous section, (Figure 3), of the 1,599 verbs classified in these two corpora, 734 of them have zeropronouns. Only 581 of them, however, are in third person and will be resolved. In Figure 4, we present a classification of these third person zero-pronouns, which have been conveniently divided into three categories: cataphoric, exophoric and anaphoric. The first category is comprised of those whose antecedent, i.e. the clause subject, comes after the verb. For example, in sentence (4) the subject, a boy, appears after the verb compro (bought).</Paragraph>
      <Paragraph position="1"> (4) [?]k Compro un ninok en el supermercado (A boyk bought in the supermarket) This kind of verb is quite common in Spanish, as can be seen in this figure (49%).</Paragraph>
      <Paragraph position="2"> This fact represents one of the main difficulties found in resolving anaphora in Spanish: the structure of a sentence is more flexible than in English. These represent intonationally marked sentences, where the subject does not occupy its usual position in the sentence, i.e. before the verb. Cataphoric zero-pronouns will not be resolved in this paper, since semantic information is needed to be able to discard all of their antecedents and to prefer those that appear within the same sentence and clause after the verb. For example, sentence (5) has the same syntactic structure than sentence (4), i.e. verb, np, pp, where the subject function of the np can only be distinguished from the object by means of semantic knowledge.</Paragraph>
      <Paragraph position="3"> (5) [?] Compro un regalo en el supermercado ([He][?] bought a present in the supermarket) The second category consists of those zero-pronouns whose antecedents do not appear, linguistically, in the text (they refer to items in the external world rather than things referred to in the text). Finally, the third category is that of pronouns that will be resolved by our computational system, i.e., those whose antecedents come before the verb: 228 zeropronouns. These pronouns would be equivalent to the full pronoun he, she, it or they.</Paragraph>
      <Paragraph position="4">  The different accuracy results are also shown in Figure 4: A success rate of 75% was attained for the 228 zero-pronouns. By &amp;quot;successful resolutions&amp;quot; we mean that the solutions offered by our system agree with the solutions offered by two human experts.</Paragraph>
      <Paragraph position="5"> For each zero-pronoun there is, on average, 355 candidates before the restrictions are applied, and 11 candidates after restrictions.</Paragraph>
      <Paragraph position="6"> Furthermore, we repeated the experiment without applying restrictions and the success rate was significantly reduced.</Paragraph>
      <Paragraph position="7"> Since the results provided by other works have been obtained on different languages, texts and sorts of knowledge (e.g. Hobbs and Lappin full parse the text), direct comparisons are not possible. Therefore, in order to accomplish this comparison, we have implemented some of these approaches in SUPAR. Although some of these approaches were not proposed for zeropronouns, we have implemented them since as our approach they could also be applied to solve this kind of pronoun. For example, with the baseline presented by Hobbs (1977) an accuracy of 49.1% was obtained, whereas, with our system, we achieved 75% accuracy. These results highlight the improvement accomplished with our approach, since Hobbs' baseline is frequently used to compare most of the work done on anaphora resolution11. The reason why Hobbs' algorithm works worse than ours is due to the fact that it carries out a full parsing of the text. Furthermore, the way to explore the syntactic tree with Hobbs' algorithm is not the best one for the Spanish language since it is nearly a free-word-order language.</Paragraph>
      <Paragraph position="8"> Our proposal has also been compared with the typical baseline of morphological agreement and proximity preference, (i.e., the antecedent 11 In Tetreault (1999), for example, it is compared with an adaptation of the Centering Theory by Grosz et al. (1995), and Hobbs' baseline out-performs it. that appears closest to the anaphor is chosen from among those that satisfy the restrictions). The result is a 48.6% accuracy rate. Our system, therefore, improves on this baseline as well.</Paragraph>
      <Paragraph position="9"> Lappin and Leass (1994) has also been implemented in our system and an accuracy of 64% was attained. Moreover, in order to compare our proposal with Centering approach, Functional Centering by Strube and Hahn (1999) has also been implemented, and an accuracy of 60% was attained.</Paragraph>
      <Paragraph position="10"> One of the improvements afforded by our proposal is that statistical information from the text is included with the rest of information (syntactic, morphologic, etc.). Dagan and Itai (1990), for example, developed a statistical approach for pronominal anaphora, but the information they used was simply the patterns obtained from the previous analysis of the text. To be able to compare our approach to that of Dagan and Itai, and to be able to evaluate the importance of this kind of information, our method was applied with statistical information12 only. If there is more than one candidate after applying statistical information, preference, and then proximity preference are applied. The results obtained were lower than when all the preferences are applied jointly: 50.8%. These low results are due to the fact that statistical information has been obtained from the beginning of the text to the pronoun. A previous training with other texts would be necessary to obtain better results.</Paragraph>
      <Paragraph position="11"> Regarding the success rates reported in Ferrandez et al. (1999) for pronominal references (82.2% for Lexesp, 84% for Spanish version of The Blue Book, and 87.3% for the English version), are higher than our 75% success rate for zero-pronouns. This reduction (from 84% to 75%) is due mainly to the lack of gender information in zero-pronouns.</Paragraph>
      <Paragraph position="12"> Mitkov (1998) obtains a success rate of 89.7% for pronominal references, working with English technical manuals. It should be pointed out, however, that he used some knowledge that was very close to the genre13 of the text. In our 12 This statistical information consists of the number of times that a word appears in the text and the number of times that it appears with a verb.</Paragraph>
      <Paragraph position="13"> 13 For example, the antecedent indicator section heading preference, in which if a NP occurs in the heading of the section, part of which is the current study, such information was not used, so we consider our approach to be more easily adaptable to different kinds of texts. Moreover, Mitkov worked exclusively with technical manuals whereas we have worked with narrative texts as well. The difference observed is due mainly to the greater difficulty found in narrative texts than in technical manuals which are generally better written. In any case, the applicability of our proposal to different genres of texts seems to have been well proven.</Paragraph>
      <Paragraph position="14"> Anyway, if the order of application of the preferences14 is varied to each different text, an 80% overall accuracy rate is attained. This fact implies that there is another kind of knowledge, close to the genre and author of the text that should be used for anaphora resolution.</Paragraph>
      <Paragraph position="15"> Conclusion In this paper, we have proposed the first algorithm for the resolution of zero-pronouns in Spanish texts. It has been incorporated into a computational system (SUPAR). In the evaluation, several baselines on pronominal anaphora resolution have been implemented, and it has achieved better results than either of them have.</Paragraph>
      <Paragraph position="16"> As a future project, the authors shall attempt to evaluate the importance of semantic information for zero-pronoun resolutions in unrestricted texts. Such information will be obtained from a lexical tool, (e.g.</Paragraph>
      <Paragraph position="17"> EuroWordNet), which could be consulted automatically. We shall also evaluate our proposal in a Machine Translation application, where we shall test its success rate by its generation of the zero-pronoun in the target language, using the algorithm described in Peral et al. (1999).</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>