<?xml version="1.0" standalone="yes"?>
<Paper uid="W99-0209">
  <Title>Orthographic Co-Reference Resolution Between Proper Nouns Through the Calculation of the Relation of &amp;quot;Replicancia&amp;quot;</Title>
  <Section position="6" start_page="63" end_page="65" type="evalu">
    <SectionTitle>
4 Experiment and Results
</SectionTitle>
    <Paragraph position="0"> We have defined the replicancia relation as an instrument for the resolution of co-reference between proper nouns, although we are perfectly aware of its limitations to identify as coreferentials nouns which are not linked by an orthographic relation, as occurs with nicknames and familiar names (Josd and Pepe, for example).</Paragraph>
    <Paragraph position="1"> In order not to conceal this limitations we have designed an experiment to evaluate the co-reference between proper nouns instead of evaluating the algorithm for the replicancia resolution, because this relation is only useful inasmuch it is capable to resolve co-reference between proper nouns.</Paragraph>
    <Paragraph position="2"> In the context of natural language processing systems evaluation is very important to decide which collection of texts is going to be used. The current trend is not to use texts specifically prepared for the evaluation, but normal texts, that is to say, similar to the texts which the system will use in its normal operation. We have chosen documents available in the World Wide Web.</Paragraph>
    <Paragraph position="3"> In our evaluation we will use a corpus manually processed composed by 100 documents with an extension scarcely over 1 MB, HTML code included. The documents come from five different newspapers, all of them spanish and available on electronic form. The newspapers, in alphabetical order, are: ABCe, El Mundo, E1 Pals Digital, El Perirdico On Line and La Vanguardia Electrrnica. The extension of the documents varies between 4kB and 25 kB, being the average around 10kB. The utilisation of these variety of sources to obtain the documents which form the Corpus is very advisable, because people who work for the same newspaper tend to write similarly; the choosing of different document sources brings us closer to the style diversity characteristic of the texts available in the World Wide Web.</Paragraph>
    <Paragraph position="4"> The result of co-reference resolution is the grouping of the document's selected objects -the proper nouns- in classes. The human analyst designs a template which includes the different instances of the same entity present in the document, and then compares it with the system's response, being this another grouping of objects in classes of co-reference. From this comparison we draw the system's quality evaluation.</Paragraph>
    <Paragraph position="5"> The evaluation proceeding chosen is based on the measures used in MUC conferences (MUC-7,  1997) (Chinchor, 1997) (Hirschman, 1997). In MUC evaluations, the results of co-reference analysis are classified according to three categories: COR - Correct. A result is correct when there is full coincidence between the contents of the template and the system's response.</Paragraph>
    <Paragraph position="6"> MIS - Missing. We consider a result absent when it appears in the template but not in the answer.</Paragraph>
    <Paragraph position="7"> SPU - Spurious. We consider a result spurious when it does not appear in the template  although it is found in the system's response. With the cardinals of these three classes we obtain two intermediate magnitudes which will be useful to calculate merit figures. These magnitudes areS: POS - Possible. The number of elements contained in the template that contribute to the final scoring. It is calculated through the following expression:</Paragraph>
    <Paragraph position="9"> included in the system's response. It is calculated as follows:</Paragraph>
    <Paragraph position="11"> Once we have gathered the data relative to the classes of responses, we are prepared to calculate the measures of the system's quality. The metrics chosen are the ones normally used in Information Retrieval systems: REC - Recall. A quantity measure of the elements of the answer-key included in the response of the system. Its expression is:</Paragraph>
    <Paragraph position="13"> elements of the response included also in the template. Its expression is:</Paragraph>
    <Paragraph position="15"> by van Rijsbergen (Rijsbergen, 1980). We use it with the control parameter B= 1 to guarantee the same weight for recall and precision. Its general expression is:</Paragraph>
    <Paragraph position="17"> From the cardinals of classes COR, MIS and SPU we obtain measures REC, PREy F, the last one with parameter B= 1. With all these data we fill in table I- 1 of Annex I, where each line shows the metrics and data of a document. Table 4.1 shows overall results.</Paragraph>
    <Paragraph position="18"> We have carried out the evaluation of the calculation of replicantes bearing in mind the use we intend to make of that relation. We have counted the cardinals of the co-reference classes among the proper nouns included in the text only in those classes whose elements adopted more than one form. Let us think, for example, in a text which had three classes of co-reference formed by the names6:  The first class refers to the entity in a single way; the second class includes two ways of referring to the entity and the third one three. For the evaluation of co-reference between nouns we consider that there are four references to the second entity and five to the third, which makes a total of nine nouns, although we know that the system is not prepared to identify as coreferentials the elements included in the third class. So, the result of the evaluation would be: 5 When the names of these measures appear in arithmetical expressions, they must be taken as the cardinals of the respective classes.</Paragraph>
    <Paragraph position="19"> 6 Each name shows the number of times it appears in the relevant text.</Paragraph>
    <Paragraph position="20">  nine (ACT) -six of them correctly located in their respective classes (COR), three located in two spurious classes (SPU) and three missing from their correct classes (MIS).</Paragraph>
    <Paragraph position="21"> Briefly, we evaluate the resolution of co-reference between proper names without taking into account how the problem is solved by our algorithm; this way, the evaluation does not reflect the quality of the algorithm in calculating the replicancia, but the resolution of the coreference between proper nouns instead, which is the aim pursued.</Paragraph>
  </Section>
class="xml-element"></Paper>