File Information
File: 05-lr/acl_arc_1_sum/cleansed_text/xml_by_section/metho/00/p00-1070_metho.xml
Size: 13,810 bytes
Last Modified: 2025-10-06 14:07:21
<?xml version="1.0" standalone="yes"?> <Paper uid="P00-1070"> <Title>Importance of Pronominal Anaphora resolution in Question Answering systems</Title> <Section position="4" start_page="0" end_page="0" type="metho"> <SectionTitle> 3 Importance of pronominal </SectionTitle> <Paragraph position="0"> information in documents Trying to measure the importance of information referenced pronominally in documents, wehave analysed several text collections used for QA task in TREC-8 Conference as well as others used frequently for IR system testing. These collections were the following: Los Angeles Times #28LAT#29, Federal Register #28FR#29, Financial Times #28FT#29, Federal Bureau Information Service #28FBIS#29, TIME, CRANFIELD, CISI, CACM, MED and LISA. This analysis consists on determining the amount and type of pronouns used, as well as the number of sentences containing pronouns in each of them. As average measure of pronouns used in a collection, we use the ratio between the quantity of pronouns and the number of sentences containing pronouns. Thismeasure approximates the level of information that is ignored if these references are not solved. Figure 1 shows the results obtained in this analysis. null As we can see, the amount and type of pronouns used in analysed collections vary depending on the subject the documents talk about. LAT, FBIS, TIME and FT collections are composed from news published in di#0Berent newspapers. The ratio of pronominal reference used in this kind of documents is very high #28from 35,96#25 to 55,20#25#29. These documents contain a great number of pronominal references in third person #28he, she, they, his, her, their#29 whose antecedents are mainly people's names. In this type of documents, pronominal anaphora resolution seems to be very necessary for a correct modelling of relations between entities. CISI and MED collections appear ranked next in decreasing ratio level order. These collections are composed by general comments about document managing, classi#0Ccation and indexing and documents extracted from medical journals respectively. Although the ratio presented by these collections #2824,94#25 and 22,16#25#29 is also high, the most important group of pronominal references used in these collections is formed by &quot;it&quot; and &quot;its&quot; pronouns. In this case,</Paragraph> </Section> <Section position="5" start_page="0" end_page="0" type="metho"> <SectionTitle> TEXT COLLECTION LAT FBIS TIME FT CISI MED CACM LISA FR CRANFIELD </SectionTitle> <Paragraph position="0"> antecedents of these pronominal references are mainly concepts represented typically by noun phrases. It seems again important solving these references for a correct modelling of relations between concepts expressed by noun-phrases. The lowest ratio results are presented by CRANFIELD collection with a 9,05#25. The reason of this level of pronominal use is due to text contents. This collection is composed by extracts of very high technical subjects. Between the described percentages we #0Cnd the CACM, LISA and FR collections.</Paragraph> <Paragraph position="1"> These collections are formed by abstracts and documents extracted from the Federal Register, from the CACM journal and from Library and Information Science Abstracts, respectively. As general behaviour, we can notice that as more technical document contents become, the pronouns &quot;it&quot; and &quot;its&quot; become the most appearing in documents and the ratio of pronominal references used decreases. Another observation can be extracted from this analysis. Distributionof pronounswithinsentences is similar in all collections. Pronouns appear scattered through sentences containing one or two pronouns. Using more than two pronouns in the same sentence is quite infrequent.</Paragraph> <Paragraph position="2"> After analysing these results an important question may arise. Is it worth enough to solve pronominal references in documents? It would seem reasonable to think that resolution of pronominal anaphora would only be accomplished when the ratio of pronominal occurrence exceeds a minimum level. However, we have to take into account that the cost of solving these references is proportional to the number of pronouns analysed and consequently, proportional to the amount of information a system will ignore if these references are not solved.</Paragraph> <Paragraph position="3"> As results above state, it seems reasonable to solve pronominal references in queries and documents for QA tasks. At least, when the ratio of pronouns used in documents recommend it. Anyway, evaluation and later analysis #28section 5#29 contribute with empirical data to conclude that applying pronominal anaphora resolution techniques improve QA systems performance.</Paragraph> </Section> <Section position="6" start_page="0" end_page="1" type="metho"> <SectionTitle> 4 Our Approach </SectionTitle> <Paragraph position="0"> Our system is made up of three modules. The #0Crst one is a standard IR system that retrieves relevant documents for queries. The second module will manage with anaphora resolution in both, queries and retrieved documents. For this purpose we use SUPAR computational system #28section 4.1#29. And the third one is a sentence-extraction QA system that interacts with SUPAR module and ranks sentences from retrieved documents to locate the answer where the correct answer appears #28section 4.2#29.</Paragraph> <Paragraph position="1"> For the purpose of evaluation an IR system has been implemented. This system is based on the standard information retrieval approach to document ranking described in Salton #281989#29. For QA task, the same approach has been used as baseline but using sentences as text unit. Each term in the query and documents is assigned an inverse document frequency #28idf #29 score based on the same corpus. This measure is computed as:</Paragraph> <Paragraph position="3"> where N is the total number of documents in the collection and df#28t#29 is the number of documents which contains term t. Query expansion consists of stemming terms using a version of the Porter stemmer. Document and sentence similarityto the querywas computed using the cosine similarity measure. The LAT corpus has been selected as test collection due to his high level of pronominal references.</Paragraph> <Section position="1" start_page="0" end_page="1" type="sub_section"> <SectionTitle> 4.1 Solving pronominal anaphora </SectionTitle> <Paragraph position="0"> In this section, the NLP Slot Uni#0Ccation is brie#0Dy described #28Ferr#13andez et al., 1999; Ferr#13andez et al., 1998#29. SUPAR's architecture consists of three independent modules that interact with one other. These modules are lexical analysis, syntactic analysis, and a resolution module for Natural Language Processing problems.</Paragraph> <Paragraph position="1"> Lexical analysis module. This module takes each sentence to parse as input, along with a tool that provides the system with all the lexical information for each word of the sentence. This tool may be either a dictionary or a part-of-speech tagger. In addition, this module returns a list with all the necessary information for the remaining modules as output. SUPAR works sentence by sentence from the input text, but stores information from previous sentences, which it uses in other modules, #28e.g. the list of antecedents of previous sentences for anaphora resolution#29.</Paragraph> <Paragraph position="2"> Syntactic analysis module. This module takes as input the output of lexical analysis module and the syntactic information represented by means of grammatical formalism Slot Uni#0Ccation Grammar #28SUG#29. It returns what is called slot structure, which stores all necessary information for following modules.</Paragraph> <Paragraph position="3"> One of the main advantages of this system is that it allows carrying out either partial or full parsing of the text.</Paragraph> <Paragraph position="4"> Module of resolution of NLP problems. In this module, NLP problems #28e.g. anaphora, extra-position, ellipsis or PPattachment#29 are dealt with. It takes the slot structure #28SS#29 that corresponds to the parsed sentence as input. The output is an SS in which all the anaphors have been resolved. In this paper, only pronominal anaphora resolution has been applied.</Paragraph> <Paragraph position="5"> The kinds of knowledge that are going to be used in pronominal anaphora resolution in this paper are: pos-tagger, partial parsing, statistical knowledge, c-command and morphologic agreement as restrictions and several heuristics such as syntactic parallelism, preference for noun-phrases in same sentence as the pronoun preference for proper nouns.</Paragraph> <Paragraph position="6"> We should remark that when wework with unrestricted texts #28as it occurs in this paper#29 we do not use semantic knowledge #28i.e. a tool suchasWorNet#29. Presently, SUPAR resolves both Spanish and English pronominal anaphora with a success rate of 87#25 and 84#25 respectively.</Paragraph> <Paragraph position="7"> SUPAR pronominal anaphora resolution di#0Bers from those based on restrictions and preferences, since the aim of our preferences is not to sort candidates, but rather to discard candidates. That is to say, preferences are considered in a similar way to restrictions, except when no candidate satis#0Ces a preference, in which case no candidate is discarded. For example in sentence: &quot;Rob was asking us about John. Ireplied that Peter saw John yesterday. James also saw him.&quot; After applying the restrictions, the following list of candidates is obtained for the pronoun him: #5BJohn, Peter, Rob#5D, which are then sorted according to their proximity to the anaphora. If preference for candidates in same sentence as the anaphora is applied, then no candidate satis#0Ces it, so the following preference is appliedon the same list of candidates. Next, preference for candidates in the previous sentence is applied and the list is reduced to the following candidates: #5BJohn, Peter#5D. If syntactic parallelism preference is then applied, only one candidate remains, #5BJohn#5D, which will be the antecedent chosen.</Paragraph> <Paragraph position="8"> Each kind of anaphora has its own set of restrictions and preferences, although they all follow the same general algorithm: #0Crst come the restrictions, after which the preferences are applied. For pronominal anaphora, the set of restrictions and preferences that apply are described in Figure 2.</Paragraph> <Paragraph position="9"> 1) Candidates in the same sentence as anaphor.</Paragraph> <Paragraph position="10"> 2) Candidates in the previous sentence 3) Preference for proper nouns.</Paragraph> <Paragraph position="11"> 4) Candidates in the same position as the anaphor with reference to the verb (before or after).</Paragraph> <Paragraph position="12"> 5) Candidates with the same number of parsed constituents as the anaphora 6) Candidates that have appeared with the verb of the anaphor more than once 7) Preference for indefinite NPs.</Paragraph> <Paragraph position="13"> The following restrictions are #0Crst applied to the list of candidates: morphologic agreement, c-command constraints and semantic consistency. Thislistis sorted by proximityto the anaphor. Next, if after applying restrictions there is still more than one candidate, the preferences are then applied, in the order shown in this #0Cgure. This sequence of preferences #28from 1 to 7#29 stops when, after having applied a preference, only one candidate remains. If after applying preferences there is still more than one candidate, then the most repeated candidates in the text are extracted from the list after applying preferences. After this is done, if there is still more than one candidate, then those candidates that have appeared most frequently with the verb of the anaphor are extracted from the previous list.</Paragraph> <Paragraph position="14"> Finally, if after having applied all the previous preferences, there is still more than one candidate left, the #0Crst candidate of the resulting list, #28the closest one to the anaphor#29, is selected.</Paragraph> </Section> <Section position="2" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 4.2 Anaphora resolution and QA </SectionTitle> <Paragraph position="0"> Our QA approach provides a second level of processing for relevant documents: Analysing matching documents and Sentence ranking.</Paragraph> </Section> <Section position="3" start_page="1" end_page="1" type="sub_section"> <SectionTitle> Analysing Matching Documents. This </SectionTitle> <Paragraph position="0"> step is applied over the best matching documents retrieved from the IR system. These documents are analysed by SUPAR module and pronominal references are solved. As result, each pronoun is associated with the noun phrase it refers to in the documents. Then, documents are split into sentences as basic text unit for QA purposes. This set of sentences is sent to the sentence ranking stage.</Paragraph> <Paragraph position="1"> Sentence Ranking. Each term in the query is assigned a weight. This weight is the sum of inverse document frequency measure of terms based on its occurrence in the LAT collection described earlier. Each document sentence is weighted the same way. The only di#0Berence with baseline is that pronouns are given the weight of the entity they refer to. As we only want to analyse the e#0Bects of pronominal reference resolution, no more changes are introduced in weighting scheme.</Paragraph> <Paragraph position="2"> For sentence ranking, cosine similarity is used between query and document sentences.</Paragraph> </Section> </Section> class="xml-element"></Paper>