File Information
File: 05-lr/acl_arc_1_sum/cleansed_text/xml_by_section/metho/04/p04-1072_metho.xml
Size: 17,835 bytes
Last Modified: 2025-10-06 14:08:59
<?xml version="1.0" standalone="yes"?> <Paper uid="P04-1072"> <Title>Splitting Complex Temporal Questions for Question Answering systemsa0</Title> <Section position="3" start_page="0" end_page="0" type="metho"> <SectionTitle> 3 Multi-layered Question-Answering </SectionTitle> <Paragraph position="0"/> <Section position="1" start_page="0" end_page="0" type="sub_section"> <SectionTitle> System Architecture Current Question Answering system architectures </SectionTitle> <Paragraph position="0"> do not allow to process complex questions. That is, questions whose answer needs to be gathered from pieces of factual information that is scattered in a document or through different documents. In order to be able to process these complex questions, we propose a multi-layered architecture. This architecture increases the functionality of the current Question-Answering systems, allowing us to solve any type of temporal questions. Moreover, this system could be easily augmented with new layers to cope with questions that need complex processing and are not temporal oriented.</Paragraph> <Paragraph position="1"> Some examples of complex questions are: a4 Temporal questions like &quot;Where did Michael Milken study before going to the University of Pennsylvania?&quot;. This kind of questions needs to use temporal information and event ordering to obtain the right answer.</Paragraph> <Paragraph position="2"> a4 Script questions like &quot;How do I assemble a bicycle?&quot;. In these questions, the final answer is a set of ordered answers.</Paragraph> <Paragraph position="3"> a4 Template-based questions like &quot;Which are the main biographical data of Nelson Mandela?&quot;.</Paragraph> <Paragraph position="4"> This question should be divided in a number of factual questions asking for different aspects of Nelson Mandela's biography. Gathering their respective answers will make it possible to answer the original question.</Paragraph> <Paragraph position="5"> These three types of question have in common the necessity of an additional processing in order to be solved. Our proposal to deal with them is to superpose an additional processing layer, one by each type, to a current General Purpose Question Answering system, as it is shown in Figure 1. This layer will perform the following steps: a4 Decomposition of the question into simple events to generate simple questions (sub null questions) and the ordering of the subquestions. null a4 Sending simple questions to a current General Purpose Question Answering system.</Paragraph> <Paragraph position="6"> a4 Receiving the answers to the simple questions from the current General Purpose Question Answering system.</Paragraph> <Paragraph position="7"> a4 Filtering and comparison between sub-answers to build the final complex answer.</Paragraph> <Paragraph position="8"> ! The main advantages of performing this multi-layered system are: a4 It allows you to use any existing general Q.A. system, with the only effort of adapting the output of the processing layer to the type of input that the Q.A. system uses.</Paragraph> <Paragraph position="9"> a4 Due to the fact that the process of complex questions is performed at an upper layer, it is not necessary to modify the Q.A. system when you want to deal with more complex questions.</Paragraph> <Paragraph position="10"> a4 Each additional processing layer is independent from each other and only processes those questions within the type accepted by that layer.</Paragraph> <Paragraph position="11"> Next, we present a layer oriented to process temporal questions according to the taxonomy shown in section 2.</Paragraph> </Section> <Section position="2" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 3.1 Architecture of a Question Answering </SectionTitle> <Paragraph position="0"> System applied to Temporality The main components of the Temporal Question Answering System are (c.f. figure 2) top-down: These components work all together for the obtainment of a final answer. The Question Decomposition Unit and the Answer Recomposition Unit are the units that conform the Temporal Q.A. layer which process the temporal questions, before and after using a General Purpose Q.A. system.</Paragraph> <Paragraph position="1"> a4 The Question Decomposition Unit is a preprocessing unit which performs three main tasks. First of all, the recognition and resolution of temporal expressions in the question. Secondly, there are different types of questions, according to the taxonomy shown in section 2.</Paragraph> <Paragraph position="2"> Each type of them needs to be treated in a different manner. For this reason, type identification must be done. After that, complex questions of types 3 and 4 only, are split into simple ones, which are used as the input of a General Purpose Question-Answering system. For example, the question &quot;Where did Bill Clinton study before going to Oxford University?&quot;, is divided into two sub-questions related through the temporal signal before: - Q1: Where did Bill Clinton study? - Q2: When did Bill Clinton go to Oxford University? a4 A General Purpose Question Answering system. Simple factual questions generated are processed by a General Purpose Question Answering system. Any Question Answering system could be used here. In this case, the SEMQA system (Vicedo and Ferr'andez, 2000) has been used. The only condition is to know the output format of the Q.A. system to accord null ingly adapt the layer interface. For the example above, a current Q.A. system returns the following answers: stage in the process. This unit builds the answer to the original question from the answers to the sub-questions and the temporal information extracted from the questions (temporal signals or temporal expressions). As a result, the correct answer to the original question is returned.</Paragraph> <Paragraph position="3"> Apart from proposing a taxonomy of temporal questions, we have presented a multi-layered Q.A. architecture suitable for enhancing current Q.A. capabilities with the possibility of adding new layers for processing different kinds of complex questions. Moreover, we have proposed a specific layer oriented to process each type of temporal questions.</Paragraph> <Paragraph position="4"> The final goal of this paper is to introduce and evaluate the first part of the temporal question processing layer: the Question Decomposition Unit.</Paragraph> <Paragraph position="5"> Next section shows the different parts of the unit together with some examples of their behavior.</Paragraph> </Section> </Section> <Section position="4" start_page="0" end_page="0" type="metho"> <SectionTitle> 4 Question Decomposition Unit </SectionTitle> <Paragraph position="0"> The main task of this unit is the decomposition of the question, which is divided in three main tasks or modules: These modules are fully explained below. Once the decomposition of the question has been made, the output of this unit is: a4 A set of sub-questions, that are the input of the General Purpose Question-Answering system. a4 Temporal tags, containing concrete dates returned by TERSEO system (Saquete et al., 2003), that are part of the input of the Answer Recomposition Unit and are used by this unit as temporal constraints in order to filter the individual answers.</Paragraph> <Paragraph position="1"> a4 A set of temporal signals that are part of the input of the Answer Recomposition Unit as well, because this information is necessary in order to compose the final answer.</Paragraph> <Paragraph position="2"> Once the decomposition has been made, the General Purpose Question-Answering system is used to treat with simple questions. The temporal information goes directly to the Answer Recomposition unit.</Paragraph> <Section position="1" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 4.1 Type Identification </SectionTitle> <Paragraph position="0"> The Type Identification Unit classifies the question in one of the four types of the taxonomy proposed in section 2. This identification is necessary because each type of question causes a different behavior (scenario) in the system. Type 1 and Type 2 questions are classified as simple, and the answer can be obtained without splitting the original question.</Paragraph> <Paragraph position="1"> However, Type 3 and Type 4 questions need to be split in a set of simple sub-questions. The types of these sub-questions are always Type 1 or Type 2 or a non-temporal question, which are considered simple questions.</Paragraph> <Paragraph position="2"> The question type is established according to the rules in figure 3:</Paragraph> </Section> <Section position="2" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 4.2 Temporal Expression Recognition and Resolution </SectionTitle> <Paragraph position="0"> This module uses TERSEO system (Saquete et al., 2003) to recognize, annotate and resolve temporal expressions in the question. The tags this module returns exhibit the following structure: Explicit dates:</Paragraph> <Paragraph position="2"> Every expression is identified by a numeric ID.</Paragraph> <Paragraph position="3"> VALDATE# and VALTIME# store the range of dates and times obtained from the system, where VALDATE2 and VALTIME2 are only used to establish ranges. Furthermore, VALTIME1 could be omitted if a single date is specified. VALDATE2, VALTIME1 and VALTIME2 are optional attributes.</Paragraph> <Paragraph position="4"> These temporal tags are the output of this module and they are used in the Answer Recomposition Unit in order to filter the individual answers obtained by the General Purpose Question-Answering system. The tags are working as temporal constraints. null Following, a working example is introduced.</Paragraph> <Paragraph position="5"> Given the next question &quot;Which U.S. ship was at- null tacked by Israeli forces during the Six Day war in the sixties?&quot;: 1. Firstly, the unit recognizes the temporal expression in the question, resolves and tags it, resulting in: <DATETIMEREF valdate1=&quot;01/01/1960&quot; valdate2=&quot;31/12/1969&quot;> in the sixties </DATETIMEREF> 2. The temporal constraint is that the date of the answers should be between the values valdate1 and valdate2.</Paragraph> </Section> <Section position="3" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 4.3 Question Splitter </SectionTitle> <Paragraph position="0"> This task is only necessary when the type of the question, obtained by the Type Identification Module, is 3 or 4. These questions are considered complex questions and need to be divided into simple ones (Type 1, Type 2). The decomposition of a complex question is based on the identification of temporal signals, which relate simple events in the question and establish an order between the answers of the sub-questions. Finally, these signals are the output of this module and are described in next subsection. null Temporal signals denote the relationship between the dates of the related events. Assuming that F1 is the date related to the first event in the question and F2 is the date related to the second event, the signal will establish an order between them. This we have named the ordering key. An example of some ordering keys is introduced in table 1.</Paragraph> </Section> </Section> <Section position="5" start_page="0" end_page="0" type="metho"> <SectionTitle> SIGNAL ORDERING KEY </SectionTitle> <Paragraph position="0"/> <Section position="1" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 4.3.2 Implementation </SectionTitle> <Paragraph position="0"> One have divided each complex question into two parts, based on the temporal signal. The former is a simple question, therefore, no transformation is required. However, the latter (the bit after the temporal signal) needs transformation into a correct question pattern, always corresponding to a &quot;When&quot; type-question. Moreover, three different kinds of question structures have been determined, being the transformation different for each of them. The implementation of this module is shown in figure 4.</Paragraph> <Paragraph position="1"> The three possible cases are: a4 The question that follows the temporal signal does not contain any verb, for example: &quot;What happened to the world oil prices after the Iraqi annexation of Kuwait?&quot; In this case, our system returns the following transformation: &quot;When did the Iraqi annexation of Kuwait occur?&quot; This case is the simplest, since the only transformation needed is adding the words &quot;When did... occur?&quot; to the second sentence.</Paragraph> <Paragraph position="2"> a4 The question that follows the temporal signal contains a verb, but this verb is a gerund tense, for example: &quot;Where did Bill Clinton study before going to Oxford University?&quot; In this case two previous steps to the transformation are necessary: 1. Extracting the subject of the previous question.</Paragraph> <Paragraph position="3"> 2. Converting the verb of the second sen null tence to infinitive tense.</Paragraph> <Paragraph position="4"> The final question returned by the system is: &quot;When did Bill Clinton go to Oxford University?&quot;. null a4 In the last type of transformation the second sentence in the question contains a tensed verb and its own subject, e.g., &quot;What did George Bush do after the U.N. Security Council ordered a global embargo on trade with Iraq?&quot; In this case, the infinitive and the tense of the sentence are obtained. Hence, the question results in the following form: &quot;When did the U.N. Security Council order a global embargo on trade with Iraq?&quot;.</Paragraph> <Paragraph position="5"> In the following example a part of the returned file of our Decomposition Unit is shown.</Paragraph> </Section> <Section position="2" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 4.4 Decomposition Unit Evaluation </SectionTitle> <Paragraph position="0"> This section presents an evaluation of the Decomposition Unit for the treatment of complex questions.</Paragraph> <Paragraph position="1"> For the evaluation a corpus of questions containing as many simple as complex questions is required.</Paragraph> <Paragraph position="2"> Due to the fact that question corpora used in TREC (TREC, ) and CLEF (CLEF, ) do not contain complex questions, the TERQAS question corpus has been chosen (Radev and Sundheim, 2002; Pustejovsky, 2002). It consists of 123 temporal questions.</Paragraph> </Section> </Section> <Section position="6" start_page="0" end_page="0" type="metho"> <SectionTitle> TOTAL TREATED SUCCESSES PRECISION RECALL F- MEASURE </SectionTitle> <Paragraph position="0"> From these, 11 were discarded due to requiring the need of a treatment beyond the capabilities of the system introduced hereby. Questions of the type: &quot;Who was the second man on the moon&quot; can not be answered by applying the question decomposition. They need a special treatment. For the aforementioned phrase, this would consist of obtaining the names of all the men having been on the moon, ordering the dates and picking the second in the ordered list of names.</Paragraph> <Paragraph position="1"> Therefore, for this evaluation, we have just been focusing on trying to resolve the 112 left. The evaluation has been made manually by three annotators.</Paragraph> <Paragraph position="2"> Four different aspects of the unit have been considered: null a4 Recognition and resolution of Temporal Expressions: In this corpus, there were 62 temporal expressions and our system was able to recognize 52, from which 47 were properly resolved by this module.</Paragraph> <Paragraph position="3"> a4 Type Identification: There were 112 temporal questions in the corpus. Each of them was processed by the module, resulting in 104 properly identified according to the taxonomy proposed in section 2.</Paragraph> <Paragraph position="4"> a4 Signal Detection: In the corpus, there were 17 questions that were considered complex (Type 3 and Type 4). Our system was able to treat and recognize correctly the temporal signal of 14 of these questions.</Paragraph> <Paragraph position="5"> a4 Question Splitter: From this set of 17 complex questions, the system was able to process 14 questions and divided properly 12 of them.</Paragraph> <Paragraph position="6"> The results, in terms of precision and recall are shown in Table 2. In the evaluation, only 19 questions are wrongly pre-processed. Errors provoking a wrong pre-processing have been analyzed thoroughly: null a4 There were 8 errors in the identification of the type of the question and they were due to: - Not treated TE or wrong TE recognition: 6 questions.</Paragraph> <Paragraph position="7"> - Wrong Temporal Signal detection: 2 questions.</Paragraph> <Paragraph position="8"> a4 There were 5 errors in the Question Splitter module: - Wrong Temporal Signal detection: 3 questions.</Paragraph> <Paragraph position="9"> - Syntactic parser problems: 2 questions.</Paragraph> <Paragraph position="10"> a4 There were 15 errors not affecting the treatment of the question by the General Purpose Question Answering system. Nevertheless, they do affect the recomposition of the final answer. They are due to: - Not treated TE or wrong TE recognition: 6 questions.</Paragraph> <Paragraph position="11"> - Wrong temporal expression resolution: 9 questions.</Paragraph> <Paragraph position="12"> Some of these questions provoke more than one problem, causing that both, type identification and division turn to be wrong.</Paragraph> </Section> class="xml-element"></Paper>