<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-0507">
  <Title>A Practical QA System in Restricted Domains</Title>
  <Section position="3" start_page="1" end_page="1" type="metho">
    <SectionTitle>
2 Overall Architecture
</SectionTitle>
    <Paragraph position="0"> The overall framework of the QA system is presented in Figure 1. The QA system consists of two major parts; the IE (Information Extractor) engine and the QA engine.</Paragraph>
    <Paragraph position="1">  We've developed the QA system for a TV broadcast schedule domain as well, which is more complex to process than the weather forecasts QA, but have not evaluated it yet. So, in this paper, we present the system for weather forecasts only.  The IE engine consists of two parts; a web crawler and a wrapper. The web crawler downloads the selected webpages from the website of the Korea Meteorological Administration (KMA) every hour. The website provides current weather conditions and 7 day-forecasts for dozens of Korean cities. The wrapper, which is a set of extraction rules, is used to extract weather information from the webpages . The extracted information is stored in the database.</Paragraph>
    <Paragraph position="2"> The QA engine performs three-phase processing: First, it analyzes natural language questions and translates the questions into Structured Query Language (SQL) statements. Second, the SQL queries are directed to a DBMS to retrieve the answers in the database. Finally, the result from the DBMS is converted to natural language sentences for output. Figure 2 depicts overall processes for the QA engine. A DBMS (Currently, Oracle Database) is used for managing extracted data. A speech processor can be merged with the system when it is used in the home agent robot, which provides speech interface. A web interface is used for providing web-based QA service with the QA system.</Paragraph>
  </Section>
  <Section position="4" start_page="1" end_page="1" type="metho">
    <SectionTitle>
3 A Practical QA System
</SectionTitle>
    <Paragraph position="0"> The question answering starts from extracting weather information from the web site. The user request is analyzed with the question analyzer and the appropriate query frame is selected, and then the request is translated into the SQL expression. The SQL query is used to retrieve the correct answer from the database, which stores weather information from the webpages. Finally, natural language</Paragraph>
    <Section position="1" start_page="1" end_page="1" type="sub_section">
      <SectionTitle>
3.1 Information Extraction
</SectionTitle>
      <Paragraph position="0"> The weather information in the webpages is semistructured. Semi-structured resources generally do not employ unrestricted natural language text, but rather exhibit a fair degree of structure (Kushmerick, 1997). Therefore, information can be accurately and easily extracted from the webpage, compared to IE from unstructured data.</Paragraph>
      <Paragraph position="1"> On the other hand, semi-structured resources are usually formatted for use by people, and contain irrelevant elements that must be ignored, such as images, advertisements, and HTML formatting tags (Figure 3). Thus information extraction from the semi-structured documents is not entirely trivial. Currently, the QA system is using hand-coded wrappers. However, we are developing an automatic process of constructing wrappers (wrapper induction) for semi-structured resources and that can detect the modification of the web page design and adapt the wrapper according to the modification, automatically, like (Sigletos et al., 2003).</Paragraph>
      <Paragraph position="2"> Presently, the IE engine extracts following infor- null mation : * Current observation: weather summary, visibility, temperature, wind, relative humidity * 7 days-forecasts : weather summary, forecast temperature (highest/lowest).</Paragraph>
    </Section>
    <Section position="2" start_page="1" end_page="1" type="sub_section">
      <SectionTitle>
3.2 Question Analysis
</SectionTitle>
      <Paragraph position="0"> First, user's request is analyzed with the query analyzer as represented in Figure 2. The analyzer extracts several keywords that describing the question, such as event word, date, time, and location, by using a dependency parser, and the user question is represented only by these extracted keywords.</Paragraph>
      <Paragraph position="1"> The named entity tagger is used to identify temporal expressions, place names, and weather events.</Paragraph>
      <Paragraph position="2"> The tagger consults the domain-dependent ontology for recognizing weather events, and the domain-independent ontology for place names. The ontology for the weather events consists of event concepts, which are similar to Synset in WORDNET (Fellbaum, 1998). For example, rain and umbrella are in same event concept in the domain ontology for weather events, because the questions about using umbrella are usually asking about raining (e.g.</Paragraph>
      <Paragraph position="3"> Will I need to bring umbrella tomorrow? and Will it be raining tomorrow?) The temporal data normalizer converts temporal expressions such as today, this weekend and now into absolute values that can be used in querying to the database.</Paragraph>
      <Paragraph position="4"> Seoul, March. 11., wide spread dust, (-/-) Seoul, March. 12., cloudy, (0/11) Seoul, March, 13., Sunny, (1/11)  from the semi-structured documents If the information on date, time, or location is not expressed in the user's request, the question analyzer infers the missing information. The inference rules, which are built based on our observation on various user questions, are domain-independent, because the omission of temporal or spatial information is common not only in weather information question, but also in questions for other domains. The user profile is used for the inference in query analysis. We observed many people omit the place name in the weather-domain question. Unlike the temporal information, it is impossible to guess the current location without any user information.</Paragraph>
      <Paragraph position="5"> Thus, we store some user-related information in the user profile. Portfolio of stocks or favorite TV programs can be stored in the user profile if the QA system processes queries on stock quote or TV schedule domain.</Paragraph>
      <Paragraph position="6"> Let's take an example of the query analysis. The following keywords are extracted from the question  Even though the time, date, and city is not explicitly mentioned in the question, the question analyzer infers the information with the user profile and the inference rules.</Paragraph>
    </Section>
    <Section position="3" start_page="1" end_page="1" type="sub_section">
      <SectionTitle>
3.3 Query Frame Decision
</SectionTitle>
      <Paragraph position="0"> Restricting the question domain and information resource, we could restrict the scope of user request.</Paragraph>
      <Paragraph position="1"> That is, there is a finite number of expected question topics. Each expected question topic is defined as a single query frame. The following are query frame examples. They are used for processing the query for the precipitation forecast for the next day, diurnal range of today, current wind direction, and current temperature, respectively.</Paragraph>
      <Paragraph position="2">  SELECT date, amprecpr, pmprecpr FROM forecast tbl WHERE $date $city date, amprecpr and pmprecpr are field names in the database table forecast tbl, which mean date, the precipitation probability of morning and afternoon of the day. The rule generates the SQL statement that means: retrieve the precipitation probability of tomorrow morning and afternoon from the DB table which stores forecast information.</Paragraph>
      <Paragraph position="3"> Here is another example, which is the SQL generation rule for [DIURNALRANGE TODAY].</Paragraph>
      <Paragraph position="4"> [DIURNALRANGE_TODAY] SELECT city, max(temp)-main(temp) FROM current tbl WHERE $date $city group by city Analyzing a question means selecting a query frame in this system. Thus, it is important to select the appropriate query frame for the user request. The selection process is a great influence on the precision of the system, while there is not much likelihood of errors in other processes, such as generating SQL query from the selected query frame, retrieving DB records, and generating an answer.</Paragraph>
      <Paragraph position="5"> As represented in Figure 2, the extracted event, temporal and spatial keywords are used for selecting an appropriate query frame. Currently, we are using a hand-coded decision tree-like classifier for selecting an appropriate query frame for the extracted keywords. If a question isn't proper for the handling  domain, the classifier rejects it. Machine learned classifier is being developed in order to substitute for the hand-coded classifier.</Paragraph>
    </Section>
    <Section position="4" start_page="1" end_page="1" type="sub_section">
      <SectionTitle>
3.4 SQL Generation
</SectionTitle>
      <Paragraph position="0"> If a query frame is selected for a question, an SQL query statement is generated from the SQL production rule of the frame. The query is sent to the DBMS to acquire the records that match to the query. Figure 4 depicts the conversion from a natural language question to its SQL expression.</Paragraph>
    </Section>
    <Section position="5" start_page="1" end_page="1" type="sub_section">
      <SectionTitle>
3.5 Answer Generation
</SectionTitle>
      <Paragraph position="0"> Based on the result of the DBMS, a natural language answer is generated. We use a rule based answer generation method. Each query frame has an answer generation pattern for the frame. For example, DIURNALRANGE TODAY has the following generation pattern.</Paragraph>
      <Paragraph position="1"> [DIURNALRANGE_TODAY] The diurnal temperature range of $date($1) is $2</Paragraph>
      <Paragraph position="3"> the queried result. $date() is the function that converts a normalized date expression to a natural language expression. Figure 5 shows the answer generated from the SQL query shown in Figure 4 (More sample outputs from the QA System are presented on the Appendix) .</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="1" end_page="1" type="metho">
    <SectionTitle>
4 Evaluation and Limitation
</SectionTitle>
    <Paragraph position="0"> We have evaluated our domain restricted QA system based on precision and recall, and investigated the limitation of the our approach to the restricted-domain QA system.</Paragraph>
    <Paragraph position="1"> For evaluation, we've collected 50 weather questions from 10 graduate students. Precision and recall rates are 90.9 % and 75.0% respectively.</Paragraph>
    <Paragraph position="2"> The low recall rate is due to some questions related to invalid date and topic. The system provides weather forecasts for 7 days from the querying day.</Paragraph>
    <Paragraph position="3"> But some of queries are asking for a future weather outlook which is out of range ( e.g. Will it be very hot summer this year? or Will it be snow on this Christmas?). Some questions asked the information that the database doesn't contain, such as UVI (ultraviolet index).</Paragraph>
    <Paragraph position="4"> The primary reason for the wrong answer is the failure of invalid topic rejection. It is due to the insufficient of weather-domain ontology data. For example, from the question What is the discomfort index calculated from the today's weather?, the keyword discomfort index was not extracted while weather was extracted, because the former was not in the ontology. So the query frame WEATHER TODAY was misselected and the system generated the wrong answer Seoul will be sunny on March 9th 2004.</Paragraph>
    <Paragraph position="5"> An error was caused by the flaw of our keyword-based query frame decision approach. For the question Can the flight for Jeju Island take off today?, the extracted keywords are  In order to know whether the flight can take off or not, the weather information of the departure city instead of the destination city (i.e. Jeju) should be returned, but our keyword based approach failed to make an appropriate query. To solve this problem, more sophisticated semantic representation, rather than the sequence of keywords, is required for the question.</Paragraph>
  </Section>
  <Section position="6" start_page="1" end_page="1" type="metho">
    <SectionTitle>
5 Related Works
</SectionTitle>
    <Paragraph position="0"> In this section, we compare our system with other QA-related approaches and briefly describe the distinctive characteristics of our system. Open-domain QA systems in QA track mostly extract answers from unstructrued documents. In the contrast, our system extracts answers from semi-structured web pages, which are pre-selected by us, because our system aims to achieve high precision with the sacrifice of the coverage of questions.</Paragraph>
    <Paragraph position="1"> Natural language front ends for databases (Copestake and Jones, 1990) and our system handle user questions similarly. However, our system has information extraction part that makes the database be updated regularly and automatically. Moreover, our system returns natural language responses to users.</Paragraph>
    <Paragraph position="2"> The START system (Katz, 1997) is a web-based QA system. It uses World Wide Web as knowledge resource. Unstructured natural language sentences are indexed in the form of ternary expressions and stored in RDB. The START system covers much wider domain of questions than ours, however, it seems that the system returns more wrong answers than ours, because we extract the answer only from semi-structured documents.</Paragraph>
    <Paragraph position="3"> The Jupiter system (Zue et al., 2000) is a conversational system that provides weather information over the phone. Based on the Galaxy architecture (Goddeau et al., 1994), Jupiter recognizes user question over the phone, parses the question with the TINA language understanding system (Seneff, 1992) and generates SQL and natural language answer with the GENESIS system (Baptist and Seneff, 2000). The generated answer is synthesized with the ENVOICE system. Even the Jupiter system deals with the same domain, ours can process a bit wider-range of weather topics. Our QA system can cover the question which requires inferences such as When is the best day for washing my car in this week? Moreover, our system has an ability of inferring missing information from the user profile and the inferring algorithm.</Paragraph>
  </Section>
class="xml-element"></Paper>