<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-2509">
  <Title>Handling Information Access Dialogue through QA Technologies - A novel challenge for open-domain question answering -</Title>
  <Section position="2" start_page="0" end_page="0" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> Open-domain question answering technologies allow users to ask a question in natural language and obtain the answer itself rather than a list of documents that contain the answer. These technologies make it possible to retrieve information itself rather than merely documents, and will lead to new styles of information access (Voorhees, 2000).</Paragraph>
    <Paragraph position="1"> The recent research on open-domain question answering concentrates on answering factoid questions one by one in isolation from each other. Such systems that answer isolated factoid questions are the most basic level of question answering technologies, and will lead to more sophisticated technologies that can be used by professional reporters and information analysts. On some stage of that sophistication, a cub reporter writing an article on a specific topic will be able to translate the main issue addressed by his report into a set of simpler questions and then pose those questions to the question answering system (Burger et al., 2001).</Paragraph>
    <Paragraph position="2"> In addition, there is a relation between multi-document summarization and question answering. In his lecture, Eduard Hovy mentioned that multi-document summarization may be able to be reduced into a series of question answering (Hovy, 2001). In SUMMAC, an intrinsic evaluation was conducted which measures the extent to which a summary provides answers to a set of obligatory questions on a given topic (Mani et al., 1998). Those suggest such question answering systems that can answer a series of related questions would surely be a useful aid to summarization work by human and by machine.</Paragraph>
    <Paragraph position="3"> Against this background, question answering systems need to be able to answer a series of questions, which have a common topic and/or share a local context. In this paper, we propose a challenge to measure objectively and quantitatively such an ability of question answering systems. We call this challenge QACIAD (Question Answering Challenge for Information Access Dialogue). In this challenge, question answering systems are used interactively to participate in dialogues for accessing information. Such information access dialogue occurs such as when gathering information for a report on a specific topic, or when browsing information of interest to the user. Actually, in QACIAD, the interaction is only simulated and systems answer a series of questions in a batch mode. Although such a simulation may neglect the inherent dynamics of dialogue, it is a practical compromise for objective evaluation and, as a result, the test sets of the challenge are reusable.</Paragraph>
    <Paragraph position="4"> Question answering systems need a wide range of abilities in order to participate in information access dialogues (Burger et al., 2001). First, the systems must respond in real time to make interaction possible. They must also properly interpret a given question within the context of a specific dialogue, and also be cooperative by adding appropriate information not mentioned explicitly by the user. Moreover, the systems should be able to pose a question for clarification to resolve ambiguity concerning the user's goal and intentions, and to participate in mixed initiative dialogue by making suggestions and leading the user toward solving the problem. Among these various capabilities, QACIAD focuses on the most fundamental aspect of dialogue, that is, interpreting a given question within the context of a specific dialogue.</Paragraph>
    <Paragraph position="5"> It measures context processing abilities of systems such as anaphora resolution and ellipses handling.</Paragraph>
    <Paragraph position="6"> This paper is organized as follows. The next chapter explains the design of QACIAD. The following three chapters discuss the reality of the challenge. First, we explain the process of constructing the test set of the challenge and introduce the results of a study conducted during this process which show the validity of QACIAD.</Paragraph>
    <Paragraph position="7"> That is, QACIAD measures valid abilities needed for participating in information access dialogues. In other words, the ability measured by the challenge is crucial to the systems for realizing information access dialogues for writing reports and summaries. Second, we show the statistics of pragmatic phenomena in the constructed test set, and demonstrate that the challenge covers a wide variety of pragmatic phenomena observed in real dialogues.</Paragraph>
    <Paragraph position="8"> Third, based on a preliminary analysis of the QACIAD run, we show that the challenge has a proper difficulty for evaluating the current state of open-domain question answering technologies. In the last two chapters, we discuss problems identified while constructing the test set and conducting the run, and draw some conclusions.</Paragraph>
  </Section>
class="xml-element"></Paper>