<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-3007">
  <Title>User-Centered Evaluation of Interactive Question Answering Systems</Title>
  <Section position="7" start_page="55" end_page="55" type="concl">
    <SectionTitle>
5 Conclusions
</SectionTitle>
    <Paragraph position="0"> We have sketched a method for evaluating interactive analytic question answering system, identified key design decisions that developers must make in conducting their own evaluations, and described the effectiveness of some of our methods.</Paragraph>
    <Paragraph position="1"> Clearly, each evaluation situation is different, and it is difficult to develop one-size-fits-all evaluation strategies, especially for interactive systems.</Paragraph>
    <Paragraph position="2"> However, there are many opportunities for developing shared frameworks and an infrastructure for evaluation. In particular, the development of scenarios and corpora are expensive and should be shared. The creation of sharable questionnaires and other instruments that are customizable to individual systems can also contribute to an infrastructure for interactive QA evaluation.</Paragraph>
    <Paragraph position="3"> We believe that important opportunities exist through interactive QA evaluation for understanding more about the interactive QA process and developing extensive theoretical and empirical foundations for research. We encourage system developers to think beyond independent system evaluation for narrow purposes, and conduct evaluations that create and inform theoretical and empirical foundations for interactive question answering research that will outlive individual systems. Although we do not have space here to detail the templates, instruments, and analytical schemas used in this study, we hope that the methods and metrics developed in connection with our study are a first step in this direction  . We plan to publish the full set of results from this study in the future.</Paragraph>
  </Section>
class="xml-element"></Paper>