<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-0502">
  <Title>Evaluation of Restricted Domain Question-Answering Systems</Title>
  <Section position="10" start_page="0" end_page="0" type="concl">
    <SectionTitle>
7 Summary
</SectionTitle>
    <Paragraph position="0"> Restricted-domain systems require a more situated evaluation than is generally provided in open-domain evaluations. A restricted-domain evaluation extends beyond domain specific test questions and collections to include the user and their task. Users of the restricted-domain KAAS system identified five areas that should be included in an evaluation: System Performance, Answers, Database Content, Display, and Expectations. Most of these evaluation dimensions could be applied to open-domain evaluations as well. Adding system performance metrics (such as answer speed) and specific task requirements may allow a convergence between open domain and restricted domain QA evaluations.</Paragraph>
  </Section>
class="xml-element"></Paper>