<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-0502">
  <Title>Evaluation of Restricted Domain Question-Answering Systems</Title>
  <Section position="3" start_page="0" end_page="0" type="intro">
    <SectionTitle>
2 Restricted-domain system
</SectionTitle>
    <Paragraph position="0"> characteristics The restricted-domain systems of today are different from the toy systems from the early years of QA (Voorhees and Tice, 2000), which might be what first comes to mind when reading the term 'restricted-domain'. Early systems like LUNAR (with a domain somewhat tangentially related to ours, namely lunar archeology) were developed by researchers in the field of natural language understanding. These early systems encoded large amounts of domain knowledge in databases. The restricted-domain systems of today are far less dependent on large knowledge bases and do not aim for language understanding per se. Rather, they use specialized extraction rules on a domain specific collection. The one thing that both types of restricted-domain systems have in common is that they are often developed with a certain goal or task in mind.</Paragraph>
    <Paragraph position="1"> As we will see later, this task orientation becomes equally important in the evaluation of these QA systems.</Paragraph>
    <Paragraph position="2"> An example of a modern-day restricted-domain system is our Knowledge Acquisition and Access System (KAAS) QA system. The KAAS was developed for use in a collaborative learning environment (Advanced Interactive Discovery Environment for Engineering Education or AIDE) for undergraduate students from two universities majoring in aeronautical engineering. While students are working within the AIDE they can ask questions and quickly get answers. The collection against which the questions are searched consists of textbooks, technical papers, and websites that have been pre-selected for relevance and pedagogical value. The KAAS system uses a two-stage retrieval model to find answers in relevant passages.</Paragraph>
    <Paragraph position="3"> Relevant passages are processed by the Center for Natural Language Processing's eQuery information extraction system using additional rules in the domain of reusable launch vehicles. Users are aided in their question formulations through domain specific query expansions.</Paragraph>
  </Section>
class="xml-element"></Paper>