<?xml version="1.0" standalone="yes"?>
<Paper uid="H91-1016">
  <Title>EVALUATION OF THE CMU ATIS SYSTEM</Title>
  <Section position="3" start_page="0" end_page="0" type="intro">
    <SectionTitle>
INTRODUCTION
</SectionTitle>
    <Paragraph position="0"> Understanding spontaneous speech presents several problems not found in transcribing read speech input. Spontaneous speech is often not fluent. It contains stutters, filled pauses, restarts, repeats, interjections, etc. Casual users do not know the lexicon and grammar used by the system. It is therefore very difficuk for a speech understanding system to achieve good coverage of the lexicon and grammar that subjects might use. Also, the task of the system is not just to produce a transcription, but to produce an action. Taking such actions requires parsing and &amp;quot;understanding&amp;quot; the utterance. Word error rate is not as important as utterance understanding rate.</Paragraph>
    <Paragraph position="1"> The Air Travel Information Service task is being used by several DARPA-funded sites to develop and evaluate speech understanding systems for database query tasks. In the ATIS task, novice users are asked to perform a task that requires getting information from the Air Travel database. This database contains information about flights and their fares, airports, aircraft, etc.</Paragraph>
    <Paragraph position="2"> The only input to the system is by voice. Users compose the questions themselves, and are allowed to phrase the queries any way they choose. No explicit grammar or lexicon is given to the subject.</Paragraph>
    <Paragraph position="3"> At Carnegie Mellon University, we have been developing a system, called Phoenix, to understand spontaneous speech \[1\] \[2\] \[3\]. We have implemented an initial version of this system for the ATIS task, This paper presents the design of the Phoenix system and its current status. We also report system evaluation results for the DARPA Feb91 test.</Paragraph>
  </Section>
class="xml-element"></Paper>