<?xml version="1.0" standalone="yes"?>
<Paper uid="H91-1017">
  <Title>Interface Bugs Ignored Test Set System % Correct % Incorrect % No Answer Total Error</Title>
  <Section position="2" start_page="0" end_page="0" type="intro">
    <SectionTitle>
ABSTRACT
</SectionTitle>
    <Paragraph position="0"> This paper describes the structure and operation of SouL, or Semantically-Oriented Understanding of Language. SouL is a knowledge intensive reasoning system which is opportunistically used to provide a more thorough, fme grained analysis of an input utterance following its processing by a case-frame speech parser. The SOUL postprocessor relies upon extensive semantic and pragmatic knowledge to correct, reject and/or clarify the outputs of the CMU PHOl/NlX case-frame parser for speech and speech transcripts. Specifically, we describe briefly both some of the linguistic phenomena which SouL addresses and how SOUL works to correct inaccurate interpretatiens produced by the PHOENIX parser. Finally, we present the results on four separate, nen-ovedapping test sets. Our &amp;quot;pilot&amp;quot; test sets include the June 1990 DARPA ATIS0 test set and two test sets composed of unseen ATIS0 data distributed in June 1990 that, unlike the DARPA test sets, contain tmrestrieted utterances.</Paragraph>
    <Paragraph position="1"> Our forth test set is the official DARPA February 1991 ATIS1 test set.</Paragraph>
    <Paragraph position="2"> These evaluations illustrate the decrease in error ram that results from SOUL's semantic and pragmatic postprocessing are most pronounced in unrestricted, as opposed to carefully constrained test sets. Specifically, a performance comparison between unrestricted mad restricted test sets in pilot experiments show that error rates are reduced by 84% as opposed to 54% when no utterances are pruned from the speaker transcripts.</Paragraph>
  </Section>
class="xml-element"></Paper>