<?xml version="1.0" standalone="yes"?>
<Paper uid="H91-1015">
  <Title>SPEECH RECOGNITION IN SRI'S RESOURCE MANAGEMENT AND ATIS SYSTEMS</Title>
  <Section position="7" start_page="97" end_page="98" type="concl">
    <SectionTitle>
SLS Evaluation
</SectionTitle>
    <Paragraph position="0"> We evaluated on DARPA's February 1991 ATIS test set using a system similar to the one described above except:  groups was used, with a test set perplexity of 43 (not counting 26 words out of vocabulary).</Paragraph>
    <Paragraph position="1"> * A template-matcher natural language component \[2\] was used to generate ATIS database queries based on the speech recognition output.</Paragraph>
    <Paragraph position="2"> We achieved the performance shown in Table 10.</Paragraph>
    <Paragraph position="3">  As can be seen, speakers CI and CM contributed significantly to the overall error rate. Furthermore, many of the errors occurred despite their relatively small bigram probabilities, indicating that the grammar is still not completely effective in overriding poor acoustic matches.</Paragraph>
    <Paragraph position="4">  The most interesting result of this evaluation (see the paper by PaUett in this proceedings) was that, though SRI along with BBN achieved the best speech recognition accuracy, and SRI along with CMU had the best natural-language-only performance, the accuracy of SRI's combined speech and natural language systems  1. NA is no answer 2. WErr or weighted error is percent no answer plus two times the percent wrong.</Paragraph>
    <Paragraph position="5"> 3. Score = 100 - Werr  was far better than that for the other sites. We attribute this to the error tolerant nature of our speech/natural-language interface. For instance, note that performance using spoken language is not much worse than the performance of the NL component given transcribed input (i.e. given a perfect speech recognition component) even though the SLS speech recognition component had a 60 percent sentence error rate (at least one word was wrong in 60 percent of the sentences). The above results indicate to us that steady progress in the speech recognition and natural language technologies, together with error-tolerant speech/natural-language interfaces can lead to practical spoken language systems in the near future.</Paragraph>
  </Section>
class="xml-element"></Paper>