<?xml version="1.0" standalone="yes"?>
<Paper uid="H89-2022">
  <Title>PRELIMINARY EVALUATION OF THE VOYAGER SPOKEN LANGUAGE SYSTEM*</Title>
  <Section position="9" start_page="164" end_page="165" type="concl">
    <SectionTitle>
SUMMARY
</SectionTitle>
    <Paragraph position="0"> In this paper we presented some results on the preliminary evaluation of the VOYAGER system. As we have stated at the onset, we are entering into a new era of research, and we do not have a clear idea of how spoken language systems should best be evaluated. However, we have chosen to explore this issue along several dimensions. We have reached the conclusion that a totally objective measure of performance may not  answer answer error error response appropriate verbose appropriate ambiguous incorrect  and (b) orthographic input.</Paragraph>
    <Paragraph position="1"> be possible now that systems have become more complex. While some objective criteria exist for individual components, overall system performance should probably incorporate subjective judgements as well. Thus far, we have not addressed the issue of efficiency, mainly because we have not focussed our attention on that issue. When VOYAGER was first developed, it ran on a Symbolics Lisp machine, and took several minutes to process a sentence. More recently, we have started to use general signal processing boards to derive the auditory-based signal representation, and a Sun workstation to implement the remainder of the SUMMIT recognition system. Currently, the system runs in about 12 times real-time. The approximate breakdown in timing is shown in Table 3. Note that the natural language component and the back end run in well under real-time. Refined algorithms, along with the availability of faster workstations and more powerful signal processing chips should enable the current VOYAGER implementation to run in real-time in the future. On the other hand, the computation is likely to increase dramatically when speech recognition and natural language are fully integrated, since many linguistic hypotheses must be pursued in parallel.</Paragraph>
  </Section>
class="xml-element"></Paper>