<?xml version="1.0" standalone="yes"?>
<Paper uid="H90-1031">
  <Title>SRI's Experience with the ATIS Evaluation</Title>
  <Section position="1" start_page="0" end_page="0" type="abstr">
    <SectionTitle>
Abstract
</SectionTitle>
    <Paragraph position="0"> SRI International participated in the June 1990 Air Travel Information System (ATIS) natural-language evaluation. This report briefly describes the system that SRI used in the evaluation, analyzes SRI's results, and makes some recommendations for changes in the database structure and data collection system to be used for future ATIS evaluations.</Paragraph>
    <Paragraph position="1"> The SRI ATIS System The natural-language processing system used by SRI in the June 1990 ATIS evaluation is a derivative of the Core Language Engine (CLE) developed at SRI's Cambridge Research Centre in Cambridge, England [I]. At present, the main processing components of SRI's ATIS system are taken from the CLE, while the grammar, semantic interpretation rules.</Paragraph>
    <Paragraph position="2"> and lexicon are substantially new. The system divides query processing into the following phases:  The syntactic and semantic rules used in the parsing and interpretation phases are expressed in a unification-based formalism. The parser is based on a left-comer parsing algorithm for context-free grammar that has been generalized to apply to unification grammar by substituting unification for identity checks in dealing with gramatical category expressions. An attribute/value notation for feature constraints is provided for the grammar writer, but this notation is compiled into ordinary term structures by assigning, for each major category symbol, an argument position for each feature that can occur with that category. Grammatical unification is then implemented simply as term unification in Prolog, which is the implementation language used in the system.</Paragraph>
    <Paragraph position="3"> In the semantic interpretation phase, logical form expressions are computed bottom-up by applying semantic interpretation rules keyed to the syntax rules. Terms in the logical form language have semantic sorts associated with them, and functors are restricted with respect to the sorts of their arguments. These sort restrictions are applied as the logical forms are constructed, acting as a filter on the structures produced by the syntactic and semantic rules. The outputs of the semantic interpretation phase are quasi-logical forms in which the scope of quantified noun phrases has not yet been determined.</Paragraph>
    <Paragraph position="4"> Quantifier scope is assigned in the next phase of processing.</Paragraph>
    <Paragraph position="5"> At this point in processing, a database-independent formal representation of the meaning of the query has been assigned.</Paragraph>
    <Paragraph position="6"> This is transformed into a database query, principally by replacing the logical-form constants and predicates derived from the lexicon with database predicates and constants. The query is then re-ordered, if necessary, to optimize database retrieval, and the answer is retrieved from the database, which is stored as a set of Prolog clauses.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
Analysis of Results
</SectionTitle>
      <Paragraph position="0"> In the blind test conducted for the June 1990 ATIS evaluation, out of 90 test queries, the SRI system produced correct answers for 25, incorrect answers for 5, and no answer for 60. Thus, the dominant factor in the performance of the SRI system was that most queries failed to get through all stages of processing. Table 1 displays the number and percentage of the queries that failed to get past various levels of processing.</Paragraph>
      <Paragraph position="1"> These numbers should be regarded at best as only an approximation of the performance of the different components of the system, for two reasons. First, no attempt has been made to judge the correctness of the output of individual system phases, only to determine whether the phase produced an answer at all. Second, the failure rate of the later phases of processing would probably have been higher if more queries had gotten past the earlier phases of processing.</Paragraph>
      <Paragraph position="2"> With these caveats, the results seem to indicate that most of the difficulties arose in the semantic interpretation phase and the database query generation phase. The grammar seemed to provide fairly good coverage of the syntactic constructions used, and the lexicon performed surprisingly well given that the vocabulary in the test was completely uncontrolled.</Paragraph>
      <Paragraph position="3"> Undoubtedly, many of the parsing and interpretation failures</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
Lexicon
</SectionTitle>
      <Paragraph position="0"> were due to the absence of some of the necessary lexical entries for particular words, but almost no words in the test material were totally absent from the lexicon.</Paragraph>
      <Paragraph position="1"> The semantic rules and the database query generator are, in fact, the parts of the system that are the most recent in origin and must be regarded as far from complete, independently of how they performed on this evaluation. Our main conclusion, then, is simply that much more work is needed on these parts of the system.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>