ANNOTATION OF ATIS DATA 
Kate Hunicke-Smith, Project Leader 
Jared Bernstein, Principal Investigator 
SRI International 
Menlo Park, California 94025 
PROJECT GOALS 
The performance of spoken lartguage systems on utterances 
from the ATIS domain is evaluated by comparing system- 
produced responses with hand-crafted (and -verified) stan- 
dard responses to the same utterances. The objective of 
SRI's annotation project is to provide SLS system develop- 
ers with the range of correct responses to human utterances 
produced during experimental sessions with ATIS domain 
interactive systems. These correct responses are then used 
in system training and evaluation. 
PLANS FOR THE COMING YEAR 
In the next year, SKI will continue to provide MADCOW 
annotation and other services to the DARPA community. In 
addition to tasks previously performed, SRI will assume a 
greater role in the resolution of training data bug reports. 
RECENT RESULTS 
Since June 1991. SRI has produced classification and 
response files for about 13,000 utterances of training data 
(4000 of these since January, 1993). A dry run system eval- 
uation and three official evaluations have been held since 
the project began in 1991. SKI has produced the standard 
responses for all of these evaluations; in all, about 3300 
ntteranc~s. 
These tests were performed according to the Common 
Answer Specification (CAS) protocol which is used in 
training. All systems are evaluated on a common set of 
data, with system responses measured against otiicial refer- 
ence answers produced at SRI in the same manner as the 
training data. 
In addition to producing the classification and standard 
response files, SKI takes an active role in the adjudication 
of test and training data bug reports, initiates nearly all of 
the changes to the Principles oflnterpretation document (a 
basic set of principles for interpreting the meaning of ATIS 
sentences agreed upon by the DARPA community), and 
continues to support NIST by modifying software and act- 
hag as a consultant regarding the correct annotation of data. 
471 
J 
