Project Summary: Linguistic Knowledge 
Sources 
for Spoken Language Understanding 
Principal Investigators 
Lynette Hirschraan and Deborah Dahl 
Unisys Defense Systems 
Center for Advanced Information Technology 
Paoli, PA 19301 
The objective of the Unisys Spoken Language Systems 
effort is to develop and demonstrate technology for the 
understanding of goal-directed spontaneous speech in- 
put. The Unisys spoken language architecfure couples 
a speech recognition system (the MIT Summit system) 
with the Unisys discourse understanding system Pundit. 
Pundit is a broad-coverage language understanding sys- 
tem used in a variety of message understanding applica- 
tions and extended to handle spoken language input. Its 
power comes from the integration of syntax, semantics 
and pragmatics (context), the ability to port r~pidly to 
new task domains, and from an open, modular architec- 
ture. Pundit is unique in its ability to handle connected 
discourse; it includes a reference resolution module that 
tracks "discourse entities" and distinguishes references 
to previously mentioned entities from the introduction of 
new entities. The Pundit front-end supports turn-taking 
dialogue and permits the system to include both ques- 
tions and answers in building an integrated discourse 
context, required for the handling of interactive commu- 
nication. Pundit has been interfaced to speech recog- 
aition systems (both Summit and the 1TT continuous 
speech recogniser) to perform applications on direction- 
finding assistance (Voyager), air travel planning (ATIS), 
and air traffic control (Right strip updating). 
Progress 
During the first year of our contract, we have ported 
the MIT Summit system and provided a loosely coupled 
interface with Pundit, via an N-best interface. Pundit 
processes the N top-scoring acoustic hypotheses until it 
finds an "understandable" hypothesis. We have been 
sble to evaluate performance of this system on a demon- 
stration task for direction finding assistance (based on 
the Mrr Voyager system). We have also paxticipated in 
the DARPA Common Evaluation Task in the Air Travel 
Information System domain. In support of the evalua- 
tion efforts, we have proposed an evaluation technique 
for automated dialogue evaluation and have developed 
techniques for "outside evaluators" to evaluate correct- 
ness of query/response pai_~. Our major results to date 
include: 
Demonstration of 70% correct test results on 
unseen data for a typed direction-assistant 
task, based on 1000-sentence training corpus, 
using the Pundit language understanding 
system plus a dialogue management front- 
end which manages the user/computer con- 
versation. 
Demonstration of a loosely coupled Spoken 
Language System (50-100x real-time); in ini- 
tial experiments, application accuracy is ap- 
proximately 50% for Spoken Language test 
data, with a false alarm rate ofless than 5%. 
Demonstration of a typed query front-end 
for the Air Travel Information System appli- 
cation, interfaced to an INGRES database. 
Metrics will be reported at the June DARPA 
Workshop. 
Development of a proposal for evaluation 
of discourse, extending the notion of the 
database comparator for use in evaluation of 
sentences in context. 
Plans 
Our plans for the coming year focus on increasing the 
performance of our spoken language system, with partic- 
ular focus on measuring and improving system usability, 
that is, the degree to which the system helps (or hin- 
ders) a user to accomplish a particulax task. This will 
requite extensions to the vocabulary (1000-2000 words), 
increase in perplexity (60-100), increased coverage (at 
least 70%), and improved error diagnosis and feedback. 
It will also require the development of metrics for user 
satisfaction and for task completion. This goal will also 
drive development ofint~active dialogue capability, pro- 
riding co-operative responses to users. We plan to de- 
velop a testbed, where we can interface multiple speech 
recognition front-ends to Pundit, via the N-best inter- 
face and the stack-decoder interface developed at Lincoln 
Labs. This will provide insight into some of the trade- 
offs between recognizer improvements and improvements 
in language understanding. 
426 
