Naval Ocean Systems Center 
Beth Sundheim, P.I. 
The Naval Ocean Systems Center (NOSC)is supporting DARPA's natural 
language research program in the area of test and evaluation. Preparations are 
being made for a task-oriented evaluation of text understanding systems, 
building off the natural language contractors' familiarity with texts taken from 
naval messages and NOSC's previous experience in carrying out system 
evaluations. The objective of the work is to devise an evaluation plan that will 
satisfy the needs for characterizing the current state of the art in theory- and 
implementation-independent ways and for setting baselines against which 
progress can be measured. 
Sets of texts have been collected to use as development (i.e. training) and test 
data, and explanatory documentation on them has been prepared. The task to 
be performed, a simulated database update task, has been defined, and 
samples of expected outputs have been prepared from texts in the 
development set. An overall evaluation plan and schedule have been set. 
Arrangements are being made to host MUCK-II (Message Understanding 
Conference 1/) in June 1989, where participants in the evaluation will meet to 
discuss the results and critique the test design. 
In FY90 NOSC plans to continue facilitating evaluation efforts, to determine 
useful new classes of naval texts to use in research and evaluation, to 
compile, analyze and distribute such texts, and to investigate opportunities for 
transitioning research into applications. It might be especially useful to 
compile a corpus for spoken language systems of voice reports (spoken naval 
messages) and corresponding written reports. 
195 
