<?xml version="1.0" standalone="yes"?>
<Paper uid="H89-1032">
  <Title>PLANS FOR A TASK-ORIENTED EVALUATION OF NATURAL LANGUAGE UNDERSTANDING SYSTEMS</Title>
  <Section position="9" start_page="201" end_page="201" type="concl">
    <SectionTitle>
SUMMARY
</SectionTitle>
    <Paragraph position="0"> This evaluation project represents a small step in the direction of benchmarking NLP systems. The results of the evaluation are not expected to be statistically significant but will begin to bring quantitative criteria to bear. They will also provide common ground for discussion of the language processing issues raised by military texts and the task-related issues of extracting, deriving and inferring needed information for a database. Developers of a number of different NLP systems that have processed military message texts are being invited to participate in this evaluation effort and will meet afterward to discuss the results and critique the test design. Although the test is designed to treat the NLP systems as black boxes, it is expected to yield a variety of meaningful measurements. It is also expected that discussion of the test results at the end of the evaluation will provide a great deal of insight not only into what systems can do but also how they do it.</Paragraph>
  </Section>
class="xml-element"></Paper>