<?xml version="1.0" standalone="yes"?>
<Paper uid="C94-1013">
  <Title>Evahmtion Metrics t'oi- Knowledge-Based Machine Translation</Title>
  <Section position="2" start_page="0" end_page="0" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> Machine Translation (MT) is considered the paradigm task of Natural Language Processing (NLP) hy some researchers because it combines almost all NLP research :treas: syntactic parsing, semantic disambigt, ation, knowledge rel)reseutation, language generation, lexical acquisition, and morphological analysis and synthesis. However, the evaluation methodologies for MT systems have heretofore centered on hlack box approaches, where global properties of tile system are evaluated, such as semantic fidelity of the translation or comprehensibility of the target langt,age output. There is a long tradition of such black-box MT evaluations (Van Slype, 1979; Nagao, 1985; JEIDA, 1989; Wilks, 1991), to the point that Yorick Wilks has stated: &amp;quot;MT Evaluation is better understood than MT&amp;quot; (Carbonell&amp;Wilks. 1991 ). While these evalt,,'ltions are extremely important, they should be augmented with detailed error analyses and with component cval uation s in ordcr to produce causal analyses l)inpointing errors and therefm'e leading to system improvement. In essence, we advocate both causal component analyses as well as gloi)al behavioral analyses, preferably when the latter is consistent with tile Iormer via composition of the component analyses.</Paragraph>
    <Paragraph position="1"> Tim advent of Knowledge Based Machine Translation (KBMT) facilitates component evaluation and error attribution because of its modular nature, though this ol)servalion by no means excludes transfer-based systems from similar aualyses. After reviewing the reasons att(I criteria for MT evaluation, this paper describes a specific evaluation methodology and its application to the KANT system, developed at CMU's Center for Machine Translation (Mitamura, et al.</Paragraph>
    <Paragraph position="2"> 1991). The KANT KBMT architecture is particularly well-suited for detailed evaluation because of its relative simplicity ':ompared to other KBMT systems, and because it has been scaled up to industrial-sized al)plications.</Paragraph>
  </Section>
class="xml-element"></Paper>