<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-0610">
  <Title>On Distance between Deep Syntax and Semantic Representation</Title>
  <Section position="7" start_page="83" end_page="83" type="evalu">
    <SectionTitle>
5 Evaluation
</SectionTitle>
    <Paragraph position="0"> It is a still open question how to evaluate systems for semantic representation. Basically, three approaches are used in similar projects: First, the coverage of the system may serve as a basis for evaluation. This criterion is used in several systems (Bos, 2005; Hor*ak, 2001; Callmeier et al., 2004). However, this criterion is far from ideal, because it's not applicable to robust systems and can not tell anything about the quality of resulting representation.</Paragraph>
    <Paragraph position="1"> Second, the consistency of the semantic representation serves as an evaluation criterion in Bos (2005). It is a desired state to have a consistent representation of texts, but there is no guarantee that a consistent semantic representation is in any sense also a good one.</Paragraph>
    <Paragraph position="2"> Third, the performance in an application (e.g., question answering system) is another criterion used for evaluating a semantic representation (Hartrumpf, 2005). A problem in this kind of evaluation is that we can not separate the evaluation of the formalism itself from the evaluation of the automatic processing tools. This problem becomes even bigger in a multilayered approach like FGD or MTT, where the overall performance depends on all participating transducers as well as on the quality of the theoretical description. However, from the user point of view, this is so far the most reliable form of semantic representation evaluation.</Paragraph>
  </Section>
class="xml-element"></Paper>