<?xml version="1.0" standalone="yes"?>
<Paper uid="W99-0407">
  <Title>FAME: a Functional Annotation Meta-scheme for multi-modal and multi-lingual Parsing Evaluation</Title>
  <Section position="2" start_page="0" end_page="0" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> Broad coverage parsing evaluation has received growing attention in the NLP community. In particular, comparative, quantitative evaluation of parsing systems has acquired a crucial role in technology assessment. In this context, it is important that evaluation be relatively independent of, or easily parametrizable relative to the following three dimensions of variation among parsing systems: * theoretical assumptions: compared systems may be based on different theoretical frameworks; null * multi-linguality: parsers are often optimally designed to deal with a particular language or family of languages; * multi-modality: systems tend to be specialized for dealing with a specific type of input, i.e. written or spoken language.</Paragraph>
    <Paragraph position="1"> As to the first point, it is important that alternative annotation schemes be evaluated (i) on the basis of the linguistic information they are intended to provide, and (ii) in terms of the utility of this information with respect to a particular task. Moreover, multi-linguality and multi-modality are crucial parameters for evaluating the robustness and portability of a given parser, with a view to the growing need for embedding NLP systems into multi-modal and multi-medial applications.</Paragraph>
    <Paragraph position="2"> An essential aspect of every evaluation campaign is the specification of an annotation scheme into which the output of the participant systems is converted and on whose basis the system performance is eventually evaluated. A suitable annotation scheme must satisfy some requirements. First of all, it should be able to represent the information relevant to a certain evaluation task in a way which is naturally conducive to quantitative evaluation. Secondly, it should easily be mappable onto different system outputs, and flexible enough to deal with multilingual phenomena and with the specific nature of both written and spoken language.</Paragraph>
    <Paragraph position="3"> The aim of this paper is to illustrate FAME, a</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
Functional Annotation Meta-scheme for Evaluation.
</SectionTitle>
      <Paragraph position="0"> We will show that it complies with the above mentioned requirements, and lends itself to effectively being used in comparative evaluation campaigns of parsing systems. There are two main features of FAME that will receive particular emphasis here: it is functional and it is a meta-seheme. We claim that these two features are essential for meeting the specific requirements of comparative parsing evaluation, while tackling issues of multi-linguality and multi-modality in a principled fashion.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>