<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-2007">
  <Title>Using an incremental robust parser to automatically generate semantic UNL graphs</Title>
  <Section position="7" start_page="6" end_page="10" type="evalu">
    <SectionTitle>
5 Evaluation
</SectionTitle>
    <Paragraph position="0"> A complete evaluation of a UNL enconverter should take into account the following possible kinds of errors: * graphs with wrong linguistic information (semantic relations, attributes, etc.), * missing information (incomplete graph due to missing relations, incomplete decorations, etc.), * graphs with wrong UWs (wrong acception or wrong lemma).</Paragraph>
    <Paragraph position="1"> Since in this article we want to emphasize the use of an incremental robust parser for creating an enconverter, we evaluated errors concerning semantic relations  , thus the first and the second points which correspond, respectively, to classic evaluation metrics of precision and recall rates.</Paragraph>
    <Paragraph position="2"> The enconverter was tested against the first 50 manually enconverted UNL graphs (1.059 words) from a corpus of legal text. The average length of the sentences was about 21 words (21,18). The semantic relations evaluated in this preliminary experiment (322 UNL expressions) were agt (44), obj (57) and mod (221).</Paragraph>
    <Paragraph position="3"> Table 3 gives the results obtained for the evaluation of this first version of the encon- null version of the enconverter.</Paragraph>
    <Paragraph position="4"> For agents, most errors come from syntactic subjects correctly identified by the parser but presenting semantic features that should had been taken into account to create aoj relations. To give an example, in the sentence &amp;quot;La culture acquiert des formes diff'erentes (...)&amp;quot;  , the parser extracts correctly the dependency subj(acquire,culture) although it is semantically encoded as aoj(acquire,culture) in UNL because the verb &amp;quot;acquire&amp;quot; is considered in this utterance as a verb of state.</Paragraph>
    <Paragraph position="5"> In the case of objects, errors on precision concern wrong scope of coordination as well as objects being a whole sentence. As for recall, there are several constructions which may be considered obj from a semantic point of view but that the parser identifies as modifiers due to their surface construction with a preposition. For example, &amp;quot;source de creativit'e&amp;quot;  is analyzed by the parser as mod(source,de,cr'eativit'e) although UNL encodes obj(source,creativity). Likewise,  The presence or absence of the different attributes was not evaluated.</Paragraph>
    <Paragraph position="6">  Culture acquires different forms (...).  Source of creativity.</Paragraph>
    <Paragraph position="7"> &amp;quot;vers l'acc'es de la diversit'eculturelle&amp;quot;  is encoded in UNL as obj(towards,access),akind of relation that the parser does not extract. A final remark concerns modifiers. As said before, the parser is not deterministic in marking modifiers: all possible combinations between a head word and its dependents are extracted. That is the main reason why precision is low and recall is high.</Paragraph>
    <Paragraph position="8"> The average of all these figures gives a global evaluation of the enconverter corresponding to</Paragraph>
    <Paragraph position="10"/>
  </Section>
class="xml-element"></Paper>