<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-2105">
  <Title>Semantic Interpretation of Prepositions for NLP Applications</Title>
  <Section position="6" start_page="32" end_page="34" type="evalu">
    <SectionTitle>
6 Evaluation
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="32" end_page="33" type="sub_section">
      <SectionTitle>
6.1 Intrinsic Evaluation
</SectionTitle>
      <Paragraph position="0"> Experiments showed that 24.2% of verb complement interpretations are equally well produced by adjunct rules (see column 5 in Table 3). Nevertheless the PP interpretation disambiguation task profits from complement information because in many of these overlap cases more than one PP interpretation was possible. Also the PP attachment disambiguation task can benefit from the complement vs. adjunct distinction because complementhoodisastrongindicatorofthecorrectattachment null place. A third argument for having such complement information is that it is important to model all roles belonging to a concept on the cognitive level; this can be easily realized by a one-to-one correspondence between cognitive roles and complements in the lexicon. Table 3 shows the number  realized as a PP or an NP. Reg. compl. means that the complement semantics specified in the lexicon is equally well produced by some PP interpretation rule and therefore viewed as being regular. cat. lexemes PP compl. lexemes with PP compl.</Paragraph>
      <Paragraph position="1"> reg. compl. mixed PP compl. lexemes with mixed PP compl.</Paragraph>
      <Paragraph position="2"> v 7006 1690 1616 24.2% 105 100 n 13111 720 684 5.6% 3750 2393 of lexemes of a given category with some PP complements and the total number of PP complements forverbsandnounsinourlexicon. Thepercentage of regular complements (24.2%) is significantly higher than the corresponding token value in the QA@CLEF corpus (17.5%, see Table 2). This indicates that regular complements often have a more optional character than irregular complements. Also in this respect, regular complements resemble adjuncts.</Paragraph>
      <Paragraph position="3"> The preposition interpretation method achieves between 84% and 89% correctness for the six prepositions supported by the hand-tagged PP corpus; for prepositions without annotated corpus data, the performance seems to drop by around 10 to 20 percent points.</Paragraph>
    </Section>
    <Section position="2" start_page="33" end_page="34" type="sub_section">
      <SectionTitle>
6.2 Extrinsic Evaluation
</SectionTitle>
      <Paragraph position="0"> One important application using the parser and the preposition interpretation described above is In-Sicht, a QA system for German text collections (Hartrumpf, 2005). To measure the impact of a deep preposition interpretation, the QA system was run twice: with and without the PP interpretation presented above. For the latter, each interpretation of a PP with preposition p, an NP c2, and syntactic head c1 was replaced by an edge with the unique artificial relation PP.p, e.g. the aus-PP rule in Fig. 2 would contain the conclusion (PP.AUS c1 c2). The QA system was evaluated on the German questions from the question answering track at CLEF (QA@CLEF) of the years 2004 and 2005. Surprisingly, the PP interpretation with unique artificial relations caused no significant performance drop. A closer look at all questions from QA@CLEF 2004 involving PPs revealed that the documents with answers almost always contained the same prepositions as the corresponding questions. Therefore we tried more difficult (and often more realistic) questions with different prepositions (and verbs or nouns).</Paragraph>
      <Paragraph position="1"> Only natural and (nearly) equivalent paraphrases were allowed. For the PP questions where the QA system delivered correct answers, 14 paraphrases were written to test the positive impact of transforming surface prepositions to their correct meaning.</Paragraph>
      <Paragraph position="2"> The evaluation of PP semantics was then performed using the paraphrases instead of the original questions. For 86% of all paraphrases, the correct answer was still found when the more distant paraphrase was used as the question for the QA system; with the artificial relations for PPs, only 14% of the paraphrases were answered correctly.</Paragraph>
      <Paragraph position="3"> This indicates clearly that NLP applications like semantic QA systems benefit from a good preposition interpretation. The paraphrases that could not be answered by the QA system with PP interpretation would need more advanced reasoning techniques to work correctly.</Paragraph>
      <Paragraph position="4"> SomeparaphrasesinvolvedPPadjuncts. Forexample, theQA@CLEFquestionqa04 055isgiven as example (5):  this question correctly. But the paraphrase (6) requires that question and documents, which differ on the surface (in-PP vs. interrogative Wann), are transformed to the same representation (expressing a temporal relation).</Paragraph>
      <Paragraph position="5">  portance of a homogenous transition between the semantics of PP adjuncts and the semantics of PP complements. Example (7) (qa04 027) contains an interrogative involving f&amp;quot;ur, which is specified as a complement of the verb anklagen ('accuse') in the current version of HaGenLex.</Paragraph>
      <Paragraph position="6">  because both questions and the relevant document sentences contain the same semantic representation (here, a single relation of justification). All these paraphrases are examples of increased recall. But also the precision of the QA system isimprovedbecauseprepositionsensemismatches between question and documents can be detected.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>