<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-0905">
  <Title>Evaluating the Performance of the OntoSem Semantic Analyzer</Title>
  <Section position="8" start_page="1" end_page="1" type="concl">
    <SectionTitle>
5 Discussion and Future Work
</SectionTitle>
    <Paragraph position="0"> The kind of evaluation that we have undertaken so far reflects our desire to understand the causes of less-than-maximum results, that is, to assign blame to the various components of the analyzer. The results clearly show that the preprocessor we have so far been using in the OntoSem system does not perform sufficiently well, and we will change the preprocessor for the  first evaluation run of OntoSem Our WSD evaluation environment differs from many WSD approaches in that it allows the &amp;quot;none of the above&amp;quot; outcome for the cases when the lexicon entries do not fit the expectations in the text even after a measure of constraint relaxation. The count of incorrectly determined word senses includes the above eventuality but also the case when the current system has to select an answer from a set of candidates none of which can be preferred on the basis of available heuristics. For future evaluations, we plan to use the version of the analyzer with additional available means of ambiguity resolution incorporated (see Figure 2 for a brief listing). In fact, we will use different combinations of the procedures for residual ambiguity resolution and recovery from &amp;quot;unexpected&amp;quot; input to dete u analysis (not only WSD but also semantic dependency determination).</Paragraph>
    <Paragraph position="1"> The evaluation of semantic dependency determination is different from that suggested by Gildea and Jurafsky (2002) who designed a system to automatically learn the semantic roles of unknown predicates. First, that system does not actually do WSD; second, it makes assumptions that our work does not: it does not use any language-independent metalanguage to record meaning and concent restrictions, a far more limited inventory than the set of all possible relations between concepts provided in our ontology.</Paragraph>
    <Paragraph position="2"> The evaluation environment we have developed reduces the amount of time necessary to produce a sense that it is a very important enabling element for larger-scale evaluation work that from this point on will become standard proc gold standard output for each of the three stages of our analysis process quite dramatically. It is in this edure in our work on building semantic analyzers</Paragraph>
  </Section>
class="xml-element"></Paper>