<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-0905">
  <Title>Evaluating the Performance of the OntoSem Semantic Analyzer</Title>
  <Section position="3" start_page="0" end_page="0" type="intro">
    <SectionTitle>
2 Generating Gold Standard TMRs
</SectionTitle>
    <Paragraph position="0"> We have developed a hum OntoSem analyzer in which the results of major stages of ontological semantic a preprocessor output, syntax output and semantic output - can be inspected and corrected by a human. For purposes of evaluation, we have used it to produce gold standard (GS) outputs for each of the three stages. The production of gold standard outputs proceeds as follows:  1. Run the OntoSem analyzer on an input text. 2. Correct preprocessor out read in text format, and we have found quickest to simply correct it by hand. It takes on average 1 minute to correct an average-</Paragraph>
    <Paragraph position="2"> Input the corrected preprocessor results into the analyzer and produce a syntactic analysis.</Paragraph>
    <Paragraph position="3"> If necessary, use a specially d editing interface to add or delete edges on the chart that presents the results of syntactic analysis, to remove spurious parses, to correct phrase and clause boundaries, and to add any missing phrase or clause parses.</Paragraph>
    <Paragraph position="4"> Feed the correct syntax back into the analyzer and obtain a semantic analysis.</Paragraph>
    <Paragraph position="5"> If necessary, correct the semantic knowledge acquisition interfaces in order side effects of this process will include the creation of a bank of gold standard TMRs as well as, possibly, less importantly, gold standard results of preprocessing and syntactic analysis. Such resources are clearly valuable as training data for statistical NLP, and a number of projects are devoted to entirely or in a large measure to their creation. The process of producing gold standard TMRs, unlike most of the resource acquisition approaches, is, to a significant degree, automated which reduces the incidence of interannotator disagreement and generally makes the process faster and cheaper.</Paragraph>
    <Paragraph position="6"> In the OntoSem research paradigm, knowledge acquisition (enhancement of the ontology, the xicon and other basic static knowledge sources) ms that do not involve knowledge acquisition f the kind OntoSem uses. However, the set of the only one we consider ractical. It is not possible for a human to produce luate the results of fully utomatic analysis (see below); as training data for auto put we evaluate several * baseline 1: same as above, except we force the first senses in our lexicon entries are</Paragraph>
    <Paragraph position="8"> put to the For prep syntax results; semantics results and evaluation results The evaluation is dard outputs and include a) the ord/phrase count; b) the number of input words le is an ongoing process. The process of creating gold standard TMRs provides an empirical impetus for knowledge acquisition. This process will not at all interfere with our evaluation regimen because our approach does not rely on having a standard test corpus. We will simply run the entire evaluation procedure (starting with the production of the gold standard TMRs) on a new corpus, analyze the results and move on to yet another corpus, and so on.</Paragraph>
    <Paragraph position="9"> This approach cannot be directly exported to those syste o gold standard TMRs produced through our evaluation process will be made freely available and can serve as the test corpus for any other semantic analyzer (word sense disambiguator and/or semantic dependency extractor). This will be our direct contribution to the resource set in the field. Of course, using this resource will involve resolving the differences in the notation and semantics between the TMR structures and any other metalanguage.</Paragraph>
    <Paragraph position="10"> This methodology for producing gold standard</Paragraph>
    <Paragraph position="12"> gold standard semantic outputs by hand because of the complexity of the knowledge, as well as the high probability of annotator disagreement due to valid semantic paraphrasing (e.g., one annotator might describe the meaning of weapons of mass destruction as the union of BIOLOGICAL-WEAPON and CHEMICAL-WEAPON, whereas another might describe it as WEAPON that has the potential to kill more than 10,000 people).</Paragraph>
    <Paragraph position="13"> In sum, gold standard outputs are used for a number of purposes: to eva a machine learning, with the goal of improving the system's static knowledge sources; to trigger manual acquisition of knowledge for lacunae; or to derive high-confidence TMRs for use in mining information for a fact repository. Last but not least, the gold standard TMRs produced according to our methodology can also be directly used in a variety of applications - from human-assisted knowledge-based MT to knowledge acquisition for general-purpose reasoning systems.</Paragraph>
  </Section>
class="xml-element"></Paper>