<?xml version="1.0" standalone="yes"?>
<Paper uid="W97-0502">
  <Title>Langer S. and Hickey M. in preparation. Using Semantic Lexicons for Intelligent Message Retrieval in a Communication Aid. Submitted to Journal of Natural Language Engineering, special issue on Natural Language Processing for Communication</Title>
  <Section position="5" start_page="13" end_page="13" type="evalu">
    <SectionTitle>
4 Evaluation
</SectionTitle>
    <Paragraph position="0"> Formal evaluation of the performance of the semantic retrieval modules is reported in Langer and Hickey (in preparation). The purpose of the evaluation was to look at the benefits of semantic expansion in terms of retrieval success. These trials have shown, that the semantic expansion enhances recall rate considerably without having a major effect on precision. A further improvement was achieved by including frequency information in the lexicon.</Paragraph>
    <Paragraph position="1"> The evaluation results for the frequency counting lexicon were considerably better than those for the first lexicon derived from WordNet without including frequencies. The frequency counts can be used to determine frequent hyponyms and hyperonyms of index terms, even if they are rather distant in the hyponymic hierarchy, because the method allows the elimination of intermediate level terms that are not interesting for query expansion. Taking into account this information resulted in a higher recall rate, without a major detoriation in precision. For a small, experimental database, 80% of the target messages were correctly retrieved (compared with about 70% with the first lexicon and 50% without query expansion).</Paragraph>
    <Paragraph position="2"> Currently we have started the second evaluation phase. A non-speaking person who has already had experience with message based systems for several years, and who uses a relatively large pool of pre-stored messages, is evaluating WordKeys in real communication settings. The main purpose of this study is to investigate the following points: * How high is the recall rate in real communication settings? * How useful is the semantic expansion module, i.e. how often does it play a crucial role for successful retrieval? Apart from these more technical points, the interface and ease of use will also be a evaluated.</Paragraph>
  </Section>
class="xml-element"></Paper>