<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-1813">
  <Title>Determining the Specificity of Terms based on Information Theoretic Measures</Title>
  <Section position="4" start_page="12" end_page="12" type="evalu">
    <SectionTitle>
3 Experiments and Evaluation
</SectionTitle>
    <Paragraph position="0"> In this section, we describe our experiments and evaluate proposed methods.</Paragraph>
    <Paragraph position="1"> We select a subtree of MeSH thesaurus for the experiment. &amp;quot;metabolic diseases(C18.452)&amp;quot; node is root of the subtree, and the subtree consists of 436 disease names which are target terms for specificity measuring. We used MEDLINE  database corpus (170,000 abstracts, 20,000,000 words) to extract statistical information.</Paragraph>
    <Paragraph position="2"> Each method was evaluated by two criteria, coverage and precision. Coverage is the fraction of the terms which have the specificity by given method. Method 2 gets relatively lower coverage than method 1, because method 2 can measure the specificity only when both the terms and their modifiers occur in corpus. Method 1 can measure the specificity whenever parts of composite words appear in corpus. Precision is the fraction of correct specificity relations values as equation (13).</Paragraph>
    <Paragraph position="4"> of R p c with correct specificity p of all R p c</Paragraph>
    <Paragraph position="6"> where R(p,c) is a parent-child relation in MeSH thesaurus. If child term c has larger specificity than that of parent term p, then the relation is said to have correct specificity. We divided parent-child relations into two types. Relations where parent term is nested in child term are categorized as type I. Other relations are categorized as type II. There are 43 relations in type I and 393 relations in type II. The relations in type I always have correct specificity provided modifier-head information method described in section 2.1 is applied.</Paragraph>
    <Paragraph position="7"> We tested prior experiment for 10 human subjects to find out the upper bound of precision.</Paragraph>
    <Paragraph position="8"> The subjects are all medical doctors of internal medicine, which is closely related division to &amp;quot;metabolic diseases&amp;quot;. They were asked to identify parent-child relationship for given term pairs. The average precisions of type I and type II were 96.6% and 86.4% respectively. We set these values as upper bound of precision for suggested methods.</Paragraph>
    <Paragraph position="9">  MEDLINE is a database of biomedical articles serviced by National Library of Medicine, USA.</Paragraph>
    <Paragraph position="10"> (http://www.nlm.nih.gov) CompuTerm 2004 Poster Session - 3rd International Workshop on Computational Terminology 89 The specificity of terms was measured with method 1, method 2, and method 3 as Table 2.</Paragraph>
    <Paragraph position="11"> Two additional methods, based on term frequency and term tf.idf, were experimented to compare compositionality based methods and term based methods.</Paragraph>
    <Paragraph position="12"> Method 1 showed better performance than term based methods. This result illustrate basic assumption of this paper that specific concepts are created by adding information to existing concepts, and new concepts are expressed as new terms by adding modifiers to existing terms. Word tf.idf based method showed better precision than word frequency based method. This result illustrate that tf.idf of words is more informative than frequency of words.</Paragraph>
    <Paragraph position="13"> Method 3 showed the best precision, 82.0%, because the two methods interacted complementary. In hybrid method, the weight value 0.8g = indicates that compositional information is more informative than contextual information for the specificity of domain specific terms.</Paragraph>
    <Paragraph position="14"> One reason of the errors is that the names of some internal nodes in MeSH thesaurus are category names rather disease names. For example, as &amp;quot;acid-base imbalance (C18.452.076)&amp;quot; is name of disease category, it doesn't occur as frequently as other real disease names. Other predictable reason is that we didn't consider various surface forms of same term. For example, although &amp;quot;NIDDM&amp;quot; is acronym of &amp;quot;non insulin dependent diabetes mellitus&amp;quot;, the system counted two terms separately. Therefore the extracted statistics can't properly reflect semantic level information.</Paragraph>
  </Section>
class="xml-element"></Paper>