<?xml version="1.0" standalone="yes"?>
<Paper uid="P05-2009">
  <Title>Learning Meronyms from Biomedical Text</Title>
  <Section position="5" start_page="49" end_page="50" type="metho">
    <SectionTitle>
3 Algorithm
</SectionTitle>
    <Paragraph position="0"> Input to the algorithm consists of existing lexical and relational resources, such as terminologies and ontologies. These are used to label text with training relations. The context of these relations are found automatically, and patterns created to describe these contexts. These patterns are generalised and used to discover new relations, which are fed back iteratively into the algorithm. The algorithm is given in</Paragraph>
    <Section position="1" start_page="49" end_page="50" type="sub_section">
      <SectionTitle>
3.1 Discovering New Terms
</SectionTitle>
      <Paragraph position="0"> Step 2e in Figure 1 labels new terms, which may be discovered as a by-product of identifying new rela- null patterns represented by tokens and parts of speech. tion instances. This is possible because there is a distinction between the lexical item used to find the pattern context (Step 2a), and the pattern element against which new relations are matched (Step 2d). For example, a pattern could be found from the context (term relation term), and expressed as (noun relation adjectivenoun). When applied to the text to learn new relation instances, sequences of tokens taking part in this relation will be found, and may be inferred to be terms for the next iteration.</Paragraph>
    </Section>
    <Section position="2" start_page="50" end_page="50" type="sub_section">
      <SectionTitle>
3.2 Implementation: PartEx
</SectionTitle>
      <Paragraph position="0"> Implementation was independent of any specific relation, but configured, as the PartEx system, to discover partOf. Relations were usually learned between terms, although this was varied in some experiments. The algorithm was implemented using the GATE NLP framework (Cunningham et al., 2002) and texts preprocessed using the tokeniser, sentence splitter, and part-of-speech (POS) tagger provided with GATE. In training, terms were labelled using MMTx, which uses lexical variant generation to map noun phrases to candidate terms and concepts attested in a terminology database. Final candidate selection is based on linguistic matching metrics, and concept resolution on filtering ambiguity from the MMTx source terminologies (Aronson, 2001).</Paragraph>
      <Paragraph position="1"> Training relations were labelled from an existing meronymy. Simple contexts of up to five tokens between the participants in the relation were identified using JAPE, a regular expression language integrated into GATE. For some experiments, relations were considered between noun phrases, labelled using LT CHUNK (Mikheev and Finch, 1997). GATE wrappers for MMTx, LT CHUNK, and other PartEx modules are freely available 4.</Paragraph>
      <Paragraph position="2"> Patterns describing contexts were expressed as shallow lexico-syntactic patterns in JAPE, and a JAPE transducer used to find new relations. A typical pattern consisted of a sequence of parts of speech and words. Pattern generalisation was minimal, removing only those patterns that were either identical to another pattern, or that had more specific lexico-syntactic elements of another pattern. To simplify pattern creation for the experiments reported here, patterns only used context between the relation participants, and did not use regular expression quantifiers. New terms found during relation discovery were labelled using a finite state machine created with the Termino compiler (Harkema et al., 2004).</Paragraph>
    </Section>
  </Section>
  <Section position="6" start_page="50" end_page="51" type="metho">
    <SectionTitle>
4 Materials and Method
</SectionTitle>
    <Paragraph position="0"> Lexical and relational resources were provided by the Unified Medical Language System (UMLS), a collection of medical terminologies 5. Term lookup in the training phase was carried out using MMTx.</Paragraph>
    <Paragraph position="1"> Experiments made particular use of The University of Washington Digital Anatomist Foundational Model (UWDA), a knowledge base of anatomy included in UMLS. Relation labelling in the training phase used a meronymy derived by computing the transitive closure of that provided with the UWDA.</Paragraph>
    <Paragraph position="2"> The UWDA gives definitions for some terms, as headless phrases that do not include the term being defined. A corpus was constructed from these, for learning and evaluation. This corpus used the first 300 UWDA terms with a definition, that had a UMLS semantic type of &amp;quot;Body Part&amp;quot;. These terms included synonyms and orthographic variants given the same definition. Complete definitions were constructed by prepending terms to definitions with the copula &amp;quot;is&amp;quot;. An example is shown in Figure 2.  Experiments were carried out using cross validation over ten random unseen folds, with 71 unique meronyms across all ten folds. Definitions were pre-processed by tokenising, sentence splitting, POS tagging and term labelling. Evaluation was carried out by comparison of relations learned in the held back fold, to those in an artificially generated gold standard (described below). Evaluation was type based, rather than instance based: unique relation instances in the gold standard were compared with unique relation instances found by PartEx, i.e. identical relation instances found within the same fold were treated as a single type. Evaluation therefore measures domain knowledge discovery.</Paragraph>
    <Paragraph position="3"> Gold standard relations were generated using the same context window as for Step 2a of the algorithm. Pairs of terms from each context were checked automatically for a relation in UWDA, and this added to the gold standard. This evaluation strategy is not ideal. First, the presence of a part and a whole in a context does not mean that they are being meronymically related (for example, &amp;quot;found in the hand and finger&amp;quot;). The number of spurious meronyms in the gold standard has not yet been ascertained. Second, a true relation in the text may not appear in a limited resource such as the UWDA (although this can be overcome through a failure analysis, as described in Section 4.1). Although a better gold standard would be based on expert mark up of the text, the one used serves to give quick feedback with minimal cost. Standard evaluation metrics were used. The accuracy of initial term and relation labelling were not evaluated, as these are identical in both gold standard creation and in experiments.</Paragraph>
    <Section position="1" start_page="51" end_page="51" type="sub_section">
      <SectionTitle>
4.1 Failure Analysis
</SectionTitle>
      <Paragraph position="0"> For some experiments, a failure analysis was carried out on missing and spurious relations. The reasons for failure were hypothesised by examining the sentence in which the relation occurred, the pattern that led to its discovery, and the source of the pattern.</Paragraph>
      <Paragraph position="1"> Some spurious relations appeared to be correct, even though they were not in the gold standard.</Paragraph>
      <Paragraph position="2"> This is because the gold standard is based on a resource which itself has limits. One of the aims of the work is to supplement such resources: the algorithm should find correct relations that are not in the resource. Proper evaluation of these relations requires care, and methodologies are currently being investigated. A quick measure of their contribution was, however, found by applying a simple methodology, based on the source texts being definitional, authoritative, and describing relations in unambiguous language. The methodology adjusts the number of spurious relations, and calculates a corrected precision. By leaving the number of actual relations unchanged, corrected precision still reflects the proportion of discovered relations that were correct relative to the gold standard, but also reflects the number of correct relations not in the gold standard. The methodology followed the steps in Figure 3.</Paragraph>
      <Paragraph position="3">  1. Examine the context of the relation.</Paragraph>
      <Paragraph position="4"> 2. If the text gives a clear statement of meronomy, the relation is not spurious.</Paragraph>
      <Paragraph position="5"> 3. If the text is clearly not a statement of meronomy, the relation is spurious.</Paragraph>
      <Paragraph position="6"> 4. If the text is ambiguous, refer to a second authoritative resource6. If this gives a clear statement of meronomy, the relation is not spurious.</Paragraph>
      <Paragraph position="7"> 5. If none of these apply, the relation is spurious.</Paragraph>
      <Paragraph position="8"> 6. Calculate corrected precision from the new number of spurious relations.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>