<?xml version="1.0" standalone="yes"?>
<Paper uid="P00-1043">
  <Title>Extracting Causal Knowledge from a Medical Database Using Graphical Patterns</Title>
  <Section position="6" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
5 Evaluation
</SectionTitle>
    <Paragraph position="0"> A total of 68 patterns were constructed for the 35 causality identifiers that occurred at least twice in the training abstracts. The patterns were applied to two sets of new abstracts downloaded from Medline: 100 new abstracts from the orig i nal four medical areas (25 abstracts from each area), and 30 abstracts from two new domains (15 each) - digestive system diseases and resp i ratory tract diseases. Each test abstract was analyzed by at least 2 of the authors to identify &amp;quot;medically relevant&amp;quot; cause and effect. A fair number of causal relations in the abstracts are trivial and not medically relevant, and it was felt that it would not be useful for the information extraction system to extract these trivial causal relations.</Paragraph>
    <Paragraph position="1"> Of the causal relations manually identified in the abstracts, about 7% are implicit (i.e. have to be inferred using knowledge-based inferen c ing) or occur across sentences. Since the focus of the study is on explicitly expressed cause and effect within a sentence, only these are included in the evaluation. The evaluation results are pr e sented in Table 4. Recall is the percentage of the slots filled by the human analysts that are co r rectly filled by the computer program. Precision is the percentage of slots filled by the computer program that are correct (i.e. the text entered in the slot is the same as that entered by the human analysts). If the text entered by the computer program is partially correct, it is scored as 0.5 (i.e. half correct). The F-measure given in Table 4 is a combination of recall and precision equally weighted, and is calculated using the formula (MUC-7):  For the 4 medical areas used for building the extraction patterns, the F-measure for the cause and effect slots are 0.508 and 0.578 respectively. If implicit causal relations are included in the evaluation, the recall measures for cause and effect are 0.405 and 0.481 respectively, yielding an F-measure of 0.47 for cause and 0.54 for e f fect. The results are not very good, but not very bad either for an information extraction task.</Paragraph>
    <Paragraph position="2"> For the 2 new medical areas, we can see in Table 4 that the precision is about the same as for the original 4 medical areas, indicating that the current extraction patterns work equally well in the new areas. The lower recall indicates that new causality identifiers and extraction patterns need to be constructed.</Paragraph>
    <Paragraph position="3"> The sources of e rrors were analyzed for the set of 100 test abstracts and are summarized in Table 5. Most of the spurious extractions (i n formation extracted by the program as cause or effect but not identified by human analysts) were actually causal relations that were not medically relevant. As mentioned earlier, the manual ide n tification of causal relations focused on med i cally relevant causal relations. In the cases where the program did not correctly extract cause and effect information identified by the analysts, half were due to incorrect parser ou t put, and in 20% of the cases, causality patterns have not been constructed for the causality ide n tifier found in the sentence.</Paragraph>
    <Paragraph position="4"> We also analyzed the instances of implicit causal relations in sentences, and found that many of them can be identified using some amount of semantic analysis. Some of them i n volve words like when , after and with that ind i cate a time sequence, for example: * The results indicate that changes to 8-OH-DPAT and clonidine-induced responses o c cur quicker with the combination treatment than with either reboxetine or sertraline treatments alone.</Paragraph>
    <Paragraph position="5"> * There are also no reports of serious adverse events when lithium is added to a monoa m ine oxidase inhibitor.</Paragraph>
    <Paragraph position="6"> * Four days after flupenthixol administration, the patient developed orolingual dyskinetic movements involving mainly tongue biting and protrusion.</Paragraph>
    <Paragraph position="7"> Table 5. Sources of Extraction Errors A. Spurious errors (the program identified cause or effect not identified by the h u man judges) A1. The relations extrac ted are not relevant to med i cine or disease. (84.1%) A2. Nominalized or adjectivized verbs are identified as causative verbs by the program because of parser error. (2.9%) A3. Some words and sentence constructions that are used to indicate cause-effect can be used to ind i cate other kinds of relations as well. (13.0%) B. Missing slots (cause or effect not e x tracted by program), incorrect text e x tracted, and partially correct extraction B1. Complex sentence structures that are not i n cluded in the pattern. (18.8%) B2. The parser gave the wrong syntactic structure of a sentence. (49.2%) B3. Unexpected sentence structure resulting in the program extracting information that is actually not a cause or effect. (1.5%) B4. Patterns for the ca usality identifier have not been constructed. (19.6%) B5. Sub-tree error. The program extracts the relevant sub-tree (of the parse tree) to fill in the cause or effect slot. However, because of the sentence construction, the sub-tree includes both the cause and effect resulting in too much text being e x tracted. (9.5%) B6. Errors caused by pronouns that refer to a phrase or clause within the same sentence. (1.3%) In these cases, a treatment or drug is associated with a treatment response or physiological event. If noun phrases and clauses in sentences can be classified accurately into treatments and trea t ment responses (perhaps by using Medline's Medical Subject Headings), then such implicit causal relations can be identified automatically. Another group o f words involved in implicit causal relations are words like receive , get and take , that indicate that the patient received a drug or treatment, for example: * The nine subjects who received p24-VLP and zidovudine had an augmentation and/or broadening of their CTL response compared with baseline (p = 0.004).</Paragraph>
    <Paragraph position="8"> Such causal relations can also be identified by semantic analysis and classifying noun phrases and clauses into treatments and treatment r e sponses. null</Paragraph>
  </Section>
class="xml-element"></Paper>