<?xml version="1.0" standalone="yes"?>
<Paper uid="P06-1128">
  <Title>Semantic Retrieval for the Accurate Identification of Relational Concepts in Massive Textbases Yusuke Miyao[?] Tomoko Ohta[?] Katsuya Masuda[?] Yoshimasa Tsuruoka+</Title>
  <Section position="6" start_page="1020" end_page="1023" type="evalu">
    <SectionTitle>
4 Evaluation
</SectionTitle>
    <Paragraph position="0"> Our system is evaluated with respect to speed and accuracy. Speed is indispensable for real-time interactive text retrieval systems, and accuracy is key for the motivation of semantic retrieval. That is, our motivation for employing semantic retrieval  for Query 4-3 (upper: keyword search, lower: semantic search) was to provide a device for the accurate identification of relational concepts. In particular, high precision is desired in text retrieval from huge texts because users want to extract relevant information, rather than collect exhaustive information.</Paragraph>
    <Paragraph position="1"> We have two parameters to vary: whether to use predicate argument structures and whether to use ontological identifiers. The effect of using predicate argument structures is evaluated by comparing &amp;quot;keyword search&amp;quot; with &amp;quot;semantic search.&amp;quot; The former is a traditional style of IR, in which sentences are retrieved by matching words in a query with words in sentences. The latter is a new feature of the present system, in which sentences are retrieved by matching predicate argument relations in a query with those in a semantically annotated textbase. The effect of using ontological identifiers is assessed by changing queries of the extended region algebra. When we use the term ontology, nominal keywords in queries are replaced with ontological identifiers in GENA and the UMLS meta-thesaurus. When we use the event expression ontology, verbal keywords in queries are replaced with event types.</Paragraph>
    <Paragraph position="2"> Table 7 is a list of queries used in the following experiments. Words in italics indicate a class of words: &amp;quot;something&amp;quot; indicates that any word can appear, and disease indicates that any disease expression can appear. These queries were selected by a biologist, and express typical relational concepts that a biologist may wish to find. Queries 1, 3, and 4 represent relations of genes/proteins, where ERK2, adiponectin, TNF, and IL6 are genes/proteins. Queries 2 and 5 describe relations concerning diseases, and Query 6 is a query that is not relevant to genes or diseases. Query 7 expresses a complex relation concerning a specific phenomena, i.e., phosphorylation, of MAP. Query 8 describes a relation concerning a gene, i.e., p53, while &amp;quot;(negative)&amp;quot; indicates that the target of retrieval is negative mentions. This is expressed by &amp;quot;not&amp;quot; modifying a predicate.</Paragraph>
    <Paragraph position="3"> For example, Query 4 attempts to retrieve sentences that mention the protein-protein interaction &amp;quot;TNF activates IL6.&amp;quot; This is converted into queries of the extended region algebra given in Figure 7.</Paragraph>
    <Paragraph position="4"> The upper query is for keyword search and only specifies the appearances of the three words. Note that the keywords are translated into the ontological identifiers, &amp;quot;activation,&amp;quot; &amp;quot;GHS019685,&amp;quot; and &amp;quot;GHS009426.&amp;quot; The lower query is for semantic search. The variables in &amp;quot;arg1&amp;quot; and &amp;quot;arg2&amp;quot; indicate that &amp;quot;GHS019685&amp;quot; and &amp;quot;GHS009426&amp;quot; are the subject and object, respectively, of &amp;quot;activation&amp;quot;. Table 8 summarizes the results of the experiments. The postfixes of query numbers denote whether ontological identifiers are used. X-1 used no ontologies, and X-2 used only the term ontology. X-3 used both the term and event expression ontologies4. Comparison of X-1 and X-2 clarifies the effect of using the term ontology. Comparison of X-2 and X-3 shows the effect of the event expression ontology. The results for X-3 indicate the maximum performance of the current system.</Paragraph>
    <Paragraph position="5"> This table shows that the time required for the semantic search for the first answer, shown as &amp;quot;time (first)&amp;quot; in seconds, was reasonably short. Thus, the present framework is acceptable for real-time text retrieval. The numbers of answers increased when we used the ontologies, and this result indicates the efficacy of both ontologies for obtaining relational concepts written in various expressions.</Paragraph>
    <Paragraph position="6"> Accuracy was measured by judgment by a biologist. At most 100 sentences were retrieved for each query, and the results of keyword search and semantic search were merged and shuffled. A biologist judged the shuffled sentences (1,839 sentences in total) without knowing whether the sen- null tence was retrieved by keyword search or semantic search. Without considering which words actually matched the query, a sentence is judged to be correct when any part of the sentence expresses all of the relations described by the query. The modality of sentences was not distinguished, except in the case of Query 8. These evaluation criteria may be disadvantageous for the semantic search because its ability to exactly recognize the participants of relational concepts is not evaluated. Table 8 shows the precision attained by keyword/semantic search and n-precision, which denotes the precision of the keyword search, in which the same number, n, of outputs is taken as the semantic search. The table also gives the relative recall of the semantic search, which represents the ratio of sentences that are correctly output by the semantic search among those correctly output by the keyword search. This does not necessarily represent the true recall because sentences not output by keyword search are excluded. However, this is sufficient for the comparison of keyword search and semantic search.</Paragraph>
    <Paragraph position="7"> The results show that the semantic search exhibited impressive improvements in precision. The precision was over 80% for most queries and was nearly 100% for Queries 4 and 5. This indicates that predicate argument structures are effective for representing relational concepts precisely, especially for relations in which two entities are involved. Relative recall was approximately 3050%, except for Query 2. In the following, we will investigate the reasons for the residual errors.</Paragraph>
    <Paragraph position="8"> Table 9 shows the classifications of the errors of Disregarding of noun phrase structures 45  lower: 151 false negatives) semantic retrieval. The major reason for false positives was that our queries ignore internal structures of noun phrases. The system therefore retrieved noun phrases that do not directly mention target entities. For example, &amp;quot;the increased mortality in patients with diabetes was caused by . . . &amp;quot; does not indicate the trigger of diabetes. Another reason was term recognition errors. For example, the system falsely retrieved sentences containing &amp;quot;p40,&amp;quot; which is sometimes, but not necessarily used as a synonym for &amp;quot;ERK2.&amp;quot; Machine learning-based term disambiguation will alleviate these errors. False negatives were caused mainly by nominal expressions such as &amp;quot;the inhibition of ERK2.&amp;quot; This is because the present system does not convert user input into queries on nominal expressions. Another major reason, phrasal verb expressions such as &amp;quot;lead to,&amp;quot; is also a shortage of our current strategy of query creation. Because semantic annotations already in- null clude linguistic structures of these expressions, the present system can be improved further by creating queries on such expressions.</Paragraph>
  </Section>
class="xml-element"></Paper>