<?xml version="1.0" standalone="yes"?>
<Paper uid="A00-1036">
  <Title>Linguistic Knowledge can Improve Information Retrieval</Title>
  <Section position="4" start_page="262" end_page="263" type="metho">
    <SectionTitle>
3 Relaxation Ranking and Specific
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="262" end_page="263" type="sub_section">
      <SectionTitle>
Passage Retrieval
</SectionTitle>
      <Paragraph position="0"> The system we are evaluating uses a technique called &amp;quot;relaxation ranking&amp;quot; to find specific passages where as many as possible of the different elements of a query occur near each other, preferably in the same form and word order and preferably closer together. Such passages are ranked by a penalty score that measures the degree of deviation from an exact match of the requested phrase, with smaller penalties being preferred. Differences in morphological form and formal subsumption of index terms by query terms introduce small penalties, while intervening words, unexplained permutations of word order, and crossing sentence boundaries introduce more significant penalties. Elements of a query that cannot be found nearby introduce substantial penalties that depend on the syntactic categories of the missing words.</Paragraph>
      <Paragraph position="1"> When the conceptual indexing system is presented with a query, the relaxation-ranking retrieval algorithm searches through the conceptual taxonomy for appropriately related concepts and uses the positions of those concepts in the indexed material to find specific passages that are likely to address the information needs of the request. This search can find relationships from base forms of words to derived forms and from more general terms to more specific terms, by following paths in the conceptual taxonomy.</Paragraph>
      <Paragraph position="2"> For example, the following is a passage retrieved by this system, when applied to the UNIX (r) operating system online documentation (the &amp;quot;man pages&amp;quot;): Query: print a message from the mail tool 6. -2.84 print mail mail mailtool Print sends copies of all the selected mail items to your default printer. If there are no selected items, mailtool sends copies of those items you axe currently...</Paragraph>
      <Paragraph position="3"> The indicated passage is ranked 6th in a returned list of found passages, indicated by the 6 in the above display. The number -2.84 is the penalty score assigned to the passage, and the subsequent words print, mail, mail, and mailtool indicate the words in the text that are matched to the corresponding content words in the input query. In this case, print is matched to print, message to mail, mail to mail, and tool to mailtool, respectively. This is followed by the content of the actual passage located. The information provided in these hit displays gives the information seeker a clear idea of why the passage was retrieved and enables the searcher to quickly skip down the hit list with little time spent looking at irrelevant passages. In this case, it was easy to  identify that the 6th ranked hit was the best one and contained the relevant information.</Paragraph>
      <Paragraph position="4"> The retrieval of this passage involved use of a semantic subsumption relationship to match message to mail, because the lexical entry for mail recorded that it was a kind of message. It used a morphological root subsumption to match tool to mailtool because the morphological analyzer analyzed the unknown word mailtool as a compound of mail and tool and recorded that its root was tool and that it was a kind of tool modified by mail. Taking away the ability to morphologically analyze unknown words would have blocked the retrieval of this passage, as would eliminating the lexical subsumption entry that recorded mail as a kind of message.</Paragraph>
      <Paragraph position="5"> Like other approaches to passage retrieval (Kaszkiel and Zobel, 1997; Salton et al., 1993; Callan, 1994), the relaxation-ranking retrieval algorithm identifies relevant passages rather than simply identifying whole documents. However, unlike approaches that involve segmenting the material into paragraphs or other small passages before indexing, this algorithm dynamically constructs relevant passages in response to requests. When responding to a request, it uses information in the index about positions of concepts in the text to identify relevant passages. In response to a single request, identified passages may range in size from a single word or phrase to several sentences or paragraphs, depending on how much context is required to capture the various elements of the request.</Paragraph>
      <Paragraph position="6"> In a user interface to the specific passage retrieval system, retrieved passages are reported to the user in increasing order of penalty, together with the rank number, penalty score, information about which target terms match the corresponding query terms, and the content of the identified passage with some surrounding context as illustrated above. In one version of this technology, results are presented in a hyper-text interface that allows the user to click on any of the presented items to see that passage in its entire context in the source document. In addition, the user can be presented with a display of portions of the conceptual taxonomy related to the terms in the request. This frequently reveals useful generalizations of the request that would find additional relevant information, and it also conveys an understanding of what concepts have been found in the material that will be matched by the query terms.</Paragraph>
      <Paragraph position="7"> For example, in one experiment, searching the on-line documentation for the Emacs text editor, the request jump to end of file resulted in feedback showing that jump was classified as a kind of move in the conceptual taxonomy. This led to a reformulated request, move to end of file, which successfully retrieved the passage 9o to end of buffer.</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="263" end_page="264" type="metho">
    <SectionTitle>
4 Experimental Evaluation
</SectionTitle>
    <Paragraph position="0"> In order to evaluate the effectiveness of the above techniques, a set of 90 queries was collected from a naive user of the UNIX operating system, 84 of which could be answered from the online documentation known as the man pages. A set of &amp;quot;correct&amp;quot; answers for each of these 84 queries was manually determined by an independent UNIX operating system expert, and a snapshot of the man pages collection was captured and indexed for retrieval. In order to compare this methodology with classical document retrieval techniques, we assign a ranking score to each document equal to the ra~king score of the best ranked passage that it contains.</Paragraph>
    <Paragraph position="1"> In rating the performance of a given method, we compute average recall and precision values at 10 retrieved documents, and we also compute a &amp;quot;success rate&amp;quot; which is simply the percentage of queries for which an acceptable answer occurs in the top ten hits. The success rate is the principal factor on which we base our evaluations, since for this application, the user is not interested in subsequent answers once an acceptable answer has been found, and finding one answer for each of two requests is a substantially better result than finding two answers to one request and none for another.</Paragraph>
    <Paragraph position="2"> These experiments were conducted using an experimental retrieval system that combined a Lisp-based language processing stage with a C++ implementation of a conceptual indexer. The linguistic knowledge sources used in these experiments included a core lexicon of approximately 18,000 words, a substantial set of morphological rules, and specialized morphological algorithms covering inflections, prefixes, suffixes, lexical compounding, and a variety of special forms, including numbers, ordinals, Roman numerals, dates, phone numbers, and acronyms. In addition, they made use of a lexical subsumption taxonomy of approximately 3000 lexical subsumption relations, and a small set of semantic entailment axioms (e.g., display entails see, but is not a kind of see). This system is described in (Woods, 1997). The database was a snapshot of the local man pages (frozen at the time of the experiment so that it wouldn't change during the experiment), consisting of approximately 1800 files of varying lengths and constituting a total of approximately 10 megabytes of text.</Paragraph>
    <Paragraph position="3"> Table 1 shows the results of comparing three versions of this technology with a textbook implementation of the standard tfid\] algorithm (Salton, 1989) and with the SearchItWMsearch application developed at Sun Microsystems, Inc., which combines a  simple morphological query expansion with a state-of-the-art commercial search engine. In the table, Recall II refers to the full conceptual indexing and search system with all of its knowledge sources and rules. The line labeled &amp;quot;w/o morph&amp;quot; refers to this system with its dynamic morphological rules turned off, and the line labeled &amp;quot;w/o knowledge&amp;quot; refers to this system with all of its knowledge sources and rules turned off. The table presents the success rate and the measured recall and precision values for 10 retrieved documents. We measured recall and precision at the 10 document level because internal studies of searching behavior had shown that users tended to give up if an answer was not found in the first ten ranked hits. We measured success rate, rather than recall and precision, for our ablation studies, because standard recall and precision measures are not sensitive to the distinction between finding multiple answers to a single request versus finding at least one answer for more requests.</Paragraph>
  </Section>
  <Section position="6" start_page="264" end_page="264" type="metho">
    <SectionTitle>
5 Discussion
</SectionTitle>
    <Paragraph position="0"> Table 1 shows that for this task, the relaxation-ranking passage retrieval algorithm without its supplementary knowledge sources (Recall II w/o knowledge) is roughly comparable in performance (42.9% versus 44.0% success rate) to a state-of-the-art commercial search engine (SearchIt) at the pure document retrieval task (neglecting the added benefit of locating the specific passages). Adding the knowledge in the core lexicon (which includes morphological relationships, semantic subsumption axioms, and entailment relationships), but without morphological analysis of unknown words (Recall II w/o morph), significantly improves these results (from 42.9% to 50.0%). Further adding the morphological analysis capability that automatically analyzes unknown words (deriving additional morphological relationships and some semantic subsumption relationships) significantly improves that result (from 50.0% to 60.7%). In contrast, we found that adding the same semantic subsumption relationships to the commercial search engine, using its provided thesaurus capability degraded its results, and results were still degraded when we added only those facts that we knew would help find relevant documents.</Paragraph>
    <Paragraph position="1"> It turned out that the additional relevant documents found were more than offset by additional irrelevant documents that were also ranked more highly.</Paragraph>
  </Section>
  <Section position="7" start_page="264" end_page="265" type="metho">
    <SectionTitle>
6 Anecdotal Evaluation of Specific
Passage Retrieval Benefits
</SectionTitle>
    <Paragraph position="0"> As mentioned above, comparing the relaxation-ranking algorithm with document retrieval systems measures only a part of the benefit of the specific passage retrieval methodology. Fully evaluating the quality and ranking of the retrieved passages involves a great many subtleties. However, two informal evaluations have been conducted that :shed some light on the benefits.</Paragraph>
    <Paragraph position="1"> The first of these was a pilot study of the technology at a telecommunications company. In that study, one user found that she could use a single query to the conceptual indexing system to find both of the items of information necessary to complete a task that formerly required searching two separate databases. The conclusion of that study was that the concept retrieval technology performs well enough to be useful to a person talking live with a customer.</Paragraph>
    <Paragraph position="2"> It was observed that the returned hits can be compared with one another easily and quickly by eye, and attention is taken directly to the relevant content of a large document: The automatic indexing was considered a plus compared with manual methods of content indexing. It was observed that an area of great potential may be in a form of knowledge management that involves organizing and providing intelligent access to small, unrelated &amp;quot;nuggets&amp;quot; of textual knowledge that are not amenable to conventional database archival or categorization.</Paragraph>
    <Paragraph position="3"> A second experiment was conducted by the Human Resources Webmaster of a high-tech company, an experienced user of search engines who used this technology to index his company's internal HR web site. He then measured the time it took him to process 15 typical HR requests, first using conventional search tools that he had available, and then using the Conceptual Indexing technology. In both cases, he measured the time it took him to either find the answer or to conclude that the answer wasn't in the indexed material. His measured times for the total suite were 55 minutes using the conventional  tools and 11 minutes using the conceptual indexing technology. Of course, this was an uncontrolled experiment, and there is some potential that information learned from searching with the traditional tools (which were apparently used first) might have provided some benefit when using the conceptual indexing technology. However, the fact that he found things with the latter that he did not find with the former and the magnitude of the time difference suggests that there is an effect, albeit perhaps not as great as the measurements. As a result of this experience, he concluded that he would expect many users to take much longer to find materials or give up, when using the traditional tools. He anticipated that after finding some initial materials, more time would be required, as users would end up having to call people for additional information. He estimated that users could spend up to an hour trying to get the information they needed...having to call someone, wait to make contact and finally get the information they needed. Using the conceptual indexing search engine, he expected that these times would be at least halved.</Paragraph>
  </Section>
class="xml-element"></Paper>