<?xml version="1.0" standalone="yes"?>
<Paper uid="H01-1045">
  <Title>Large scale testing of a descriptive phrase finder</Title>
  <Section position="4" start_page="0" end_page="0" type="metho">
    <SectionTitle>
2. THE SYSTEM
</SectionTitle>
    <Paragraph position="0"> The Web-based DPF was composed of two parts: a front-end to an existing Web search engine, which fetched documents; and the system that located sentences holding descriptive phrases.</Paragraph>
    <Paragraph position="1"> The Web front end simply routed queries to a Web search engine (Google), and the text of the top 600 documents returned by the engine was fetched, split into sentences (using a locally developed sentence splitter), and those sentences holding the query term were passed onto the DPF.</Paragraph>
    <Paragraph position="2"> It ranked sentences on a score calculated from multiple sources of evidence. A detailed description of the DPF is found in [3]. The primary clue to there being a descriptive phrase in a sentence was the presence of a key phrase within it. An example key phrase was &amp;quot;such as&amp;quot;, which may be found in the sentence: &amp;quot;He used several search engines such as AltaVista, HotBot and WebTop to compare the performance&amp;quot;. If such a sentence were returned to a user who entered the query &amp;quot;WebTop&amp;quot;, they would determine it was a search engine. Specifically, the DPF is searching for the key phrase in proximity to a query noun (qn) to locate a descriptive phrase (dp) e.g.</Paragraph>
    <Paragraph position="3">  * ... dp such as qn ...</Paragraph>
    <Paragraph position="4"> other key phrases used, some suggested by [2], were * ... such dp as qn ...</Paragraph>
    <Paragraph position="5"> * ... qn (and  |or) other dp ...</Paragraph>
    <Paragraph position="7"> The phrases form the key part of the DPF as they identify well sentences likely to contain descriptions of qn. While the number of times a particular qn appears in a sentence with a key phrase are small, by searching a large corpus, like the Web, the chances of finding a few (accurately identified) descriptions of qn in the form required are high.</Paragraph>
    <Paragraph position="8"> Based on results from a testing phase, certain key phrases were found more accurate at locating a descriptive phrase than others.</Paragraph>
    <Paragraph position="9"> Consequently, when ranking matching sentences, different scores were assigned depending on the accuracy of the key phrase found within. Since unfamiliar words tend to be explained or rephrased at the early part of a document, sentence position was also a factor in the rank score, with earlier sentences given preference.</Paragraph>
    <Paragraph position="10"> Finally, cross-document information was taken into account.</Paragraph>
    <Paragraph position="11"> Across all the matching sentences for a particular query, the occurrence of all the terms within the sentences was noted. It was anticipated that terms occurring more frequently within the set of sentences were likely to belong to descriptions.</Paragraph>
    <Paragraph position="12"> Consequently, sentences holding a high number of commonly occurring words were given further preference in the ranking. The last two pieces of information not only improved the accuracy of ranking, but also enabled the system to produce reasonable results when no key phrases were matched. A training phase where the optimum balance between the sources of information was run on existing training data created from the LA Time corpus described in [3].</Paragraph>
    <Paragraph position="13"> It may be reasonable to question why such a simple approach to extracting information from free-text sources be taken when more principled NLP-based techniques are well-established (e.g. [4], [5]). There are a number of reasons: * Any simple approach is likely to be much faster than one that requires operations such as parsing.</Paragraph>
    <Paragraph position="14"> * We believe that the use of simple but accurate methods searching over very large corpora provides a new means of determining lexical relations from corpora that are worthy of further exploration.</Paragraph>
  </Section>
  <Section position="5" start_page="0" end_page="0" type="metho">
    <SectionTitle>
3. INITIAL STUDY
</SectionTitle>
    <Paragraph position="0"> A pilot study was conducted, searching ten queries using the top hundred documents returned by Google. Of the ten queries, six had the best description located in the top two ranked sentences, two more queries had a good description in the top two. For all queries, a sentence holding a descriptive phrase was returned in the top five ranked sentences.</Paragraph>
  </Section>
  <Section position="6" start_page="0" end_page="0" type="metho">
    <SectionTitle>
4. DEFINING RELEVANCE
</SectionTitle>
    <Paragraph position="0"> In this and the previous evaluation described in [3], relevance was defined as a sentence that told the user anything about the query term: a liberal view of relevance (described here as binary relevance). The results from the pilot, under this interpretation, showed the system performed well. Consequently a more stringent form of relevance was devised. A sample answer for each query was solicited from users: for example, &amp;quot;the Prime Minister of Great Britain&amp;quot; for Tony Blair. Those key answers were taken as an acceptable criterion of highly relevant descriptive phrases. Sentences ranked by the system were then compared to the key answer. Correctness of DPs is not enough for this aim. Only a DP that described a query as well as a key answer was regarded as relevant. To illustrate, the sentence &amp;quot;Tony Blair is the current Prime Minister of the United Kingdom.&amp;quot; was regarded as relevant, but &amp;quot;Tony Blair is a political leader&amp;quot; was not.</Paragraph>
  </Section>
  <Section position="7" start_page="0" end_page="0" type="metho">
    <SectionTitle>
5. THE MAIN EXPERIMENT
</SectionTitle>
    <Paragraph position="0"> A total of 146 queries were tested in the main experiment: 50 of which were evaluated based on key answers; 96 using binary evaluation. In the binary test, the DPF returned a relevant (descriptive) sentence in the top twenty sentences for all 96 queries. On average sixteen of the sentences returned were relevant to each query. The minimum number of relevant was six and maximum was twenty. Across the 96 queries, at least one relevant sentence was found in the top five for every tested query.</Paragraph>
    <Paragraph position="1"> This is a significant improvement over the previously reported experimental results where 90% of queries were answered in the top five.</Paragraph>
    <Paragraph position="2"> Using more stringent key answer based relevance, the system succeeded in retrieving at least one relevant sentence in the top five for 66% of the queries, at least one in the top ten for 82%, and one in the top twenty for 88%.</Paragraph>
    <Paragraph position="3"> These results show that the DPF searching the Web (1 billion documents) works dramatically better than the previous experiment using LA Times (100,000 documents). As was shown in previous work, the size of the collection impacts on the effectiveness of the system. This is because by searching a larger collection, there is a better chance of locating a relevant descriptive phrase in the format of one of the searched for key phrases. However in the previous work, there appeared to be an upper bound on the accuracy of the descriptive phrases alone. By searching a much larger collection it is speculated that the cross document term occurrence statistics used contributed significantly to improving the effectiveness of the system.</Paragraph>
  </Section>
  <Section position="8" start_page="0" end_page="0" type="metho">
    <SectionTitle>
6. CONCLUSION
</SectionTitle>
    <Paragraph position="0"> An existing descriptive phrase system was adapted to work with a Web search engine to locate phrases describing query words. The system was found to be highly effective at locating good descriptions: finding at least one high quality descriptive phrase in the top 10 returned sentences for 82% of test queries.</Paragraph>
  </Section>
class="xml-element"></Paper>