<?xml version="1.0" standalone="yes"?>
<Paper uid="C04-1135">
  <Title>Extracting Hyponyms of Prespecified Hypernyms from Itemizations and Headings in Web Documents</Title>
  <Section position="5" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
4 Experimental Results
</SectionTitle>
    <Paragraph position="0"> To evaluate our procedure, we had to provide a set of proper hypernyms for which HEAIH would find hyponyms. This was a rather difficult task. There are many nouns that cannot be hypernyms. We assumed that the Japanese noun sequences or nouns that occupied the position of X in the patterns &amp;quot;Xa&amp;quot; (table of X) &amp;quot;Xwp&amp;quot; (guide to X) &amp;quot;EwX&amp;quot; (successive (or chronological list of) X) and &amp;quot;X&amp;quot; (wellknown X) in corpora were appropriate as hypernyms.</Paragraph>
    <Paragraph position="1"> (Despite this filtering, there were some inappropriate hypernyms in the set of hypernyms subjected to the procedures in our experiments. These inappropriate hypernyms included expressions whose hyponyms change drastically according to the situation in which the expressions are used. Examples are &amp;quot;recommended products.&amp;quot; One cannot determine the possible hyponyms without knowing who is recommending. We judged any hyponymy relations including such hypernyms as being unacceptable. ) We downloaded 1:00PS106 Japanese HTML documents (1.26 GB without tags), applied the above patterns and found 8,752 expressions. Then, we randomly picked out 100 hypernym candidates from 869 expressions that occurred with the above patterns more than three times, and 100 hypernym candidates from the remaining 7,883 expressions. These 200 hypernym candidates became the input for our procedure. As mentioned, we downloaded a maximum of 25 pages for each hypernym, and extracted 3In HEAIH, the hypernym x may not be included in the set of nouns for which we obtained a co-occurrence vector since x is simply given to the procedure from outside, and the procedure may not be able to compute the sim values. In that case, we replace x with the longest suffix of x that is contained in the set of nouns for which co-occurrence vectors were obtained. The head final characteristic of the Japanese language justifies this replacement.</Paragraph>
    <Paragraph position="2"> Z(laboratories, 34)*,H(health food/beverage, 18)*, (welfare facilities, 13)*,;(functionalities, 12),N (parks in cities, 10)*,3(stores/shops, 10)*,(emperors, 7)*, (districts, 6)*,(businesses, 6),(legacies, 6)*,{M (offered products, 5),C'(participant companies, 5),^(works of art, 5)*,(parts of machines, 5),~G(Japan's top three something, 4), (novels, 4)*,(club activities, 3)*, M(fortune telling websites, 3)*, MS(rules of business, 3), (time attack, 3),(commands, 3), (recommended products, 2), \(producers,2),(poems, 2)*, (cities or markets, 2)*,(alpine plants, 2)*,(names of teams, 2)*,(side businesses, 2),S(jobs, 2), E(things, 1),[(Japanese versions, 1),(animals, 1)*, (specialties, 1),p(introductions, 1), H(novelists, 1)*, (questions, 1),(data, 1),OP(working at home, 1),(students' clubs, 1)*,q(venues, 1)(names of railway stations, 1)*,J(dept. of multimedia),  3,211 itemizations from them. (We restricted the itemizations to the ones containing less than or equal to 30 items.) Then, we picked out 2,034 itemizations and used them in our evaluation. The choice was made in the following manner. First, for each hypernym candidate, the itemizations were sorted in ascending order of the distance between the occurrence of the hypernym candidate and the itemization in the downloaded page. Then, the itemizations in the top 65% were chosen for each hypernym.4. This selection was made to eliminate the itemizations located extremely far from the given hypernyms and to keep the number of itemizations close to 2,000, which was the number of itemizations used in Shinzato and Torisawa, 2004.</Paragraph>
    <Paragraph position="3"> Recall that HEAIH (and AHRAI) require two different types of document sets: global document sets and local document sets. As a global document set, we used the downloaded 1:00 PS 106 HTML documents used to obtain hypernyms given to the HEAIH. As a local document set for each hyponym candidate, we downloaded the top 100 documents in the ranking produced by a search engine. In addition, we used 5:72PS106 Japanese HTML documents (6.27 GB without tags) to obtain co-occurrence vectors to calculate the semantic similarities between expressions. To derive co-occurrence vectors, we parsed the documents by using a downgraded version of an existing parser (Kanayama et al., 2000) and collected co-occurrences from the parsing results.</Paragraph>
    <Paragraph position="4"> As mentioned, we obtained 200 pairs of a hypernym and an HCS as the final HEAIH output. All the hypernyms appearing in the output are listed in  the number of HCSs that the procedure produced with the hypernym. In the 200 pairs, 48 hypernyms appeared. The HCSs were taken from 119 distinct websites, and the maximum number of the HCSs taken from a single site was 7. The resulting pairs of hypernym candidates and hyponym candidates were checked by the authors according to the definition of the hypernym given in Miller et al., 1990; i.e., we checked if the expression &amp;quot;a hyponym candidate is a kind of a hypernym candidate.&amp;quot; is acceptable. Figure 5 shows some examples of the hypernym-HCS pairs that were obtained by HEAIH.</Paragraph>
    <Paragraph position="5"> A hyponym candidates in the HCSs is marked by &amp;quot;*&amp;quot; if it is a proper hyponym of the hypernym in the pair. We then computed the precision, which was the ratio of correct hypernym-hyponym pairs against all the pairs obtained from the top 200 pairs of an HCS and its hypernym candidate. The graph in Figure 6 plots the precision obtained by HEAIH, along with the precisions of the alternative methods as we explain later. The x-axis of the graph indicates the number of hypernym-hyponym pairs obtained from the top j pairs of an HCS and its hypernym candidate, while the y-axis indicates the precision.</Paragraph>
    <Paragraph position="6"> More precisely, the curve plots the points denoted by hPjh=1 jCij;(Pjh=1 correct(Ch;x0h))=(Pjh=1 jChj)i, where the output of the HEAIH is denoted by fhx0h;Chig200h=1 and 1 * j * 200. correct(Ch;x0h) indicates the number of hyponym candidates in Ch that are true hyponyms of the hypernym x0h.</Paragraph>
    <Paragraph position="7"> We compared the performances of the following five alternative methods with that of HEAIH.</Paragraph>
    <Paragraph position="8"> Alternative 1 Produce pairs consisting of a given hypernym and a hyponym candidate in an HCS if the given hypernym is a suffix of the hyponym candidate. Note that Japanese is a head final language and that suffixes of hyponym candidates are good candidates to be hypernyms.</Paragraph>
    <Paragraph position="9"> Alternative 2 Extract hyponymy relations by applying lexicosyntactic patterns to the documents in the local document sets for our method. We used  hypernymhyponym,hyponym, .*w.* hypernym, hyponym .*wOs.* hypernym, hyponym .*th.* hypernym, hyponym .*sr(zjw)? hypernym, hyponym .*qzy.* hypernym, hyponym .*q(Mjt)O.* hypernym, hyponym .* (jhj) .* hypernym  patterns proposed in previous work (Imasumi, 2001; Ando et al., 2003) (Figure 7). Note that these are regular expressions and may overgenerate hyponymy relations; however, they do not miss the relations acquired through more sophisticated methods such as those with parsers.</Paragraph>
    <Paragraph position="10"> Alternative 3 Extract hyponymy relations by looking for lexicosyntactic patterns with an existing search engine. The patterns used were basically the same as those used in Alternative 2. However, the expression &amp;quot;.*&amp;quot; was eliminated from the patterns and the disjunctions &amp;quot;j&amp;quot; were expanded to simple strings since the engine would not accept regular expressions. In addition, the pattern &amp;quot;hypernym hyponym&amp;quot; was not used because the brackets &amp;quot; &amp;quot; were not treated properly by the engine.</Paragraph>
    <Paragraph position="11"> Alternative 4 Original AHRAI.</Paragraph>
    <Paragraph position="12"> Alternative 5 Produce hypernym-hyponym pairs according to only the distance between the headings including the hypernym and the itemizations including HCSs. Recall that Hd(x) is the set of strings likely to be headings of itemizations for a given hypernym x. This alternative method computes the distance in bytes between the position of a member of Hd(x) in a downloaded document and the position of the itemization including an HCS. The pairs of an itemization and a given hypernym are then sorted according to this distance to produce the 200 pairs with the smallest distance as pairs of hypernyms and the corresponding HCSs. Note that we assumed a heading must appear before an HCS.</Paragraph>
    <Paragraph position="13">  We checked if the above alternatives can acquire the correct pairs of a hypernym and a hyponym obtained by HEAIH. In other words, we counted how many correct pairs produced by HEAIH were also acquired by the alternatives when using the same document set. Note that all the alternative methods except for Alternative 5 were applied only to the 200 pairs of a hypernym and an HCS that were the final HEAIH output. The results are presented in Figure 6. The curves indicate the ratios of correct hyponymy relations that are acquired by an alternative against all the relations produced by HEAIH. As for Alternatives 1-4, we plotted the graph assuming the pairs of hypernym candidates and hyponym candidates were sorted in the same order as the order obtained by our procedure. In the case of Alternative 5, the 2,034 pairs of a hypernym candidates and an HCS, which were the results of Step B in HEAIH, were sorted according to the distance between headings and itemizations, and only the top 200 pairs were produced as the final output. The results suggest that our method can acquire a significant number of hyponymy relations that the alternatives miss.</Paragraph>
    <Paragraph position="14"> We then conducted a fairer comparison between HEAIH and Alternative 4 (or AHRAI). There are some hypernyms that can never be produced by AHRAI since these hypernyms are not considered in AHRAI. Recall that we computed the score hS for the nouns in a set N, which contained the 155,345 nouns most frequently observed in the downloaded 5:72PS106 documents in our experiments. If a given hypernym was not included in N, AHRAI could not produce that hypernym. In addition, some of the given hypernyms are actually noun sequences (or complex nouns) and cannot be members of N. On the other hand, HEAIH can acquire a hypernym not included in N if the hypernym contains substrings included in N. Thus, we also compared the performance under the assumption that only the hypernyms included in N could be true hypernyms. The results are presented in Figure 8. &amp;quot;Alternative 4&amp;quot; refers to the performance of AHRAI, while &amp;quot;HEAIH (restricted)&amp;quot; indicates the performance of HEAIH  when the produced hypernyms were restricted to the members of N. They show that HEAIH still out-performed AHRAI. In addition, the curve &amp;quot;AHRAI (full)&amp;quot; shows the performance of AHRAI when we accept the hypernyms that were not given to the HEAIH and all the 2,034 pairs of a hypernym candidate and an HCS were sorted according to the original score for AHRAI to produce the top 200 pairs. In this case, AHRAI outperformed HEAIH, though the difference is small.</Paragraph>
    <Paragraph position="15"> In the next set of experiments, we compared HEAIH and Alternatives 1-5 in a slightly different setting. Recall that Figure 4 gave the list of hypernyms in the HEAIH output and the number of HCSs that the procedure produced with each hypernym. The data was not balanced very evenly. While the procedure found 34 HCSs for laboratories, it provided only one HCS for animals. We tried to reevaluate these methods by using more balanced data. From the data, we eliminated the pairs of a hypernym and an HCS that were not included in the top five for each hypernym in the ranking of the HEAIH output. In other words, each hypernym could have a maximum of only five HCSs in the evaluation data.</Paragraph>
    <Paragraph position="16"> This reduced the influence by dominant hypernyms.</Paragraph>
    <Paragraph position="17"> In addition, we removed problematic hypernyms from the evaluation data. The preserved hypernyms are marked by '*' in Figure 4. We preserved only the hypernyms that could have proper nouns, names of species, or trade names as their hyponyms.5 In addition, there are inappropriate hypernyms such as those for which we could not determine their hyponyms without knowing the situation in which the hypernyms are used, as mentioned before. We eliminated 5Evidently, this condition was more restrictive than we expected with regard to hypernyms, and some intuitively acceptable hypernyms were not preserved. Examples are &amp;quot;jobs&amp;quot; and &amp;quot;business&amp;quot; (For their Japanese translation, we could not find hyponyms which were either proper nouns, names of species, or trade names). We made this restriction simply to keep the condition simple and to reduce borderline cases of proper hypernyms. Note that some of the eliminated hypernyms, such as &amp;quot;jobs&amp;quot; and &amp;quot;business&amp;quot;, were treated as proper hypernyms in the first comparison in Figure 6.</Paragraph>
    <Paragraph position="18"> such hypernyms too. We also removed &amp;quot;things&amp;quot; because it was too general. As a result of these changes, the evaluation data contained 73 pairs of a hypernym and an HCS. The comparison using this data is shown in Figure 9. HEAIH still acquired a large number of correct hyponymy relations that the alternative methods miss.</Paragraph>
  </Section>
class="xml-element"></Paper>