<?xml version="1.0" standalone="yes"?>
<Paper uid="A97-1043">
  <Title>An Automatic Extraction of Key Paragraphs Based on Context Dependency</Title>
  <Section position="4" start_page="291" end_page="292" type="metho">
    <SectionTitle>
3 Term Weighting
</SectionTitle>
    <Paragraph position="0"> Every sense of words in articles for extracting key paragraphs is automatically disambiguated in advance. This is because to disambiguate word-senses in articles might affect the accuracy of context dependent (domain specific) key paragraphs retrieval, since the meaning of a word characterises the domain in which it is used. Word-sense disambiguation (WSD in short) is a serious problem for NLP, and a variety of approaches have been proposed for solving it (Brown, 1991), (Yarowsky, 1992). Our disambignation method is based on Niwa's method which uses the similarity between a sentence containing a polysemous noun and a sentence of dictionarydefinition (Niwa, 1994). Furthermore, we linked nouns which are disambignated with their semantically similar nouns mainly in order to cope with the problem of a phrasal lexicon. A phrasal lexicon such as Atlantic Seaboard, New England gives a negative influence for keywords retrieval, since it can not be regarded as units, i.e. each word which is the element of a phrasal lexicon is assigned to each semantic code (Fukumoto, 1996).</Paragraph>
    <Paragraph position="1"> To the results of WSD and linking methods, we then applied a term weighting method to extract keywords. There have been several term weighting based on word frequencies, such as TF(Term Frequency), IDF(Inverse Document Frequency), TF*IDF, WIDF(Weighted Inverse Document Frequency) (Luhn, 1957), (Sparck, 1973), (Salton, 1983), (Tokunaga, 1994). We used Watan- null Formula (1) shows the value of X 2 of the word i in the domain j. zij in (1) is the frequency of word i in the domain j. mij in (1) is shown in formula (2).</Paragraph>
    <Paragraph position="3"> In formula (2), k is the number of different words and l is the number of the domains. A larger value of X~ means that the word i appears more frequently in thJe domain j than in the other.</Paragraph>
  </Section>
  <Section position="5" start_page="292" end_page="292" type="metho">
    <SectionTitle>
4 An Extraction of Keywords
</SectionTitle>
    <Paragraph position="0"> The first step to extract keywords is to calculate X 2 for each word in the Paragraph, the Article, and the Domain. We used formula (1) to calculate the value of xP~j, xAi2j, and xD~y, where xP~i, xAi~, and xD~; indicate which word i appears most frequently in t~e context j of Paragraph, Article, and Domain, respectively. For example, xPi2j is shown in formula  (3) by using formula (1).</Paragraph>
    <Paragraph position="2"> In formula (3), xlj is the frequency of word i in the context j of Paragraph. miy in formula (3) is shown in (2) where k is the number of different words and l is the number of contexts in Paragraph.</Paragraph>
    <Paragraph position="3"> The second step is to calculate the degree of word /in Paragraph (xP~), Article (xA~), and Domain (xD~). We defined the degree of word i in Paragraph, Article, and Domain as the deviation value of k contexts in Paragraph, Article, and Domain, respectively. Here, k is the number of contexts in Paragraph, Article, and Domain, respectively. For example, the deviation value of the word i in Paragraph is defined as follows:</Paragraph>
    <Paragraph position="5"> In formula (4), k is the number of contexts in Paragraph, and mi is the mean value of the total frequency of word i in Paragraph which consists of k contexts.</Paragraph>
    <Paragraph position="6"> The last step to extract keywords is to calculate the context dependency of word i using formula (4). We recall that if i satisfies both 1 and 2 in section 2, the word i is regarded as a keyword.</Paragraph>
    <Paragraph position="8"> Formulae (5) and (6) shows i, and 2 in section 2, respectively. In formulae (5) and (6), xP~, xA~, and xD~ are the deviation value of a set of Paragraph, Article, and Domain, respectively.</Paragraph>
  </Section>
  <Section position="6" start_page="292" end_page="293" type="metho">
    <SectionTitle>
5 An Extraction of Key Paragraphs
</SectionTitle>
    <Paragraph position="0"> The procedure for extracting key paragraphs has the following three stages: Stage One: Representing every paragraph as a vector The goal of this stage is to represent every paragraph in an article as a vector. Using a term weightmg method, every paragraph in an article would be represented by vector of the form</Paragraph>
    <Paragraph position="2"> where n is the number of nouns in an article and Niy is as follows; {o 1(Nj) Nis= 0 where I(Nj) is a frequency with which the noun Nj appears in paragraph Pi.</Paragraph>
    <Paragraph position="3"> Stage Two: Clustering method Given a vector representation of paragraphs P1, * .., P,~ as in formula (7), a similarity between two paragraphs Pi, Pj in an article would be obtained by using formula (8). The similarity of Pi and Pj is measured by the inner product of their normalised vectors and is defined as follows: Nj does not appear in Pi Nj is a keyword and appears in Pi Nj is not a keyword and appears</Paragraph>
    <Paragraph position="5"> The greater the value of Sim(Pi, Pi) is, the more similar these two paragraphs are. For a set of paragraphs P1, &amp;quot;&amp;quot; &amp;quot;, Pm of an article, we calculate the semantic similarity value of all possible pairs of paragraphs. The clustering algorithm is applied to the sets and produces a set of semantic clusters, which are ordered in the descending order of their semantic similarity values. We adopted non-overlapping, group average method in our clustering technique (Jardine, 1968).</Paragraph>
    <Paragraph position="6"> Stage Three: Extraction of key paragraphs The sample results of clustering is shown in Table  'Num' in Table 1 shows the order of clusters which we have obtained and the number shown under 'Cluster' shows the paragraph numbers. In Table 1, if the number of keywords which belonging to the third paragraph is larger than that of the fourth, the order of key paragraphs is 3 ) 4 &gt; 1 ~ 2, otherwise, 4 &gt; 3 ~ 1 ) 2.</Paragraph>
  </Section>
  <Section position="7" start_page="293" end_page="294" type="metho">
    <SectionTitle>
6 Experiments
</SectionTitle>
    <Paragraph position="0"> We have conducted three experiments to examine the effect of our method. The first experiment, Key-words Experiment, is concerned with the keywords extracting technique and with verifying the effect of our method which introduces context dependency.</Paragraph>
    <Paragraph position="1"> The second experiment, Key Paragraphs Experiment, shows how the extracted keywords can be used to extract key paragraphs. In the third experiment, Comparison to Other Related Work, we applied Zechner's key sentences method (Zechner, 1996) to key paragraphs extraction (we call this method_A), and compared it with our method.</Paragraph>
    <Section position="1" start_page="293" end_page="293" type="sub_section">
      <SectionTitle>
6.1 Data
</SectionTitle>
      <Paragraph position="0"> The corpus we have used is the 1988, 1989 Wall Street J~urnal (Liherman, 1991) in ACL/DCI CD-ROM which consists of about 280,000 part-of-speech tagged sentences (BriU, 1992). Wall Street Journal consists of many articles, and each article has a title name. These titles are classified into 76 different domains. We selected 10 different domains and used them as Domain. As a test data, we selected 50 articles each of which belongs to one of these 10 domains. The selected domain names and the number of articles are shown in Table 2.</Paragraph>
      <Paragraph position="1">  STK: stock market 5 RET: retailing 1 ARO: aerospace 5 ENV: environment 3 PCS: stones, gold 9 CMD: farm products 3 There are 3,802 different nouns in 50 articles. As a result of WSD and linking methods for these articles, we have obtained 3,707 different nouns.</Paragraph>
    </Section>
    <Section position="2" start_page="293" end_page="293" type="sub_section">
      <SectionTitle>
6.2 Keywords Experiment
</SectionTitle>
      <Paragraph position="0"> Formulae (5) and (6) are applied to 50 articles which are the results of WSD and linking methods, and as a result, we have obtained 1,047 keywords in all. The result of keyword extraction is shown in Table 3.</Paragraph>
      <Paragraph position="2"> In Table 3, z in 'z(y)' of 'Paragraph' shows the number of paragraphs in an article, 'y' shows the number of articles. For example, 3(1) shows that there is one article which consists of three paragraphs. Recall and Precision in Table 3 are as follows; Number of correct keywords Recall = Number of keywords which are selected by human Number of correct keywords Precision = Number of keywords which are selected in our method Recall and Precision in Table 3 show the means in each paragraph. The denominator of Recall is made by three human judges; i.e. when more than one human judged the word as a keyword, the word is regarded as a keyword.</Paragraph>
    </Section>
    <Section position="3" start_page="293" end_page="294" type="sub_section">
      <SectionTitle>
6.3 Key Paragraphs Experiment
</SectionTitle>
      <Paragraph position="0"> For each article, we extracted 10 ,~50 % of its paragraphs as key paragraphs. The results of key paragraphs experiment are shown in Table 4.</Paragraph>
      <Paragraph position="1"> In Table 4, 10 ,~ 50 % indicates the extraction ratio used. 'Para.' shows the number of paragraphs which humans judged to be key paragraphs, and 'Correct' shows the number of these paragraphs which the method obtained correctly. Evaluation is performed by three human judges. When more than one human judges a paragraph as a key paragraph, the paragraph is regarded as a key paragraph. '*' in Table 4 shows that the number of the correct data is smaller than that of an extraction ratio. For example, in Table 4, the number of paragraphs of 20 % out of 22 is 4. However, the number of paragraphs that more than one human judged the paragraph  as a key paragraph was only two. Therefore, 2 is marked with a '*'.</Paragraph>
    </Section>
    <Section position="4" start_page="294" end_page="294" type="sub_section">
      <SectionTitle>
6.4 Comparison to Other Related Work
</SectionTitle>
      <Paragraph position="0"> Zechner proposed a method to extract key sentences in an article by using simple statistical method; i.e.</Paragraph>
      <Paragraph position="1"> TF*IDF term weighting method. In order to show the applicability of our method, we applied Zechner's key sentences method to key paragraphs extraction and compared it with our method. In Zechner's method, the sum over all TF*IDF values of the content words for each sentence are calculated, and the sentences are sorted according to their weights.</Paragraph>
      <Paragraph position="2"> Finally a particular number of sentences are extracted as key sentences. The data we used consists of 1.92 sentences par a paragraph and was not so many sentences within a paragraph. Then, in order to apply his method to key paragraphs extraction, we calculated the sum over all sentences for each paragraph, and sorted the paragraphs according to their weights. From these, we extracted a certain number of paragraphs (method_A). In our method, every sense of words in articles for extracting key paragraphs is disambiguated in advance and linking method is performed. In order to examine where the performance comes from, we also compared our method to the method which WSD and linking method are not applied. The result is shown in Table 5.</Paragraph>
      <Paragraph position="3"> In Table 5, '%' shows the extraction ratio, 10 ,-~ 50% and 'Para.' shows the number of paragraphs corresponding to each 'Percentage'. 'Our method', 'not WSD', and 'metbod_h' shows the results using our  method, the method which WSD and linking are not applied, and method_A, respectively.</Paragraph>
    </Section>
  </Section>
  <Section position="8" start_page="294" end_page="297" type="metho">
    <SectionTitle>
7 Discussion
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="294" end_page="295" type="sub_section">
      <SectionTitle>
7.1 Keywords Experiment
Effectiveness of the Method
</SectionTitle>
      <Paragraph position="0"> According to Table 3, Recall and Precision values range from 46.5/50.2 to 100.0/89.4, the mean being 78.7/78.6. This shows that our method is effective even in a restricted domain such as financial articles, e.g. Wall Street Journal, although the test set was small (50 articles). Furthermore, the correct ratio does not depend on the number of paragraphs in an article. This shows that our context dependency model is applicable for different size of the samples.</Paragraph>
      <Paragraph position="1">  paragraphs was 14. As a result, the result of the extraction of key paragraphs shown in Table 4 was also worst (56.5%). The possible causes of the error were summarised the following two points:  In Table 6, each value of 'Paragraph', 'Article', and 'Domain', shows each X 2 value. 'Total average' shows the mean of all keywords. 'word237' and 'word238' are representative words which are the result of linking noun with their semantically similar nouns. According to Table 6, we can observe that in 'Paragraph', for example, some words whose X 2 values are slightly higher than the average (1,772) exist.</Paragraph>
      <Paragraph position="2"> For example, the X 2 value of 'word237' is 4.633 and slightly higher than 1.772. However, 'word237' satisfies the formulae of context dependency. As a result, 'word237' is regarded as a keyword, while this is not. When the extracted ratio was 10%, there were four articles whose correct ratio did not attained 100%.</Paragraph>
      <Paragraph position="3"> Of these, three articles are classified into this type of the error.</Paragraph>
      <Paragraph position="4"> From the above observation, we can estimate that the formulae of context dependency are weak constraints in some domains, while they are still effective even in a restricted domain. In order to get more accuracy, some other constraints such as loca-tion heuristics (Baxendale, 1958) or upper-case word feature (Kupiec, 1995) might be necessary to be introduced into our framework.</Paragraph>
      <Paragraph position="5"> (2) The error of WSD When the extracted ratio was 10%, there was one article out of four articles which could not be extracted correctly because of the error of WSD. The test article and the results of it was shown in Figure , In Figure 2, the headline shows the title name. The numbers show the paragraph number, and the underlined words are keywords which are extracted in our method. The bottom shows the result of key paragraphs extraction. According to Figure 2, when the extraction ratio was 50%, the paragraphs 3 and 4 were extracted and the paragraph 1 was not extracted, although it is a key paragraph. The key-words and their frequencies of appearance in paragraph 1, 3, and 4 are shown in Table 7.</Paragraph>
      <Paragraph position="6">  exchangel, offer4, notes, shares, stockS, amount4, tradingl, stockl, cents According to Table 7, 'crystal' and 'oil in paragraph 1 are disambiguated incorrectly and were replaced by 'crystal4' and 'oi14', respectively, while 'crystal' should have been replaced by 'crystal2' and 'oi1' with 'oi13'. Therefore, the number of words which appear in both paragraph 3 and 4 was larger than any other pair of paragraphs. As a result, paragraph 3 and 4 are the most semantically similar paragraphs and 1 was not extracted as a key paragraph.</Paragraph>
      <Paragraph position="7"> In our method, the correct ratio of key paragraphs extraction strongly depends on the results of WSD. The correct ratio of our WSD was 78.4% (Fukumoto, 1996). In order to get higher accuracy, it is necessary to improve our WSD method.</Paragraph>
    </Section>
    <Section position="2" start_page="295" end_page="296" type="sub_section">
      <SectionTitle>
7.2 Key Paragraphs Experiment
Effectiveness of the Method
</SectionTitle>
      <Paragraph position="0"> In Key Paragraphs Experiment, the overall results were positive, especially when the ratio of extraction was 10,,,30%. The ratios of correct judgements in these cases were significantly high; i.e. 92.5%, 91.3%, and 89.2%, respectively. This demonstrates the applicability of the degree of context dependency. null Limitations of the Method When the ratio of extraction was higher than 30%, the results was 72.7% and 60.4%. Furthermore, the more paragraphs are in an article, the smaller the number of correct judgements. One possible cause of these results is that the clustering method might have a negative effect on extracting key paragraphs.  In the field of text summarisation, a vector model was often used for extracting key sentence or key paragraph (Tokunaga, 1994), (Zechner, 1996). In this model, the sentences with term weighting are sorted according to their weights and this information is used to extract a certain ratio of highest weighted paragraph in an article. We implemented this model and compared it with our clustering technique. The results are shown in Table 8.</Paragraph>
      <Paragraph position="1">  In Table 8, '%' shows the extraction ratio, 10 ~ 50% and 'Para.' shows the mfmber of total paragraphs corresponding to each '%'. 'Our method', and 'Vector model' shows the results of our method, and using vector model, respectively.</Paragraph>
      <Paragraph position="2"> Table 8 shows that the results using our method are highly than those of using the vector model. In our method, when the extraction ratio was more than 30%, the correct ratio decreased. This phenomena is also observed in the vector model. From the observation, we can estimate that the cause of the results was not our clustering technique. Examining the results of human judges, when the number of paragraphs was more than 14, the number of paragraphs marked with a '*' is large. This shows that it is too difficult even for a human to judge whether a paragraph is a key paragraph or not. From the observation, for these articles, there are limitations to our method based on context dependency.</Paragraph>
      <Paragraph position="3"> Other Heuristics As we discussed in Keywords Experiment, it might be considered that some heuristics such as location of paragraphs are introduced into our method to get a higher accuracy of keywords and key paragraphs extraction, even in these articles. Table 9 shows the location of key paragraphs extracted using our method and extracted by humans. The extraction ratio described in Table 9 is 30%.</Paragraph>
      <Paragraph position="4">  In Table 9, each paragraph (First, Mid-position, and Last paragraph) includes the paragraphs around it. According to Table 9, in human judgement, 39 out of 50 articles' key paragraphs are located in the first parts, and the ratio attained 78.0%. This shows that using only location heuristics (the key paragraph tends to be located in the first parts) is a weak constraint in itself, since the results of our method showed that the correct ratio attained 89.2%. However, in our method, 2 articles are not extracted correctly, while the key paragraph is located in the first parts of these articles. From the observation, in a corpus such as Wall Street Journal, utilising a location heuristics is useful for extracting key paragraphs. null</Paragraph>
    </Section>
    <Section position="3" start_page="296" end_page="297" type="sub_section">
      <SectionTitle>
7.3 Comparison to Other Related Work
</SectionTitle>
      <Paragraph position="0"> According to Table 5, the average ratio of our method and method_A was 74.7%, and 48.6%, respectively. This shows that method_A is not more effective than our method. This is because most of nouns do not contribute to showing the characteristic of each domain for given articles. In the test data which consists of 3,802 different nouns, 2,171 nouns appeared in only one article and the frequency of each of them is one. We recall that in method_A,  when word i appears in only one article and the frequency of i is one, the value of TF*IDF equals to log50. There are 2,955 out of 3,802 nouns whose TF*IDF value is less than log50, and the percentage attained at 77.7%. This causes the fact that most of nouns do not contribute to showing the characteristic of each domain for given articles.</Paragraph>
      <Paragraph position="1"> Comparing the difference ratio of 'Our method' and 'not WSD' to that of 'not WSD' and method_A, the former was 10.5% and the latter was 15.6%.</Paragraph>
      <Paragraph position="2"> Therefore, our context dependency model contributes the extraction of key paragraphs, although WSD and linking are still effective.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>