<?xml version="1.0" standalone="yes"?>
<Paper uid="W03-1610">
  <Title>Optimizing Synonym Extraction Using Monolingual and Bilingual Resources</Title>
  <Section position="5" start_page="213" end_page="213" type="evalu">
    <SectionTitle>
4 Evaluation
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="213" end_page="213" type="sub_section">
      <SectionTitle>
4.1 The Gold Standard
</SectionTitle>
      <Paragraph position="0"> The simplest evaluation measure is direct comparison of the extracted synonyms with the manually created thesaurus. However, the thesaurus coverage is a problem. In this paper, we combined two thesauri as a gold stardard: WordNet 1.6</Paragraph>
    </Section>
    <Section position="2" start_page="213" end_page="213" type="sub_section">
      <SectionTitle>
http://www.cogsci.princeton.edu/~wn/) and Roget
(Roget's II: The New Thesaurus, 1995.
</SectionTitle>
      <Paragraph position="0"> http://www.bartleby.com/thesauri/).</Paragraph>
      <Paragraph position="1"> In WordNet, one synset consists of several synonyms which represent a single sense. Therefore, a polysemous word occurs in more than one synsets. For example, the polysemous word &amp;quot; abandon&amp;quot; occur in five different synsets: (abandon, forsake, desolate, desert, lurch) (vacate, empty, abandon) (abandon, give up, give) (abandon, give up) (abandon) For a given word, we combine its synonyms from all synsets including the word. Thus, we get the synonyms of the word &amp;quot; abandon&amp;quot; as follows: abandon forsake, desolate, desert, lurch, vacate, empty, give up, give For synonyms in Roget, we also combine the synonyms in different synsets into one set as we do for WordNet. Thus, we get the synonyms of the word &amp;quot; abandon&amp;quot; as follows: abandonbreak off, desist, discontinue, give up, leave off, quit, relinquish, remit, stop, desert, forsake, leave, throw over, abdicate, cede, demit, forswear, hand over, quitclaim, render, renounce, resign, surrender, waive, yield, give over, forgo, lay down Combining the results of WordNet and Roget, we can get the synonyms of the word &amp;quot; abandon&amp;quot; as follows.</Paragraph>
      <Paragraph position="2"> abandon desolate, lurch, vacate, empty, give, abdicate, break off, cede, demit, desert, desist, discontinue, forgo, forsake, forswear, give up, give over, hand over, lay down, lay off, leave off, leave, quit, quitclaim, relinquish, remit, stop, swear off, throw over, render, renounce, resign, surrender, waive, yield</Paragraph>
    </Section>
    <Section position="3" start_page="213" end_page="213" type="sub_section">
      <SectionTitle>
4.2 Evaluation Measures
</SectionTitle>
      <Paragraph position="0"> The evaluation metrics are precision, recall, and f-measure. If we use S to indicate the synonyms that our method extracts for a word and GS to denote the synonyms of the word in WordNet and Roget, the methods to calculate the precision, recall, and f-measure of our methods are shown in Equation (7), (8), and (9). To investigate the results of more than one word, we calculate the average precision, recall and f-measure, which sum the individual values divided by the number of the investigated words.</Paragraph>
    </Section>
    <Section position="4" start_page="213" end_page="213" type="sub_section">
      <SectionTitle>
4.3 Test Set
</SectionTitle>
      <Paragraph position="0"> In order to evaluate our methods, we build up a test set which includes three parts:  (a) high-frequency words: occurring more than 100 times; (b) middle-frequency words: occurring more than 10 times and not greater than 100 times; (c) low-frequency words: occurring no greater  words have synonyms both in our results extracted from the three resources and in the thesauri WordNet and Roget. The statistics of the test set are shown in Table 5.</Paragraph>
    </Section>
    <Section position="5" start_page="213" end_page="213" type="sub_section">
      <SectionTitle>
4.4 Experimental Results
</SectionTitle>
      <Paragraph position="0"> In this section, we compare the extracted synonyms of the nouns and verbs in the test set with those in WordNet and Roget. For each method, we select those as synonyms whose similarity scores with the investigated word are larger than a given threshold. A development set is used to determine the thresholds of each method. The thresholds for getting highest f-measure scores on the development set are selected. In our experiments, we get 0.04, 0.04, 0.1 and 0.04 for Method 1, Method 2, Method 3 and the combined approach, respectively.</Paragraph>
      <Paragraph position="1"> The evaluation results for the individual extractors and the ensemble extractor are shown in Table 6 and Table 7. We set a1=0.4, a2=0.4 and a3=0.2 in Equation (6) for the ensemble to combine the results from the three resources. The weights are also obtained with the development set.</Paragraph>
      <Paragraph position="2"> In order to examine the performance of each method in more details, we also get the precisions and recalls under different thresholds. Figure 1 and Figure 2 shows the precision values under different recall values (different thresholds) for all nouns and verbs, respectively.</Paragraph>
      <Paragraph position="3"> Among all of the methods, the method combining all of the three resources gets the best results in terms of both precision and recall. The effect is similar to the ensemble methods for synonym  extraction in (Curran 2002). However, our method uses an ensemble of different resources instead of one single resource. During the experiments, we also find the ensemble combining all of the three extractors outperforms the ensembles only combining any two of the three extractors. This indicates that the extractors using the three different resources are complementary to each other. For example, the extractor using the monolingual dictionary gets a high precision and the extractor using the bilingual corpus gets a high recall. Although the extractor using the monolingual corpus achieved much lower precision and recall on synonym extraction, it is still useful to be included in the ensemble. This shows that the monolingual corpus is complementary to the other two resources on synonym extraction. The success of our method also indicates that our ensemble method by weighting all extractors is effective for synonym extraction.</Paragraph>
      <Paragraph position="4"> Among the methods only using one kind of resource, Method 2, which uses the bilingual corpus, has the highest f-measure scores on both nouns and verbs. From the results in Figure 1 and Figure 2, we can see that the coverage of synonyms extracted by Method 2 is the highest. Although it has lower precisions than Method 1 under low recalls, its precisions are higher than those of Method 1 under higher recalls. This shows that Method 2 can get a good compromise between precision and recall. We also note that the maximum recall of Method 2 is much larger than that of Method 1. This is because (1) in Method 1, the words used in the definitions are highly limited. Thus, the coverage of the synonyms is limited; (2) the advantage of Method 2 is that the coverage of extracted synonyms is high because it can extract the synonyms not occurring in the corpus. It is different from the method in (Barzilay and Mckeown, 2001; Shimohata and Sumita, 2002), which can only extract the synonyms in the bi-lingual corpus.</Paragraph>
      <Paragraph position="5"> The performance of Method 3 is the worst. It is caused by two factors: (1) the context model of Method 3 introduces much noise because of the errors of the parser; (2) this method is unable to distinguish synonyms, antonyms, and similar words because they tend to have similar contexts.</Paragraph>
      <Paragraph position="6"> From the contexts it uses, method 3 is suitable to extract related words which have the similar usages from the view of syntax.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>