<?xml version="1.0" standalone="yes"?>
<Paper uid="P06-2075">
  <Title>Integrating Pattern-based and Distributional Similarity Methods for Lexical Entailment Acquisition</Title>
  <Section position="8" start_page="91904" end_page="91904" type="evalu">
    <SectionTitle>
4.2 Results
</SectionTitle>
    <Paragraph position="0"> The numbers of candidate entailment pairs collected for the test terms are shown in Table 2.</Paragraph>
    <Paragraph position="1"> These figures highlight the markedly complementary yield of the two acquisition approaches, where only about 10% of all candidates were identified by both methods. On average, 120 candidate entailment pairs were acquired for each target term.</Paragraph>
    <Paragraph position="2"> The SVM classifier was trained on a quite small annotated sample of 700 candidate entailment pairs of the 10 training terms. Table 3 presents comparative results for the classifier, for each of the two sets of candidates produced by each method alone, and for the union of these two sets (referred as Naive Combination). The results were computed for an annotated random sample of about 400 candidate entailment pairs of the test terms. Following common pooling evaluations in Information Retrieval, recall is calculated relatively to the total number of correct entailment pairs acquired by both methods together.</Paragraph>
    <Paragraph position="3">  The first two rows of the table show quite moderate precision and recall for the candidates of each separate method. The next row shows the great impact of method combination on recall, relative to the amount of correct entailment pairs found by each method alone, validating the complementary yield of the approaches. The integrated classifier, applied to the combined set of candidates, succeeds to increase precision substantially by 21 points (a relative increase of almost 60%), which is especially important for many precision-oriented applications like Information Retrieval and Question Answering. The precision increase comes with the expense of some recall, yet having F1 improved by 9 points.</Paragraph>
    <Paragraph position="4"> The integrated method yielded on average about 30 correct entailments per target term. Its classification accuracy (percent of correct classifications) reached 70%, which nearly doubles the naive combination's accuracy.</Paragraph>
    <Paragraph position="5"> It is impossible to directly compare our results with those of other works on lexical semantic relationships acquisition, since the particular task definition and dataset are different. As a rough reference point, our result figures do match those of related papers reviewed in Section 2, while we notice that our setting is relatively more difficult since we excluded the easier cases of proper nouns. (Geffet and Dagan, 2005), who exploited the distributional similarity approach over the web to address the same task as ours, obtained higher precision but substantially lower recall, considering only distributional candidates. Further research is suggested to investigate integrating their approach with ours.</Paragraph>
    <Section position="1" start_page="91904" end_page="91904" type="sub_section">
      <SectionTitle>
4.3 Analysis and Discussion
</SectionTitle>
      <Paragraph position="0"> Analysis of the data confirmed that the two methods tend to discover different types of relations. As expected, the distributional similarity method contributed most (75%) of the synonyms that were correctly classified as mutually entailing pairs (e.g. assault - abuse in Table 4). On the other hand, about 80% of all correctly identified hyponymy relations were produced by the pattern-based method (e.g. abduction - abuse).</Paragraph>
      <Paragraph position="1"> The integrated method provides a means to determine the entailment direction for distributional similarity candidates which by construction are non-directional. Thus, amongst the (nonsynonymous) distributional similarity pairs classified as entailing, the direction of 73% was correctly identified. In addition, the integrated method successfully filters 65% of the non-entailing co-hyponym candidates (hyponyms of the same hypernym), most of them originated in the distributional candidates, which is a large portion (23%) of all correctly discarded pairs.</Paragraph>
      <Paragraph position="2"> Consequently, the precision of distributional similarity candidates approved by the integrated system was nearly doubled, indicating the additional information that patterns provide about distributionally similar pairs.</Paragraph>
      <Paragraph position="3"> Yet, several error cases were detected and categorized. First, many non-entailing pairs are context-dependent, such as a gap which might constitute a hazard in some particular contexts, even though these words do not entail each other in their general meanings. Such cases are more typical for the pattern-based approach, which is sometimes permissive with respect to the relationship captured and may also extract candidates from a relatively small number of pattern occurrences. Second, synonyms tend to appear less frequently in patterns. Consequently, some synonymous pairs discovered by distributional similarity were rejected due to insufficient pattern matches. Anecdotally, some typos and spelling alternatives, like privatization privatisation, are also included in this category as they never co-occur in patterns.</Paragraph>
      <Paragraph position="4"> In addition, a large portion of errors is caused by pattern ambiguity. For example, the pattern &amp;quot;NP1, a|an NP2&amp;quot;, ranked among the top IS-A patterns by (Pantel et al., 2004), can represent both apposition (entailing) and a list of co-hyponyms (non-entailing). Finally, some misclassifications can be attributed to technical web-based processing errors and to corpus data sparseness.</Paragraph>
      <Paragraph position="5">  integrated method, illustrating Section 4.3. The columns specify the method that produced the candidate pair. Asterisk indicates a non-entailing pair.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>