<?xml version="1.0" standalone="yes"?>
<Paper uid="W05-1207">
  <Title>Discovering entailment relations using &amp;quot;textual entailment patterns&amp;quot;</Title>
  <Section position="3" start_page="37" end_page="39" type="metho">
    <SectionTitle>
2 The method
</SectionTitle>
    <Paragraph position="0"> Discovering entailment relations within texts implies the understanding of two aspects: firstly, how these entailment relations are usually expressed and, secondly, when an entailment relation may be considered stable and commonly shared. Assessing the first aspect requires the investigation of which are the prototypical textual forms that describe entailment relations. We will call them textual entailment patterns. These patterns (analysed in Sec. 2.2) will enable the detection of point-wise entailment assertions, that is, candidate verb pairs that still need a further step of analysis in order to be considered true entailment expressions. In fact, some of these candidates may be not enough stable and commonly shared in the language to be considered true entailments. To better deal with this second aspect, methods for statistically analysing large corpora are needed (see later in Sec. 2.3).</Paragraph>
    <Paragraph position="1"> The method we propose may be used in either: (1) recognising if entailment holds between two verbs, or, (2) extracting from a corpus C all the implied entailment relations. In recognition, given a verb pair, the related textual entailment expressions are derived as instances of the textual entailment patterns and, then, the statistical entailment indicators on a corpus C are computed to evaluate the stability of the relation. In extraction, the corpus C should be scanned to extract textual expressions that are instances of the textual entailment patterns. The resulting pairs are sorted according to the statistical entailment indicators and only the best ranked are retained as useful verb entailment pairs.</Paragraph>
    <Section position="1" start_page="37" end_page="38" type="sub_section">
      <SectionTitle>
2.1 An intuition
</SectionTitle>
      <Paragraph position="0"> Our method stems from an observation: verb logical subjects, as any verb role filler, have to satisfy specific preconditions as the theory of selectional restrictions suggests. Then, if in a given sentence a verb v has a specific logical subject x, its selectional restrictions imply that the subject has to satisfy some preconditions p, that is, v(x) - p(x). This can be read also as: if x has the property of doing the action  v this implies that x has the property p. For example, if the verb is to eat, the selectional restrictions of eat would imply, among other things, that its subject is an animal. If the precondition p is &amp;quot;having the prop-erty of doing an action a&amp;quot;, the constraint may imply that the action v entails the action a, that is, v - a.</Paragraph>
      <Paragraph position="1"> As for selectional restriction acquisition, the previous observation can enable the use of corpora as enormous sources of candidate entailment relations among verbs. For example &amp;quot;John McEnroe won the match...&amp;quot; can contribute to the definition of the selectional restriction win(x) - human(x) (since John McEnroe is a human), as well as to the induction (or verification) of the entailment relation between win and play, since John McEnroe has the property of playing. However, as the example shows, classes relevant for acquiring selectional preferences may be more explicit than active properties useful to derive entailment relations (i.e., it is easier to derive that John McEnroe is a human than that he has the property of playing).</Paragraph>
      <Paragraph position="2"> This limitation can be overcome when agentive nouns such as runner play subject roles in some sentences. Agentive nouns usually denote the &amp;quot;doer&amp;quot; or &amp;quot;performer&amp;quot; of some action a. This is exactly what is needed to make clearer the relevant property of the noun playing the logical subject role, in order to discover entailment. The action a will be the one entailed by the verb heading the sentence. For example, in &amp;quot;the player wins&amp;quot;, the action play evocated by the agentive noun player is entailed by win.</Paragraph>
    </Section>
    <Section position="2" start_page="38" end_page="39" type="sub_section">
      <SectionTitle>
2.2 Textual entailment patterns
</SectionTitle>
      <Paragraph position="0"> As observed for the isa relations in (Hearst, 1992) local and simple inter-sentential patterns may carry relevant semantic relations. As we saw in the previous section, this also happens for entailment relations. Our aim is thus to search for an initial set of textual patterns that describe possible linguistic forms expressing entailment relations between two verbs (vt,vh). By using these patterns, actual point-wise assertions of entailment can be detected or verified in texts. We call these prototypical patterns textual entailment patterns.</Paragraph>
      <Paragraph position="1"> The idea described in Sec. 2.1 can be straight-forwardly applied to generate textual entailment patterns, as it often happens that verbs can undergo an agentive nominalization (hereafter called personification), e.g., play vs. player. Whether or not an entailment relation between two verbs (vt,vh) holds according to some writer can be verified looking for sentences with expressions involving the agentive nominalization of the hypothesis verb vh. Then, the procedure to verify if entailment between two verbs (vt,vh) holds in a point-wise assertion is: whenever it is possible to personify the hypothesis vh, scan the corpus to detect the expressions where the personified hypothesis verb is the subject of a clause governed by the text verb vt.</Paragraph>
      <Paragraph position="2"> Given the two investigated verbs (vt,vh) we will refer to this first set of textual entailment patterns as personified patterns Ppers(vt,vh). This set will contain the following textual patterns:</Paragraph>
      <Paragraph position="4"> where pers(v) is the noun deriving from the personification of the verb v and elements such as l|f1,...,fN are the tokens generated from lemmas l by applying constraints expressed via the features f1,...,fN.</Paragraph>
      <Paragraph position="5"> For example, in the case of the verbs play and win, the related set of textual entailment expressions derived from the patterns will be Ppers(win,play) = { &amp;quot;player wins&amp;quot;, &amp;quot;players win&amp;quot;, &amp;quot;player won&amp;quot;, &amp;quot;players won&amp;quot; }. In the experiments hereafter described, the required verbal inflections (except personification) have been obtained using the publicly available morphological tools described in (Minnen et al., 2001) whilst simple heuristics have been used to personify verbs1.</Paragraph>
      <Paragraph position="6"> As the statistical measures introduced in the following section are those usually used for studying co-occurrences, two more sets of expressions, Fpers(v) and F(v), are needed to represent the single events in the pair. These are defined as:</Paragraph>
      <Paragraph position="8"> 1Personification, i.e. agentive nominalization, has been obtained adding &amp;quot;-er&amp;quot; to the verb root taking into account possible special cases such as verbs ending in &amp;quot;-y&amp;quot;. A form is retained as a correct personification if it is in WordNet.</Paragraph>
    </Section>
    <Section position="3" start_page="39" end_page="39" type="sub_section">
      <SectionTitle>
2.3 Measures to estimate the entailment
</SectionTitle>
      <Paragraph position="0"> strength The above textual entailment patterns define point-wise entailment assertions. In fact, if pattern instances are found in texts, the only conclusion that may be drawn is that someone (the author of the text) sustains the related entailment pairs. A sentence like &amp;quot;Painter draws on old techniques but creates only decorative objects.&amp;quot; suggests that painting entails drawing. However, it may happen that these correctly detected entailments are accidental, that is, the detected relation is only valid for that given text. For example, the text fragment &amp;quot;When a painter discovers this hidden treasure, other people are immediately struck by its beauty.&amp;quot; if taken in insulation suggests that painting entails discovering, but this is questionable. Furthermore, it may also happen that patterns detect wrong cases due to ambiguous expressions like &amp;quot;Painter draws inspiration from forest, field&amp;quot; where the sense of the verb draw is not the one expected.</Paragraph>
      <Paragraph position="1"> In order to get rid of these wrong verb pairs, an assessment of point-wise entailment assertions over a corpus is needed to understand how much the derived entailment relations are shared and commonly agreed. This validation activity can be obtained by both analysing large textual collections and applying statistical measures relevant for the task.</Paragraph>
      <Paragraph position="2"> Before introducing the statistical entailment indicators, some definitions are necessary. Given a corpus C containing samples, we will refer to the absolute frequency of a textual expression t in the corpus C with fC(t). The definition is easily extended to a set of expressions T as follows:</Paragraph>
      <Paragraph position="4"> Given a pair vt and vh we may thus define the following entailment strength indicators S(vt,vh), related to more general statistical measures.</Paragraph>
      <Paragraph position="5"> The first relevance indicator, Sf(vt,vh), is related to the probability of the textual entailment pattern as it is. This probability may be represented by the frequency, as the fixed corpus C makes constant the total number of pairs:</Paragraph>
      <Paragraph position="7"> where logarithm is used to contrast the effect of the Zipf's law. This measure is often positively used in terminology extraction (e.g., (Daille, 1994)).</Paragraph>
      <Paragraph position="8"> Secondly, another measure Smi(vt,vh) related to point-wise mutual information (Fano, 1961) may be also used. Given the possibility of estimating the probabilities through maximum-likelihood principle, the definition is straightforward:</Paragraph>
      <Paragraph position="10"> where p(x) = fC(x)/fC(.). The aim of this measure is to indicate the relatedness between two elements composing a pair. Mutual information has been positively used in many NLP tasks such as collocation analysis (Church and Hanks, 1989), terminology extraction (Damerau, 1993), and word sense disambiguation (Brown et al., 1991).</Paragraph>
    </Section>
  </Section>
  <Section position="4" start_page="39" end_page="40" type="metho">
    <SectionTitle>
3 Experimental Evaluation
</SectionTitle>
    <Paragraph position="0"> As many other corpus linguistic approaches, our entailment detection model relies partially on some linguistic prior knowledge (the expected structure of the searched collocations, i.e., the textual entailment patterns) and partially on some probability distribution estimation. Only a positive combination of both these two ingredients can give good results when applying (and evaluating) the model.</Paragraph>
    <Paragraph position="1"> The aim of the experimental evaluation is then to understand, on the one side, if the proposed textual entailment patterns are useful to detect entailment between verbs and, on the other, if a statistical measure is preferable with respect to the other. We will here evaluate the capability of our method to recognise entailment between given pairs of verbs.</Paragraph>
    <Paragraph position="2"> We carried out the experiments using the web as the corpus C where to estimate our two textual entailment measures (Sf and Smi) and GoogleTM as a count estimator. The findings described in (Keller and Lapata, 2003) seem to suggest that count estimations we need in the present study over Subject-Verb bigrams are highly correlated to corpus counts.</Paragraph>
    <Paragraph position="3"> As test bed we used existing resources: a non trivial set of controlled verb entailment pairs is in fact contained in WordNet (Miller, 1995). There, the entailment relation is a semantic relation defined at the synset level, standing in the verb subhierarchy. Each  pair of synsets (St,Sh) is an oriented entailment relation between St and Sh. WordNet contains 415 entailed synsets. These entailment relations are consequently stated also at the lexical level. The pair (St,Sh) naturally implies that vt entails vh for each possible vt [?] St and vh [?] Sh. It is then possible to derive from the 415 entailment synset a test set of 2,250 verb pairs. As the proposed model is applicable only when hypotheses can be personified, the number of the pairs relevant for the experiment is thus reduced to 856. This set is hereafter called the True Set (TS).</Paragraph>
    <Paragraph position="4"> As the True Set is our starting point for the evaluation, it is not possible to produce a natural distribution in the verb pair space between entailed and not-entailed elements. Then, precision, recall, and f-measure are not applicable. The only solution is to use a ROC (Green and Swets, 1996) curve mixing sensitity and specificity. What we then need is a Control Set (CS) of verb pairs that in principle are not in entailment relation. The Control Set has been randomly built on the basis of the True Set: given the set of all the hypothesis verbs H and the set of all the text verbs T of the True Set, control pairs are obtained randomly extracting one element from H and one element from T. A pair is considered a control pair if it is not in the True Set. For comparative purposes the Control Set has the same cardinality of the True Set. However, even if the intersection between the True Set and the Control Set is empty, we are not completely sure that the Control Set does not contains any pair where the entailment relation holds. What we may assume is that this last set at least contains a smaller number of positive pairs.</Paragraph>
    <Paragraph position="5"> Sensitivity, i.e. the probability of having positive answers for positive pairs, and specificity, i.e. the probability of having negative answers for negative pairs, are then defined as:</Paragraph>
    <Paragraph position="7"> where p((vh,vt) [?] TS|S(vh,vt) &gt; t) is the probability of a candidate pair (vh,vt) to belong to TS if the test is positive, i.e. the value S(vh,vt) of the entailment detection measure is greater than t, while p((vh,vt) [?] CS|S(vh,vt) &lt; t) is the probability of belonging to CS if the test is negative. The ROC curve (Sensitivity vs. 1 [?] Specificity) naturally follows (see Fig. 1).</Paragraph>
    <Paragraph position="8"> Results are encouraging as textual entailment patterns show a positive correlation with the entailment relation. Both ROC curves, the one related to the frequency indicator Sf (f in figure) and the one related to the mutual information SMI (MI in figure), are above the Baseline curve. Moreover, both curves are above the second baseline (Baseline2) applicable when it is really possible to use the indicators. In fact, textual entailment patterns have a non-zero frequency only for 61.4% of the elements in the True Set. This is true also for 48.1% of the elements in the Control Set. The presence-absence in the corpus is then already an indicator for the entailment relation of verb pairs, but the application of the two indicators can help in deciding among elements that have a non-zero frequency in the corpus. Finally, in this case, mutual information appears to be a better indicator for the entailment relation with respect to the frequency.</Paragraph>
  </Section>
class="xml-element"></Paper>