Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language
Processing (HLT/EMNLP), pages 371–378, Vancouver, October 2005. c©2005 Association for Computational Linguistics
A Semantic Approach to Recognizing Textual Entailment
Marta Tatu and Dan Moldovan
Language Computer Corporation
Richardson, TX 75080, USA
marta,moldovan@languagecomputer.com
Abstract
Exhaustive extraction of semantic infor-
mation from text is one of the formidable
goals of state-of-the-art NLP systems. In
this paper, we take a step closer to this
objective. We combine the semantic in-
formation provided by different resources
and extract new semantic knowledge to
improve the performance of a recognizing
textual entailment system.
1 Recognizing Textual Entailment
While communicating, humans use different expres-
sions to convey the same meaning. Therefore, nu-
merous NLP applications, such as, Question An-
swering, Information Extraction, or Summarization
require computational models of language that rec-
ognize if two texts semantically overlap. Trying to
capture the major inferences needed to understand
equivalent semantic expressions, the PASCAL Net-
work proposed the Recognizing Textual Entailment
(RTE) challenge (Dagan et al., 2005). Given two text
fragments, the task is to determine if the meaning of
one text (the entailed hypothesis, H) can be inferred
from the meaning of the other text (the entailing text,
T).
Given the wide applicability of this task, there
is an increased interest in creating systems which
detect the semantic entailment between two texts.
The systems that participated in the Pascal RTE
challenge competition exploit various inference el-
ements which, later, they combine within statisti-
cal models, scoring methods, or machine learning
frameworks. Several systems (Bos and Markert,
2005; Herrera et al., 2005; Jijkoun and de Rijke,
2005; Kouylekov and Magnini, 2005; Newman et
al., 2005) measured the word overlap between the
two text strings. Using either statistical or Word-
Net’s relations, almost all systems considered lexical
relationships that indicate entailment. The degree of
similarity between the syntactic parse trees of the
two texts was also used as a clue for entailment by
several systems (Herrera et al., 2005; Kouylekov and
Magnini, 2005; de Salvo Braz et al., 2005; Raina
et al., 2005). Several groups used logic provers to
show the entailment between T and H (Bayer et
al., 2005; Bos and Markert, 2005; Fowler et al.,
2005; Raina et al., 2005) and some of them made
use of world knowledge axioms to increase the logic
prover’s power of inference (Bayer et al., 2005; Bos
and Markert, 2005; Fowler et al., 2005).
In this paper, we describe a novel technique which
employs a set of semantic axioms in its attempt to
exhaustively extract semantic knowledge from texts.
In order to show the contribution that our semantic
information extraction method brings, we append it
as an additional module to an already existing sys-
tem that participated in the RTE challenge. Our sys-
tem (Fowler et al., 2005),  rst, transforms the text
T and the hypothesis H into semantically enhanced
logic forms, and, then, the integrated logic prover
tries to prove or disprove the entailment using a
set of world-knowledge axioms (die of blood loss
→ bleed to death), linguistic rewriting rules which
break down complex syntactic structures, like co-
ordinating conjunctions, and WordNet-based lexical
chains axioms (buy/VB/1 → pay/VB/1).
371
2 Approach
We believe that a logic-based semantic approach is
highly appropriate for the RTE task1. Text T seman-
tically entails H if its meaning logically implies the
meaning of H. Because the set of semantic relations
encoded in a text represents its meaning, we need to
identify all the semantic relations that hold between
the constituents of T and, subsequently, between the
constituents of H to understand the meaning of each
text. It should be noted that state-of-the-art seman-
tic parsers extract only some of the semantic rela-
tions encoded in a given text. To complete this in-
formation, we need semantic axioms that augment
the extracted knowledge and, thus, provide a better
coverage of the text’s semantics. Once we gather
this information, we state that text T entails hypoth-
esis H if and only if we  nd similar relations be-
tween a concept from T and a semantically analo-
gous concept from H. By analogous concepts, we
mean identical concepts, or words connected by a
chain of SYNONYMY, HYPERNYMY or morphologi-
cal derivation relations in WordNet.
Because the set of semantic elements identi ed by
a semantic parser does not necessarily convey the
complete meaning of a sentence, we shall use a set
of semantic axioms to infer the missing pieces of in-
formation. By combining two semantic relations or
by using the FrameNet’s frame elements identi ed
in a given text, we derive new semantic information.
In order to show if T entails H, we analyze their
meanings. Our approach to semantic entailment in-
volves the following stages:
1. We convert each text into logic form (Moldovan
and Rus, 2001). This conversion includes part-of-
speech tagging, parse tree generation, and name en-
tity recognition.
2. Using our semantic parser, we identify some of
the semantic relations encoded in the analyzed texts.
We note that state-of-the-art semantic parsers can-
not discover all the semantic relations conveyed im-
plicitly or explicitly by the text. This problem com-
promises our system’s performance. To obtain the
complete set of semantic relations that represents the
meaning of the given texts, we introduce a new step
in our algorithm.
1After all, the entailment, inference, and equivalence terms
originated from logic.
3. We add semantic axioms to the already created
set of world knowledge, NLP, and WordNet-based
lexical chain (Moldovan and Novischi, 2002) ax-
ioms that assist the logic prover in its search for
proofs. We developed semantic axioms that show
how two semantic relations can be combined. This
will allow the logic prover to combine, whenever
possible, semantic instances in order to infer new se-
mantic relationships. The instances of relations that
participate in semantic combinations can be either
provided by the text or annotated between WordNet
synsets. We also exploit other sources of semantic
information from the text. For example, the frames
encoded in the text sentence provide information
which complements the meaning given by the se-
mantic relations. Our second type of axioms derive
semantic relations between the frame elements of a
given FrameNet frame.
We claim that the process of applying the seman-
tic axioms, given the semantic relations detected by
a semantic parser, will capture the complete seman-
tic information expressed by a text fragment. In this
paper, we show the usefulness of this procedure for
the RTE task, but we are convinced that it can be used
by any system which plans to extract the entire se-
mantic information from a given text.
4. We load the COGEX logic prover (Moldovan
et al., 2003) which operates by  reductio ad absur-
dum with H’s negated form and T’s predicates.
These clauses are weighted in the order in which
they should be chosen to participate in the search.
To ensure that H will be the last clause to partici-
pate, we assign it the largest value. The logic prover
searches for new inferences that can be made using
the smallest weight clauses. It also assigns a value
to each inference based on the axiom it used to de-
rive it. This process continues until the set of clauses
is empty. If a refutation is found, the proof is com-
plete. If a contradiction cannot be found, then the
predicate arguments are relaxed and, if the argument
relaxation fails, then predicates are dropped until a
proof by refutation is found. Its score will be com-
puted by deducting points for each argument relax-
ation and predicate removal. If this value falls below
a threshold, then T does not entail H. Otherwise, the
(T,H) pair is a true entailment.
We present a textual entailment example to show
the steps of our approach. This proof will not
372
T John and his son, George, emigrated with Mike, John’s uncle, to US in 1969.
LFT John(x1) ∧ son(x2) ∧ George(x3) ∧ ISA(x3, x2) ∧ KIN(x1, x3) ∧ emigrate(e1) ∧ AGT(x1, e1) ∧ AGT(x2, e1) ∧ Mike(x4) ∧
uncle(x5) ∧ ISA(x4, x5) ∧ KIN(x1, x4) ∧ US(x6) ∧ LOC(e1, x6) ∧ 1969(x7) ∧ TMP(e1, x7).
TDeparting [John and his son, George,]Theme.fe emigrated [with Mike, John’s uncle,]Cotheme.fe to [US]Goal.fe in [1969]Time.fe.
TKinship [John]Ego.fe and his son, [George,]Alter.fe emigrated with [Mike]Alter.fe, [John]Ego.fe’s uncle, to US in 1969.
TAxiom1 KIN(w1, w2) ↔ KIN(w2, w1)
KIN(x1, x3) → KIN(x3, x1) (KIN(John, George) → KIN(George, John))
TAxiom2 KIN ◦ KIN = KIN (KIN(w1, w2) ∧ KIN(w2, w3) → KIN(w1, w3))
KIN(x3, x1) ∧ KIN(x1, x4) → KIN(x3, x4) (KIN(George, Mike))
TAxiom3 DEPARTING F → LOC(Theme.fe, Goal.fe) (LOC(John, US) ∧ LOC(George, US))
TAxiom4 DEPARTING F → LOC(Cotheme.fe, Goal.fe) (LOC(Mike, US))
TSemantics KIN(John, George), KIN(John, Mike), KIN(George, Mike), LOC(John, US), LOC(George, US), LOC(Mike, US),
TMP(emigrate, 1969), AGT(John, emigrate), AGT(George, emigrate)
H George and his relative, Mike, came to America.
LFH George(x1) ∧ relative(x2) ∧ Mike(x3) ∧ ISA(x3, x2) ∧ KIN(x1, x3) ∧ come(e1) ∧ AGT(x1, e1) ∧ AGT(x2, e1) ∧ America(x4)
∧ LOC(e1, x2)
HArriving [George and his relative, Mike,]Theme.fe came to [America]Goal.fe.
HKinship [George]Ego.fe and his relative, [Mike]Alter.fe, came to America.
HAxiom1 ARRIVING F → LOC(Theme.fe, Goal.fe) (LOC(George, America) ∧ LOC(Mike, America))
HSemantics KIN(George, Mike), LOC(George, America), LOC(Mike, America)
Table 1: Entailment proof example. Table 2 lists the semantic relations and their abbreviations. Sections 3.2
and 4.1 will detail the semantics behind the axioms TAxiom1,TAxiom2,TAxiom3,TAxiom4, and HAxiom1.
make use of any world knowledge axioms. Let
the text T be John and his son, George, emigrated
with Mike, John’s uncle, to US in 1969 and the en-
tailed hypothesis H George and his relative, Mike,
came to America. Our system transforms each
text into its corresponding semantically enhanced
logic form (LFT and LFH in Table 1). Then,
the logic prover uses the newly added semantic ax-
ioms to derive extra semantic information from T
and H (for example, George and Mike are rela-
tives, but T does not explicitly specify this), af-
ter another preprocessing step which identi es the
frame elements of each frame encoded in the two
texts (TDeparting, TKinship, HArriving, HKinship).
In our example, the axioms TAxiom1 and TAxiom2
denote the symmetry and the transitivity of the KIN-
SHIP relation. TAxiom3, TAxiom4 and HAxiom1 are
the frame-related axioms used by the logic prover.
The TSemantics and HSemantics rows (Table 1) sum-
marize the meaning of T and H. We note that half of
these semantic instances were extracted using the se-
mantic axioms. Once the lexical chains between the
concepts in T and the ones from H are computed,
the entailment becomes straightforward. We repre-
sented, graphically, the meaning of the two texts in
Figure 1. We also show the links between the analo-
gous concepts that help prove the entailment.
In the coming sections of the paper, we detail the
process of semantic axiom generation. We start with
a summary of the axioms that combine two semantic
relations.
Figure 1: TSemantics and HSemantics. The solid ar-
rows represent the relations identi ed by the seman-
tic parser. The dotted arrows symbolize the lexical
chains between concepts in T and their analogous
concepts in H (UST and AmericaH belong to the
same WordNet synset). The dash arrows denote the
relations inferred by combining two semantic rela-
tions. The long dash arrows indicate the relations
between frame elements.
3 Semantic Calculus
3.1 Semantic relations
For this study, we adopt a revised version of the se-
mantic relation set proposed by (Moldovan et al.,
2004). Table 2 enumerates the semantic relations
that we consider2.
2See (Moldovan et al., 2004) for de nitions and examples.
373
POSSESSION (POS) MAKE-PRODUCE (MAK) RECIPIENT (REC) THEME-PATIENT (THM)
KINSHIP (KIN) INSTRUMENT (INS) FREQUENCY (FRQ) RESULT (RSL)
PROPERTY-ATTRIBUTE (PAH) LOCATION-SPACE (LOC) INFLUENCE (IFL) STIMULUS (STI)
AGENT (AGT) PURPOSE (PRP) ASSOCIATED WITH (OTH) EXTENT (EXT)
TEMPORAL (TMP) SOURCE-FROM (SRC) MEASURE (MEA) PREDICATE (PRD)
DEPICTION (DPC) TOPIC (TPC) SYNONYMY-NAME (SYN) CAUSALITY (CSL)
PART-WHOLE (PW) MANNER (MNR) ANTONYMY (ANT) JUSTIFICATION (JST)
HYPERNYMY (ISA) MEANS (MNS) PROBABILITY OF EXISTENCE (PRB) GOAL (GOL)
ENTAIL (ENT) ACCOMPANIMENT (ACC) POSSIBILITY (PSB) BELIEF (BLF)
CAUSE (CAU) EXPERIENCER (EXP) CERTAINTY (CRT) MEANING (MNG)
Table 2: The set of semantic relations
3.2 Combinations of two semantic relations
Our goal is to devise semantic axioms for combina-
tions of two relations, R1 and R2, by observing the
semantic connection between the w1 and w3 words
for which there exists at least one other word, w2,
such that R1(w1,w2) and R2(w2,w3) hold true3.
Harabagiu and Moldovan (1998) tackled the prob-
lem of semantic combinations, for the  rst time.
Their set of relations included the WordNet1.5 anno-
tations and 12 relationships derived from the Word-
Net glosses4. In our research, unlike (Harabagiu and
Moldovan, 1998), the semantic combinations use the
relations identi ed in text with a rather minimal con-
tribution from the WordNet relations.
Harabagiu and Moldovan (1998) also investi-
gate the number of possible semantic combinations.
Based on their properties, we can have up to eight
combinations between any two semantic relations
and their inverses, not counting the combinations
between a semantic relation and itself5. For in-
stance, given an asymmetric relation and a sym-
metric one which share the same part-of-speech for
their arguments, we can produce four combinations.
ISA ◦ ANT, ISA−1 ◦ ANT, ANT ◦ ISA, and ANT ◦
ISA−1 are the four possible distinct combinations
between HYPERNYMY and ANTONYMY.  ◦ sym-
bolizes the semantic composition between two rela-
tions compatible with respect to the part-of-speech
of their arguments: for any two concepts, w1 and w3,
(Ri◦Rj)(w1,w3) if and only if∃w2, a third concept,
such that Ri(w1,w2) and Rj(w2,w3) hold. By R−1,
3R(x,y) indicates that relation R holds between x and y.
4This set includes the AGENT, OBJECT, INSTRUMENT, BEN-
EFICIARY, PURPOSE, ATTRIBUTE, REASON, STATE, LOCA-
TION, THEME, TIME, and MANNER relations.
5Harabagiu and Moldovan (1998) lists the exact number of
possible combinations for several WordNet relations and part-
of-speech classes.
we denote the inverse of relation R: if R(x,y), then
R−1(y,x).
While analyzing the combinations, we observed
some regularities within the semantic composition
process. For example, R−11 ◦ R−12 = (R2 ◦ R1)−1
for any, not necessarily distinct, semantic relations
R1 and R26. If one of the relations is symmet-
ric (R−1 = R), the statement is still valid. Using
(R−1)−1 = R and the previous equality, we can re-
duce by half the number of semantic combinations
that we have to compute for R1 negationslash= R2.
We plan to create a 40 × 40 matrix with all the
possible combinations between any two semantic
relations from the set we consider. Theoretically,
we can have up to 27,556 semantic combinations,
but only 25.79% of them are possible7 (for exam-
ple, MNR(r,v) and SYN(n,n) cannot be combined).
Many combinations are not semantically signi cant
either because they are very rare, like, KIN(n,n)
◦ TMP(n,v), or because they do not result into
one of the 40 relations, for instance, PAH(a,n) ◦
AGT(n,v)8. We identi ed two approaches to the
problem mentioned above. The  rst tries to  ll one
matrix cell at a time in a consecutive manner. The
second approach tries to solve the semantic combi-
nations we come upon in text corpora. As a result,
we analyzed the RTE development corpus and we de-
vised rules for some of the Ri◦Rj combinations that
we encountered. We validated these axioms by man-
6The equality holds only if the two composition terms exist.
7On average, each semantic relation has 2.075 pairs of argu-
ments. For example, SRC can connect two nouns (US investor),
or an adjective and a noun (American investor) and, depending
on its arguments, SRC will participate in different combinations.
Out of the 27,556 combinations, only 7,109 are syntactically
possible.
8n, v, a, and r stand for noun, verb, adjective, and adverb,
respectively. As an example, R(n,n) means that relation R can
connect two nouns.
374
LOCATION ◦ PART-WHOLE = LOCATION (LOCATION(x,l1) ∧ PART-WHOLE(l1,l2) → LOCATION(x,l2))
Example: John lives in Dallas, Texas.
LOCATION(John,Dallas) and PART-WHOLE(Dallas,Texas) imply that LOCATION(John,Texas).
ISA ◦ ATTRIBUTE = ATTRIBUTE (ISA(x,y) ∧ ATTRIBUTE(y,a) → ATTRIBUTE(x,a))
Example: Mike is a rich man.
If ISA(Mike,man) and ATTRIBUTE(man,rich), then ATTRIBUTE(Mike,rich).
Similar statements can be made for other  attributes : LOCATION, TIME, SOURCE, etc.
ISA ◦ LOCATION = LOCATION (ISA(x,y) ∧ LOCATION(y,l) → LOCATION(x,l))
Example: The man in the car, George, is an old friend of mine.
ISA(George,man) and LOCATION(man,car) → LOCATION(George,car)
KINSHIP ◦ KINSHIP = KINSHIP (KINSHIP(x,y) ∧ KINSHIP(y,z) → KINSHIP(x,z))
See example in Section 2.
THEME ◦ ISA−1 = THEME (THEME(e,y) ∧ ISA(x,y) → THEME(e,x))
Example: Yesterday, John ate some fruits: an apple and two oranges.
THEME(eat,fruit) ∧ ISA(apple,fruit) → THEME(eat,apple)
THEME ◦ PART-WHOLE−1 = THEME (THEME(e,y) ∧ PART-WHOLE(x,y) → THEME(e,x))
Example: Five Israelis, including two children, were killed yesterday.
THEME(kill,Israeli) ∧ PART-WHOLE(child,Israeli) → THEME(kill,child)
Similar statements can be made for all the thematic roles: AGENT, EXPERIENCER, INSTRUMENT, CAUSE, LOCATION, etc.
AGENT ◦ ISA−1 = AGENT (AGENT(e,y) ∧ ISA(x,y) → AGENT(e,x))
AGENT ◦ PART-WHOLE−1 = AGENT (AGENT(e,y) ∧ PART-WHOLE(x,y) → AGENT(e,x))
Table 3: Examples of semantic combination axioms
ually checking all the LA Times corpus (w1,w3)
pairs which satisfy (Ri◦Rj)(w1,w3). We have iden-
ti ed 64 semantic axioms that show how semantic
relations can be combined. These axioms use re-
lations such as PART-WHOLE, ISA, LOCATION, AT-
TRIBUTE, or AGENT. We listed several example
rules in Table 3. The 64 axioms can be applied in-
dependent of the concepts involved in the semantic
composition. We have also identi ed rules that can
be applied only if the concepts that participate sat-
isfy a certain condition or if the relations are of a
certain type. For example, LOC ◦ LOC = LOC only if
the LOC relation shows inclusion (John is in the car
in the garage → LOC(John,garage). John is near
the car behind the garage negationslash→ LOC(John,garage)).
4 FrameNet Can Help
The Berkeley FrameNet project9 (Baker et al., 1998)
is a lexicon-building effort based on the theory of
frame semantics which de nes the meanings of lexi-
cal units with respect to larger conceptual structures,
called frames. Individual lexical units point to spe-
ci c frames and establish a binding pattern to spe-
ci c elements within the frame. FrameNet describes
the underlying frames for different lexical units and
examines sentences related to the frames using the
BNC corpus. The result is an XML database that
9http://framenet.icsi.berkeley.edu
contains a set of frames, a set of frame elements for
each frame, and a set of frame annotated sentences.
4.1 Frame-based semantic axioms
With respect to a given target, the frame ele-
ments contribute to the understanding of the sen-
tence. But they only link each argument to the
target word (for example, THM(theme,target)
or AGT(theme,target), LOC(place,target), etc.).
Often enough, we can  nd relations between the
frame elements of a given frame. These new in-
stances of semantic relations take as arguments the
frame elements of a certain frame, when they are ex-
pressed in the text. For example, given the DEPART-
ING frame, we can say that the origin of the theme
is the source (SRC(theme,source)) and that the
new location of the theme is the goal frame element
(LOC(theme,goal)). Moreover, if the text speci-
 es the cotheme frame element, then we can make
similar statements about it (SRC(cotheme,source)
and LOC(cotheme,goal)). These new relation in-
stances increase the semantic information that can
be derived from text.
So far, we manually inspected 54 frames and ana-
lyzed the relationships between their frame elements
by examining their de nitions and the annotated cor-
pus provided with the FrameNet data. For each
frame, we retained only the rules independent of the
375
CLOTHING PARTS F → PW(subpart,clothing)
CLOTHING PARTS F → PW(material,subpart)
Example:  Hello, Hank they said from the depths of the [fur]Material [collars]Subpart,Target of [their]Wearer [coats]Clothing.
PW(fur,collar) and PW(collar,coat)
CLOTHING F → PAH(descriptor,garment) ∨ PAH(descriptor,material)
Example: She didn’t bring heels with her so she decided on [gold]Descriptor [leather]Material [ ip- ops]Garment,Target.
PAH(gold,leather) ∨ PAH(gold,flip−flop)
KINSHIP F → KIN(ego,alter)
Example: The new subsidiary is headed by [Rupert Soames]Alter, [son]Target [of the former British Ambassador to France and
EC vice-president]Ego.
KIN(Rupert Soames, the former British Ambassador to France and EC vice-president)
GETTING F → POS(recipient,theme)
GETTING F →¬ POS(source,theme) (only if the source is a person)
Example: In some cases, [the BGS libraries]Recipient had [obtained]Target [copies of theses]Theme [from the authors]Source
[by purchase or gift]Means, and no loan records were available for such copies.
POS(the BGS libraries, copies of theses) and ¬ POS(authors, copies of theses)
GETTING F → SRC(theme,source) (if the source is not a person)
Example: He also said that [Iran]Recipient [acquired]Target [ ghter-bomber aircraft]Theme [from countries other than the USA
and the Soviet Union]Source.
SRC(fighter-bomber aircraft, countries other than the USA and the Soviet Union)
Table 4: Frames-related semantic rules
frame’s lexical units. We identi ed 132 semantic ax-
ioms that hold in most cases10. We show some ex-
amples in Table 4.
4.2 Context importance
There are cases when the rules that we identi ed
should not be applied. Let’s examine the sen-
tence John intends to leave the kitchen. If we
consider only the DEPARTING frame and its cor-
responding rules, without looking at the context,
then our conclusions (¬ LOC(John,kitchen) and
SRC(John,kitchen)) will be false. This sentence
states an intention of motion, not the actual action.
Therefore, our semantic axioms apply only when the
context they are in, allows it. To overcome this prob-
lem, we do not apply the axioms for target words
found in planning contexts, contexts related to be-
liefs, intentions, desires, etc. As an alternative, we
keep track of plans, intentions, desires, etc. and,
if, later on, we con rm them, then we apply the
semantic axioms. Also, when we analyze a sen-
tence, the frame whose rules we apply needs to be
chosen carefully. For example, in the sentence [A
boat]Agent [carrying]Target [would-be Moroccan il-
legal emigrants]Theme [from UK]Path start [to Spain
]Path end sank in the Strait of Gibraltar on June 8,
the CARRYING frame’s axioms do not apply. The
boat nor the emigrants reach Spain (the path end of
10Section 4.2 lists some exception cases.
the motion) because the boat sank. Here, the rules
given by sink.v’s frame should be given priority over
the carry.v’s rules. We can generalize and conclude
that, given a sentence that contains more than one
target (therefore, maybe multiple frames), the dom-
inant frame, the one whose rules should be applied,
is the frame given by the predicative verb. In the
previous sentence, the dominant frame is the one
given by sink.v and its rules should be applied before
the axioms of the CARRYING frame. It should be
noted that some of the axioms semantically related
to the CARRYING frame still apply (for example,
SRC(emigrants,UK) or SRC(boat,UK)). Unlike
LOC(emigrants,Spain), the previous relations do
not con ict with the semantics given by sink.v and
its location (the Strait of Gibraltar).
5 Experimental Results
5.1 The RTE data
The benchmark corpus for the RTE task consists of
seven subsets with a 50%-50% split between the
positive entailment examples and the negative ones.
Each subgroup corresponds to a different NLP appli-
cation: Information Retrival (IR), Comparable Doc-
uments (CD), Reading Comprehension (RC), Ques-
tion Answering (QA), Information Extraction (IE),
Machine Translation (MT), and Paraphrase Acquisi-
tion (PP). The RTE data set includes 1367 English
(T,H) pairs from the news domain (political, eco-
376
Semantic Axioms CD IE IR MT PP QA RC
T F T F T F T F T F T F T F
Test data (%)
applied to all Ts 13.33 21.33 26.66 10 6.66 4.44 11.66 10 8 0 15.38 7.69 21.43 17.14
applied to all Hs 1.33 9.33 5 10 0 0 1.66 1.66 8 0 1.53 0 0 1.43
solution for (T, H) 9.33 0 20 0 4.44 0 10 1.66 0 0 10.77 1.53 10 5.71
Development data (%)
applied to all Ts 22 27.08 34.28 5.71 8.57 8.57 18.51 18.51 5.12 9.3 28.88 0 9.61 9.8
applied to all Hs 4 8.33 5.71 2.85 0 2.85 7.4 3.7 0 0 2.22 0 0 1.96
solution for (T, H) 10 2.08 22.85 5.71 5.71 2.85 18.51 3.7 2.56 0 20 0 7.69 0
Table 5: The impact of the semantic axioms on each NLP application data set. T and F stand for True and
False entailments, respectively.
nomical, etc.). The development set consists of 567
examples and the test set contains the remaining 800
pairs.
5.2 Semantic axiom applicability
We measured the applicability of our set of semantic
rules, by counting the number of times they extract
new semantic information from text. Table 6 shows,
in percentages, the coverage of the semantic axioms
when applied to the texts T and the hypotheses H.
We also show the number of times the semantic rules
solve a (T,H) entailment without employing any
other type of axioms.
Semantic Axioms True False Overall
(True and False)
Test data (%)
applied to all Ts 15.75 11.75 13.75
applied to all Hs 2.00 3.74 2.87
both Ts and Hs 8.87 7.75 8.31
solution for (T,H) 10.25 1.50 5.87
Development data (%)
applied to all Ts 18.02 11.26 14.64
applied to all Hs 2.47 2.81 2.65
both Ts and Hs 10.25 7.04 8.64
solution for (T,H) 12.36 1.76 7.05
Table 6: Applicability on the RTE data
Clearly, because the texts T convey much more
information than H, they are the ones that bene t
the most from our semantic axioms. The hypotheses
H are more straightforward and a semantic parser
can extract all their semantic information. Also,
the rules tend to solve more positive (T,H) entail-
ments. Because there are seven subsets correspond-
ing to different NLP applications that make up the
RTE data, we analyzed the contribution of our se-
mantic axioms to each of the seven tasks. Table 5
shows the axioms’ impact on each type of data. The
logic-based approach proves to be useful to tasks
like Information Extraction, Reading Comprehen-
sion, or Comparable Documents, and it doesn’t seem
to be the right choice for the more lexical-orientated
applications like Paraphrase Acquisition, Machine
Translation, and Information Retrieval.
5.3 RTE performance
To show the impact of our semantic axioms, we
measured the contribution they bring to a system that
participated in the RTE challenge. The ACC and F
columns (Table 7) show the performance of the sys-
tem before and after we added our semantic rules to
the list of axioms needed by the logic prover.
Task Original Enhanced
ACC F ACC F
Test-IR .478 .472 .5 .505
Test-CD .78 .736 .847 .819
Test-RC .514 .558 .6 .636
Test-QA .485 .481 .523 .537
Test-IE .483 .603 .575 .687
Test-MT .542 .444 .567 .49
Test-PP .45 .585 .44 .576
Test .551 .561 .604 .621
Development .63 .619 .718 .714
Table 7: The accuracy(ACC) and f-measure(F) per-
formance values of our system
The results show that richer semantic connectiv-
ity between text concepts improve the performance
of a semantic entailment system. The overall accu-
racy increases with around 5% on the test data and
almost 8% on the development set. We obtained per-
formance improvements for all application settings,
except for the Paraphrase Acquisition task. For this
application, we obtained the smallest axiom cover-
age (Table 5). The impact of the semantic axioms
on each NLP application data set correlates with the
377
improvement that the addition of the rules brought
to the system’s accuracy.
Our error analysis showed that the system did not
take full advantage of our semantic axioms, because
the semantic parser did not identify all the seman-
tic relations needed as building blocks by the ax-
ioms. We noticed a signi cant decrease in the logic
prover’s usage of world-knowledge axioms.
6 Conclusion
In this paper, we present a logic-based semantic ap-
proach for the recognizing textual entailment task.
The system participating in the RTE competition
used a set of world-knowledge, NLP, and lexical
chain-based axioms and an in-house logic prover
which received as input the logic forms of the two
texts enhanced with semantic relation instances. Be-
cause the state-of-the-art semantic parsers cannot
extract the complete semantic information encoded
in text, the need for semantic calculus in NLP be-
came evident. We introduce semantic axioms that
either combine two semantic instances or label rela-
tions between the frame elements of a given frame.
Preliminary statistical results show that incorporat-
ing semantic rules into the logic prover can double
the semantic connectivity between the concepts of
the analyzed text. Our process of identifying more
semantic instances leads to a smaller dependency of
the logic-based RTE system on world knowledge ax-
ioms, while improving its overall accuracy.
References
Collin Baker, Fillmore Charles, and John Love. 1998.
The Berkeley FrameNet project. In Proceedings of
COLING/ACL.
Samuel Bayer, John Burger, Lisa Ferro, John Henderson,
and Alexander Yeh. 2005. MITRE’s Submissions to
the EU Pascal RTE Challenge. In Proceedings of the
PASCAL RTE Challenge.
Johan Bos and Katja Markert. 2005. Combining Shal-
low and Deep NLP Methods for Recognizing Textual
Entailment. In Proceedings of the PASCAL RTE Chal-
lenge.
Ido Dagan, Oren Glickman, and Bernardo Magnini.
2005. The PASCAL Recognising Textual Entailment
Challenge. In Proceedings of the PASCAL RTE Chal-
lenge.
Rodrigo de Salvo Braz, Roxana Girju, Vasin Pun-
yakanok, Dan Roth, and Mark Sammons. 2005. An
Inference Model for Semantic Entailment in Natural
Language. In Proceedings of the PASCAL RTE Chal-
lenge.
Abraham Fowler, Bob Hauser, Daniel Hodges, Ian Niles,
Adrian Novischi, and Jens Stephan. 2005. Applying
COGEX to Recognize Textual Entailment. In Pro-
ceedings of the PASCAL RTE Challenge.
Sanda Harabagiu and Dan Moldovan. 1998. Knowledge
Processing on Extended WordNet. In WordNet: an
Electronic Lexical Database and Some of its Applica-
tions.
Jess Herrera, Anselmo Peas, and Felisa Verdejo. 2005.
Textual Entailment Recognision Based on Depen-
dency Analysis and WordNet. In Proceedings of the
PASCAL RTE Challenge.
Valentin Jijkoun and Maarten de Rijke. 2005. Recogniz-
ing Textual Entailment Using Lexical Similarity. In
Proceedings of the PASCAL RTE Challenge.
Milen Kouylekov and Bernardo Magnini. 2005. Rec-
ognizing Textual Entailment with Tree Edit Distance
Algorithms. In Proceedings of the PASCAL RTE Chal-
lenge.
Dan Moldovan and Adrian Novischi. 2002. Lexical
Chains for Question Answering. In Proceedings of
COLING.
Dan Moldovan and Vasile Rus. 2001. Logic Form Trans-
formation of WordNet and its Applicability to Ques-
tion Answering. In Proceedings of ACL.
Dan Moldovan, Christine Clark, Sanda Harabagiu, and
Steve Maiorano. 2003. COGEX A Logic Prover
for Question Answering. In Proceedings of the
HLT/NAACL.
Dan Moldovan, Adriana Badulescu, Marta Tatu, Daniel
Antohe, and Roxana Girju. 2004. Models for the Se-
mantic Classi cation of Noun Phrases. In Proceed-
ings of HLT/NAACL, Computational Lexical Seman-
tics workshop.
Eamonn Newman, Nicola Stokes, John Dunnion, and Joe
Carthy. 2005. UCD IIRG Approach to the Textual
Entailment Challenge. In Proceedings of the PASCAL
RTE Challenge.
Rajat Raina, Aria Haghighi, Christopher Cox, Jenny
Finkel, Jeff Michels, Kristina Toutanova, Bill Mac-
Cartney, Marie-Catherine de Marneffe, Christopher
Manning, and Andrew Ng. 2005. Robust Textual
Inference using Diverse Knowledge Sources. In Pro-
ceedings of the PASCAL RTE Challenge.
378
