Proceedings of the Workshop on How Can Computational Linguistics Improve Information Retrieval?, pages 9–16,
Sydney, July 2006. c©2006 Association for Computational Linguistics
Hybrid Systems for Information Extraction and Question 
Answering 
Rodolfo Delmonte 
Ca' Bembo, San Trovaso 1075 
Università "Ca Foscari" 
30123 - VENEZIA 
Tel. 39-041-2345717/12 - Fax. 39-041-2345703 
E-mail: delmont@unive.it - website: project.cgm.unive.it 
 
Abstract 
Information Extraction, Sumarization and 
Question Answering al manipulate natural 
language texts and should benefit from the use 
of NLP techniques. Statistical techniques have 
til now outperformed symbolic procesing of 
unrestricted text. However, Information 
Extraction and Question Answering require by 
far more acurate results of what is curently 
produced by Bag-Of-Words aproaches. 
Besides, we se that such tasks as Semantic 
Evaluation of Text Entailment or Similarity – 
as required by the RTE Chalenge, impose a 
much stricter performance in semantic terms to 
tel true from false pairs. We wil speak in 
favour of a hybrid system, a combination of 
statistical and symbolic procesing with 
reference to a specific problem, that of 
Anaphora Resolution which loms large and 
dep in text procesing. 
1. Introduction 
Although ful syntactic and semantic analysis of open-
domain natural language text is beyond curent 
technology, a number of papers have ben recently 
published [1,2,3] showing that, by using probabilistic or 
symbolic methods, it is posible to obtain dependency-
based representations of unlimited texts with god recal 
and precision. Consequently, we believe it should be 
posible to augment the manual-anotation-based 
aproach with automaticaly built anotations by 
extracting a limited subset of semantic relations from 
unstructured text. In short, shalow/partial text 
understanding on the level of semantic relations, an 
extended label including Predicate-Argument Structures 
and other syntacticaly and semanticaly derivable head 
modifiers and adjuncts. This aproach is promising 
because it atempts to adres the wel-known 
shortcomings of standard “bag-of-words” (BOWs) 
information retrieval/extraction techniques without 
requiring manual intervention: it develops curent NLP 
technologies which make heavy use of statisticaly and 
FSA based aproaches to syntactic parsing. 
GETARUNS [4,5,6], a text understanding system (TUS), 
developed in colaboration betwen the University of 
Venice and the University of Parma,  can perform 
semantic analysis on the basis of syntactic parsing and, 
after performing anaphora resolution, builds a quasi 
logical form with flat indexed Augmented Dependency 
Structures (ADSs). In adition, it uses a centering 
algorithm to individuate the topics or discourse centers 
which are weighted on the basis of a relevance score. 
This logical form can then be used to individuate the best 
sentence candidates to answer queries or provide 
apropriate information. 
This paper is organized as folows: in section 2 below e 
discus why dep linguistic procesing is neded in 
Information Retrieval and Information Extraction; in 
section 3 we present GETARUNS, the NLP system and 
the Uper Module of GETARUNS; in section 4 we 
describe two experiments with state-of-the-art 
benchmark corpora. 
2 Ternary Expresions as Predicate-
Argument Structures 
Researchers like Lin, Katz and Litkowski have started to 
work in the direction of using NLP to populate a 
database of RDFs, thus creating the premises for the 
automatic creation of ontologies to be used in the IR/IE 
tasks. However, in no way RDFs and ternary expresions 
may constitute a formal tol suficient to expres the 
complexity of natural language texts. 
RDFs are asertions about the things (people, Webpages 
and whatever) they predicate about by aserting that they 
have certain properties with certain values. If we may 
agre with the fact that this is natural way of dealing with 
data handled by computers most frequently, it also a fact 
that this is not equivalent as being useful for natural 
language. The misconception sems to be deply 
embeded in the nature of RDFs as a whole: they are 
directly comparable to atribute-value pairs and DAGs 
which are also the formalism used by most recent 
linguistic unification-based gramars. From the logical 
and semantic point of view RDFs also resemble very 
closely first order predicate logic constructs: but we must 
remember that FOPL is as such insuficient to describe 
natural language texts. 
Ternary expresions(T-expresions), <subject relation 
object>. 
Certain other parameters (adjectives, posesive nouns, 
prepositional phrases, etc.) are used to create aditional 
T-expresions in which prepositions and several special 
words may serve as relations. For instance, the folowing 
simple sentence 
 
(1) Bil surprised Hilary with his answer 
9
 
wil produce two T-expresions: 
 
(2) <Bil surprise Hilary> with answer> 
  <answer related-to Bil> 
 
In Litkowski’s system the key step in their question-
answering prototype was the analysis of the parse tres to 
extract semantic relation triples and populate the 
databases used to answer the question. A semantic 
relation triple consists of a discourse entity, a semantic 
relation which characterizes the entity's role in the 
sentence, and a governing word to which the entity 
stands in the semantic relation. The semantic relations in 
which entities participate are intended to capture the 
semantic roles of the entities, as generaly understod in 
linguistics. This includes such roles as agent, theme, 
location, maner, modifier, purpose, and time. Surogate 
place holders included are "SUBJ," "OBJ", "TIME," 
"NUM," "ADJMOD," and the prepositions heading 
prepositional phrases. The governing word was generaly 
the word in the sentence that the discourse entity stod in 
relation to. For "SUBJ," "OBJ," and "TIME," this was 
generaly the main verb of the sentence. For prepositions, 
the governing word was generaly the noun or verb that 
the prepositional phrase modified. For the adjectives and 
numbers, the governing word was generaly the noun that 
was modified. 
2.1 Ternary Expressions are better than the 
BOWs approach, but… 
People working advocating the supremacy of the Tes 
aproach were reacting against the Bag of Words 
aproach of IR/IE in which words were wrongly 
regarded to be entertaining a meaningful relation simply 
on the basis of topological criteria: normaly the distance 
criteria or the more or les proximity betwen the words 
to be related. Intervening words might have already ben 
discarded from the input text on the basis of stopword 
filtering. Stopwords list include al gramatical close 
type words of the language considered useles for the 
main purpose of IR/IE practitioners sen that they canot 
be used to denote concepts. Stopwords constitute what is 
usualy regarded the noisy part of the chanel in 
information theory. However, it is just because the 
redundancy of the information chanel is guaranted by 
the presence of gramatical words that the mesage gets 
apropriately computed by the subject of the 
comunication proces, i.e. human beings. Besides, 
entropy is not to be computed in terms of number of 
words or leters of the alphabet, but in number of 
semantic and syntactic relation entertained by open clas 
words (nouns, verbs, adjectives, adverbials) basicaly by 
virtue of closed clas words. Redundancy should then be 
computed on the basis of the ambiguity intervening when 
enumerating those relations, a very hard task to 
acomplish which has never ben atemped yet, at least 
to my knowledge. 
What people working with TEs noted was just the 
problem of encoding relations apropriately, at least 
some of these relations. The IR/IE BOWs aproach 
sufers (at least) from Reversible Arguments Problem 
(se [7]) 
- What do frogs eat? vs What eats frogs? 
The verb “eat” entertains asymetrical relations with its 
SUBJect and its OBJect: in one case we talk of the 
“eater”, the SUBJect and in another case of the “eate”, 
the OBJect. Other similar problems ocur with TEs when 
the two elements of the relation have the same head, as 
in: 
-The president of Rusia visited the president of China. 
Who visited the president? 
The question wil not be properly answered in lack of 
some clarification dialogue intervening, but the 
coresponding TEs should have more structure to be able 
to represent the internal relations of the two presidents. 
The asymetry of relation in transitive constructions 
involving verbs of acomplishments and achievements 
(or simply world-changing events) is however further 
complicated by a number of structural problems which 
are typicaly found in most languages of the world, the 
first one and most comon being Pasive constructions: 
i.John kiled Tom. 
i.Tom was kiled by a man. 
Who kiled the man? 
Answer to the question would be answered by “John” in 
case the information available was represented by 
sentence in i., but it would be answered by “Tom” in case 
the information available was represented by sentence i. 
Obviously this would hapen only in lack of suficient 
NLP elaboration: a to shalow aproach would not be 
able to capture presence of a pasive structure. We are 
here refering to “Chunk”-based aproaches those in 
which the object of computation is constituted by the 
creation of Noun Phrases and no atempt is made to 
compute clause-level structure. 
There is a certain number of other similar structure in 
texts which must be regarded as inducing into the same 
type of miscomputation: i.e. taking the surface order of 
NPs as indicating the dep intended meaning. In al of 
the folowing constructions the surface subject is on the 
contrary the dep object thus the Afected Theme or 
argument that sufers the efects of the action expresed 
by the governing verb rather than the Agent: 
 
Inchoatized structures; Ergativized structures; 
Impersonal structures 
 
Other important and typical structures which constitute 
problematic cases for a surface chunks based TEs 
aproach to text computation are the folowing ones in 
which one of the arguments is mising and Control 
should be aplied by a governing NP, they are caled in 
one definition Open Predicative structures and they are 
 
Relative clauses; Fronted Adjectival adjunct clauses; 
Infinitive clauses; Fronted Participial clauses,; 
Gerundive Clauses; Eliptical Clauses; Cordinate 
constructions 
 
In adition to that there is one further problem and is 
definable as the Factuality Prejudice: by colecting 
10
keywords and TEs people aply a Factuality 
Presuposition to the text they are mining: they believe 
that al terms being recovered by the search represent real 
facts. This is however not true and the problem is related 
to the posibility to detect in texts the presence of such 
semantic indicators as those listed here below: 
 
Negation; Quantification; Opaque contexts (wish, 
want); Future, Subjunctive Mode; Modality; 
Conditionals 
 
Finaly there is a discourse related problem and is the 
Anaphora Resolution problem which is the hardest to 
be tackled by NLP: it is a fact that anaphoric relations are 
the building blocks of cohesivenes and coherence in 
texts. Whenever an anaphoric link is mised one relation 
wil be asigned to a wrong refering expresion thus 
presumably jeopardising the posibility to answer a 
related question apropriately. This is we believe the 
most relevant topic to be put forward in favour of the 
ned to have symbolic computational linguistic 
procesing (besides statistical procesing). 
3 GETARUNS – the NLUS 
GETARUN, the System for Natural Language 
Understanding, produces a semantic representation in 
xml format, in which each sentence of the input text is 
divided up into predicate-argument structures where 
arguments and adjuncts are related to their apropriate 
head. Consider now a simple sentence like the folowing: 
(1) John went into a restaurant 
GETARUNS represents this sentence in diferent 
maners acording to whether it is operating in Complete 
or in Shalow modality. In turn the operating modality is 
determined by its ability to compute the curent text: in 
case of failure the system wil switch automaticaly from 
Complete to Partial/Shalow modality. 
The system wil produce a representation inspired by 
Situation Semantics[14] where reality is represented in 
Situations which are colections of Facts: in turn facts are 
made up of Infons which are information units 
characterised as folows: 
  Infon(Index, 
 Relation(Property), 
 List of Arguments - with Semantic Roles, 
 Polarity - 1 afirmative, 0 negation, 
 Temporal Location Index, 
 Spatial Location Index) 
In adition each Argument has a semantic identifier 
which is unique in the Discourse Model and is used to 
individuate the entity uniquely. Also propositional facts 
have semantic identifiers asigned, thus constituting 
second level ontological objects. They may be 
“quantified” over by temporal representations but also by 
discourse level operators, like subordinating conjunctions 
and a performative operator if neded. Negation on the 
contrary is expresed in each fact. 
In case of failure at the Complete level, the system wil 
switch to Partial and the representation wil be deprived 
of its temporal and spatial location information. In the 
curent version of the system, we use Complete modality 
for tasks which involve short texts (like the students 
sumaries and text understanding queries), where text 
analyses may be supervisioned and updates to the 
gramar and/or the lexicon may be neded. For 
unlimited text from the web we only use partial modality. 
Evaluation of the two modalities are reported in a section 
below. 
3.1 The Parser and the Discourse Model 
As said above, the query building proces neds an 
ontology which is created from the translation of the 
Discourse Model built by GETARUNS in its 
Complete/Partial Representation. GETARUNS, is 
equiped with thre main modules: a lower module for 
parsing where sentence strategies are implemented; a 
midle module for semantic interpretation and discourse 
model construction which is cast into Situation 
Semantics; and a higher module where reasoning and 
generation takes place. The system works in Italian and 
English. 
Our parser is a rule-based deterministic parser in the 
sense that it uses a lokahead and a Wel-Formed 
Substring Table to reduce backtracking. It also 
implements Finite State Automata in the task of tag 
disambiguation, and produces multiwords whenever 
lexical information alows it. In our parser we use a 
number of parsing strategies and graceful recovery 
procedures which folow a strictly parameterized 
aproach to their definition and implementation. A 
shalow or partial parser is also implemented and always 
activated before the complete parse takes place, in order 
to produce the default baseline output to be used by 
further computation in case of total failure. In that case 
partial semantic maping wil take place where no 
Logical Form is being built and only refering 
expresions are aserted in the Discourse Model – but se 
below. 
3.2 Lexical Information 
The output of gramatical modules is then fed onto the 
Binding Module(BM) which activates an algorithm for 
anaphoric binding in LFG (se [13]) terms using f-
structures as domains and gramatical functions as entry 
points into the structure. We show here below the 
architecture of the system. The gramar is equiped with 
a lexicon containing a list of 300 wordforms derived 
from Pen Trebank. 
However, morphological analysis for English has also 
ben implemented and used for OV words. The system 
uses a core fuly specified lexicon, which contains 
aproximately 10,00 most frequent entries of English. 
In adition to that, there are al lexical forms provided by 
a fuly revised version of COMLEX. In order to take into 
acount phrasal and adverbial verbal compound forms, 
we also use lexical entries made available by UPen and 
TAG encoding. Their gramatical verbal syntactic codes 
have then ben adapted to our formalism and is used to 
generate an aproximate subcategorization scheme with 
an aproximate aspectual clas asociated to it. 
11
 
 
Fig. 1. GETARUNS’ LFG-Based Parser 
 
Fig. 2. GETARUNS’ Discourse Level Modules
 
Semantic inherent features for Out of Vocabulary words, 
be they nouns, verbs, adjectives or adverbs, are provided 
by a fuly revised version of WordNet – 270,00 lexical 
entries - in which we used 75 semantic clases similar to 
those provided by CoreLex. Subcategorization 
information and Semantic Roles are then derived from a 
carefuly adapted version of FrameNet and VerbNet. Our 
“training” corpus is made up of 20,00 words and 
contains a number of texts taken from diferent genres, 
portions of the UPen Trebank corpus, test-suits for 
gramatical relations, and sentences taken from 
COMLEX manual. An evaluation caried out on the 
Susan Corpus related GREVAL testsuite made of 50 
sentences has ben reported lately [12] to have achieved 
90% F-measure over al major gramatical relations. We 
achieved a similar result with the shalow cascaded 
parser, limited though to only SUBJect and OBJect 
relations on LFG-XEROX 70 corpus. 
3.3 The Upper Module 
GETARUNS, as shown in Fig.2 has a linguisticaly-
based semantic module which is used to build up the 
Discourse Model. Semantic procesing is strongly 
modularized and distributed amongst a number of 
diferent submodules which take care of Spatio-
Temporal Reasoning, Discourse Level Anaphora 
Resolution, and other subsidiary proceses like Topic 
Hierarchy which wil impinge on Relevance Scoring 
when creating semantic individuals. These are then 
aserted in the Discourse Model (hence the DM), which 
is then used to solve nominal coreference together with 
WordNet. Semantic Maping is performed in two steps: 
at first a Logical Form is produced which is a structural 
maping from DAGs onto of unscoped wel-formed 
formulas. These are then turned into situational 
semantics informational units, infons which may become 
facts or sits. 
In each infon, Arguments have each a semantic identifier 
which is unique in the DM and is used to individuate the 
entity. Also propositional facts have semantic identifiers 
asigned thus constituting second level ontological 
objects. They may be “quantified” over by temporal 
representations but also by discourse level operators, like 
subordinating conjunctions. Negation on the contrary is 
expresed in each fact. Al entities and their properties 
are aserted in the DM with the relations in which they 
are involved; in turn the relations may have modifiers - 
sentence level adjuncts and entities may also have 
modifiers or atributes. Each entity has a polarity and a 
couple of spatiotemporal indices which are linked to 
main temporal and spatial locations if any exists; else 
they are linked to presumed time reference derived from 
tense and aspect computation. Entities are maped into 
semantic individuals with the folowing ontology: on first 
ocurence of a refering expresion it is aserted as an 
INDividual if it is a definite or indefinite expresion; it is 
aserted as a CLAS if it is quantified (depending on 
quantifier type) or has no determiner. Special individuals 
are ENTs which are asociated to discourse level 
anaphora which bind relations and their arguments. 
Finaly, we have LOCs for main locations, both spatial 
and temporal. Whenever there is cardinality determined 
by a digit, its number is plural or it is quantified 
(depending on quantifier type) the refering expresion is 
aserted as a SET. Cardinality is simply infered in case 
of naked plural: in case of colective nominal expresion 
it is set to 10, otherwise to 5. On second ocurence of 
the same nominal head the semantic index is recovered 
from the history list and the system checks whether it is 
the same refering expresion: 
- in case it is definite or indefinite with a predicative role 
and no atributes nor modifiers, nothing is done; 
- in case it has diferent number - singular and the one 
present in the DM is a set or a clas, nothing hapens; 
- in case it has atributes and modifiers which are 
diferent and the one present in the DM has none, 
nothing hapens; 
- in case it is quantified expresion and has no 
cardinality, and the one present in the DM is a set or a 
clas, again nothing hapens. 
In al other cases a new entity is aserted in the DM 
which however is also computed as being included in (a 
superset of) or by (a subset of) the previous entity. 
The uper module of GETARUNS has ben evaluated on 
the basis of its ability to perform anaphora resolution and 
to individuate refering expresions, with a corpus of 
40,00 words: it achieved 74% F-measure. 
 
12
4. Two experiments with GETURANS 
As an example of the shalow system we discus here 
below the analysis of a newspaper article which as would 
usualy be the case has a certain number of pronominal 
expresions, which modify the relevance of lexical 
descriptions in the overal procesing for the search of 
either “Named Entities” or simply entities individuated 
by comon nouns. If the count is based solely on lexical 
lemata and not on the presence of coreferential 
pronominal expresions, the results wil be heavily 
biased and certainly wrong. Here is the text: 
 
1.Thursday, 25th June 201 
National Parties and the Internet 
by Joana Crawford 
2.A survey of how national parties used the internet as a 
campaigning tol during the election wil brand their eforts 
"bleak and dispiriting" - despite the pre-campaign hype of an 
"e-election". 
3.Researchers from Salford University studied websites from 
al the major parties during the general election, as wel as 
loking at every site put up by local candidates. 
4.Their conclusions - to be presented tomorow at a special 
conference organised by the Institute for Public Policy 
Research - could influence how future political contests, 
including the forthcoming Euro debate, are caried out on the 
web. 
5.The report finds that none of the major thre parties alowed 
mesage boards or chat roms for users to post their opinions 
on the sites. 
6.It states: "Parties were acused of simply engaging in online 
propaganda with boring content and largely ignoring 
interactivity." 
7.The report concludes: "The new media is a way for them to 
get closer to the public without necesarily alowing the public 
to become overly familiar in return. 
8.The authors - Rachel Gibson and Stephen Ward - go on to 
state that this may be because parties stil regard the web as an 
electionering tol, rather than as a democratic device. 
9.They said: "Very few ofered original material, or changed 
their sites noticeably over the course of the campaign. 
10.Inded, a large majority of local sites were realy no more 
than static electronic brochures." 
1.They dub this "rather disapointing", but praise the Liberal 
Democrats as "clearly the most active" with around 150 sites. 
The report concludes: "Parties, as with the general public, ned 
incentives to use the technology. 
12.As yet, there sems more to lose and les to gain if they 
make mistakes experimenting with the technology." 
 
We highlighted pronominal expressions in bold. In 
a BOWs approach, the count for most relevant 
topics is solely based on lexical descriptions and 
“party, internet” are computed as the most 
important key-words. However, after the text has 
been passed by the partial semantic analysis, 
“researcher, author” come up as important topics. 
We report here below the output of the Anaphora 
Resolution module: in interaction with the 
Discourse Model where semantic indices are 
asserted for each entity. Sentence numbers are taken 
from the text. We report Anaphora Resolution 
decisions: in particular in sentences where a 
pronoun is coreferred to an antecedent, the 
antecedent is set as current Main Topic and its 
semantic ID is used. 
1. state(1, change) 
topics: main:party, secondary: internet 
topics(1, main, id1; secondary, id2; potential, id3) 
2. state(2, continue) 
topics: main:party, secondary: survey 
topics(2, main, id1; secondary, id7; potential, id2) 
3. state(3, retaining) 
topics: main: researcher, secondary: party 
topic(3, main, id18; secondary, id1; , id19) 
4. Anaphora Resolution: their resolved as researcher 
state(4, continue) 
topics: main: researcher, secondary: contest 
topics(4, main, id18; secondary, id26; potential, id27) 
5. state(5, retaining) 
topics: main: report, secondary: researcher 
topics(5, main, id7; secondary, id18; potential, id1) 
6. Anaphora Resolution: it resolved as report 
state(6, continue) 
topics: main: report, secondary: party 
topics(6, main, id7; secondary, id1; potential, id40) 
7. state(7, continue) 
topics: main: report, secondary: party 
topics(7, main, id7; secondary, id1; potential, id2) 
8. The authors - Rachel Gibson and Stephen Ward - go 
on to state that this may be because parties stil regard the 
web as an electionering tol, rather than as a democratic 
device. 
Anaphora Resolution: this resolved as 'discourse bound' 
state(8, retaining) 
topics: main: author, secondary: report 
topics(8, main, id54; secondary, id7; potential, id5) 
9. Anaphora Resolution: they resolved as author 
state(9, continue) 
topics: main: author, secondary: material 
topics(9, main, id54; secondary, id61; potential, id62) 
10. state(10, continue) 
topics: main: author, secondary: site 
topics(10, main, id54; secondary, id67; potential, id68) 
1. Anaphora Resolution: this  resolved as  'discourse 
bound'; they resolved as author 
state(1, retaining) 
topics: main: author, secondary: active 
topics(1, main, id54; secondary, id71; potential, id72) 
12. Anaphora Resolution: they resolved as party 
state(12, continue) 
topics: main: party, secondary: mistake 
topics(12, main, id1; secondary, id78) 
4.1 The First Experiment: Anaphora Resolution 
in Technical Manuals 
We downloaded the only frely available corpus 
anotated with anaphoric relations, i.e. Wolverhampton’s 
Manual Corpus made available by Prof. Ruslan Mitkov 
on his website. The corpus contains text from Manuals at 
the folowing adres, 
htp:/clg.wlv.ac.uk/resources/corpus.html 
13
 
 
Text Type Referring 
Exps 
Coreferring 
Exps 
Total 
Words 
AIWA 1629 716 6818 
ACCESS 1862 513 9381 
PANASONIC 1263 537 4829 
HINARI 673 292 2878 
URBAN 453 81 22 
WINHELP 672 206 2935 
CDROM 194 279 10568 
Totals 8496 2624 39631 
Table 2. General data of Worlverhampton’s 
coreference annotated corpora 
 
 
Text Type Refering 
Exps % W 
Corefering 
Exps % RE 
AIWA 23.89 43.21 
ACCESS 19.84 27.01 
PANASONIC 26.15 42.51 
HINARI 23.38 29,2 
URBAN 20.38 17.8 
WINHELP 2.89 27.14 
CDROM 18.39 14.24 
Means 21.43 30.8 
Table 3. Proportion of coreferential expressions to 
referring expressions 
 
 
Fig. 3. Comparing GETARUNS output to WMC 
 
 
We reported in Tab. 2 the general data of the Coreference 
Corpus. As can be easily noted, there is no direct 
relationship existing betwen the number of refering 
expresions and the number of corefering expresions. 
We asume that the higher the number of corefering 
expresions in a text the higher is the cohesion achieved. 
Thus the text identified as CDROM has a very smal 
number of corefering expresions if compared to the 
total number of refering expresions. The proportion of 
refering expresions to words and of corefering 
expresions to refering expresions is reported in percent 
value in table 3. where the most highly cohesive texts are 
highlighted in italics; highly non cohesive texts are 
highlighted in bold: 
The final results are reported in the folowing figure 
where we plot Precision and Recal for each text and then 
the comprehensive values. 
 
 
Fig. 4. Precision and Recal for the WMC 
 
4.2 GETARUNS approach to WEB-Q/A 
Totaly shalow aproaches when compared to ours wil 
always be lacking suficient information for semantic 
procesing at propositional level: in other words, as 
hapens with our “Partial” modality, there wil be no 
posibility of checking for precision in producing 
predicate-argument structures. 
Most systems would use some Word Matching algorithm 
to count the number of words apearing in both question 
and the sentence being considered after striping 
stopwords: usualy two words wil match if they share 
the same morphological rot after some steming has 
taken place. Most QA systems presented in the literature 
rely on the clasification of words into two clases: 
function and content words. They don't make use of a 
Discourse Model where input text has ben transformed 
via a rigorous semantic maping algorithm: they rather 
aces taged input text in order to sort best matched 
words, phrases or sentences acording to some scoring 
function. It is an acepted fact that introducing or 
increasing the amount of linguistic knowledge over crude 
IR-based systems wil contribute substantial 
improvements. In particular, systems based on simple 
Named-Entity identification tasks are to rigid to be able 
to match phrase relations constraints often involved in a 
natural language query. 
We raise a number of objections to these aproaches: 
first objection is the imposibility to take into acount 
pronominal expresions, their relations and properties as 
belonging to the antecedent, if no head transformation 
has taken place during the analysis proces. 
Another objection comes from the treatment of the 
Question: it is usualy the case that QA systems divide 
the question to be answered into two parts: the Question 
14
Target represented by the wh- word and the rest of the 
sentence; otherwise the words making up the yes/no 
question are taken in their order, and then a match takes 
place in order to identify most likely answers in relation 
to the rest/whole of the sentence except for stopwords. 
However, it is just the semantic relations that ned to be 
captured and not just the words making up the question 
that mater. Some systems implemented more 
sophisticated methods (notably [8;9;10]) using syntactic-
semantic question analysis. This involves a robust 
syntactic-semantic parser to analyze the question and 
candidate answers, and a matcher that combines word- 
and parse-tre-level information to identify answer 
pasages more precisely. 
4.3 A Prototype Q/A system for the web 
We experimented our aproach over the web using 450 
factoid questions from TREC. On a first run the base 
system only used an of-the-shelf tager in order to 
recover main verb from the query. In this way we 
managed to get 67% corect results, by this meaning that 
the corect answer was contained in the best five snipets 
selected by the BOWs system on the output of Gogle 
API. However, only 30% of the total corect results had 
the right snipet ranked in position one. 
Then we aplied GETARUNS shalow on the best five 
snipets with the intent of improving the automatic 
ranking of the system and have the best snipet always 
position as first posibility. Here below is a figure 
showing the main components for GETARUNS based 
analysis. 
We wil present two examples and discus them in some 
detail. The questions are the folowing ones: 
Q: Who was elected president of South Africa in 194? 
 A: Nelson Mandela 
Q: When was Abraham Lincoln born? 
 A: Lincoln was born February_12_1809 
The answers produced by our system are indicated after 
each question. Now consider the best five snipets as 
filtered by the BOWs system: 
 
 
Fig. 5. System Architecture for QA 
 
who/WP was/VBD elected/VBN president/N of/IN south/J 
africa/N in/IN 194/CD 
Main keywords: president south africa 194 
Verb rots: elect 
Gogle search: elected president south africa 194 
1.On June 2, 199, Mbeki, the pragmatic deputy 
president of South Africa and leader of the 
African National Congres, was elected president 
in a landslide, having already asumed many of 
Mandela's governing responsibilities shortly 
after Mandela won South Africa's first 
democratic election in 194. 
2.Washington ? President Bil Clinton anounced 
yesterday a doubling in US asistance South 
Africa of $60-milion (R2 160-milion) over 
thre years, and said his wife Hilary would 
atend Nelson Mandela's inauguration as the 
country's first black president. 
3.Nelson Mandela, President of the African 
National Congres (ANC), casting the balot in 
his country's first al-race elections, in April 
194 at Ohlange High Schol near Durban, South 
Africa. 
4.Newly-elected President Nelson Mandela 
adresing the crowd from a balcony of the Town 
Hal in Pretoria, South Africa on May 10, 194. 
5.The CDF boycoted talks in King Wiliam's Town 
yesterday caled by the South African government 
and the Transitional Executive Council to smoth 
the way for the peaceful reincorporation of the 
homeland into South Africa folowing the 
resignation of Oupa Gqozo as president. 
 
Notice snipet n.1 where two presidents are present and 
two dates are reported for each one: however the relation 
“president” is only indicated for the wrong one, Mbeki 
and the system rejects it. The answer is colected from 
snipet no.4 instead. As a mater of fact, after computing 
the ADM, the system decides to rerank the snipets and 
use the contents of snipet 4 for the answer. Now the 
second question: 
 
when/WRB was/VBD abraham/N lincoln/N born/VBN 
Main keywords: abraham lincoln 
Verb rots: bear  
Gogle search: abraham lincoln born 
1. Abraham Lincoln was born in a log cabin in 
Kentucky to Thomas and Nancy Lincoln. 
2. Two months later on February 12, 1809, 
Abraham Lincoln was born in a one-rom log cabin 
near the Sinking Spring. 
3. Abraham Lincoln was born in a log cabin near 
Hodgenvile, Kentucky. 
4.Lincoln himself set the date of his birth at 
feb_ 12, 1809, though some have atempted to 
disprove that claim . 
5. A. Lincoln ( February 12, 1809 April 15, 1865 
) was the 16/th president of the United States 
of America. 
 
In this case, snipet n.2 is selected by the system as the 
one containing the required information to answer the 
question. In both cases, the answer is built from the 
ADM, so it is not precisely the case that the snipets are 
selected for the answer: they are nonetheles reranked to 
make the answer available. 
5. System Evaluation 
After runing with GETARUNS, the 450 questions 
recovered the whole of the original corect result 67% 
from first snipet. 
The complete system has ben tested with a set of texts 
derived from newspapers, narative texts, children 
stories. The performance is 75% corect. However, 
updating and tuning of the system is required for each 
15
new text whenever a new semantic relation is introduced 
by the parser and the semantics does not provide the 
apropriate maping. For instance, consider the case of 
the constituent "holes in the tre", where the syntax 
produces the apropriate structure but the semantics does 
not map "holes" as being in a LOCATion semantic 
relation with "tre". In lack of such a semantic role 
information a dumy "MODal" wil be produced which 
however wil not generate the adequate semantic 
maping in the DM and the meaning is lost. 
As to the partial system, it has ben used for DUC 
sumarization contest, i.e. it has run over aproximately 
1 milion words, including training and test sets, for a 
number of sentences totaling over 50K. We tested the 
"Partial" modality with an aditional 90,00 words texts 
taken from the testset made available by DUC 202 
contest. On a preliminary perusal of samples of the 
results, we calculated 85% Precision on parsing and 70% 
on semantic maping. However evaluating ful results 
requires a manualy anotated database in which al 
linguistic properties have ben carefuly decided by 
human anotators. In lack of such a database, we are 
unable to provide precise performance data. The system 
has also ben used for the RTE Chalenge and 
performance was over 60% corect [1]. 
6. Conclusions 
Results reported in the experiment above have ben 
limited to the ability of the system to cope with what has 
always ben regarded as the toughest task for an NLP 
system to cope with. We have not adresed the problem 
of question answering for lack of space. 
Would it be posible for computers the recognize the 
layout of a Web page, much in the same maner as a 
human? Much like the development of the Semantic Web 
itself, early eforts to integrate natural language 
technology with the Semantic Web wil no doubt be slow 
and incremental. By weaving natural language into the 
basic fabric of the Semantic Web, we can begin to create 
an enormous network of knowledge easily acesible by 
both machines and humans alike. Furthermore, we 
believe that natural language querying capabilities wil 
be a key component of any future Semantic Web system. 
By providing “natural” means for creating and acesing 
information on the Semantic Web, we can dramaticaly 
lower the barier of entry to the Semantic Web. Natural 
language suport gives users a whole new way of 
interacting with any information system, and from a 
knowledge enginering point of view, natural language 
technology divorces the majority of users from the ned 
to understand formal ontologies. As we have tried to 
show in the paper, this cals for beter NLP tols where a 
lot of efort has to be put in order to alow for complete 
and shalow techniques to coalesce smothly into one 
single system. GETARUNS represents such a hybrid 
system and its performance is steadily improving. 
In the future we intend to adres the problem of using 
the database of TEs created by our system in asnswering 
a more extended set of natural language queries than 
what has ben tried sofar. 

References 
1. Dan Klein and Christopher D. Maning: Acurate 
Unlexicalized Parsing. ACL, (203) 423-430 
2. D. Lin.: Dependency-based evaluation of MINIPAR. In 
Procedings of the Workshop on Evaluation of Parsing 
Systems at LREC 198. Granada, Spain, (198) 
3. Sleator, Daniel, and Davy Temperley: "Parsing English with 
a Link Gramar." Procedings of IWPT ’93, (193) 
4. Delmonte R.: Parsing Preferences and Linguistic Strategies, 
in LDV-Forum - Zeitschrift fuer Computerlinguistik und 
Sprachtechnologie - "Comunicating Agents", Band 17, 
1,2, (200) 56-73 
5. Delmonte R.: Parsing with GETARUN, Proc.TALN200, 7° 
confèrence anuel sur le TALN, Lausane, (200) 13-146 
6. Delmonte R., D. Bianchi: From Dep to Partial 
Understanding with GETARUNS, Proc. ROMAND 202, 
Università Roma2, Roma, (202) 57-71 
7. Boris Katz, Jimy J. Lin, Sue Felshin: The START 
Multimedia Information System: Curent Technology and 
Future Directions, In Procedings of the International 
Workshop on Multimedia Information Systems (MIS 
202) 
8. Hovy, E., U. Hermjakob, & C. Lin.: The Use of External 
Knowledge in Factoid QA. In E. M. Vorhes & D. K. 
Harman (eds.), The Tenth Text Retrieval Conference 
(TREC 201). (202) 64-652 
9. Litkowski, K. C.: Syntactic Clues and Lexical Resources in 
Question-Answering. In E. M. Vorhes & D. K. Harman 
(eds.), The Ninth Text Retrieval Conference (TREC-9). 
(201) 157-16 
10. Litkowski, K. C.: CL Research Experiments in TREC-10 
Question-Answering. In E. M. Vorhes & D. K. Harman 
(eds.), The Tenth Text Retrieval Conference (TREC 201). 
(202) 12-131 
1. Delmonte R., Sara Toneli, Marco Aldo Picolino Boniforti, 
Antonela Bristot, Emanuele Pianta: VENSES – a 
Linguisticaly-Based System for Semantic Evaluation, RTE 
Chalenge Workshop, Southampton, PASCAL - European 
Network of Excelence, (205) 49-52 
12. Delmonte R.: Evaluating GETARUNS Parser with 
GREVAL Test Suite, Proc. ROMAND - 20th International 
Conference on Computational Linguistics - COLING, 
University of Geneva, (204) 32-41. 
13. Bresnan J.(ed.): The Mental Representation of 
Gramatical Relations, IT Pres, Cambridge 
Mas., 1982) 
14. Barwise J., J.M.Gawron, G.Plotkin, S.Tutiya(eds.): 
Situation Theory and its Aplications, Vol.2, CSLI 
Lecture Notes No.26, (191) 
