Answer Mining from On-Line Documents
Marius Pas¸ca and Sanda M. Harabagiu
Department of Computer Science and Engineering
Southern Methodist University
Dallas, TX 75275-0122
a0 mars,sanda
a1 @engr.smu.edu
Abstract
Mining the answer of a natural lan-
guage open-domain question in a large
collection of on-line documents is made
possible by the recognition of the ex-
pected answer type in relevant text pas-
sages. If the technology of retriev-
ing texts where the answer might be
found is well developed, few studies
have been devoted to the recognition of
the answer type.
This paper presents a unified model of
answer types for open-domain Ques-
tion/Answering that enables the discov-
ery of exact answers. The evaluation
of the model, performed on real-world
questions, considers both the correct-
ness and the coverage of the answer
types as well as their contribution to an-
swer precision.
1 Introduction
Answer mining, a.k.a. textual Ques-
tion/Answering (Q/A), represents the task of
discovering the answer to an open-domain nat-
ural language question in large text collections.
Answer mining became a topic of significant
recent interest, partly due to the popularity of In-
ternet Q/A services like AskJeeves and partly due
to the recent evaluations of domain-independent
Q/A systems organized in the context of the
Text REtrieval Conference (TREC)1. The TREC
1The Text REtrieval Conference (TREC) is a series of
evaluations of fully automatic Q/A systems
specified two restrictions: (1) there is at least
one document in the test collection that contains
the answer to a test question; and (2) the answer
length is either 50 contiguous bytes (short an-
swers) or 250 contiguous bytes (long answers).
These two requirements intentionally simplify
the answer mining task, since the identification
of the exact answer is left to the user. However,
given that the expected information is recognized
by inspecting text snippets of relatively small
size, the TREC Q/A task took a step closer
to information retrieval rather than document
retrieval. Moreover, the techniques developed
to extract text snippets where the answers might
lie paved the way to a unified model for answer
mining.
To find the answer to a question several steps
must be taken, as reported in (Abney et al., 2000)
(Moldovan et al., 2000) (Srihari and Li, 2000):
a2 First, the question semantics needs to be cap-
tured. This translates into identifying (i) the
expected answer type and (ii) the question
keywords that can be used to retrieve text
passages where the answer may be found.
a2 Secondly, the index of the document collec-
tion must be used to identify the text pas-
sages of interest. The retrieval method either
employs special operators or simply modi-
fies boolean or vector retrieval. Since the
expected answer type is known at the time
workshops organized by the National Institute of Standards
and Technology (NIST), designed to advance the state-of-
the-art in information retrieval (IR)
of the retrieval, the quality of the text pas-
sages is greatly improved by filtering out
those passages where concepts of the same
category as the answer type are not present.
a2 Thirdly, answer extraction takes place by
combining several features that take into ac-
count the expected answer type.
Since the expected answer type is the only in-
formation used in all the phases of textual Q/A,
its recognition and usage is central to the perfor-
mance of answer mining.
For an open-domain Q/A system, establishing
the possible answer types is a challenging prob-
lem. Currently, most of the systems recognize the
answer type by associating the question stem (e.g.
What, Who, Why or How) and one of the concepts
from the question to a predefined general cate-
gory, such as PERSON, ORGANIZATION, LOCA-
TION, TIME, DATE, MONEY or NUMBER. Since
many of these categories are represented in texts
as named entities, their recognition as possible
answers is enabled by state-of-the-art Named En-
tity (NE) recognizers, devised to work with high
precision in Information Extraction (IE) tasks. To
allow for NE-supported answer mining, a large
number of semantic categories corresponding to
various names must be considered, e.g. names of
cars, names of diseases, names of dishes, names
of boats, etc. Furthermore, a significant number
of entities are not unique, therefore do not bear
names, but are still potential answers to an open-
domain question. Additionally, questions do not
focus only on entities and their attributes; they
also ask about events and their related entities.
In this paper we introduce a model of answer
types that accounts for answers to questions of
various complexity. The model enables several
different formats of the exact answer to open-
domain questions and considers also the situation
when the answer is produced from a number of
different document sources. We define formally
the answer types to open-domain questions and
extend the recognition of answer types beyond the
question processing phase, thus enabling several
feed-back mechanisms derived from the process-
ing of documents and answers.
The main contribution of the paper is in provid-
ing a unified model of answer mining from large
collections of on-line documents that accounts for
the processing of open-domain natural language
questions of varied complexity. The hope is that a
coherent model of the textual answer discovery
could help developing better text mining meth-
ods, capable of acquiring and rapidly prototyping
knowledge from the vast amount of on-line texts.
Additionally, such a model enables the develop-
ment of intelligent conversational agents that op-
erate on open-domain tasks.
We first present a background of Q/A systems
and then define several classes of question com-
plexity. In Section 3 we present the formal answer
type model whereas in Section 4 we show how to
recognize the answer type of open-domain ques-
tions and use it to mine the answer. Section 5
presents the evaluation of the model and summa-
rizes the conclusions.
2 Background
Open-Domain Question/Answering
To search in a large collection of on-line docu-
ments for the answer to a natural language ques-
tion we need to know (1) what we are looking for,
i.e. the expected answer type; and (2) where the
answer might be located in the collection. Fur-
thermore, knowing the answer type and recog-
nizing a text passage where the answer might be
found is not sufficient for extracting the exact an-
swer. We also need to know the dependencies
between the answer type and the other concepts
from the question or the answer. For example, if
the answer type of the TREC question
QT: How many dogs pull a sled in the Iditarod?
is known to be a number, we also need to be aware
that this number must quantify the dogs harnessed
to a sled in the Iditarod games and not the number
of participants in the games.
Capturing question or answer dependencies
can be cast as a straightforward process of
mapping syntactic trees to sets of binary head-
modifier relationships, as first noted in (Collins,
1996). Given a parse tree, the head-child of each
syntactic constituent can be identified based on a
simple set of rules used to train syntactic parsers,
cf. (Collins, 1996). Dependency relations are es-
tablished between each leaf corresponding to the
head child and the leaves of its constituent sib-
VP
SQ
WP
PP
What do most tourists visit in
Question ET1:
Reims
RBSVBP NNS VB IN NNP
NPNPWHNP
SBARQParse:
What visit Reimstouristsmost
Question Dependecies (ET1):
What do most tourists visit in Reims?
(a)
(b)
Figure 1: Example of TREC test question
lings that are not stop words, as illustrated by the
mapping of Figure 1(a) into Figure 1(b). Unlike in
IR systems, question stems are considered content
words. When question dependencies are known
(Harabagiu et al., 2000) proposed a technique of
identifying the answer type based on the semantic
category of the question stem and eventually of its
most connected dependent concept. For example,
in the case of question ET1, illustrated in Figure 1,
the answer type is determined by the ambiguous
question stem what and the verb visit. The answer
type is the object of the verb visit, which is a place
of attraction or entertainment, defined by the se-
mantic category LANDMARK. The answer type
replaces the question stem, generating the follow-
ing dependency graph, that can be later unified
with the answer dependency graph:
mostLANDMARK tourists visit Reims
However syntactic dependencies vary across
question reformulations or equivalent answers
made possible by the productive nature of natural
language. For example, the dependency structure
of ET2, a reformulation of question ET1 differs
from the dependency structure of ET1:
Due to the fact that verbs see and visit are syn-
onyms (cf. WordNet (Miller, 1995)) and pronoun
I can be read a possible visitor, the dependency
Question ET2:
ILANDMARK Reimssee
What could I see in Reims?
structures of ET1 and ET2 can be mapped one into
another. The mapping is produced by unifying the
two structures when lexical and semantic alterna-
tions are allowed. Possible lexical alternations are
synonyms or morphological alternations. Seman-
tic alternations consist of hypernyms, entailments
or paraphrases. The unifying mapping of ET1 and
ET2 shows that the two questions are equivalent
only when I refers to a visitor; other readings of
ET2 being possible when the referent is an investi-
gator or a politician. In each of the other readings,
the answer type of the question would be differ-
ent. The unifying mapping of ET1 and ET2 is:
ReimsI -->tourists see/visitLANDMARK
Similarly, a pair of equivalent answers is rec-
ognized when lexical and semantic alternations of
the concepts are allowed. This observation is cru-
cial for answer mining because:
1. it establishes the dependency relations as the
basic processing level for Q/A; and
2. it defines the search space based on alterna-
tions of the question and answer concepts.
Consequently, lexical and semantic alternations
are incorporated as feedback loops in the architec-
ture of open-domain Q/A systems, as illustrated
in Figure 2.
To locate answers, text passages are retrieved
based on keywords assembled from the question
dependency structure. At the time of the query, it
is unknown which keywords can be unified with
answer dependencies. However, the relevance of
the query is determined by the number of result-
ing passages. If too many passages are gener-
ated, the query was too broad, thus is needs a
specialization by adding a new keyword. If too
few passages were retrieved the query was too
specific, thus one keyword needs to be dropped.
The relevance feedback based on the number of
retrieved passages ends when no more keywords
can be added or dropped. After this, the unifi-
cations of the question and answer dependencies
Answer
Dependencies KB
KB
Answer Type
Keywords
Index
Relevance Feedback (# Passages)
Lexical Alternations
Semantic Unifications Abductive Justification
Question On-line Documents Answer
Dependencies
Question
Text Passages
Semantic Alternations
Answer Fusion
Lexico
Semantic
Figure 2: A diagram of the feedbacks supporting Open-Domain Q/A
is produced and the lexical alternations imposed
by unifications are added to the list of keywords,
making possible the retrieval of new, unseen text
passages, as illustrated in Figure 2.
The unification of dependency structures al-
lows erroneous answers when the resulting map-
ping is a sparse graph. To justify the correct-
ness of the answer an abductive proof backchain-
ing from the answer to the question must be
produced. Such abductive mechanisms are de-
tailed in (Harabagiu et al., 2000). Moreover, the
proof relies on lexico-semantic knowledge avail-
able from WordNet as well as rapidly formated
knowledge bases generated by mechanisms de-
scribed in (Chaudri et al., 2000). The justification
process brings forward semantic alternations that
are added to the list of keywords, the feedback
destination of all loops represented in Figure 2.
Mining the exact answer does not always end
after extracting the answer type from a correct
text snippet because often they result only in par-
tial answers that need to be fused together. The
fusion mechanisms are dictated by the answer
type.
Question Complexity
Open-Domain natural language questions can
also be of different complexity levels. Gener-
ally, the test questions used in the TREC evalu-
ations were qualified as fact-based questions (cf.
(Voorhees and Tice, 2000)) as they mainly were
short inquiries about attributes or definitions of
some entity or event. Table 1 lists a sample of
TREC test questions.
The TREC test set did not include any question
Where is Romania located? Europe
Who wrote ”Dubliners”? James Joyce
What is the wingspan of a condor? 9 feet
What is the population of Japan? 120 million
What king signed the Magna Carta? King John
Name a flying mammal. bat
Table 1: TREC test questions and their exact an-
swers (boldfaced)
that can be modeled as Information Extraction
(IE) task. Typically, IE templates model queries
regarding who did an event of interest, what was
produced by that event, when and where and even-
tually why. The event of interest is a complex
event, like terrorism in Latin America, joint ven-
tures or management successions. An example of
template-modeled question is:
What management successions occurred at
IBM in 1999?
In addition, questions may also ask about de-
velopments of events or trends that are usually
answered by a text summary. Since data produc-
ing these summaries can be sourced in different
documents, summary fusion techniques as pro-
posed in (Radev and McKeown, 1998) can be em-
ployed. Template-based questions and summary-
asking inquiries cover most of the classes of ques-
tion complexity proposed in (Moldovan et al.,
2000). Although the topic of natural language
open-domain question complexity needs further
study, we consider herein the following classes of
questions:
a2 Class 1: Questions inquiring about entities,
events, entity attributes (including number),
event themes, event manners, event condi-
tions and event consequences.
a2 Class 2: Questions modeled by templates,
including questions that focus only on one
of the template slots (e.g. “What managers
were promoted last year at Microsoft?”).
a2 Class 3: Questions asking for a sum-
mary that is produced by fusing template-
based information from different sources
(e.g. “What happened after the Titanic
sunk?”).
Since (Radev and McKeown, 1998) describes the
summary fusion mechanisms, Class 3 of ques-
tions can be reduced in this paper to Class 2,
which deals with the processing of the template.
3 A Model of Answer Types
This section describes a knowledge-based model
of open-domain natural language answer types
(ATs). In particular we formally define the an-
swer type through a quadruple
a0a2a1a4a3a6a5CATEGORY, DEPENDENCY, NUMBER,
FORMATa7.
The CATEGORY is defined as one of the following
possibilities:
1. one of the tops of a predefined ANSWER
TAXONOMY or one of its nodes;
2. DEFINITION;
3. TEMPLATE; or
4. SUMMARY.
For expert Q/A systems, this list of categories can
be extended. The DEPENDENCY is defined as the
question dependency structure when the CATE-
GORY belongs to the ANSWER TAXONOMY or is
a DEFINITION. Otherwise it is a template auto-
matically generated. The NUMBER is a flag indi-
cating whether the answer should contain a single
datum or a list of elements. The FORMAT defines
the text span of the exact answer. For example, if
the CATEGORY is DIMENSION, the FORMAT is
a8 Number
a9
a8 Measuring Unit
a9 .
The ANSWER TAXONOMY was created in
three steps:
Step 1 We devise a set of top categories modeled
after the semantic domains encoded in the Word-
Net database, which contains 25 noun categories
and 15 verb categories. The top of each WordNet
hierarchy corresponding to every semantic cate-
gory was manually inspected to select the most
representative nodes and add them to the tops of
he ANSWER TAXONOMY. Furthermore we have
added open semantic categories corresponding to
named entities. For example Table 2 lists the
named entity categories we have considered in
our experiments. Many of the tops of the AN-
SWER TAXONOMY are further categorized, as il-
lustrated in Figure 3. In total, we have considered
33 concepts as tops of the taxonomy.
D
ATER URATIOND
ERCENTAGEP
OUNTCEGREED
VALUEUMERICALN OCATIONL
COUNTRY
IMENSION TOWNPROVINCE O THERLOCATION
Figure 3: Two examples of top answer hierar-
chies.
Step 2 The additional categorization of the top
ANSWER TAXONOMY generates a many-to-
many mapping of the Named Entity categories in
the tops of the ANSWER TAXONOMY. Figure 4
illustrates some of the mappings.
date time organization city
product price country money
human disease phone number continent
percent province other location plant
mammal alphabet airport code game
bird reptile university dog breed
number quantity landmark dish
Table 2: Named Entity Categories.
ERSON
MONEY
PEEDS
DURATION
MOUNTA
A NSWER TYPE ENTYTY
P
number
N AMED CATEGORY
human
money
price
quantity
Figure 4: Mappings of answer types in named en-
tity categories.
Step 3: Each leaf from the top of the ANSWER
TAXONOMY is connected to one or several Word-
N ALUEV
dew
point
temperature
body
temperature
zero
absolute perimeter,
girth
size
largeness
bigness
circumference
distance,length
wingspread
wingspan,
light time
altitude
duration,
UMERICAL
length
N
longevity
longness
ATIONALITY
of the trip from...?
What is the duration
of an active volcano get?
EGREED TEMPERATURE DURATION COUNT SPEED DIMENSION
OCATIONL
What is the wingspan
of a condor? in diameter?
How big is our galaxyHow hot does the inside
Figure 5: Fragment of the ANSWER TAXONOMY.
Net subherarchies. Figure 5 illustrates a fragment
of the ANSWER TAXONOMY comprising several
WordNet subhierarchies.
4 Answer Recognition and Extraction
In this section we show how, given a question and
its dependency structure, we can recognize its an-
swer type and consequently extract the exact an-
swer. Here we describe four representative cases.
Case 1: The CATEGORY of the answer type is
DEFINITION when the question can be matched
by one of the following patterns:
(Q-P1):What a0 isa1area2a4a3 phrase to definea5 ?
(Q-P2):What is the definition of a3 phrase to definea5 ?
(Q-P3):Who a0 isa1wasa1area1werea2a6a3 person name(s)a5 ?
The format of the DEFINITION answers is sim-
ilarly dependent on a set of patterns, determined
as the head of the a8 Answer phrasea9 :
(A-P1):a7a3 phrase to definea5
a0 is
a1area2a9a8a10a3 Answer phrasea5
(A-P2):a7a3 phrase to definea5 , a0 aa1thea1an a3 Answer phrasea5a4a2a9a8
(A-P3):a7a3 phrase to definea5 –a8a10a3 Answer phrasea5
Case 2: The dependency structure of the ques-
tion indicates that a special instance of a concept
is sought. The cues are given either by the pres-
ence of words kind, type, name or by the ques-
tion stems what or which connected to the object
of a verb. Table 3 lists a set of such questions
and their corresponding answers. In this case the
answer type is given by the subhierarchy defined
by the node from the dependency structure whose
adjunct is either kind, type, name or the question
stem. In this situation the CATEGORY does not
belong to the top of the ANSWER TAXONOMY,
but it is rather dynamically created by the inter-
pretation of the dependency graph.
For example, the dynamic CATEGORY bridge,
generated for Q204 from Table 3, contains 14
member instances, including viaduct, rope bridge
and suspension bridge. Similarly, question Q581
generates a dynamic CATEGORY flower, with 470
member instances, comprising orchid, petunia
and sunflower. For dynamic categories all mem-
ber instances are searched in the retrieved pas-
sages during answer extraction to detect candidate
answers.
Case 3: In all other cases, the concept related
to the question stem in the question dependency
graph is searched through the ANSWER TAXON-
OMY, returning the answer type as the top of it
hierarchy. Figure 5 illustrates several questions
and their answer type CATEGORY.
Case 4: Whenever the semantic dependencies of
several correct answers can be mapped one into
another, we change the CATEGORY of the answer
type into TEMPLATE. The slots of the actual tem-
plate are determined by a three step procedure,
that we illustrate with a walk-through example
corresponding to the question What management
successions occurred at IBM in 1999?:
Step 1: For each pair of extracted candidate
Q204: What type of bridge is the Golden Gate Bridge?
Answer: the Seto Ohashi Bridge, consisting of six suspension bridges in the style of Golden Gate Bridge.
Q267: What is the name for clouds that produce rain?
Answer: Acid rain in Cheju Island and the Taean peninsula is carried by rain clouds from China.
Q503: What kind of sports team is the Buffalo Sabres?
Answer: Alexander Mogilny hopes to continue his hockey career with the NHL’s Buffalo Sabres.
Q581: What flower did Vincent Van Gogh paint?
Answer: In March 1987, van Gogh’s “Sunflowers” sold for $39.9 million at Christie’s in London
Table 3: TREC test questions and their answers. The exact answer is emphasized.
nominate/assignOrganization
(b)
PositionPerson
Organizationresign/leave PositionPerson
(a)
Position
Person1 Person2 Organization Positionreplace/succeed
Person1 Person2 Organization
Figure 6: Dependencies that generate templates.
answers unify the dependency graphs and find
common generalizations whenever possible. Fig-
ure 6(a) illustrates some of the mappings.
Step 2: Identify across mappings the common
categories and the trigger-words that were used
as keywords. In Figure 6(a) the trigger words are
boldfaced.
Step 3: Collect all common categories in a tem-
plate and use their names as slots. Figure 6(b)
illustrates the resulting template.
This procedure is a reverse-engineering of the
mechanisms used generally in Information Ex-
traction (IE), where given a template, linguistic
patterns are acquired to identify the text frag-
ments having relevant information. In the case
of answer mining, the relevant text passages are
known. The dependency graphs help finding the
linguistic rules and are generalized in a template.
To be able to generate the template we also
need to have a way of extracting the text where
the answer dependencies are detected. For this
purpose we have designed a method that em-
ploys a simple machine learning mechanism: the
perceptron. For each text passage retrieved by
the keyword-based query we define the following
seven features:
a0a2a1a4a3a6a5a8a7a4a9 the number of question words matched in
the same phrase as the answer type CATEGORY;
a0a10a1a4a3a6a5a8a7a11a7 the number of question words matched in
the same sentence as the answer type CATEGORY;
a0a12a1a4a3a6a5a14a13a15a9 : a flag set to 1 if the answer type CATE-
GORY is followed by a punctuation sign, and set
to 0 otherwise;
a0a16a1a4a3a6a5a8a17a19a18a21a20a23a22 : the number of question words
matches separated from the answer type CATE-
GORY by at most three words and one comma;
a0a24a1a4a3a6a5a8a7a11a22a25a7 : the number of question words occur-
ring in the same order in the answer text as in the
question;
a0a26a1a4a3a6a5a8a27a28a20a23a22 : the average distance from the an-
swer type CATEGORY to any of the question word
matches;
a0a29a1a4a3a6a5a8a30a10a31a32a22 : the number of question words
matched in the answer text.
To train the perceptron we annotated the correct
answers of 200 of the TREC test questions. Given
a pair of answers, in which one of the answers is
correct, we compute a relative comparison score
using the formula:
a33a35a34a37a36a38a40a39a42a41a43a45a44a47a46a28a48a6a49a50a48a52a51a54a53a2a33a35a34a37a36a55a48a6a49a50a48a57a56a32a46a28a58a11a59a32a51a54a53a2a33a35a34a37a36a55a58a11a59
a56a60a46a28a61a63a62a4a64a65a49a66a51a54a53a2a33a35a34a37a36a55a61a63a62a4a64a65a49a26a56a32a46a28a48a6a59a67a51a54a53a2a33a35a34a37a36a55a48a6a59
a56a60a46a28a48a6a48a12a51a54a53a2a33a35a34a37a36a55a48a6a48a57a56a32a46a28a68a70a69a10a49a71a51a72a53a2a33a35a34a37a36a55a68a70a69a10a49
a56a60a46a28a73a74a64a65a49a66a51a54a53a2a33a35a34a37a36a55a73a74a64a65a49a75a56a67a76a78a77a79a33a35a34a81a80a42a77a79a82a35a36a83
The perceptron learns the seven weights as well
as the value of the threshold used for future tests
on the remaining 693 TREC questions. Whenever
the relative score is larger than the threshold, a
passage is extracted as a candidate answer. In our
experiments, the performance of the perceptron
surpassed the performance of decision trees for
answer extraction.
5 Evaluations and Conclusion
To evaluate our answer type model we used 693
TREC test questions on which we did not train the
perceptron. Table 4 lists the breakdown of the an-
swer type CATEGORIES recognized by our model
as well as the coverage and precision of the recog-
nition. Currently our ANSWER TAXONOMY en-
codes 8707 concepts from 129 WordNet hierar-
chies, covering only 81% of the expected answer
types. This shows that we have to continue en-
coding more top concepts in the taxonomy and
link them to more WordNet concepts.
The recognition mechanism had better preci-
sion than coverage in our experiments. Moreover
a relationship between the coverage of answer
type recognition and the overall performance of
answer mining, as illustrated in Table 4. Some of
the test questions are listed in Tables 1 and 3. The
experiments were conducted by using 736,794
on-line documents from Los Angeles Times, For-
eign Broadcast Information Service, Financial
Times AP Newswire, Wall Street Journal and San
Jose Mercury News.
CATEGORY (# Questions) Precision Coverage
DEFINITION (64) 91% 84%
Top ANSWER
TAXONOMY (439) 79% 74%
Dynamic answer
category (17) 86% 79%
TEMPLATE (14) 93% 65%
# ANSWER Answer Type Q/A Precision
Taxonomy Coverage
Tops
8 44% 42%
22 56% 55%
33 83% 78%
Table 4: Evaluation results.
The experiments show that open-domain natu-
ral language questions of varied degrees of com-
plexity can be answered consistently from vast
amounts of on-line texts. One of the appli-
cations of a unified model of answer mining
is the development of intelligent conversational
agents (Harabagiu et al., 2001).
Acknowledgement
This research was supported in part by the
Advanced Research and Development Activity
(ARDA) grant 2001*H238400*000 and by the
National Science Foundation CAREER grant
CCR-9983600.

References
S. Abney, M. Collins, and A. Singhal. 2000. Answer
extraction. In Proceedings of ANLP-2000, pages
296–301, Seattle, Washington.
V.K. Chaudri, M.E. Stickel, J.F. Thomere, and R.J.
Waldinger. 2000. Reusing prior knowledge: Prob-
lems and solutions. In Proceedings of AAAI-2000,
Austin, Texas.
M. Collins. 1996. A new statistical parser based on
bigram lexical dependencies. In Proceedings of the
ACL-96, pages 184–191, Copenhagen, Denmark.
S. Harabagiu, M. Pas¸ca, and S. Maiorano. 2000. Ex-
periments with open-domain textual question an-
swering. In Proceedings of COLING-2000, Saar-
brucken, Germany.
S. Harabagiu, M. Pasca, and F. Lacatusu. 2001. Dia-
logue management for interactive question answer-
ing. In Proceedings of FLAIRS-2001. To appear.
G. Miller. 1995. WordNet: a lexical database. Com-
munications of the ACM, 38(11):39–41.
D. Moldovan, S. Harabagiu, M. Pas¸ca, R. Mihalcea,
R. Gˆırju, R. Goodrum, and V. Rus. 2000. The
structure and performance of an open-domain ques-
tion answering system. In Proceedings of ACL-
2000, Hong Kong.
D. Radev and K. McKeown. 1998. Generating natu-
ral language summaries from multiple on-line re-
sources. Computational Linguistics, 24(3):469–
500.
R. Srihari and W. Li. 2000. A question answering sys-
tem supported by information extraction. In Pro-
ceedings of ANLP-2000, Seattle, Washington.
E.M. Voorhees and D.M. Tice. 2000. Building a
question-answering test collection. In Proceedings
of the 23rd International Conference on Research
and Development in Information Retrieval (SIGIR-
2000), Athens, Greece.
