Cooperative Question Answering in Restricted Domains:
the WEBCOOP Experiment
Farah Benamara
Institut de Recherches en Informatique de Toulouse, IRIT
118, route de Narbonne,
31062, Toulouse, France
benamara@irit.fr
Abstract
We present an experiment for designing a logic
based QA system, WEBCOOP, that integrates
knowledge representation and advanced rea-
soning procedures to generate cooperative re-
sponses to natural language queries on the web.
The system is first developed for the tourism
domain. We then examine how and under what
conditions this system can be re-used for other
domains.
1 Introduction
The current trend in Question Answering is to-
wards the processing of large volumes of open-
domain texts (e.g. documents extracted from
the World Wide Web). Open domain QA is
a hard task because no restriction is imposed
either on the question type or on the user’s
vocabulary. This is why, most of the eﬀorts
(Voorhees, 2003) are focused on answering fac-
toid style questions (and to a little extend, def-
inition questions) using shallow text processing
which is roughly based on pattern extraction or
information retrieval techniques. However, QA
should support the integration of deeper modes
of language understanding as well as more elab-
orated reasoning schemas for more complex QA
strategies, in order to provide, for example,
better answer ranking, answer justification, re-
sponses to unanticipated questions or to resolve
situations in which no answer is found in the
data sources. Cooperative answering systems
are typically designed to deal with such situa-
tions, by providing non-misleading, and useful
answers to a query. (Grice, 1975) maxims of
conversation namely the quality, quantity, rela-
tion and style maxims are frequently used as a
basis for designing cooperative answering sys-
tems. An overview of cooperative answering
techniques is given in (Gaasterland et al., 1994).
In COGEX (Moldovan et al., 2003), a recent
QA system, authors used automated reasoning
for QA and showed that it is feasible, eﬀec-
tive and scalable. This logical prover aims at
checking and extracting all kinds of lexical re-
lationships between the question and its can-
didate answers using world knowledge axioms,
supplied by WordNet glosses, as well as rewrit-
ing rules representing equivalent classes of lin-
guistic patterns. Such inference techniques (e.g.
lexical equivalence, unification on logical repre-
sentations of texts) are not suﬃcient for provid-
ing intelligent or cooperative responses. Indeed,
advanced strategies for QA requires, as we ex-
plain in this paper, the integration of reasoning
components operating over a variety of knowl-
edge bases, encoding common sense knowledge
as well as knowledge specific to a variety of do-
mains.
We relate in this paper, an experiment for de-
signing a logic based QA system, WEBCOOP,
that integrates knowledge representation and
advanced reasoning procedures to generate co-
operative responses to natural language (NL)
queries on the web. This experiment is first car-
ried out on a relatively restricted domain that
includes a number of aspects of tourism (ac-
commodation and transportation, which have
very diﬀerent characteristics on the web). The
tourism domain is in fact half way between an
open domain and a closed domain (e.g. weather
forecast, Unix technical manuals). The tourism
domain has a kernel roughly around accom-
modation and transportation, but it also in-
cludes satellite domains, such as history, secu-
rity, health, immigration, ecology, etc. Those
satellite domains are only partly considered,
from the point of view of the ‘kernel’ domains.
We also observe that there is, in fact, a kind
of continuum between the notions of open do-
main and closed domain, via restricted domains
which makes quite fuzzy the definition of what
a restricted domain is.
Besides the technical functionalities of WEB-
COOP, the main goal of this paper is to eval-
uate the diﬀerent facets of the portability of
WEBCOOP. Three major points are at stake:
(1) resources, in term of language resources
and kinds of knowledge required, (2) coopera-
tive procedures involved, such as identifying and
explaining user false presuppositions, relaxing
constraints or providing intensional responses,
and finally (3) the intelligibility of the system
outputs (such as hyperlinks, short responses or
list of answers), considering that answers should
also include a trace of the inferences drawn.
In the next sections, we briefly present the
WEBCOOP architecture focusing on the kinds
of knowledge and cooperative procedures in-
volved. Then, we analyze the main characteris-
tics of the tourism domain and outline its main
features as a restricted domain. Then, we ana-
lyze the portability of this type of QA system
to other restricted domains. Finally, we propose
an evaluation methodology based on experimen-
tal psychology for the point (3) cited in the last
paragraph.
2 The WEBCOOP Architecture
2.1 A Corpus Based Approach
To have a more accurate perception of how co-
operativity is realized in man-man communi-
cation, we collected a corpus of question an-
swer pairs (QA pairs) found in a number of
web sites dedicated to diﬀerent kinds of large
public domains. 60% of the corpus is dedicated
to tourism (our implementation being based on
this application domain), 22% to health and the
other QA pairs are dedicated to sport, shopping
and education. The analysis of this corpus aims
at identifying the external form and the concep-
tual categories of questions, as well as categoriz-
ing the diﬀerent cooperative functions deployed
by humans in their discourse. Our main claim is
that an automatic cooperative QA system could
be induced from natural productions without
loosing too much of the cooperative contents
produced by humans. We noted that human
responses are much more diverse than any ma-
chine could produce in the near future. Never-
theless, it is possible to normalize these forms
to more stereotyped utterances.
2.2 The Architecture
The general architecture of the system (figure
1) is inspired from our corpus analysis. Our
system being a direct QA system, it does not
have any user model.
In WEBCOOP, NL responses are produced
Figure 1: The WEBCOOP architecture
from first order logical formulas constructed
from reasoning processes carried out by an infer-
ence engine. Our approach requires the develop-
ment of a knowledge extractor from web pages
(Benamara and Saint Dizier, 2004b) (viewed as
a passage retrieval component) and the elabo-
ration of a robust question parser. We assume
that the most relevant documents to the user’s
question are found using standard information
retrieval techniques and that the relevant para-
graphs that respond to the question keywords
are correctly extracted from those documents
(Harabagiu and Maiorano, 1999). Then, our
knowledge extractor transforms each relevant
paragraphs into a logical representation. The
WEBCOOP inference engine has to decide, via
cooperative rules, what is relevant and how to
organize it in a way that allows for the realiza-
tion of a coherent and informative response.
Responses are structured in two parts. The
first part contains explanation elements in nat-
ural language. It is a first level of coopera-
tivity that reports user misconceptions in re-
lation with the domain knowledge (answer ex-
planation). The second part is the most im-
portant and the most original. It reflects the
know-how of the cooperative system, going be-
yond the cooperative statements given in part
one. It is based on intensional description tech-
niques and on intelligent relaxation procedures
going beyond classical generalization methods
used in AI. This component also includes ad-
ditional dedicated cooperative rules that make
a thorough use of the domain ontology and of
general knowledge. In WEBCOOP, responses
provided to users are built in web style by in-
tegrating natural language generation (NLG)
techniques with hypertexts in order to produce
dynamic responses (Dale et al., 1998).
We claim that responses in natural language
must make explicit in some way, via explana-
tions and justifications, the mechanisms that led
to the answer. For each type of inference used in
WEBCOOP, we define general and underspec-
ified natural language templates (Reiter, 1995)
that translate the reasoning mechanisms in ac-
cessible terms. A template is composed of three
parts, S, F, and R,where:
-S are specified elements,
-F are functions that choose for each concept
in the ontology, its appropriate lexicalization,
- R are logical formulas representing the rest
of the response to be generated.
The underspecified elements, F and R,de-
pend on the question, on local semantic factors
and on the type of solution elaborated. Their
generation relies on ontological knowledge, gen-
eral linguistic knowledge and lexicalisation and
aggregation functions. Templates have been in-
duced from a number of QA pairs found in large
public domaines. Responses have been normal-
ized without loosing too much of their accuracy
in order to get stereotyped response forms us-
able in NL generation frameworks. A large por-
tion of underspecified elements, within a tem-
plate, is presented as an hyperlink to the user
as illustrated in the examples in the next sec-
tion. Here is an example of a template dedi-
cated to one of our relaxation schemas. It is
used when the question focus is relaxed using
its sister nodes in the ontology. Specified ele-
ments are in italic:
un autre type de lexicalisation(mother node):
(lexicalisation(sister node))
+
R.
1
At the moment, in WEBCOOP we have 28
basic templates.
2.3 Two Typical Examples
The following examples illustrate WEBCOOP
outputs.
Example1. Suppose one wishes to rent a 15
person country cottage in Corsica and (1) that
observations made on the related web pages or
(2) that a constraint or a regulation, indicates
that the maximum capacity of a country cottage
in Corsica is 10 persons (figure 1).
The first part of the response relates the de-
tection of a false presupposition or the viola-
1
A template fragment of the form (fragment)
+
,in-
dicates that that fragment occurs in the generated re-
sponse at least one time.
tion of an integrity constraint for respectively
cases (1) and (2) above. Case (2) entails the
production of the following message, generated
by a process that evaluates the question logical
formula against the knowledge base: A chalet
capacity is less than 10 persons in Corsica. In
Figure 2: Detection of a misconception and
query relaxation
a second step, the know-how component of the
cooperative system generates a set of flexible so-
lutions as shown in the figure above, since the
first part of the response is informative but not
really productive for the user. The three flexible
solutions proposed emerge from know-how co-
operative rules based on relaxation procedures
designed to be minimal and conceptually rel-
evant. The first flexible solution is based on
a cardinality relaxation, while in the last two
solutions, relaxation operates gradually on con-
cepts such as the type of accommodation (hotel
or pension) or the region (possibly a close-by
region, with similar characteristics), via the do-
main model and the ontology. Dynamically cre-
ated links are underlined. The user can then, at
will, get more precise information, dynamically
generated from the data base of indexed web
pages. For technical details on how relaxed re-
sponses are elaborated and generated in NL see
(Benamara and Saint Dizier, 2004a).
Example 2. Suppose a user asks for means
of transportation to go to Geneva airport.In
WEBCOOP, we have a variable-depth inten-
sional calculus which allows us, experimentally,
to tune the degree of intensionality of responses
in terms of the abstraction level in the ontology
of the generalizes. This choice is based on a con-
ceptual metrics that determines the ontological
proximity between two concepts. The goal is to
have a level of abstraction adequate for the user.
A supervisor manages both the abstraction level
and the display of the elaborated intensional an-
swers (IA). The retrieved IA are structured in
two parts. First, the generation of a response
with generalizations and exceptions: all trains,
buses and taxis go to the airport. Then, a sorted
list of the retrieved extensional answers is gen-
erated according to the frequency and to the
cost of transportation. This strategy avoids the
problem of having to guess the user’s intent. For
technical details on how IA are elaborated and
generated in NL see (Benamara, 2004).
Figure 3: Variable depth intensional answers
3 Sources of Knowledge and
Inference mechanisms in
WEBCOOP
3.1 Knowledge Representation for the
Tourism Domain: a Typology
A first question about knowledge, for automat-
ing the production of cooperative responses,
concerns the type and the typology of knowl-
edge involved and where such knowledge can
be best represented: in databases, in knowledge
bases, in texts (involving knowledge extraction
or fragments of text extractions). So far, the
diﬀerent forms of knowledge we have identified
are, roughly:
1. general-purpose, factual information
(places, distances, proper names, etc.),
2. descriptive information like flight sched-
ules, hotel fares, etc. that we find in general
in databases,
3. common sense knowledge and constraints
such as: for a given trip, the arrival time
is greater that the departure time,
4. hierarchical knowledge: such as a hotel is
a kind of tourist accommodation.This
knowledge is often associated with prop-
erties that define the object, for example
a restaurant is characterized by its type of
food, category, localization, etc.
5. procedures or instructions that describe
how to prepare a trip or how to book a
room in a given hotel category.
6. definitions,
7. regulations, warnings,
8. classification criteria of objects according
to specific properties such as sorting hotels
according to their category.
9. interpretation functions, for example, of
fuzzy terms (e.g. expensive, far from the
beach).
Items 8 and 9 have a quite diﬀerent nature,
but they are closely related to the domain at
stake.
3.2 Knowledge Representation in
WEBCOOP
Let us now consider how these forms of knowl-
edge are represented. WEBCOOP has two
main forms for encoding knowledge: (1) general
knowledge and domain knowledge represented
by means of a deductive knowledge base, that
includes facts, rules and integrity constraints
and (2) a large set of indexed texts, where in-
dexes are logical formulae. Our semantic rep-
resentation is based on a simplified version of
the Lexical Conceptual Structure (LCS). Let us
review these below.
The kernel-satellite structure of the tourism
domain requires that we study, for this appli-
cation, portability and data integration aspects
for each satellite domain. At this level of com-
plexity there is no ready-made method that we
can use; furthermore, most of the work is done
manually. The results of the integration reflect
our own intuitions coupled with and applied on
generic data available on the web.
a. The knowledge base iscodedinProlog.
It includes basic knowledge, e.g. country names
coded as facts or distance graphs between
towns, coded as facts and rules. It also includes
rules which play at least two roles: data
abstraction (e.g. to describe the structure of an
object, besides e.g. part-of descriptions found
in the ontology):
hotel stay cost(Hotel ID, NbNights, Total)
:- hotel(Hotel ID, Night rate),
Total is NbNights * Night rate.
and the encoding of conditional situations:
book flight(A) :-
person(A), age(A, AG), AG > 17.
which says that you can book a flight if you
are at least 18 years old. Finally the knowledge
base contains integrity constraints. For exam-
ple, the constraint:
constraint([chalet(X), capacity(X,C), C>
10], fail).
indicates that ‘a chalet cannot accommodate
more than 10 persons’.
The ontology, described below, contains data
which can be interpreted as facts (e.g. hierarchi-
cal relations), rules or integrity constraints (as
simple as domain constraints for property val-
ues). Currently, our KB contains 170 rules and
47 integrity constraints, which seems to cover a
large number of situations.
b. The ontology is basically conceptual
where nodes are associated with concept lexi-
calizations and essential properties. Each node
is represented by the predicate :
onto-node(concept, lex, properties)
where concept is described using properties
and lex are possible lexicalisations of concept.
Most lexicalisations are entries in the lexicon
(except for paraphrases), where morphological
and grammatical aspects are described. For
example, for hotel,wehave:
onto-node(hotel, [[htel], [htel,
rsidence]], [night-rate, nb-of-rooms]).
There are several well-designed public domain
ontologies on the net. Our ontology is inspired
from two existing French ontologies, that we
had to customize: TourinFrance
2
and the
bilingual (French and English) thesaurus of
tourism and leisure activities
3
which includes
2800 French terms. We manually integrated
these ontologies in WEBCOOP (Doan et al.,
2002) by removing concepts that are either too
specific (i.e. too low level), like some basic
aspects of ecology or rarely considered, as e.g.
the economy of tourism. We also removed quite
surprising classifications like sanatorium under
tourist accommodation. We finally reorganized
some concept hierarchies, so that they ‘look’
more intuitive for a large public. Finally, we
found that some hierarchies are a little bit odd,
forexample,wefoundatthesamelevelaccom-
modation capacity and holiday accommodation
whereas, in our case, we consider that capacity
is a property of the concept tourist accommoda-
2
www.tourinfrance.net
3
www.iztzg.hr/indokibiblioteka/THESAUR.PDF
tion. We have, at the moment, an organization
of 1000 concepts in our tourism ontology which
describe accommodation and transportation
and a few other satellite elements (geography,
health, immigration).
c. The lexicon contains nouns, verbs and
adjectives related to the tourism domain, ex-
tracted from both corpora and ontologies. The
lexicon contains also determiners, connectors
and prepositions. The lexicon is constructed
directly from the revised ontologies for nouns.
Nouns contain basic information (e.g. pred-
icative or not, count/mass, deverbal) coded by
hand, their ’semantic’ type, directly character-
ized by their ancestor in the ontology, and a
simple semantic representation. Verbs are those
found in our corpora. We have a large verb
KB (VOLEM project)(Fernandez et al., 2002)
of 1700 verbs in French, Spanish and Catalan.
The verb lexicon is extracted from this KB al-
most without modification. For tourism, in-
cluding request verbs, we have 150 verbs. Since
verbs are central in NLG, it is crucial that they
get much information, in our system: thematic
roles, selectional restrictions, syntactic alterna-
tions, Wordnet classification, and semantic rep-
resentation (a conceptual representation, a sim-
plification of the Lexical Conceptual Structure).
d. Indexed texts. Our knowldge extractor,
which is based on the domain ontology, trans-
forms each text fragment into the following
logical representation : text(F, http) where
F is a first-order formula that represents
knowledge extracted (in general) from a web
page, with address http (or explicit text).
For example, indexed texts about airport
transportations in various countries have the
following form:
text(route(50) ∧to(50,cointrin)∧
bymeansof(50,Y)∧tramway(Y )∧airport(cointrin)
∧localization(cointrin,in(geneva)), www.gva.ch).
Indexed paragraphs also describe categories
such as: procedures, regulations, warnings or
classifications. Texts identified as such are in-
dexed by indicating (1) the category in which
they fall, (2) a keyword or a formula that iden-
tifies the nature of the procedure, regulation,
etc., and (3) the text itself, generally used as
such in a response.
e. Query representation and evalua-
tion. Processing a query allows for the iden-
tification of: the type of the query (yes/no,
Boolean or entity, etc.), the question focus and
the construction of its semantic representation
in first-order logic. For example, the question:
what are the means of transportation to go to
Geneva airport ? has the following logical rep-
resentation: (entity,meansoftransportation(Y ),
route(X)∧to(X,Z)∧
bymeansof(X,Y) ∧meansoftransportation(Y )∧
airportof(Z,geneva))
Given a fragment of text, we infer that it is an
answer to a question by two diﬀerent ways: (1)
from the deductive knowledge base, in that case,
responses are variable instances or (2) from the
indexed text base, and in that case, responses
are formulae which unify with the query for-
mula. In this latter case, roughly, unification
proceeds as follows. Let Q (conjunction of terms
q
i
) be the question formula and F (conjunction
of f
j
) be a formula associated with an indexed
text. F is a response to Q iﬀ for all q
i
there is
an f
j
such that:
(i) q
i
unifies with f
j
or
(ii) q
i
subsumes, via the concept ontology,
f
j
(e.g. means-of-transportation(Y) subsumes
tramway(Y)), or
(iii) q
i
rewrites, via rules of the knowl-
edge base, into a conjunction of f
j
, e.g.:
airportof(Z,geneva) rewrites into: airport(Z) ∧
localisation(Z,in(geneva)).
3.3 Inference Needs for Providing
Cooperative Responses
We develop a general typology of cooperative
functions. The aim is to identify the types
and sources of knowledge associated with each
of these functions. In terms of portability, we
think that annotating in QA corpora of a spe-
cific domain the various cooperative functions
used should help identify the needs in terms of
knowledge for the development of each coop-
erative function. It remains, then, to evaluate
the validity and the adequacy of the inference
schemas, but these can only be evaluated a pos-
teriori, whereas the types of knowledge can be
evaluated a priori.
Another perspective is that, given the de-
scription of the forms of knowledge associated
with an application, it may be possible to antic-
ipate what kinds of cooperative functions could
be implemented for this application.
We decompose cooperative functions into two
main classes: Response Elaboration (ER)
and Additional Information (ADR).The
first class includes response units that propose
alternatives to the question whereas the latter
contains a variety of complements of informa-
tion, which are useful but not absolutely neces-
sary such as precision, suggestion or warnings.
Figure 4 shows the diﬀerent kinds of knowl-
edge involved for each of the cooperative func-
tions that belong to the ER class
4
:
Figure 4: Cooperative functions and related knowl-
edge
In the tourism domain, queries are very di-
verse in form and contents. From that point
of view, they are closer to open domains than
to closed domains, as advocated in the intro-
duction. Questions about tourism, as revealed
by our corpora studies, include false presuppo-
sitions (FP), misunderstandings (MIS), concept
relaxations (RR), intensional responses (IR).
For the moment, we investigate only questions
of type boolean and questions about entities
and we use the inference schemas: FP, MIS,
RR and IR cited above. We think it is impor-
tant to make explicit in the response the types
of knowledge used in the inferences and to show
how they are organized and lexicalized. As de-
scribed in example 1 of section 2.3, the expla-
nation given in italic in the response :another
accommodation type:hotel,pension, indicates
that a relaxation based on the ontological type
of the concept chalet was carried out.
4 Evaluation of WEBCOOP
It is clear that an evaluation in the TREC
style is not relevant for our approach. We have
two forms of evaluations: (1) the evaluation of
the portability of the system w.r.t. the forms
of knowledge involved and the applicability of
the inference schemas and (2) the evaluation of
4
Indirect responses, for example: is your camping
close to the highway?, can be indirectly, but coopera-
tively responded: yes, but that highway is quiet at night.
the linguistic and cognitive adequacy of the re-
sponses produced by the system.
4.1 Evaluating System Portability
Porting WEBCOOP to other large-public ap-
plications, given the complexity of the system,
is quite challenging.
4.1.1 The lexicon and the Ontology
First, we claim that the syntax of questions and
the template-based approach used for producing
responses are relatively stable. At the language
level, the main task is to define an appropriate
lexicon, in relation with the domain ontology.
This task may be somewhat facilitated by the
existence of shared resources, however these are
quite rare for French. In general, we observe
that some resources are common to all applica-
tions (e.g. communication or possession verbs),
or prepositions, while others are totally specific,
with dedicated senses and usages. Creating an
application lexicon is costly, in particular when
NL generation is involved. To give an evaluation
of the complexity, an application like tourism re-
quires about 150 verbs and about 1800 nouns.
Among verbs, 100 are generic verbs, with stan-
dard senses. Describing verbs is complex, but
their number is quite modest. Most nouns are
not predicative, therefore, their lexicon can be
partly deduced from the domain ontology.
There are many domain ontologies on the
web. Although constructed by domain experts,
they turn out not to be necessarily adequate for
providing responses to a large public of non-
specialists. The main diﬃculties are to cus-
tomize these ontologies and to manage their co-
herence in order to produce a domain ontology
which leads to coherent and adequate responses,
as explained in section 3.2.
4.1.2 The Inference Schemas
In terms of cooperative functions, our experi-
ence is that most applications require the same
types of functions, but with various degrees of
importance. For example, some application will
be subject to more cases of misunderstandings
than others, depending, e.g. on the complexity
of their associated knowledge and on the type
of services expected by users. Similarly, the
inference procedures used in WEBCOOP have
been designed with a certain level of generic-
ity. They should be portable provided that the
knowledge resources of the new domain can be
implemented using WEBCOOP format, which
is quite generic. But, besides QA annotations,
which is a very useful perspective, the adequacy
of inferences can only be evaluated a posteriori.
In a future stage, we plan to use what Barr
and Klavans (Barr and Klavans, 2001) call com-
ponent performance evaluation which consists
of assessing the performance of system compo-
nents and determining their impact on the over-
all system performance.
4.2 Evaluating Response intelligibility
Finally, since WEBCOOP produces responses
in NL, some of which on a template basis (dif-
ferent from TREC which simply reproduces text
extracts), it is important to evaluate the porta-
bility of those templates. We propose a method
based on experimental psychology, that aims at
evaluating the cooperative responses generated
in the know-how component of WEBCOOP.
Our methodology involves the following steps:
- Evaluating templates within a single do-
main (tourism in our case). This goal includes
two main parts :
1. intra-templates which aims at evaluat-
ing:
- response intelligibility in terms of (1)
the adequacy of the response w.r.t the user
intent, and of (2) the justifications and ex-
planations mechanisms provided that led to
the answer.
-thereadability of the responses in terms
of (3) the linguistic surface generation of
both the underspecified terms and the dif-
ferent lexicalization choices made within
each templates, and in terms of (4) the ad-
equacy of our hyperlinks generation heuris-
tics.
2. inter-templates which aims at evaluat-
ing:
-thedisplay order relevance.Ifwego
back to the example 1 in section 2.3, the re-
sponses are displayed following the inverse
reading order of the question constraints
i.e. chalet is the last concept to be relaxed
in the question. This evaluation can also
be useful for identifying other kinds of cor-
relation between the answers display and
the constraints order in the question.
-thegeneral fluency in terms of syntaxi-
cal regularities of the responses generated
by each template.
-thevisual aspect of the responses : enu-
merations vs. paragraphs.
- Evaluating templates portability to other
large public domains like health and education.
We have developed the experimental proto-
cols associated to the relevance of explanation
(point 2 cited above) and to the display order
relevance. Interpretation results are ongoing.
5 Conclusion and Perspectives
We reported in this paper an experiment for de-
signing a logic based QA system, WEBCOOP,
that integrates knowledge representation and
advanced reasoning procedures to generate co-
operative responses to natural language queries
on the web. We claim that restricted domains
are more suitable than open domains to con-
duct research in advanced techniques on QA
because those systems require deeper modes of
language understanding, more elaborated rea-
soning schemas paired with a variety of knowl-
edge forms and sources.
WEBCOOP is applied to the tourism do-
main, it is a challenging and rewarding expe-
rience because the tourism domain is half way
between open domain applications and closed
domains, allowing us to better perceive these
two perspectives.
Our corpus based approach allows to iden-
tify the type of knowledge associated with each
cooperative function. The annotation of cor-
pora constitutes, in our sense, a good evalua-
tion method for the study of the portability of
WEBOOP to other restricted domains.
Finally, since an evaluation in TREC style is
not relevant for our approach, we have: (1) the
evaluation of the portability of the system w.r.t.
the forms of knowledge involved and the appli-
cability of inference schemas and (2) the eval-
uation of the linguistic and cognitive adequacy
of the responses produced by the system. We
are now evaluating the portability of generation
templates to the health and education domains
and the accuracy of cooperative functions.

References
V. Barr and J. Klavans. 2001. Verification and
validation of language processing systems: Is
it evaluation? In ACL 2001 Workshop on
Evaluation Methodologies for Language and
Dialogue Systems, July, pages 34 – 40.
F. Benamara and P. Saint Dizier. 2004a. Ad-
vanced relaxation for cooperative question
answering. New Directions in Question An-
swering, Chapter 21, Mark T. Maybury, edi-
tor, AAAI/MIT Press. To appear.
F. Benamara and P. Saint Dizier. 2004b.
Knowledge extraction from the web: an ex-
periment and an analysis of its portability.
Vivek, 15(1):3–15.
F. Benamara. 2004. Generating intensional an-
swers in intelligent question answering. In
Proceeding of INLG 04, the International
Conference on Natural Language Generation,
Brighton, UK.
R. Dale, J. Oberlander, M. Milosavljevic, and
A. Knott. 1998. Integrating natural lan-
guage generation and hypertext to produce
dynamic documents. Interacting with Com-
puters, 11(2):109–135.
A. Doan, J. Madhavan, P. Domingos, and
A. Halevy. 2002. Learning to map between
ontologies on the semantic web. In Pro-
ceedings of the 11th international conference
on World Wide Web, pages 662–673. ACM
Press.
A. Fernandez, P. Saint-Dizier, G. Vazquez,
M. Kamel, and F. Benamara. 2002. The
volem project : a framework for the construc-
tion of advanced multilingual lexicons. In
Language Technology. Springer Verlag, Lec-
ture Notes.
T. Gaasterland, P. Godfrey, and J. Minker.
1994. An Overview of Cooperative Answer-
ing. In Papers in Non-standard Queries and
Non-standard Answers, in series Studies in
Logic and Computation. Oxford, Clarendon
Press.
H. Grice. 1975. Logic and Conversation.In
Cole and Morgan editors, Academic Press.
S. Harabagiu and S. Maiorano. 1999. Finding
Answers in Large Collections of Texts: Para-
graph Indexing + Abductive Inference. In
AAAI Fall Symposium on Question Answer-
ing Systems, November, pages 63–71.
D. Moldovan, C. Clark, S. Harabagiu, and
S. Maiorano. 2003. Cogex: A logic prover for
question answering. In Language Technology,
pages 87–93. Proceedings of HLT-NAACL,
Edmonton.
E. Reiter. 1995. Cogex: A logic prover for ques-
tion answering. In NLG versus Templates,
pages 87–93. In Proceedings of 7th European
Workshop on Natural Language Generation,
Leiden, The Netherlands.
E. M. Voorhees. 2003. Cogex: A logic prover
for question answering. In Overview of the
TREC 2002 Question Answering Track.Pro-
ceedings of TREC-11, NIST.
