Proceedings of the ACL-SIGLEX Workshop on Deep Lexical Acquisition, pages 28–37,
Ann Arbor, June 2005. c©2005 Association for Computational Linguistics
Automatically Learning Qualia Structures from the Web
Philipp Cimiano & Johanna Wenderoth
Institute AIFB
University of Karlsruhe
Abstract
Qualia Structures have many applications
within computational linguistics, but currently
there are no corresponding lexical resources
such as WordNet or FrameNet. This paper
presents an approach to automatically learn
qualia structures for nominals from the World
Wide Web and thus opens the possibility to ex-
plore the impact of qualia structures for natural
language processing at a larger scale. Further-
more, our approach can be also used support a
lexicographer in the task of manually creating
a lexicon of qualia structures. The approach is
based on the idea of matching certain lexico-
syntactic patterns conveying a certain seman-
tic relation on the World Wide Web using stan-
dard search engines. We evaluate our approach
qualitatively by comparing our automatically
learned qualia structures with the ones from the
literature, but also quantitatively by presenting
results of a human evaluation.
1 Introduction
Qualia Structures have been originally introduced by
(Pustejovsky, 1991) and are used for a variety of purposes
in Natural Language processing such as the analysis of
compounds (Johnston and Busa, 1996), co-composition
and coercion (Pustejovsky, 1991) as well as for bridging
reference resolution (Bos et al., 1995). Further, it has also
been argued that qualia structures and lexical semantic
relations in general have applications in information re-
trieval (Voorhees, 1994; Pustejovsky et al., 1993). One
major bottleneck however is that currently Qualia Struc-
tures need to be created by hand, which is probably also
the reason why there are no practical system using qualia
structures, but a lot of systems using globally available re-
sources such as WordNet (Fellbaum, 1998) or FrameNet1
1http://framenet.icsi.berkeley.edu/
as source of lexical/world knowledge. The work de-
scribed in this paper addresses this issue and presents
an approach to automatically learning qualia structures
for nominals from the Web. The approach is inspired
in recent work on using the Web to identify instances
of a relation of interest such as in (Markert et al., 2003)
and (Cimiano and Staab, 2004). These approaches are
in essence a combination of the usage of lexico-syntactic
pattens conveying a certain relation of interest such as in
(Hearst, 1992), (Charniak and Berland, 1999), (Iwanska
et al., 2000) or (Poesio et al., 2002) with the idea of us-
ing the web as a big corpus (Resnik and Smith, 2003),
(Grefenstette, 1999), (Keller et al., 2002).
The idea of learning Qualia Structures from the Web is
not only a very practical, it is in fact a principled one.
While single lexicographers creating qualia structures -
or lexicon entries in general - might take very subjective
decisions, the structures learned from the Web do not mir-
ror the view of a single person, but of the whole world as
represented on the World Wide Web. Thus, an approach
learning qualia structures from the Web is in principle
more reliable than letting lexicographers craft lexical en-
tries on their own. Obviously, on the other hand, using
an automatic web based approach yields also a lot of in-
appropriate results which are due to 1) errors produced
by the linguistic analysis (e.g. part-of-speech tagging), 2)
idiosyncrasies of ranking algorithms of search machines,
3) the fact that the Web or in particular search engines
are to a great extent commercially biased, 4) the fact that
people also publish erroneous information on the Web,
and 5) lexical ambiguities. Because of these reasons our
aim is in fact not to replace lexicographers, but to support
them in the task of creating qualia structures on the basis
of the automatically learned qualia structures. The pa-
per is structured as follows: Section 2 introduces qualia
structures and describes the specific qualia structures we
aim to acquire. Section 3 describes our approach in detail
and section 4 presents a quantitative and qualitative eval-
uation of our approach. Before concluding, we discuss
some related work in Section 5.
28
2 Qualia Structures
According to Aristotle, there are four basic factors or
causes by which the nature of an object can be described
(cf. (Kronlid, 2003)):
a0 the material cause, i.e. the material an object is
made of
a0 the agentive cause, i.e. the source of movement, cre-
ation or change
a0 the formal cause, i.e. its form or type
a0 the final cause, i.e. its purpose, intention or aim
In his Generative Lexicon (GL) framework (Puste-
jovsky, 1991) reused Aristotle’s basic factors for the de-
scription of the meaning of lexical elements. In fact he in-
troduced so called Qualia Structures by which the mean-
ing of a lexical element is described in terms of four roles:
a0 Constitutive: describing physical properties of an
object, i.e. its weight, material as well as parts and
components
a0 Agentive: describing factors involved in the bringing
about of an object, i.e. its creator or the causal chain
leading to its creation
a0 Formal: describing that properties which distinguish
an object in a larger domain, i.e. orientation, magni-
tude, shape and dimensionality
a0 Telic: describing the purpose or function of an object
Most of the qualia structures used in (Pustejovsky,
1991) however seem to have a more restricted interpre-
tation. In fact, in most examples the Constitutive role
seems to describe the parts or components of an object,
while the Agentive role is typically described by a verb
denoting an action which typically brings the object in
question into existence. The Formal role normally con-
sists in typing information about the object, i.e. its hyper-
nym or superconcept. Finally, the Telic role describes the
purpose or function of an object either by a verb or nom-
inal phrase. The qualia structure for knife for example
could look as follows (cf. (Johnston and Busa, 1996)):
Formal: artifact tool
Constitutive: blade,handle,...
Telic: cut act
Agentive: make act
Our understanding of Qualia Structure is in line with this
restricted interpretation of the qualia roles. Our aim is to
automatically acquire Qualia Structures from the Web for
nominals, looking for (i) nominals describing the type of
the object, (ii) verbs defining its agentive role, (iii) nomi-
nals describing its parts or components and (iv) nouns or
verbs describing its intended purpose.
3 Approach
Our approach to learning qualia structures from the
Web is on the one hand based on the assumption that
instances of a certain semantic relation can be learned by
matching certain lexico-syntactic patterns more or less
reliably conveying the relation of interest in line with
the seminal work of (Hearst, 1992), who defined the
following patterns conveying a hypernym relation:
(1) a1a3a2a5a4 such as a1a6a2a8a7 , a1a3a2a10a9 , ..., a1a3a2a10a11a13a12a14a7 (anda15or)
a1a3a2a10a11 2
(2) such a1a3a2a5a4 as a1a3a2a8a7 , a1a3a2a10a9 , ... a1a3a2a10a11a16a12a17a7 (anda15or) a1a6a2a10a11
(3) a1a6a2a18a7 , a1a3a2a10a9 , ..., a1a3a2a10a11 (anda15or) other a1a3a2a5a4
(4) a1a6a2a19a4 , (includinga15especially) a1a3a2a20a7 , a1a3a2a10a9 , ..., a1a3a2a10a11a16a12a17a7
(anda15or) a1a3a2a10a11
According to Hearst, from such patterns we can derive
that for all a1a3a2a10a21a23a22a25a24a6a26a28a27a29a26a31a30a8a22a33a32a35a34a37a36a39a38a41a40a37a30a14a34a16a42a44a43a45a1a3a2a5a21a23a22a46a1a3a2 a4a41a47 .
For example, for the expression: Bruises, wounds,
broken bones or other injuries, we would extract:
hypernym(bruise,injury), hypernym(broken bone,injury)
and hypernym(wound,injury). However, it is well known
that Hearst-style patterns occur rarely, such that it seems
intuitive to match them on the Web. So in our case we
are looking not only for the hypernym relation (com-
parable to the Formal-Relation) but for similar patterns
conveying a Constitutive, Telic or Agentive relation. As
currently there is no support for searching using regular
expressions in standard search engines such as Google or
Altavista3, our approach consists of 5 phases (compare
Figure 1):
1. generate for each qualia role a set of so called clues,
i.e. search engine queries indicating the relation of
interest
2. download the snippets of the 10 first Google hits
matching the generated clues 4
3. part-of-speech-tagging of the downloaded snippets
4. match regular expressions conveying the qualia role
of interest
5. weight the returned qualia elements according to
some measure
The outcome of this process are then so called
Weighted Qualia Structures (WQSs) in which every
2a48a50a49a52a51 stands for a noun phrase.
3An exception is certainly the Linguist’s Search Engine
(Resnik and Elkiss, 2003)
4The reason for using only the 10 first hits is to maintain
efficiency. With the current setting the systems needs between
3 and 10 minutes to generate the qualia structure for a given
nominal
29
qualia element in a certain role is weighted according to
some measure. The patterns in our pattern library are ac-
tually tuples a43a36a5a22 a1 a47 wherea36 is a regular expression defined
over part-of-speech tags and a1 a function a1a3a2a5a4a7a6 a40a37a27 a30a9a8a11a10
a4a12a6
a40a37a27 a30a9a8 called the clue. Given a nominal
a6 and a clue a1 , the
query a1 a43 a6 a47 is sent to the Google API and we download the
abstracts of the first a30 documents matching this query and
then process the abstracts to find instances of pattern a36 .
For example, given the clue a13
a43a15a14 a47a17a16a19a18
a4a7a20a9a1
a32a22a21
a4
a18a24a23 a43a15a14 a47 and
the instance computer we would download a30 abstracts
matching the query f(computer), i.e. ”such as comput-
ers”. Hereby a23 a43a15a14 a47 is a function returning the plural form
of x. We implemented this function as a lookup in a lexi-
con in which plural nouns are mapped to their base form.
With the use of such clues, we thus download a number
of Google-abstracts in which a corresponding pattern will
probably be matched thus restricting the linguistic analy-
sis to a few promising pages. The downloaded abstracts
are then part-of-speech tagged using QTag (Tufis and Ma-
son, 1998). Then we match the corresponding pattern a36
in the downloaded snippets thus yielding candidate qualia
elements as output. In our approach we then calculate the
weight of a candidate qualia element a38 for the term a6 we
want to compute the qualia structure for by the Jaccard
Coefficient:
a25a27a26a28a26 a8a30a29 a38a32a31 a27
a6a33a4
a43 a38a35a34
a6
a47
a25a27a26a28a26 a8a30a29 a38a32a31 a27
a6a33a4
a43 a38 a47 a34 a25a27a26a28a26 a8a30a29 a38a32a31 a27
a6a33a4
a43
a6
a47a37a36 a25a27a26a28a26 a8a38a29a45a38a32a31 a27
a6a33a4
a43a45a38a39a34
a6
a47
The result is then a Weighted Qualia Structure (WQS) in
which for each role the qualia elements are weighted ac-
cording to this Jaccard coefficient. In what follows we
describe in detail the procedure for acquiring qualia el-
ements for each qualia role. In particular, we describe
in detail the clues and lexico-syntactic patterns used. In
general, the patterns have been crafted by hand, testing
and refining them in an iterative process, paying attention
to maximize their coverage but also accuracy.
In general it is important to mention that by this approach
we are not able to detect and separate multiple meanings
of words, i.e. to handle polysemy, which is appropriately
accounted for in the framework of the Generative Lexi-
con (Pustejovsky, 1991).
3.1 The Formal Role
To derive qualia elements for the Formal role, we first
download for each of the clues in Table 1 the first 10
abstracts matching the clue and then process them offline
matching the patterns defined over part-of-speech-tags5
thus yielding up to 10 different qualia element candidates
per clue. The patterns are specified in form of regular
expressions, whereby the part-of-speech tags are always
5We use the well-known Penn Treebank tagset described at
http://www.computing.dcu.ie/a40 acahill/tagset.html.
Figure 1: General Approach
given in square brackets after the token. Further, besides
using the traditional regular expression operators such
as a34 , a41 and a42 , we also use Perl-like symbols such as
a43a32a44 denoting any alphabetic character as well as [a-z]
denoting the set of all lower case letters.
As there are 4 different clues for the Formal role, we thus
yield up to 40 qualia elements as potential candidates
to fill the Formal role. In general, we paid attention to
create clues relying on indefinite articles as we found
out that they produce more general and reliable results
than when using definite articles. In order to choose the
correct indefinite article – a or an – or even using no
article at all, we implemented some ad-hoc heuristics
checking if the first letter of the term in question is a
vowel and checking if the term is used more often with
an article or without an article on the Web by a set of
corresponding Google queries. The alternative ’(a/an/?)’
means that we use either the indefinite article ’a’ ’an’
or no article depending on the results of the above
mentioned Google queries.
A general question raised also by Hearst (Hearst, 1992)
is how to deal with NP modification. Hearst’s conclusion
is that this depends on the application. In our case we
mainly remove adjective modifiers, keeping only the
heads of noun phrases as candidate qualia elements.
The lemmatized heads of the NPa45 noun phrase are then
regarded as qualia role candidates for the Formal role.
These candidates are then weighted using the above
defined Jaccard Coefficient measure. Hereby, a noun
phrase is an instance matching the following regular
expression:
NP:=[a-z]+[DT]? ([a-z]+[JJ])+? ([a-z]+[NN(S?)])+,
where the head is the underlined expression, which
is lemmatized and considered as a candidate qualia
element. After some initial experiments we decided not
to use the patterns ’X is Y’ and ’X is a kind of Y’ such
as in a book is an item or a book is a kind of publication
30
as well as the pattern ’Y, including X’ (compare (Hearst,
1992)) as we found that in our settings they delivered
quite spurious results.
Clue Pattern
such as a23 a43 a6 a47 NPa45 ,? such[DT] as[IN] NP
especially a23 a43 a6 a47 NPa45 ,? especially[RB] NP
a23 a43
a6
a47 or other NP or[CC] other[JJ] NP
a45
a23 a43
a6
a47 and other NP and[CC] other[JJ] NP
a45
Table 1: Clues and Patterns for the Formal role
3.2 The Constitutive Role
The procedure for finding elements of the Constitutive
role is similar to the one described above for the Formal
role. The corresponding clues and patterns are given in
Table 2. As above, the candidate qualia elements are then
the lemmatized heads of the noun phrase NPa0 .
Clue Pattern
(a/an)? a6 is made NP is[VBZ] made[VBN]
up of up[RP] of[IN] NPa0
a23 a43
a6
a47 are made up of NP are[VBP] made[VBN]
up[RP] of[IN] NPa0
(a/an)? a6 is made of NP are[VBP] made[VBN]
of[IN] NPa0
a23 a43
a6
a47 are made of NP are[VBP] made[VBN]
of[IN] NPa0
(a/an)? a6 comprises NP comprises[VBZ] NPa0
a23 a43
a6
a47 comprise NP comprise[VBP] NP
a0
(a/an)? a6 consists of NP consists[VBZ] of[IN] NPa0
a23 a43
a6
a47 consist of NP consist[VBP] of[IN] NP
a0
Table 2: Clues and Patterns for the Constitutive Role
As an additional heuristic, we test if the lemmatized
head of NPa0 is an element of the following list contain-
ing nouns denoting an indication of amount: a1 variety,
bundle, majority, thousands, million, millions, hundreds,
number, numbers, set, sets, series, rangea2 and further-
more this NPa0 is followed by the preposition ’of’. In
that case we would take the head of the noun phrase after
the preposition ’of’ as potential candidate of the Consti-
tutive role. For example, when considering a conversa-
tion is made up of a series of observable interpersonal
exchanges, we would take exchange as a potential qualia
element candidate instead of series.
3.3 The Telic Role
The Telic Role is in principle acquired in the same way as
the Formal and Constitutive roles with the exception that
the qualia element is not only the head of a noun phrase,
but also a verb or a verb followed by a noun phrase. Table
3 gives the corresponding clues and patterns. In particu-
lar, the returned candidate qualia elements are the lem-
matized underlined expressions in PURP:=a43 w+[VB] NP
a15 NP a15 be[VB]
a43 w+[VBD]).
Clue Pattern
purpose of a a6 is purpose[NN] of[IN]
NPa4 is[VBZ] (to[TO])? PURP
purpose of a23 a43 a6 a47 is purpose[NN] of[IN]
NPa4 is[VBZ] (to[TO])? PURP
(a/an)? a6 is used to (Aa15aa15Ana15an) NPa4 is[VBZ]
used[VBN] to[TO] PURP
a23 a43
a6
a47 are used to NPa4 are[VBZ] used[VBN]
to[TO] PURP
Table 3: Clues and Patterns for the Telic Role
3.4 The Agentive Role
As mentioned in (Hearst, 1992), it is not always as
straightforward to find lexico-syntactic patterns reliably
conveying a certain relation. In fact, we did not find any
patterns reliably identifying qualia elements for the Agen-
tive role. Certainly, it would have been possible to find
the source of the creation by using patterns such as X is
made by Y or X is produced by Y. However, we found
that these patterns do not reliably convey a verb describ-
ing how an object is brought into existence. The fact
that it is far from straightforward to find patterns indi-
cating an Agentive role is further corroborated by the re-
search in (Yamada and Baldwin, 2004), in which only
one pattern indicating a qualia relation is used, namely
’NN BE V[+en]’ in order to match passive constructions
such as the book was written. On the other hand it is
clear that constructing a reliable clue for this pattern is
not straightforward given the current state-of-the-art con-
cerning search engine queries. Nevertheless, in order to
also get results for the Agentive role, we apply a different
method here. Instead of issuing a query which is used to
search for possible candidates for the role, we take advan-
tage of the fact that the verbs which describe how some-
thing comes into being, particularly artificial things, are
often quite general phrases like ”make, produce, write,
build...”. So instead of generating clues as above, we
calculate the value a3a5a4a6a4a8a7a10a9a12a11a14a13
a21a16a15a18a17a20a19a18a21a23a22
a3a25a24a27a26a29a28a31a30a33a32a5a24 a32a27a24a27a34a36a35a38a37a40a39
a15a42a41
a3a27a4a20a4a43a7a10a9a12a11a14a13
a21a12a15a18a17a20a19a16a15a42a41
for the nominal we want to acquire a qualia structure for
as well as the following verbs: build, produce, make,
write, plant, elect, create, cook, construct and design. If
this value is over a threshold (0.0005 in our case), we as-
sume that it is a valid filler of the Agentive qualia role.
4 Evaluation
We evaluate our approach for the lexical elements knife,
beer, book, which are also discussed in (Johnston and
31
Busa, 1996) or (Pustejovsky, 1991), as well as computer,
an abstract noun, i.e. conversation, as well as two very
specific multi-term words, i.e. natural language process-
ing and data mining. We give the automatically learned
weighted Qualia Structures for these entries in Figures 3,
4, 5 and 6. The evaluation of our approach consists on
the one hand of a discussion of the weighted qualia struc-
tures, in particular comparing them to the ideal struc-
tures form the literature. On the other hand, we also
asked a student at our institute to assign credits to each
of the qualia elements from 0 (incorrect) to 3 (totally cor-
rect) whereby 1 credit meaning ’not totally wrong’ and 2
meaning ’still acceptable’.
4.1 Quantitative Evaluation
The distribution of credits for each qualia role and term
is given in Table 4. It can be seen that with three
exceptions: beera10 formal, booka10 agentive as well as
beera10 constitutive, ’3’ is the mark assigned in most
cases to the automatically learned qualia elements. Fur-
ther, for almost every query term and qualia role, at
least 50% of the automatically learned qualia structures
have a mark of ’2’ or ’3’ – the only exceptions being
beera10 formal with 45.45%, booka10 agentive with 33.33%
and beera10 constitutive with 28.57%. In general this
shows that the automatically learned qualia roles are in-
deed reasonable. Considering the average over all the
terms (’All’ in the table), we observe that the qualia role
which is recognized most reliably is the Telic one with
73.15% assignments of credit ’3’ and 75.93% of cred-
its ’2’ or ’3’, followed by the Agentive role with 71.43%
assignments of credit 3. The results for the Formal and
Constitutive role are still reasonable with 62.09% assign-
ments of credit ’3’ and 66.01% assignments of credits
’2’ or ’3’ for the Formal role; and respectively 61.61%
and 64.61% for the Constitutive role. The worst results
are achieved for the Constitutive role due to the fact that
26.26% of the qualia elements are regarded as totally
wrong. Table 5 supports the above claims and shows
the average credits assigned by the human evaluator per
query term and role. It shows again that the roles with
the best results are the Agentive and Telic roles, while the
Formal and Constitutive roles are not identified as accu-
rately. This is certainly due to the fact that the patterns
for the Telic role are much less ambiguous than the ones
for the Formal and Constitutive roles. Finally, we also
discuss the correlation between the credits assigned and
the Jaccard Coefficient. Figure 2 shows this correlation.
While for the Formal role the correlation is as expected,
i.e. the higher the credit assigned, the higher also the Jac-
card Coefficient, for the Constitutive and Telic roles this
correlation is unfortunately less clear, thus making the
task of finding a cut-off threshold more difficult.
4.2 Qualitative Evaluation & Discussion
In this section we provide a more subjective evaluation
of the automatically learned qualia structures by compar-
ing them to ideal qualia structures discussed in the liter-
ature wherever possible. In particular, we discuss more
in detail the qualia structure for book, knife and beer and
leave the detailed assessment of the qualia structures for
computer, natural language processing, data mining and
conversation to the interested reader.
For book, the first four candidates of the Formal role,
i.e. product, item, publication and document are very ap-
propriate, but alluding to the physical object meaning of
book as opposed to the meaning in the sense of informa-
tion container (compare (Pustejovsky, 1991). As candi-
dates for the Agentive role we have make, write and cre-
ate which are appropriate, write being the ideal filler of
the Agentive role according to (Pustejovsky, 1991). For
the Constitutive role of book we get – besides it at the
first position which could be easily filtered out – sign
(2nd position), letter (3rd position) and page (6th posi-
tion), which are quite appropriate. The top four candi-
dates for the Telic role are give, select, read and purchase.
It seems that give is emphasizing the role of a book as a
gift, read is referring to the most obvious purpose of a
book as specified in the ideal qualia structures of (Puste-
jovsky, 1991) as well as (Johnston and Busa, 1996) and
purchase denotes the more general purpose of a book, i.e.
to be bought.
The first element of the Formal role of knife unfortunately
denotes the material it is typically made of, i.e. steel, but
the next 5 elements are definitely appropriate: weapon,
item, kitchenware, object and instrument. The ideal ele-
ment artifact tool (compare (Johnston and Busa, 1996))
can be found at the 10th position. The results are inter-
esting in that on the one hand the most prominent mean-
ing of knife according to the web is the one of a weapon.
On the other hand our results are more specific, classify-
ing a knife as kitchenware instead of merely as an arti-
fact tool. Very interesting are the specific and accurate
results at the end of the list. The reason why they appear
at the end is that the Jaccard Coefficient ranks them lower
because they are more specific, thus appearing less fre-
quently. This shows that using some other measure less
sensitive to frequency could yield more accurate results.
The fillers of the Agentive role produce, make and create
seem all appropriate, whereby make corresponds exactly
to the ideal filler for the Agentive role as mentioned in
(Johnston and Busa, 1996). The results for the Constitu-
tive role contain not only parts but also materials a knife
is made of and thus contain more information than the
typical qualia structures assumed in the literature. The
best results are (in this order) blade, metal, steel, wood
and handle at the 6th position. In fact, in the ideal qualia
structure in (Johnston and Busa, 1996) blade and han-
32
Formal
0 1 2 3
Book 2/17 (11.76%) 4/17 (23.52%) 1/17 (5.88%) 10/17 (58.82%)
Computer 8/28 (28.57%) 1/28 (3.57%) 2/28 (7.14%) 17/28 (60.71%)
Knife 3/16 (18.75%) 0/16 (0%) 0/16 (0%) 13/16 (81.25%)
Beer 12/22 (54.54%) 0/22 (0%) 2/22 (9.09%) 8/22 (36.36%)
Data Mining 6/25 (24%) 0/25 (0%) 0/25 (0%) 19/25 (76%)
Natural Language Processing 2/15 (13.33%) 1/15 (6.66%) 0/15 (0%) 12/15 (80%)
Conversation 10/30 (33.33%) 4/30 (13.33%) 0/30 (0%) 16/30 (53.33%)
All 43/153 (28.10%) 11/153 (7.19%) 6/153 (3.92%) 95/153 (62.09%)
Agentive
Book 0/3 (0%) 2/3 (66.66%) 0/3 (0%) 1/3 (33.33%)
Computer 0/1 (0%) 0/1 (0%) 0/1 (0%) 1/1 (100%)
Knife 0/3 (0%) 0/3 (0%) 0/3 (0%) 3/3 (100%)
Beer 0/3 (0%) 1/3 (33.33%) 0/3 (0%) 2/3 (66.66%)
Data Mining 0/1 (0%) 0/1 (0%) 0/1 (0%) 1/1 (100%)
Natural Language Processing 0/1 (0%) 0/1 (0%) 0/1 (0%) 1/1 (100%)
Conversation 1/2 (50%) 0/2 (0%) 0/2 (0%) 1/2 (50%)
All 1/14 (7.14%) 3/14 (21.43%) 0/14 (0%) 10/14 (71.43%)
Constitutive
Book 8/29 (27.58%) 4/29 (13.79%) 1/29 (3.44%) 16/29 (55.17%)
Computer 6/26 (23.07%) 1/26 (3.84%) 0/26 (0%) 19/26 (73.07%)
Knife 4/15 (26.66%) 0/15 (0%) 0/15 (0%) 11/15 (73.33%)
Beer 5/7 (71.42%) 0/7 (0%) 0/7 (0%) 2/7 (28.57%)
Data Mining 0/1 (0%) 0/1 (0%) 0/1 (0%) 1/1 (100%)
Natural Language Processing
Conversation 3/21 (14.28%) 4/21 (19.04%) 0/21 (0%) 14/21 (66.66%)
All 26/99 (26.26%) 9/99 (9%) 3/99 (3%) 61/99 (61.61%)
Telic
Book 3/22 (13.63%) 2/22 (9.09%) 3/22 (13.63%) 14/22 (63.63%)
Computer 0/27 (0%) 3/27 (11.11%) 0/27 (0%) 24/27 (88.88%)
Knife 5/18 (27.77%) 0/18 (0%) 0/18 (0%) 13/18 (72.22%)
Beer
Data Mining 2/22 (9.09%) 4/22 (18.18%) 0/22 (0%) 16/22 (72.72%)
Natural Language Processing 1/6 (16.66%) 0/6 (0%) 0/6 (0%) 5/6 (83.33%)
Conversation 6/13 (46.15%) 0/13 (0%) 0/13 (0%) 7/13 (53.84%)
All 17/108 (15.74%) 9/108 (8.33%) 3/108 (2.78%) 79/108 (73.15%)
Table 4: Distribution of credits for each role and term
Formal Agentive Constitutive Telic
Book 2.12 1.67 1.86 2.27
Computer 2 3 2.23 2.78
Knife 2.44 3 2.2 2.17
Beer 1.27 2.33 0.96 n.a.
Data Mining 2.28 3 3 2.36
Natural Language Processing 2.47 3 n.a. 2.5
Conversation 1.73 1.5 2.19 1.62
All 1.99 2.36 2.02 2.33
Table 5: Average credits for each role
33
 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 0  0.5  1  1.5  2  2.5  3
Jaccard Coefficient
Credit
FormalConstitutive
Telic
Figure 2: Average Jaccard Coefficient value per credit
dle are mentioned as fillers of the Constitutive role, while
there are no elements describing the materials of which a
knife is made of. Finally, the top four candidates for the
Telic role are kill, slit, cut and slice, whereby cut corre-
sponds to the ideal filler of the qualia structure for knife
as mentioned in (Johnston and Busa, 1996).
Considering the qualia structure for beer, it is surpris-
ing that no purpose has been found. The reason is that
currently no results are returned by Google for the clue
a beer is used to and the four snippets returned for the
purpose of a beer contain expressions of the form the
purpose of a beer is to drink it which is not matched
by our patterns as it is a pronoun and not matched by
our NP pattern (unless it is matched by an error as in the
Qualia Structure for book in Figure 4). Considering the
results for the Formal role, the elements drink (1st), al-
cohol (2nd) and beverage (4th) are much more specific
than liquid as given in (Pustejovsky, 1991), while thing
at the 3rd position is certainly too general. Furthermore,
according to the automatically learned qualia structure,
beer is made of rice, malt and hop, which are perfectly
reasonable results. Very interesting are the results con-
coction and libation for the Formal role of beer, which
unfortunately were rated low by our evaluator (compare
Figure 3).
Overall, the discussion has shown that the results pro-
duced by our method are reasonable when compared to
the qualia structures from the literature. In general, our
method produces in some cases additional qualia candi-
dates, such as the ones describing the material a knife is
typically made of. In other cases it discovers more spe-
cific candidates, such as for example weapon or kitchen-
ware as elements of the Formal role for knife instead of
the general term artifact tool.
5 Related Work
There is quite a lot of work related to the use of lin-
guistic patterns to discover certain ontological relations
from text. Hearst’s (Hearst, 1992) seminal work had the
aim of discovering taxonomic relations from electronic
dictionaries. The precision of the is-a-relations learned
is 61/106 (57.55%) when measured against WordNet
as gold standard, which is comparable to our results.
Hearst’s idea has been reapplied by different researchers
with either slight variations in the patterns used (Iwan-
ska et al., 2000), to acquire knowledge for anaphora res-
olution (Poesio et al., 2002), or to discover other kinds
of semantic relations such as part-of relations (Char-
niak and Berland, 1999) or causation relations (Girju and
Moldovan, 2002).
Instead of matching these patterns in a large text collec-
tion, some researchers have recently turned to the Web
to match these patterns such as in (Cimiano and Staab,
2004) or (Markert et al., 2003). (Cimiano and Staab,
2004) for example aim at learning instance-of as well as
taxonomic (is-a) relations. This is very related to the ac-
quisition of the Formal role proposed here. (Markert et
al., 2003) aim at acquiring knowledge for anaphora res-
olution, while (Etzioni et al., 2004) aim at learning the
complete extension of a certain concept. For example,
they aim at finding all the actors in the world.
Our approach goes further in that it not only learns typing,
superconcept or instance-of relations, but also Constitu-
tive and Telic relations.
There also exist approaches specifically aiming at learn-
ing qualia elements from corpora based on machine
learning techniques. (Claveau et al., 2003) for example
use Inductive Logic Programming to learn if a given verb
is a qualia element or not. However, their approach goes
not as far as learning the complete qualia structure for a
lexical element in an unsupervised way as presented in
our approach. In fact, in their approach they do not dis-
tinguish between different qualia roles and restrict them-
selves to verbs as potential fillers of qualia roles. (Ya-
mada and Baldwin, 2004) present an approach to learn-
ing Telic and Agentive relations from corpora analyzing
two different approaches: one relying on matching cer-
tain lexico-syntactic patterns as in the work presented
here, but also a second approach consisting in training
a maximum entropy model classifier. Their conclusion is
that the results produced by the classification approach
correlate better with two hand-crafted gold standards.
34
The patterns used by (Yamada and Baldwin, 2004) differ
substantially from the ones used in this paper, which is
mainly due to the fact that search engines do not provide
support for regular expressions and thus instantiating a
pattern as ’V[+ing] Noun’ is impossible in our approach
as the verbs are unknown a priori.
Finally, (Pustejovsky et al., 1993) present an interesting
framework for the acquisition of semantic relations from
corpora not only relying on statistics, but guided by the-
oretical lexicon principles.
6 Conclusion
We have presented an approach to automatically learn-
ing Qualia Structures from the Web. Such an approach is
especially interesting either for lexicographers aiming at
constructing lexicons, but even more for natural language
processing systems relying on deep lexical knowledge as
represented by qualia structures. We have in particular
shown that the qualia structures learned by our system
are reasonable. In general, it is valid to claim that our
system is the first one automatically producing complete
qualia structures for a given nominal.
Our system can be tested online at http://km.aifb.uni-
karlsruhe.de/pankow/qualia/. Further work will aim at
improving the system but also at using the automatically
learned structures within NLP applications.
Acknowledgments The work reported in this paper
has been partially supported by the SmartWeb project6,
funded by the German Ministry of Research. Thanks also
to Laura Goebes for assisting in the evaluation of the sys-
tem.

References
J. Bos, P. Buitelaar, and M. Mineur. 1995. Bridging
as coercive accomodation. In E. Klein, S. Manand-
har, W. Nutt, and J. Siekmann, editors, Working Notes
of the Edinburgh Conference on Computational Logic
and Natural Language Processing (CLNLP-95).
E. Charniak and M. Berland. 1999. Finding parts in
very large corpora. In Proceedings of the 37th Annual
Meeting of the ACL, pages 57–64.
P. Cimiano and S. Staab. 2004. Learning by googling.
SIGKDD Explorations, 6(2), December.
V. Claveau, P. Sebillot, C. Fabre, and P. Bouillon. 2003.
Learning semantic lexicons from a part-of-speech and
semantically tagged corpus using inductive logic pro-
gramming. Journal of Machine Learning Research,
(4):493–525.
O. Etzioni, M. Cafarella, D. Downey, S. Kok, A.-M.
Popescu, T. Shaked, S. Soderland, D.S. Weld, and
A. Yates. 2004. Web-scale information extraction in
KnowItAll (preliminary results). In Proceedings of the
13th World Wide Web Conference, pages 100–109.
C. Fellbaum. 1998. WordNet, an electronic lexical
database. MIT Press.
R. Girju and M. Moldovan. 2002. Text mining for causal
relations. In Proceedings of the FLAIRS Conference,
pages 360–364.
G. Grefenstette. 1999. The WWW as a resource for
example-based MT tasks. In Proceedings of ASLIB’99
Translating and the Computer 21.
M.A. Hearst. 1992. Automatic acquisition of hyponyms
from large text corpora. In Proceedings of the 14th In-
ternational Conference on Computational Linguistics,
pages 539–545.
L.M. Iwanska, N. Mata, and K. Kruger. 2000. Fully
automatic acquisition of taxonomic knowledge from
large corpora of texts. In L.M. Iwanksa and S.C.
Shapiro, editors, Natural Language Processing and
Knowledge Processing, pages 335–345. MIT/AAAI
Press.
M. Johnston and F. Busa. 1996. Qualia structure and the
compositional interpretation of compounds.
F. Keller, M. Lapata, and O. Ourioupina. 2002. Using
the web to overcome data sparseness. In Proceedings
of EMNLP-02, pages 230–237.
F. Kronlid. 2003. Modes of explanation - aristotelian
philosophy and pustejovskyan linguistics. Ms. Univer-
sity of Gteborg.
K. Markert, N. Modjeska, and M. Nissim. 2003. Us-
ing the web for nominal anaphora resolution. In
EACL Workshop on the Computational Treatment of
Anaphora.
M. Poesio, T. Ishikawa, S. Schulte im Walde, and
R. Viera. 2002. Acquiring lexical knowledge for
anaphora resolution. In Proceedings of the 3rd Con-
ference on Language Resources and Evaluation.
J. Pustejovsky, P. Anick, and S. Bergler. 1993. Lexi-
cal semantic techniques for corpus analysis. Compu-
tational Lingustics, Special Issue on Using Large Cor-
pora II, 19(2):331–358.
J. Pustejovsky. 1991. The generative lexicon. Computa-
tional Linguistics, 17(4):209–441.
P. Resnik and A. Elkiss. 2003. The linguist’s
search engine: Getting started guide. Tech-
nical Report LAMP-TR-108/CS-TR-4541/UMIACS-
TR-2003-109, University of Maryland, College Park,
November.
P. Resnik and N. Smith. 2003. The web as a parallel
corpus. Computational Lingusitics, 29(3):349–380.
D. Tufis and O. Mason. 1998. Tagging Romanian
Texts: a Case Study for QTAG, a Language Indepen-
dent Probabilistic Tagger. In Proceedings of the First
International Conference on Language Resources and
Evaluation (LREC), pages 589–96.
E.M. Voorhees. 1994. Query expansion using lexical-
semantic relations. In Proceedings of the 17th annual
international ACM SIGIR conference on Research and
development in information retrieval, pages 61–69.
I. Yamada and T. Baldwin. 2004. Automatic discov-
ery of telic and agentive roles from corpus data. In
Proceedings of the The 18th Pacific Asia Conference
on Language, Information and Computation (PACLIC
18).
