Automatic Retrieval and Clustering of Similar Words 
Dekang Lin 
Department of Computer Science 
University of Manitoba 
Winnipeg, Manitoba, Canada R3T 2N2 
lindek@ cs.umanitoba.ca 
Abstract 
Bootstrapping semantics from text is one of the 
greatest challenges in natural language learning. 
We first define a word similarity measure based on 
the distributional pattern of words. The similarity 
measure allows us to construct a thesaurus using a 
parsed corpus. We then present a new evaluation 
methodology for the automatically constructed the- 
saurus. The evaluation results show that the the- 
saurns is significantly closer to WordNet than Roget 
Thesaurus is. 
1 Introduction 
The meaning of an unknown word can often be 
inferred from its context. Consider the following 
(slightly modified) example in (Nida, 1975, p.167): 
(1) A bottle of tezgiiino is on the table. 
Everyone likes tezgiiino. 
Tezgiiino makes you drunk. 
We make tezgiiino out of corn. 
The contexts in which the word tezgiiino is used 
suggest that tezgiiino may be a kind of alcoholic 
beverage made from corn mash. 
Bootstrapping semantics from text is one of the 
greatest challenges in natural language learning. It 
has been argued that similarity plays an important 
role in word acquisition (Gentner, 1982). Identify- 
ing similar words is an initial step in learning the 
definition of a word. This paper presents a method 
for making this first step. For example, given a cor- 
pus that includes the sentences in (1), our goal is to 
be able to infer that tezgiiino is similar to "beer", 
"wine", "vodka", etc. 
In addition to the long-term goal of bootstrap- 
ping semantics from text, automatic identification 
of similar words has many immediate applications. 
The most obvious one is thesaurus construction. An 
automatically created thesaurus offers many advan- 
tages over manually constructed thesauri. Firstly, 
the terms can be corpus- or genre-specific. Man- 
ually constructed general-purpose dictionaries and 
thesauri include many usages that are very infre- 
quent in a particular corpus or genre of documents. 
For example, one of the 8 senses of "company" in 
WordNet 1.5 is a "visitor/visitant", which is a hy- 
ponym of "person". This usage of the word is prac- 
tically never used in newspaper articles. However, 
its existance may prevent a co-reference recognizer 
to rule out the possiblity for personal pronouns to 
refer to "company". Secondly, certain word us- 
ages may be particular to a period of time, which 
are unlikely to be captured by manually compiled 
lexicons. For example, among 274 occurrences of 
the word "westerner" in a 45 million word San Jose 
Mercury corpus, 55% of them refer to hostages. If 
one needs to search hostage-related articles, "west- 
emer" may well be a good search term. 
Another application of automatically extracted 
similar words is to help solve the problem of data 
sparseness in statistical natural language process- 
ing (Dagan et al., 1994; Essen and Steinbiss, 1992). 
When the frequency of a word does not warrant reli- 
able maximum likelihood estimation, its probability 
can be computed as a weighted sum of the probabil- 
ities of words that are similar to it. It was shown in 
(Dagan et al., 1997) that a similarity-based smooth- 
ing method achieved much better results than back- 
off smoothing methods in word sense disambigua- 
tion. 
The remainder of the paper is organized as fol- 
lows. The next section is concerned with similari- 
ties between words based on their distributional pat- 
terns. The similarity measure can then be used to 
create a thesaurus. In Section 3, we evaluate the 
constructed thesauri by computing the similarity be- 
tween their entries and entries in manually created 
thesauri. Section 4 briefly discuss future work in 
clustering similar words. Finally, Section 5 reviews 
related work and summarize our contributions. 
768 
2 Word Similarity 
Our similarity measure is based on a proposal in 
(Lin, 1997), where the similarity between two ob- 
jects is defined to be the amount of information con- 
tained in the commonality between the objects di- 
vided by the amount of information in the descrip- 
tions of the objects. 
We use a broad-coverage parser (Lin, 1993; Lin, 
1994) to extract dependency triples from the text 
corpus. A dependency triple consists of two words 
and the grammatical relationship between them in 
the input sentence. For example, the triples ex- 
tracted from the sentence "I have a brown dog" are: 
(2) (have subj I), (I subj-of have), (dog obj-of 
have), (dog adj-mod brown), (brown 
adj-mod-of dog), (dog det a), (a det-of dog) 
We use the notation IIw, r, w'll to denote the fre- 
quency count of the dependency triple (w, r, w ~) in 
the parsed corpus. When w, r, or w ~ is the wild 
card (*), the frequency counts of all the depen- 
dency triples that matches the rest of the pattern are 
summed up. For example, Ilcook, obj, *11 is the to- 
tal occurrences of cook-object relationships in the 
parsed corpus, and I1., *, *11 is the total number of 
dependency triples extracted from the parsed cor- 
pus. 
The description of a word w consists of the fre- 
quency counts of all the dependency triples that 
matches the pattern (w,., .). The commonality be- 
tween two words consists of the dependency triples 
that appear in the descriptions of both words. For 
example, (3) is the the description of the word 
"cell". 
(3) Ilcell, subj-of, absorbll=l 
Ilcell, subj-of, adapt\[l=l 
Ilcell, subj-of, behavell=l 
\[Icell, pobj-of, in11=159 
\[\[cell, pobj-of, insidell=16 
Ilcell, pobj-of, intoll=30 
Ilcell, nmod-of, abnormalityll=3 
Ilcell, nmod-of, anemiall=8 
Ilcell, nmod-of, architecturell=l 
\[\[cell, obj-of, attackl\[=6 
\[\[cell, obj-of, bludgeon\[\[=l 
\[Icell, obj-of, callll=l 1 
Hcell, obj-of, come froml\[=3 
Ilcell, obj-of, containll--4 
Ilcell, obj-of, decoratell=2 
...*** 
I\[cell, nmod, bacteriall=3 
Ilcell, nmod, blood vesselH=l 
IIcell, nmod, bodYll=2 
Ilcell, nmod, bone marrowll=2 
Ilcell, nmod, burialH=l 
Ilcell, nmod, chameleonll=l 
Assuming that the frequency counts of the depen- 
dency triples are independent of each other, the in- 
formation contained in the description of a word is 
the sum of the information contained in each indi- 
vidual frequency count. 
To measure the information contained in the 
statement IIw, r, w' H=c, we first measure the amount 
of information in the statement that a randomly se- 
lected dependency triple is (w, r, w') when we do 
not know the value of IIw, r,w'll. We then mea- 
sure the amount of information in the same state- 
ment when we do know the value of II w, r, w' II. The 
difference between these two amounts is taken to be 
the information contained in Hw, r, w' \[l=c. 
An occurrence of a dependency triple (w, r, w') 
can be regarded as the co-occurrence of three 
events: 
A: a randomly selected word is w; 
B: a randomly selected dependency type is r; 
C: a randomly selected word is w ~. 
When the value of Ilw, r,w'll is unknown, we 
assume that A and C are conditionally indepen- 
dent given B. The probability of A, B and C co- 
occurring is estimated by 
PMLE( B ) PMLE( A\[B ) PMLE( C\[B ), 
where PMLE is the maximum likelihood estimation 
of a probability distribution and 
P.LE(B) = II*,*,*ll' 
P.,~E(AIB ) = II*,~,*ll ' 
P, LE(CIB) = 
When the value of Hw, r, w~H is known, we can 
obtain PMLE(A, B, C) directly: 
PMLE(A, B, C) = \[\[w, r, wll/\[\[*, *, *H 
Let I(w,r,w ~) denote the amount information 
contained in Hw, r,w~\]\]=c. Its value can be corn- 
769 
simgindZe(Wl, W2) = ~'~(r,w)eTCwl)NTCw2)Are{subj.of.obj-of} min(I(Wl, r, w), I(w2, r, w) ) 
simHindte, (Wl, W2) = ~,(r,w)eT(w,)nT(w2) min(I(wl, r, w), I(w2, r, w)) 
\]T(Wl)NT(w2)I simcosine(Wl,W2) = x/IZ(w~)l×lZ(w2)l 
2x IT(wl)nZ(w2)l simDice(Wl, W2) = iT(wl)l+lT(w2) I 
simJacard (Wl, W2) = T(wl )OT(w2)l T(wl) + T(w2)l-IT(Wl)rlT(w2)l 
Figure 1: Other Similarity Measures 
puted as follows: 
I(w,r,w') 
= _ Iog(PMLE(B)PMLE(A\]B)PMLE(CIB)) 
--(-- log PMLE(A, B, C)) 
- log IIw,r,wfl×ll*,r,*ll 
-- IIw,r,*ll xll*,r,w'll 
It is worth noting that I(w,r,w') is equal to 
the mutual information between w and w' (Hindle, 
1990). 
Let T(w) be the set of pairs (r, w') such that 
log Iw'r'w'lr×ll*'r'*ll is positive. We define the sim- wlr~* X *~r~w ! 
ilarity sim(wl, w2) between two words wl and w2 
as follows: 
)"~(r,w)eT(w, )NT(w~)(I(Wl, r, w) + I(w2, r, w) ) 
~-,(r,w)eT(wl) I(Wl, r, w) q- ~(r,w)eT(w2) I(w2, r, w) 
We parsed a 64-million-word corpus consisting 
of the Wall Street Journal (24 million words), San 
Jose Mercury (21 million words) and AP Newswire 
(19 million words). From the parsed corpus, we 
extracted 56.5 million dependency triples (8.7 mil- 
lion unique). In the parsed corpus, there are 5469 
nouns, 2173 verbs, and 2632 adjectives/adverbs that 
occurred at least 100 times. We computed the pair- 
wise similarity between all the nouns, all the verbs 
and all the adjectives/adverbs, using the above sim- 
ilarity measure. For each word, we created a the- 
saurus entry which contains the top-N ! words that 
are most similar to it. 2 The thesaurus entry for word 
w has the following format: 
w (pos) : Wl, 81, W2, 82,. • • , WN, 8N 
where pos is a part of speech, wi is a word, 
si=sim(w, wi) and si's are ordered in descending 
'We used N=200 in our experiments 
2The resulting thesaurus is available at: 
http://www.cs.umanitoba.caflindek/sims.htm. 
order. For example, the top-10 words in the noun, 
verb, and adjective entries for the word "brief" are 
shown below: 
brief (noun): affidavit 0.13, petition 0.05, memo- 
randum 0.05, motion 0.05, lawsuit 0.05, depo- 
sition 0.05, slight 0.05, prospectus 0.04, docu- 
ment 0.04 paper 0.04 .... 
brief(verb): tell 0.09, urge 0.07, ask 0.07, meet 
0.06, appoint 0.06, elect 0.05, name 0.05, em- 
power 0.05, summon 0.05, overrule 0.04 .... 
brief (adjective): lengthy 0.13, short 0.12, recent 
0.09, prolonged 0.09, long 0.09, extended 0.09, 
daylong 0.08, scheduled 0.08, stormy 0.07, 
planned 0.06 .... 
Two words are a pair of respective nearest neigh- 
bors (RNNs) if each is the other's most similar 
word. Our program found 543 pairs of RNN nouns, 
212 pairs of RNN verbs and 382 pairs of RNN 
adjectives/adverbs in the automatically created the- 
saurus. Appendix A lists every 10th of the RNNs. 
The result looks very strong. Few pairs of RNNs in 
Appendix A have clearly better alternatives. 
We also constructed several other thesauri us- 
ing the same corpus, but with the similarity mea- 
sures in Figure 1. The measure simHinate is the 
same as the similarity measure proposed in (Hin- 
dle, 1990), except that it does not use dependency 
triples with negative mutual information. The mea- 
sure simHindle,, is the same as simHindle except that 
all types of dependency relationships are used, in- 
stead of just subject and object relationships. The 
measures simcosine, simdice and simdacard are ver- 
sions of similarity measures commonly used in in- 
formation retrieval (Frakes and Baeza-Yates, 1992). 
Unlike sim, simninale and simHinater, they only 
770 
210g P(c) ,~ simwN(wl, w2) = maxc~ eS(w~)Ac2eS(w2) (maxcesuper(c~)nsuper(c2) log P(cl )+log P(c2) ! 
21R(~l)nR(w2)l simRoget(Wl, W2) = IR(wx)l+lR(w2)l 
where S(w) is the set of senses of w in the WordNet, super(c) is the set of (possibly indirect) 
superclasses of concept c in the WordNet, R(w) is the set of words that belong to a same Roget 
category as w. 
Figure 2: Word similarity measures based on WordNet and Roget 
make use of the unique dependency triples and ig- 
nore their frequency counts. 
3 Evaluation 
In this section, we present an evaluation of automat- 
ically constructed thesauri with two manually com- 
piled thesauri, namely, WordNetl.5 (Miller et al., 
1990) and Roget Thesaurus. We first define two 
word similarity measures that are based on the struc- 
tures of WordNet and Roget (Figure 2). The simi- 
larity measure simwN is based on the proposal in 
(Lin, 1997). The similarity measure simRoget treats 
all the words in Roget as features. A word w pos- 
sesses the feature f if f and w belong to a same 
Roget category. The similarity between two words 
is then defined as the cosine coefficient of the two 
feature vectors. 
With simwN and simRoget, we transform Word- 
Net and Roget into the same format as the automat- 
ically constructed thesauri in the previous section. 
We now discuss how to measure the similarity be- 
tween two thesaurus entries. Suppose two thesaurus 
entries for the same word are as follows: 
'tO : '//31~ 81~'//12~ 82~... ~I)N~S N 
Their similarity is defined as: 
(4) 
sis  
For example, (5) is the entry for "brief (noun)" in 
our automatically generated thesaurus and (6) and 
(7) are corresponding entries in WordNet thesaurus 
and Roget thesaurus. 
(5) brief (noun): affidavit 0.13, petition 0.05, 
memorandum 0.05, motion 0.05, lawsuit 0.05, 
deposition 0.05, slight 0.05, prospectus 0.04, 
document 0.04 paper 0.04. 
(6) brief (noun): outline 0.96, instrument 0.84, 
summary 0.84, affidavit 0.80, deposition 
0.80, law 0.77, survey 0.74, sketch 0.74, 
resume 0.74, argument 0.74. 
(7) brief (noun): recital 0.77, saga 0.77, 
autobiography 0.77, anecdote 0.77, novel 
0.77, novelist 0.77, tradition 0.70, historian 
0.70, tale 0.64. 
According to (4), the similarity between (5) and 
(6) is 0.297, whereas the similarities between (5) 
and (7) and between (6) and (7) are 0. 
Our evaluation was conducted with 4294 nouns 
that occurred at least 100 times in the parsed cor- 
pus and are found in both WordNetl.5 and the Ro- 
get Thesaurus. Table 1 shows the average similarity 
between corresponding entries in different thesauri 
and the standard deviation of the average, which 
is the standard deviation of the data items divided 
by the square root of the number of data items. 
Since the differences among simcosine, simdice and 
simJacard are very small, we only included the re- 
sults for simcosine in Table 1 for the sake of brevity. 
It can be seen that sire, Hindler and cosine are 
significantly more similar to WordNet than Roget 
is, but are significantly less similar to Roget than 
WordNet is. The differences between Hindle and 
Hindler clearly demonstrate that the use of other 
types of dependencies in addition to subject and ob- 
ject relationships is very beneficial. 
The performance of sim, Hindler and cosine are 
quite close. To determine whether or not the dif- 
ferences are statistically significant, we computed 
their differences in similarities to WordNet and Ro- 
get thesaurus for each individual entry. Table 2 
shows the average and standard deviation of the av- 
erage difference. Since the 95% confidence inter- 
771 
Table I: Evaluation with WordNet and Roget 
WordNet 
Roget 
sim 
Hindle~ 
cosine 
Hindle 
average 
0.178397 
0.212199 
0.204179 
0.199402 
0.164716 
~av~ 
0.001636 
0.001484 
0.001424 
0.001352 
0.001200 
Roget 
average 
WordNet 0.178397 
sim 0.149045 
Hindler 0.14663 
cosine 0.135697 
Hindle 0.115489 
aav 8 
0.001636 
0.001429 
0.001383 
0.001275 
0.001140 
vals of all the differences in Table 2 are on the posi- 
tive side, one can draw the statistical conclusion that 
simis better than simnindle ~, which is better than 
simcosine. 
Table 2: Distribution of Differences 
sim-Hindle~ 
sim-cosine 
Hindler-cosine 
sim-Hindle~ 
sim-cosine 
Hindle~-cosine 
WordNet 
average ffavg 
0.008021 0.000428 
0.012798 0.000386 
0.004777 0.000561 
Roget 
average trav8 
0.002415 0.000401 
0.013349 0.000375 
0.010933 0.000509 
4 Future Work 
Reliable extraction of similar words from text cor- 
pus opens up many possibilities for future work. For 
example, one can go a step further by constructing a 
tree structure among the most similar words so that 
different senses of a given word can be identified 
with different subtrees. Let wl,..., Wn be a list of 
words in descending order of their similarity to a 
given word w. The similarity tree for w is created 
as follows: 
• Initialize the similarity tree to consist of a sin- 
gle node w. 
• For i=l, 2 ..... n, insert wi as a child of wj 
such that wj is the most similar one to wi 
among {w, Wl ..... wi-1}. 
For example, Figure 3 shows the similarity tree for 
the top-40 most similar words to duty. The first 
number behind a word is the similarity of the word 
to its parent. The second number is the similarity of 
the word to the root node of the tree. 
duty 
responsibility 0.21 
role 0.12 0.ii 
I action 0.ii 
0.21 
0.i0 
change 0.24 0.08 
l__.rule 0.16 0.08 
l__.restriction 0.27 0.08 
I I ban 0.30 0.08 I 
l__.sanction 0.19 0.08 
I schedule 0.Ii 0.07 I 
regulation 0.37 0.07 
challenge 0.13 0.07 
l__.issue 0.13 0.07 
I reason 0.14 0.07 
I matter 0.28 0.07 
measure 0.22 0.07 ' 
obligation 0.12 0.10 
power 0.17 0.08 
I jurisdiction 0.13 0.08 
I right 0.12 0.07 
I control 0.20 0.07 
I ground 0.08 0.07 
accountability 0.14 0.08 
experience 0.12 0.07 
post 0.14 0.14 
job 0.17 0.I0 
l__work 0.17 0.i0 
I training 0.Ii 0.07 
position 0.25 0.10 
task 0.10 0.10 
I chore 0.ii 0.07 
operation 0.10 0.10 
I function 0.i0 0.08 
I mission 0.12 0.07 
I I patrol 0.07 0.07 I 
staff 0.i0 0.07 
penalty 0.09 0.09 
I fee 0.17 0.08 
I tariff 0.13 0.08 
I tax 0.19 0.07 
reservist 0.07 0.07 
Figure 3: Similarity tree for "duty" 
Inspection of sample outputs shows that this al- 
gorithm works well. However, formal evaluation of 
its accuracy remains to be future work. 
5 Related Work and Conclusion 
There have been many approaches to automatic de- 
tection of similar words from text corpora. Ours is 
772 
similar to (Grefenstette, 1994; Hindle, 1990; Ruge, 
1992) in the use of dependency relationship as the 
word features, based on which word similarities are 
computed. 
Evaluation of automatically generated lexical re- 
sources is a difficult problem. In (Hindle, 1990), 
a small set of sample results are presented. In 
(Smadja, 1993), automatically extracted colloca- 
tions are judged by a lexicographer. In (Dagan et 
al., 1993) and (Pereira et al., ! 993), clusters of sim- 
ilar words are evaluated by how well they are able 
to recover data items that are removed from the in- 
put corpus one at a time. In (Alshawi and Carter, 
1994), the collocations and their associated scores 
were evaluated indirectly by their use in parse tree 
selection. The merits of different measures for as- 
sociation strength are judged by the differences they 
make in the precision and the recall of the parser 
outputs. 
The main contribution of this paper is a new eval- 
uation methodology for automatically constructed 
thesaurus. While previous methods rely on indirect 
tasks or subjective judgments, our method allows 
direct and objective comparison between automati- 
cally and manually constructed thesauri. The results 
show that our automatically created thesaurus is sig- 
nificantly closer to WordNet than Roget Thesaurus 
is. Our experiments also surpasses previous experi- 
ments on automatic thesaurus construction in scale 
and (possibly) accuracy. 
Acknowledgement 
This research has also been partially supported by 
NSERC Research Grant OGP121338 and by the In- 
stitute for Robotics and Intelligent Systems. 

References 
Hiyan Alshawi and David Carter. 1994. Training 
and scaling preference functions for disambiguation. 
Computational Linguistics, 20(4):635-648, Decem- 
ber. 
Ido Dagan, Shaul Marcus, and Shaul Markovitch. 1993. 
Contextual word similarity and estimation from sparse 
data. In Proceedings of ACL-93, pages 164-171, 
Columbus, Ohio, June. 
Ido Dagan, Fernando Pereira, and Lillian Lee. 1994. 
Similarity-based estimation of word cooccurrence 
probabilities. In Proceedings of the 32nd Annual 
Meeting of the ACL, pages 272-278, Las Cruces, NM. 
Ido Dagan, Lillian Lee, and Fernando Pereira. 1997. 
Similarity-based method for word sense disambigua- 
tion. In Proceedings of the 35th Annual Meeting of 
the ACL, pages 56-63, Madrid, Spain. 
Ute Essen and Volker Steinbiss. 1992. Cooccurrence 
smoothing for stochastic language modeling. In Pro- 
ceedings oflCASSP, volume 1, pages 161-164. 
W. B. Frakes and R. Baeza-Yates, editors. 1992. In. 
formation Retrieval, Data Structure and Algorithms. 
Prentice Hall. 
D. Gentner. 1982. Why nouns are learned before verbs: 
Linguistic relativity versus natural partitioning. In 
S. A. Kuczaj, editor, Language development: Vol. 2. 
Language, thought, and culture, pages 301-334. Erl- 
baum, Hillsdale, NJ. 
Gregory Grefenstette. 1994. Explorations in Auto- 
matic Thesaurus Discovery. Kluwer Academic Press, 
Boston, MA. 
Donald Hindle. 1990. Noun classification from 
predicate-argument structures. In Proceedings of 
ACL-90, pages 268-275, Pittsburg, Pennsylvania, 
June. 
Dekang Lin. 1993. Principle-based parsing without 
overgeneration. In Proceedings of ACL-93, pages 
112-120, Columbus, Ohio. 
Dekang Lin. 1994. Principarman efficient, broad- 
coverage, principle-based parser. In Proceedings of 
COLING-94, pages 482-488. Kyoto, Japan. 
Dekang Lin. 1997. Using syntactic dependency as local 
context to resolve word sense ambiguity. In Proceed- 
ings of ACL/EACL-97, pages 64-71, Madrid, Spain, 
July. 
George A. Miller, Richard Beckwith, Christiane Fell- 
baum, Derek Gross, and Katherine J. Miller. 1990. 
Introduction to WordNet: An on-line lexical database. 
International Journal of Lexicography, 3(4):235-244. 
George A. Miller. 1990. WordNet: An on-line lexi- 
cal database. International Journal of Lexicography, 
3(4):235-312. 
Eugene A. Nida. 1975. ComponentialAnalysis of Mean- 
ing. The Hague, Mouton. 
F. Pereira, N. Tishby, and L. Lee. 1993. Distributional 
Clustering of English Words. In Proceedings of ACL- 
93, pages 183-190, Ohio State University, Columbus, 
Ohio. 
Gerda Ruge. 1992. Experiments on linguistically based 
term associations. Information Processing & Man- 
agement, 28(3):317-332. 
Frank Smadja. 1993. Retrieving collocations from text: 
Xtract. Computational Linguistics, 19(1): 143-178. 
