J 
J 
,II 
Parallel Translations as Sense Discriminators 
Nancy IDE 
Department of Computer Science 
Vassar College 
124 Raymond Avenue 
Poughkeepsie, NY 12604-0520 USA 
lde@cs vassar edu 
Abstract 
This article reports the results of a 
prehmlnary analysis of translation 
equivalents in four languages from different 
language famdles, extracted from an on-hne 
parallel corpus of George Orwell's Nmeteen 
Eighty-Four The goal of the study is to 
determine the degree to which translatmn 
equivalents for different meamngs of a 
polysemous word In Enghsh are lexlcahzed 
differently across a variety of languages, and 
to detelmme whether this information can be 
used to structure or create a set of sense 
distinctions useful in natural language 
processing apphcatmns A coherence Index 
is computed that measures the tendency for 
different senses o1 the same English word to 
be lexlcahzed differently, and flora this data 
a clustering algorithm is used to create sense 
hierat chles 
Introduction 
It ~s well known that the most nagging issue for 
word sense disamblguanon (WSD) Is the 
definmon of just what a word sense is At its 
base, the problem Is a philosophical and 
linguistic one that is far from being resolved 
However, work in automated language 
processing has led to effotts to flnd practical 
means to dlstmgmsh word senses, at least to the 
degree that they are useful for natural language 
processing tasks such as summarization, 
document retrieval, and machine translataon 
Several criteria have been suggested and 
exploited to automatically determine the sense 
of a word m context (see Ide and V6roms, 1998), 
including syntactic behavior, semantic and 
pragmatic knowledge, and especially in more 
recent empirical studies, word co-occurrence 
within syntactic relations (e g, Hearst, 1991, 
Yarowsky, 1993), words co-occurring m global 
context (e g, Gale et al, 1993, Yarowsky, 1992 
Schutze, 1992, 1993), etc No clear criteria have 
emerged, however, and the problem continues to 
loom large for WSD work 
The notion that cross-hngual comparison can be 
useful fol sense dlsamblguauon has served as a 
basis for some recent work on WSD Foi 
example, Brown et al (1991)and Gale et al 
(1992a, 1993) used the parallel, aligned Hansard 
Corpus of Canadian Parhamentary debates foi 
WSD, and Dagan et al (1991) and Dagan and 
Ital (1994) used monohngual corpora of Hebrew 
and German and a bilingual dictionary These 
studies rely on the assumption that the mapping 
between words and word senses vanes 
significantly among languages For example, the 
word duty in English t~anslates into French as 
devoir m ~ts obhgatlon sense, and tmpOt m ~ts 
tax sense By determining the translation 
52 
,..--.,.~ .- 
eqmvalent ot duty in a parallel French text, the 
correct sense of the Enghsh word is identified 
These studies exploit th~s lnformatmn m order to 
gather co-occurrence data for the different 
senses, which ts then used to dtsamb~guate new 
texts In related work, Dywk (1998) used 
patterns of translational relatmns in an Enghsh- 
Norwegian paralle ! corpus (ENPC, Oslo 
Umverslty) to define semantic propemes such as 
synonymy, ambtgmty, vagueness, and semantic 
helds and suggested a derivation ot- semantic 
representations for signs (eg, lexemes), 
captunng semantm relatmnshlps such as 
hyponymy etc, fiom such translatmnal relatmns 
Recently, Resnlk and Yarowsky (1997) 
suggested that fol the purposes ot WSD, the 
different senses of a wo~d could be detelmlned 
by considering only sense d~stmctmns that are 
lextcahzed cross-hngmstlcally In particular, 
they propose that some set of target languages 
be ~dent~fied, and that the sense d~stmctmns to 
be considered for language processing 
appllcatmns and evaluatmn be restricted to those 
that are reahzed lexlcally in some minimum 
subset of those languages This idea would seem 
to p~ovtde an answer, at least m part, to the 
problem of determining different senses of a 
word mtumvely, one assumes that ff another 
language lexlcahzes a word m two or more 
ways, there must be a conceptual monvatmn If 
we look at enough languages, we would be 
likely to fred the s~gmficant lexlcal differences 
that dehmtt different senses of a word 
However, th~s suggestmn raises several 
questions Fo~ instance, ~t ~s well known that 
many amb~gumes are preserved across 
languages (for example, the French tntdrYt and 
the Enghsh interest), especmlly languages that 
are relatively closely related Assuming this 
problem can be overcome, should differences 
found m closely related languages be given 
lesser (or greater) weight than those found m 
more distantly related languages 9 More 
generally, which languages should be considered 
for this exermse 9 All languages 9 Closely related 
languages9 Languages from different language 
famlhes '~ A mixture of the two 9 How many 
languages, and of which types, would be 
"enough" to provide adequate lnfotmanon tot 
this purpose~ 
There ts also the questmn ot the crlterm that 
would be used to estabhsh that a sense 
distinction is "lexlcahzed cross-hngu~stmally" 
How consistent must the d~stlnCtlOn be 9 Does it 
mean that two concepts are expressed by 
mutually non-lntetchangeable lexmal items in 
some slgmficant number ot other languages, or 
need tt only be the case that the option ot a 
different lexlcahzatlon exists m a certain 
percentage of cases 9 
Another conslderatmn ts where the cross-hngual 
mformatlon to answer these questmns would 
come from Using bdmgual dictionaries would 
be extremely tedmus and error-prone, g~ven the 
substantial d~vergence among d~ctlonanes in 
terms of the kinds and degree of sense 
dlstmctmns they make Resmk and Yalowsky 
(1997) suggest EutoWordNet (Vossen, 1998) as 
a possible somce of mformatmn, but, given that 
EuroWordNet ts pttmatdy a lexmon and not a 
corpus, ~t is subject to many of the same 
objections as for bl-hngual dictionaries 
An alternative would be to gather the 
reformation from parallel, ahgned corpma 
Unlike bilingual and muttt-hngual dictionaries, 
translatmn eqmvalents xn parallel texts a~e 
determined by experienced translatols, who 
evaluate each instance ot a word's use m context 
rather than as a part of the meta-hngmst~c 
actlvlty of classifying senses for mclusmn in a 
dictionary However, at present very few parallel 
ahgned corpora exist The vast majority ot these 
are bl-texts, mvolwng only two languages, one 
of which is very often English Ideally, a serious 
53 
evaluation of Resnik and Yarowsky's proposal 
would include parallel texts m languages from 
several different language families, and, to 
maximally ensure that the word m question is 
used in the exact same sense across languages, ~t 
would be preferable that the same text were used 
over all languages in the study The only 
currently avadable parallel corpora for more 
than two languages are Olwell's Nmeteen 
Eighty-Four (Erjavec and Ide, 1998), Plato's 
Repubhc (Erjavec, et al, 1998), the MULTEXT 
Journal .o/ the Commt.~ston corpus (Ide and 
V6roms, 1994), and the Bible (Resnlk, et al, m 
press) It is likely that these corpora do not 
provide enough appropriate data to reliably 
determine sense distinctions Also, ~t Is not clear 
how the lexlcahzatlon of sense distractions 
across languages Is affected by genre, domain, 
style, etc 
Thls paper attempts to provide some prehmlnary 
answers to the questions outhned above, In order 
to eventually determine the degree to which the 
use of parallel data ts vmble to determine sense 
distinctions, and, ff so, the ways in which th~s 
reformation might be used Given the lack of 
lalge parallel texts across multiple languages, 
the study is necessarily hmlted, however, close 
exammanon of a small sample of parallel data 
can, as a first step, provide the basis and 
dlrectmn for more extensive studies 
1 Methodology 
I have conducted a small study using parallel, 
aligned versmns ot George Orwell's Nineteen 
Etghtv-Fo,lr (Euavec and Ide, 1998)m five 
languages Enghsh, Slovene, Estonian, 
Romanlan, and Czech I The study therefole 
Involves languages from four language families 
The O~well parallel corpus also includes vers|ons o) 
Ntneteen-E~gho Four m Hungarian, Bulgarmn, 
Latwan, Llthuaman, Se~bmn, and Russmn 
(Germanic, Slavic, Fmno-Ugrec, and Romance), 
two languages from the same family (Czech and 
Slovene), as well as one non-Indo-European 
language (Estoman) 
Nmeteen Eighty-Four Is a text of about 100,000 
words, translated directly from the original 
English m each of the other languages The 
parallel versions of the text are sentence-aligned 
to the English and tagged for part of speech 
Although Nineteen Eighty-Four is a work of 
fiction, Orwell's prose IS not highly stylized and, 
as such, it provides a reasonable sample ot 
modern, ordinary language that ~s not tied to a 
given topic or sub-domain (such as newspapers, 
technical reports, etc ) Furthermore, the 
translations of the text seem to be relatively 
faithful to the original for instance, over 95% ot 
the sentence alignments in the full pmallel 
corpus of seven languages are one-to-one 
(Prlest-Dorman, et al, 1997) 
Nine ambiguous English words were considered 
hard, head, country, hne, promise, shght, seize, 
scrap, float The first four were chosen because 
they have been used in other dlsamb~guatlon 
studies, the latter five were chosen from among 
the words used m the Senseval dlsamblguatlon 
exercise (Kllgamff and Palmer, forthcoming) In 
all cases, the study was necessarily hmlted to 
words that occurred frequently enough in the 
Orwell text to warrant consideration 
F~ve hundred forty-two sentences conta|nmg an 
occurrence or occurrences (Including 
morphological variants) of each of the nine 
words were extracted from the Enghsh text, 
together w~th the parallel sentences m which 
they occur m the texts ot the four comparison 
languages (Czech, Estonian, Romantan, 
Slovene) As Walks and Stevenson (1998) have 
pointed out, pa~t-of-speech tagging 
accomplishes a good portion of the work ot 
semantic dlsamb~guatmn, therefore occmrences 
of wolds that appemed in the data in more than 
54 
one part of speech were grouped separately 2 
The Enghsh occurrences were then grouped 
usmg the sense distinctions m WordNet, 
(version 1 6) \[Miller et al, 1990, Fellbaum, 
1998\]) The sense categonzatmn was performed 
by the author and two student assistants, results 
from the three were compared and a final, 
mutually agreeable set of sense assignments 
was estabhshed 
For each of the four comparison languages, the 
corpus of sense-grouped parallel sentences were 
sent to a llngmst and natl,ve speaker of the 
comparison language The hngmsts were asked 
to provide the lexlcal item m each parallel 
sentence that corresponds to the ambiguous 
Enghsh word If inflected, they were asked to 
provide both the inflected form and the root 
form In addttmn, the lmgmsts were asked to 
indicate the type of translatmn, according to the 
dtstmctmns given m Table 1 
For over 85% of the Enghsh word occurrences 
(corresponding to types 1 and 2 m Table 1), a 
specific lexlcal item or items could be identified 
as the translation equivalent for the 
corresponding Enghsh word For comparison 
purposes, each translanon equivalent was 
represented by ~ts lemma (or the lemma of the 
toot form in the case of derivatives) and 
associated w~th the WordNet sense to which it 
corresponds 
In order to determine the degree to which the 
assigned sense dlstlncttons correspond to 
translation eqmvalents, a coherence index ( Cl) 
was computed that measures how often each pmr 
of senses is translated usmg the same word as 
well as the consistency with which a g~ven se,ls,z 
~s translated with the same word ~ Note that the 
z The adJective and adverb senses of hard are 
consadeied together because the distinction is not 
consistent across the translations used m the study 
Note that the CI ~s similar to semanuc entropy 
(Melamed, 1997) However, Melamed computes 
CIs do not determine whether or not a sense 
dtstmctton can be lextcahzed in the target 
language, but only the degree to whmh they are 
lexicahzed differently m the translated text 
However, tt can be assumed that the CIs provide 
a measure of the tendency to lex~cahze different 
WordNet senses differently, which can m turn 
be seen as an mdtcatmn of the degree to which 
the distraction ts vahd 
For each ambiguous word, the CI Is computed 
for each pair of senses, as follows 
S<q t> 
Cl(sqS, ) = '=1 
m rnrt 
where 
@ n ~s the number of comparison languages 
under consideration, 
nl~q and m,, are the nt~mber of occurrences ol- 
sense sqand sense s~ m the Enghsh corpus, 
respectively, including occurrences that 
have no idenufiable translation, 
s<~ ~>m ts the number of times that senses q 
and r are translated by the same lex~cal Item 
m language t, i e, 
 x=y 
t ~tJan ~( q ), r~oan~( r ) 
The CI ts a value between 0 and 1, computed by 
examining clusters of occurrences translated by 
the same word In the othel languages If sense 
and sense ) are consistently translated w~th the 
same wo~d in each comparison language, then 
Cl(s, s~) = 1, if they are translated with a 
different word m every occurrence, Cl(s, ~) = 0 
In general, the CI for pans of different senses 
provides an index of thmr relatedness, t e, the 
greater the value of Cl(s, sj), the more frequently 
occurrences of-sense t and sense j are translated 
with the same lextcal item When t = j, we 
entropy tOl wold types, lather than word senses 
55 
obtain a measure of the coherence of a ~lven sense 
Type Meaning 
1 A slngle lexlcal Item is used to translate the En@izsh equivalent (possibly a 
2 The English word is translated by a phrase of two or more words or a compound, 
meaning as the slngle English word 
3 The En@izsh word is not lexzcalized in the translation 
4 A pronoun is substituted for the English word In the translation 
An English phrase contalnmng the ambiguous word Is translated by a single 
language which has a broader or more specific meanlng, or by a phrase in whl 
corresponding to the English word Is not explicltl~ lexlcallzed 
Table 1 Translation types and their trequencles 
% 
dizen 
whl%h h 
6% 
6% 
6% 
of 
s p same 
Word # Description 
hard 1 1 difficult 
2 
head i 
i 
i 
1 
Table 2 
1 2 _meta~horlcally hard 
_\] 3 not yielding to pressure , 
1 4 very strong or ~lgorous, ar 
2 I wlth force or vigor (adv) 
3 earnestly, intently (adv) 
i_ ~art of the body ..... 
3 intellect 
4 _r~le_!r, ch,%ef 
7 front, front part 
WoldNet senses ot hard and head 
CIs were also computed for each language 
individually as well as for different language 
groupings Romaman, Czech, and Estonian 
(three different language families) Czech and 
Slovene (same family), Romaman, Czech, 
Slovene (Indo-European, and Estonian (non- 
Indo-European) 
To better visualize the relationship between 
senses, a hierarchical clustering algorithm was 
applied to the CI data to generate trees reflecting 
sense proximity 4 Finally, in order to determine 
the degree to which the linguistic relaUon 
between languages may affect coherence, a 
correlation was run among CIs for all pairs of 
the four target languages 
Fol example, Table 2 gives the senses of hard 
and head that occurred in the data s The CI data 
.s 'sobS' hard and head are given in Tables 3 and 4 
~uous CIs measuring the aff, mty of a sense with 
itself--that is, the tendency for all occurrences 
of that sense to be translated wlth the same 
word--show that all of the s,x senses of ha,d 
have greatel internal consistency tfian athmty 
with other senses, with senses 1 1 ("dlff|cult" - 
CI = 56) and 13 (,'not soft,, - ci = 63) 
registenng the h,ghest internal consistency 6 The 
same holds true for three of the four senses of 
head, while the CI for senses 1 3 ("Intellect") 
and 1 1 ("part of the body") is higher than the CI 
for 1 3/1 3 
WordNet 
Sense 
2 1 
2 3 
1 4 
1 3 
1 1 
1 2 
21 23 1 4 13 
0 50 
o 13 i ool 
0 O0 0 25 i O0 
0 04 0 50 0 17 0 56 
0 19 0 00 0 00 0 00 
0 00 0 00 0 25 0 21 
Table 3 CIs for hard 
I i 12 
0,,63 
0 00 0 50 
2 Results 
Although the data sample is small, It gives some 
insight into ways m which a larger sample might 
contribute to sense discrimination 
4 Developed by Andleas Stolcke 
Results tor all words m the study are avadable at 
http//www cs vassar edu/~~de/wsd/cross-hng html 
6 Senses 2 3 and 1 4 have CIs ot 1 because each ot 
these senses exists m a single occurrence m the 
corpus, and have theretote been dlscarded horn 
consideration ot CIs to~ individual senses We a~e 
currently mvesugatmg the use oI the Kappa staUst~c 
(Carletta, 1996) to normahze these sparse data 
56 
WordNet 
Sense 1 1 1 3 1 4 1 7 
1 1 0 69 
1 3 0 53 0 45 
1 4 0 12 0 07, 0 50 
1 7 0 40 0 001 0 00 1 00 
Table 4 CIs for head 
Figure 2 shows the sense clusters for hard 
generated from the CI data 7 The senses fall into 
two mare clusters, w~th the two most internally 
consistent senses (1 1 and 1 3) at the deepest 
level of each ot the respecuve groups The two 
adverbml forms 8 are placed in separate groups, 
leflectmg thmr semantic proximity to the 
different adjecuval meanings of hard The 
clusters for head (Figure 2) stmdarly show two 
dlstmct groupings, each anchored in the two 
senses with the h~ghest internal consistency and 
the lowest mutual CI ("part of the body" (1 1) 
and "ruler, chief" (1 4)) 
The h~erarchtes apparent m the cluster graphs 
make intuitive sense Structured hke dictmnary 
enmes, the clusters for hard and head might 
appeal as m F~gure 1 This ts not dissimilar to 
actual dlctLonary entries for hard and head, for 
example, the enmes for hard in four differently 
constructed dlctmnanes ( Colhns Enghsh (CED), 
Longman's (LDOCE), OxJotd Advanced 
Learner's (OALD), and COBUILD) all hst the 
"'d~fficult" and "not soft" senses first and second, 
whmh, since most dictionaries hst the most 
common Ol frequently used senses hrst, reflects 
the gross dlwslon apparent m the clusters 
Beyond this, ~t ~s difficult to assess the 
7 Foi the purposes ot the cluster analys~s, CIs of l 00 
resulting from a single occurrrence were normahzed 
to 5 
8 Because ~oot to, ms were used m the analysis, no 
dzstlncUon m UanslaUon eqmvalents was made tor 
part ot speech 
correspondence between the senses In the 
dictionary entries and the clusters The 
remamlng WordNet senses are scattered at 
various places within the entries or, m some 
cases, split across various senses The 
h~erarchlcal relatmns apparent m the clusters are 
not reflected m the d~cttonary enmes, smce the 
senses are for the most part presented in flat, 
hnear hsts However, It is interesting to note that 
the first five senses of hard In the COBUILD 
d~cuonary, which is the only d~cttonary in the 
group constructed on the bas~s of colpus 
examples 9 and presents senses m ruder of 
frequency, correspond to hve of the six 
WordNet senses in thls study WordNet's 
"metaphorically hard" is spread over multiple 
senses in the COB UILD, as it.is In the other 
d~ctlonarles 
HARD 
HEAD 
I 1 dlfflcult 
2 vlgorously 
II 1 a not soft 
b strong 
2 a earnestly 
b metaphorlcally hard 
I 1 a part of the body 
b zntellect 
2 front, front part 
II ruler, chlef 
Flgme 1 Clusteis tol hard and head suuctured as 
dlcuonary entt ~es 
The results tor dlftment language groupings 
show that the tendency to lextcahze senses 
differently is not aftected by language d~stance 
(Table 5) In fact, the mean CI fol Estonian, the 
only non-Indo-European language m the study, 
~s lower than that for any other group, mdmatmg 
that WordNet sense dtstmctmns are slightly less 
hkely to be lexlcahzed differently m Estonian 
9 Edmons ot the LDOCE (1987 vexsmn) and OALD 
(1985 version) dictlonalles consulted m this study 
ple-date edmons ol those same d~ctlonanes based on 
colpus evidence 
57 
Correlations of CIs for each language pair 
(Table 5) also show no relationship between the 
degree to which sense d~stmcuons are 
lexlcahzed differently and language distance 
This is contrary to results obtained by Resmk 
and Yarowsky (subm,tted), who, using a memc 
slmdar to the one used in this study, found that 
that non-Indo-European languages tended to 
lexlcallze English sense d~stmctlons more than 
Indo-European languages, especially at finer- 
grained levels However, their translation data 
was generated by native speakers presented with 
Isolated sentences in English, who were asked to 
provide the translation for a given word In the 
sentence It is not clear how this data compares 
to translations generated by trained translators 
working with full context 
Lanquaqe qroup Averaqe CI 
ALL 0 27 
RO/ES/SL 0 28 
SL/CS 0 28 
RO/SL/CS 0 27 
ES 0 26 
Table 5 Average CI values 
Lanqs Hard Country Llne Head Ave 
ES/CS 0 86 0 72 0 68 0 69 0 74 
RO/SL 0 73 0 78 0 68 1 00 0 80 
RO/CS 0 83 0 66 0 67 0 72 0 72 
SL/CS 0 88 0 51 0 72 0 71 0 71 
RO/ES 0 97 0 26 0 70 0 98 0 73 
ES/SL 0 73 0 59 0 90 0 99 0 80 
Table 6 CI correlauon tor the tour target languages 
I -I 
I ........................... I 
I 
m~nlmum dlstance = 0 249399 
m~nlmum d~stance = 0 434856 
mlnlmum dlstance = 0 555158 
mlnlmum dlstance = 0 602972 
m~nlmum dlstance = 0 761327 
I .................... >21 
I .................... >ii 
I ......... >23 
l ......... >13 
l ............... >14 
I ............... >12 
(13) (23) 
(12) (1,4) 
(ii) (21) 
(1412) (2313) 
( 2 3 1 3 1 4 1 2 ) ( 2 111 ) 
Figure 2 Cluster tree and distance measures tor the sm senses of hard 
I ......................... >14 
-i I ................ > i i 
I---- ..................... 1 J ................ > i 3 
I ...................... >17 
mlnlmum dlstance = 0 441022 
mlnlmum dlstance = 0 619052 
mln~mum dlstance = 0 723157 
(13) (ll) 
(17) (1113) 
(111317) (14) 
F,gure 3 Cluster tree and dmtance measures tot the tout senses ot head 
58 
Conclusion 
The small sample m this study suggests that 
cross-hngual lexlcahzat~on can be used to define 
and structure sense d~stmct~ons The cluster 
graphs above provide mformat~on about 
relations among WordNet senses that could be 
used, for example, to determine the granularity 
of sense differences, whtch m turn could be used 
in tasks such as machine translatton, mtormaUon 
retrieval, etc For example, it is hkely that as 
sense dtstmcttons become finer, the degree of 
error ~s less severe Resmk and Yarowsky 
(1997) suggest that confusing freer-grained 
sense dtstmctlons should be penahzed less 
severely than confusing grosser d~stmct~ons 
when evaluatmg the performance of sense 
dtsambtguatt0n systems The clusters also 
provide insight into the lexlcallzatlon of sense 
dtstmcttons related by various semantic relations 
(metonymy, meronymy, etc ) across languages, 
for instance, the "part of the body" and 
"intellect" senses of head are lex~cahzed with 
the same ~tem a s~gnlficant portion of the t~me 
across all languages, reformation that could be 
used m machine translatton In addtt~on, cluster 
data such as that presented here could be used m 
lexicography, to determine a mole detaded 
hierarchy of relations among senses in 
dtct~onary entries 
It is less clear how cross-hngual reformation can 
be used to determine sense d~st~nctlons 
independent of a pre-deflned set, such as the 
WordNet senses used here In an effort to 
explore how thts mlght be done, I have used the 
small sample from thts study to create word 
groupmgs from "back translations" (l e, 
additional translations m the original language 
ot the translations m the target language) and 
developed a metric that uses th~s mformatton to 
determine relatedness between occurrences, 
whtch ~s m turn used to cluster occurrences into 
sense groups I have also compared sets of back 
translations for words representing the various 
WordNet senses, which provtde word groups 
s~mdar to WordNet synsets Interestingly, there 
ts virtually no overlap between the WordNet 
synsets and word groups generated from back 
translations The results show, however, that 
sense dlstmctlons useful for natural language 
processing tasks such as machme translanon 
could potentsally be determined, ot at least 
influenced, by constdeHng this mformatton The 
automatically generated synsets themselves may 
also be useful m the same apphcatlons; where 
WordNet synsets (and ontologtes) have been 
used tn the past 
More work needs to be done on the topic of 
cross-hngual sense determination, utthzmg 
substantially larger parallel corpora that include 
a variety ot language types as well as texts fiom 
several genres This small study explores a 
possible methodology to apply when such 
resources become avatlable 
Acknowledgements 
The author would hke to gratefully acknowledge 
the contrtbut~on of those who provided the 
translatton mfotmat~on Tomaz Eua~ec 
(Slovene), Kadrt Muxschnek (Estonian), 
Vladtmlr Petkevtc (Czech), and Dan Tubs 
(Romanlan), as well as Dana Fleut and Darnel 
Khne, who helped to transcrtbe and evaluate the 
data Special thanks to Dan Melamed and 
Hlnrtch Schutze for their helpful comments 
59 
\[\] 
\[\] 
in 
\[\] 
in 
i 
i 
Hg 
nn 
i 
an 
i 
am 

References 

Ca~letta, Jean (1996) Assessing Agreement on 
Classthcatton Tasks The Kappa Stat~st~t. 
Computational Lmgulstlcs, 22(2), 249-254 

Dagan, Ido and Ita~, Alon (1994) Wo~d sense 
dlsambxguat~on using a second language 
monohngual corpus Computattonal Ltngmsttcs, 
20(4), 563-596 

Dagan, Ido, Ital, Alon, and Schwall, Ulnke (1991) 
Two languages a~e more mformattve than one 
Proceedings of the 29th Annual Meettng of the 
Assoctatton for Computattonal Ltngutsttcs, 18-21 
June 1991, Berkeley, Cahfornm, 130-137 

Dyvtk, Helge (1998) Translations as Semantic 
Mirrors Proceedmgs of Workshop W13 
Multzlmguahty in the Lextcon II, The 13th Biennial 
European Conference on Arttftctal lntelhgence 
(ECA198), Brighton, UK, 24-44 

Eqavec, Tomaz and Ide, Nancy (1998) The 
MULTEXT-EAST Corpus Proceedlng~ of the 
Fltst International Conference on Language 
Resources and Evaluatton, 27-30 May 1998, 
Granada, 971-74 

Erjavec, Tomaz, Lawson, Ann, and Romary, Laurent 
(1998) East meets West Producing Multflmgual 
Resources m a European Context Pioceedtngs of 
the Ftrst Internattonal Conference on Language 
Resources and Evaluation, 27-30 May 1998, 
Gtanada, 981-86 

Fellbaum, Chttstmne (ed) (1998) WordNet An 
Electrontc Lexlcal Database MIT Press, 
Cambridge, Massachusetts 

Gale, Wdham A, Church, Kenneth W and 
Yatowsky, Davtd (1993) A method tor 
dlsamblguatmg word senses m a large cmpus 
Computers and the Humamtles, 26, 415-439 

,...; Hearst, M'attl A (1991) Noun homograph 
• ' dlsamblguatlon using local:'~.'0ntext m large 
corpora Proceedtngs of the 7th Annual Conference 
of the Umver~lt~ of Waterloo Centre for the New 
OED and Text ReaeaJch, Oxford, Umted Kingdom, 
1-19 

Ide, Nancy and V61oms, Jean (1998) Word sense 
d~samb~guat~on The state of the alt Computational 
Lmgut~ttc~, 24 1, 1-40 

Kdgar~ttt, Adam and Palmer, Ma~tha, Eds 
(forthcoming) Proceedmgs ot the Senseval Word 
Sense D~samb~guatlon Workshop, Specml double 
~ssue ot- Computer~ and the Humamttes, 33 4-5 

Leacock, Claudia, Towell, Geoffrey and Voorhees, 
Ellen (1993) Corpus-based stattstlcal sense 
resolution Proceedtng~ of the ARPA Human 
Language Technology Worsl~shop, San Francisco, 
Morgan Kautman 

Melamed, I Dan (1997) Measuring Semantic 
Entropy ACL-SIGLEX Workshop Taggmg Tert 
wtth Lextcal Semanttcs Why, What, and How ~ 
April 4-5, 1997, Washington, D C, 41-46 

Mtllet, George A, Beckwlth, Richard T Fellbaum. 
Chnstlane D, Gross, Derek and Miller, Katherine 
J (1990) WordNet An on-hne lexlcal database 
Internattonal Journal of Lextcograph~, 3(4), 235-244 

Pnest-Dorman, Gteg, Eqavec, Tomaz, lde. Nancy 
and Petkevlc, Vladtmfi (1997) Corpus Malkup 
COP Ploject 106 MULTEXT-East Dehverable 
D2 3 F Available at 

Resmk, Phdlp, B~oman Olsen, Man and Dlab, Mona 
(1999) Creatmg a Parallel Corpus from the Book 
ot 2000 Tongues Compute~3 and the Humamttea, 
33 1-2, 129-53 

Resmk, Phdlp and Yarowsky, David (submitted) 
Dlstmgmshmg systems and dtstmgutsbmg sense,, 
New evaluation methods for word sense 
dlsambiguatton Submltted to Natural Language 
Engozeermg 

Resmk, Phdlp and Yalowsky, David (1997) A 
perspectwe on wo~d sense d~samblguat~on methods 
and thell evaluatton ACL-SIGLEX Workshop 
Tagging Text with Lexlcal Semantics Wh), ~¢¢hat, 
and How)I April 4-5, 1"997, Washington, D C, 79-86 

Schutze, Hmtlch (1992) Dimensions ot meanmg 
Proceedings of Supetcomputtng '92 IEEE 
Computer Society P~ess. Los Alamxtos, Cahtorma, 
787-796 

Schutze, Hlnnch (1993) Word space In Hanson. 
Stephen J, Cowan, Jack D, and Gales, C Lee 
(Eds) Advance~ tn Neural Infotmatton Pro~e~auzg 
Systems 5, Morgan Kauffman, San Mateo, 
Cahforma, 5, -895-902 

Vossen, P1ek (ed) (1998) EuroWoldNet A 
Multdmgual Database w~th Lexlcal Semantic 
Networks Kluwe~ Academic P~ess, Dotdrecht 
Reprinted from Computers and the Humamttes, 
32-2-3 

Wdks, Yorick and Stevenson, Mark (1998) Word 
Sense Dtsamblguatlon Using Optimized 
Combinations olr Knowledge Sources Proceedmgs 
of COLING/A CL-98, Montreal, August, 1998 

Yarowsky, David (1992) Word sense dlsambtguatlon 
using statistical models of Roget's categories 
trained on large corpora Proceedings of the 14th 
lnternattonal Conference on Computattonal 
Ltngul.~ttcs, COLING'92, 23-28 August, Nantes, 
France, 454-460 

Yarowsky, David (1993) One sense per collocation 
Pioceedtngs of the ARPA Human Language 
Technology Workshop, Princeton, New Jersey, 
266-271 
