<?xml version="1.0" standalone="yes"?>
<Paper uid="P04-1067">
  <Title>A Geometric View on Bilingual Lexicon Extraction from Comparable Corpora</Title>
  <Section position="3" start_page="0" end_page="0" type="metho">
    <SectionTitle>
2 Standard approach
</SectionTitle>
    <Paragraph position="0"> Bilingual lexicon extraction from comparable corpora has been studied by a number of researchers, (Rapp, 1995; Peters and Picchi, 1995; Tanaka and Iwasaki, 1996; Shahzad et al., 1999; Fung, 2000, among others). Their work relies on the assumption that if two words are mutual translations, then their more frequent collocates (taken here in a very broad sense) are likely to be mutual translations as well. Based on this assumption, the standard approach builds context vectors for each source and target word, translates the target context vectors using a general bilingual dictionary, and compares the translation with the source context vector:  1. For each source word v (resp. target word w), build a context vector !v (resp. !w ) consisting in the measure of association of each word e (resp. f) in the context of v (resp. w), a(v; e).</Paragraph>
    <Paragraph position="1"> 2. Translate the context vectors with a general bilingual dictionary D, accumulating the contributions from words that yield identical translations. null 3. Compute the similarity between source word v  and target word w using a similarity measures, such as the Dice or Jaccard coefficients, or the cosine measure.</Paragraph>
    <Paragraph position="2"> As the dot-product plays a central role in all these measures, we consider, without loss of generality, the similarity given by the dot-product between !v and the translation of !w :</Paragraph>
    <Paragraph position="4"> Because of the translation step, only the pairs (e; f) that are present in the dictionary contribute to the dot-product.</Paragraph>
    <Paragraph position="5"> Note that this approach requires some general bilingual dictionary as initial seed. One way to circumvent this requirement consists in automatically building a seed lexicon based on spelling and cognates clues (Koehn and Knight, 2002). Another approach directly tackles the problem from scratch by searching for a translation mapping which optimally preserves the intralingual association measure between words (Diab and Finch, 2000): the underlying assumption is that pairs of words which are highly associated in one language should have translations that are highly associated in the other language. In this latter case, the association measure is defined as the Spearman rank order correlation between their context vectors restricted to &amp;quot;peripheral tokens&amp;quot; (highly frequent words). The search method is based on a gradient descent algorithm, by iteratively changing the mapping of a single word until (locally) minimizing the sum of squared differences between the association measure of all pairs of words in one language and the association measure of the pairs of translated words obtained by the current mapping.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
2.1 Geometric presentation
</SectionTitle>
      <Paragraph position="0"> We denote by si; 1 i p and tj; 1 j q the source and target words in the bilingual dictionary D. D is a set of n translation pairs (si; tj), and may be represented as a p q matrix M, such that Mij = 1 iff (si; tj) 2 D (and 0 otherwise).2 Assuming there are m distinct source words e1; ; em and r distinct target words f1; ; fr in the corpus, figure 1 illustrates the geometric view of the standard method.</Paragraph>
      <Paragraph position="1"> The association measure a(v; e) may be viewed as the coordinates of the m-dimensional context vector !v in the vector space formed by the orthogonal basis (e1; ; em). The dot-product in (1) only involves source dictionary entries. The corresponding dimensions are selected by an orthogonal 2The extension to weighted dictionary entries Mij 2 [0; 1] is straightforward but not considered here for clarity. projection on the sub-space formed by (s1; ; sp), using a p m projection matrix Ps. Note that (s1; ; sp), being a sub-family of (e1; ; em), is an orthogonal basis of the new sub-space. Similarly, ! w is projected on the dictionary entries (t1; ; tq) using a q r orthogonal projection matrix Pt. As M encodes the relationship between the source and target entries of the dictionary, equation 1 may be rewritten as: S(v; w) = h !v ; !tr(w)i = (Ps !v )&gt; M (Pt !w) (2) where &gt; denotes transpose. In addition, notice that M can be rewritten as S&gt;T, with S an n p and T an n q matrix encoding the relations between words and pairs in the bilingual dictionary (e.g. Ski is 1 iff si is in the kth translation pair). Hence: S(v; w)= !v&gt;P&gt;s S&gt;TPt !w =hSPs !v ; TPt !wi (3) which shows that the standard approach amounts to performing a dot-product in the vector space formed by the n pairs ((s1; tl); ; (sp; tk)), which are assumed to be orthogonal, and correspond to translation pairs.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
2.2 Problems with the standard approach
</SectionTitle>
      <Paragraph position="0"> There are two main potential problems associated with the use of a bilingual dictionary.</Paragraph>
      <Paragraph position="1"> Coverage. This is a problem if too few corpus words are covered by the dictionary. However, if the context is large enough, some context words are bound to belong to the general language, so a general bilingual dictionary should be suitable. We thus expect the standard approach to cope well with the coverage problem, at least for frequent words.</Paragraph>
      <Paragraph position="2"> For rarer words, we can bootstrap the bilingual dictionary by iteratively augmenting it with the most probable translations found in the corpus.</Paragraph>
      <Paragraph position="3"> Polysemy/synonymy. Because all entries on either side of the bilingual dictionary are treated as orthogonal dimensions in the standard methods, problems may arise when several entries have the same meaning (synonymy), or when an entry has several meanings (polysemy), especially when only one meaning is represented in the corpus.</Paragraph>
      <Paragraph position="4"> Ideally, the similarities wrt synonyms should not be independent, but the standard method fails to account for that. The axes corresponding to synonyms si and sj are orthogonal, so that projections of a context vector on si and sj will in general be uncorrelated. Therefore, a context vector that is similar to si may not necessarily be similar to sj.</Paragraph>
      <Paragraph position="5"> A similar situation arises for polysemous entries.</Paragraph>
      <Paragraph position="6"> Suppose the word bank appears as both financial institution (French: banque) and ground near a river</Paragraph>
      <Paragraph position="8"> (French: berge), but only the pair (banque, bank) is in the bilingual dictionary. The standard method will deem similar river, which co-occurs with bank, and argent (money), which co-occurs with banque.</Paragraph>
      <Paragraph position="9"> In both situations, however, the context vectors of the dictionary entries provide some additional information: for synonyms si and sj, it is likely that !si and !sj are similar; for polysemy, if the context vectors !banque and !bank have few translations pairs in common, it is likely that banque and bank are used with somewhat different meanings. The following methods try to leverage this additional information.</Paragraph>
      <Paragraph position="10"> 3 Extension of the standard approach The fact that synonyms may be captured through similarity of context vectors3 leads us to question the projection that is made in the standard method, and to replace it with a mapping into the sub-space formed by the context vectors of the dictionary entries, that is, instead of projecting !v on the sub-space formed by (s1; ; sp), we now map it onto the sub-space generated by ( !s1; ; !sp). With this mapping, we try to find a vector space in which synonymous dictionary entries are close to each other, while polysemous ones still select different neighbors. This time, if !v is close to !si and !sj , si and sj being synonyms, the translations of both si and sj will be used to find those words w close to v.</Paragraph>
      <Paragraph position="11"> Figure 2 illustrates this process. By denoting Qs, respectively Qt, such a mapping in the source (resp.</Paragraph>
      <Paragraph position="12"> target) side, and using the same translation mapping (S; T) as above, the similarity between source and target words becomes: S(v; w)=hSQs !v ; TQt !wi= !v&gt;Q&gt;s S&gt;TQt !w (4) A natural choice for Qs (and similarly for Qt) is the following m p matrix:</Paragraph>
      <Paragraph position="14"> eral studies, e.g. (Grefenstette, 1994; Lewis et al., 1967).</Paragraph>
      <Paragraph position="15"> but other choices, such as a pseudo-inverse of Rs, are possible. Note however that computing the pseudo-inverse of Rs is a complex operation, while the above projection is straightforward (the columns of Q correspond to the context vectors of the dictionary words). In appendix A we show how this method generalizes over the probabilistic approach presented in (Dejean et al., 2002). The above method bears similarities with the one described in (Besanc,on et al., 1999), where a matrix similar to Qs is used to build a new term-document matrix. However, the motivations behind their work and ours differ, as do the derivations and the general framework, which justifies e.g. the choice of the pseudo-inverse of Rs in our case.</Paragraph>
    </Section>
  </Section>
  <Section position="4" start_page="0" end_page="0" type="metho">
    <SectionTitle>
4 Canonical correlation analysis
</SectionTitle>
    <Paragraph position="0"> The data we have at our disposal can naturally be represented as an n (m + r) matrix in which the rows correspond to translation pairs, and the columns to source and target vocabularies:</Paragraph>
    <Paragraph position="2"> where (s(k); t(k)) is just a renumbering of the translation pairs (si; tj).</Paragraph>
    <Paragraph position="3"> Matrix C shows that each translation pair supports two views, provided by the context vectors in the source and target languages. Each view is connected to the other by the translation pair it represents. The statistical technique of canonical correlation analysis (CCA) can be used to identify directions in the source view (first m columns of C) and target view (last r columns of C) that are maximally correlated, ie &amp;quot;behave in the same way&amp;quot; wrt the translation pairs. We are thus looking for directions in the source and target vector spaces (defined by the orthogonal bases (e1; ; em) and (f1; ; fr)) such that the projections of the translation pairs on these directions are maximally correlated. Intuitively, those directions define latent semantic axes</Paragraph>
    <Paragraph position="5"> that capture the implicit relations between translation pairs, and induce a natural mapping across languages. Denoting by s and t the directions in the source and target spaces, respectively, this may be formulated as:</Paragraph>
    <Paragraph position="7"> As in principal component analysis, once the first two directions ( 1s; 1t ) have been identified, the process can be repeated in the sub-space orthogonal to the one formed by the already identified directions. However, a general solution based on a set of eigenvalues can be proposed. Following e.g. (Bach and Jordan, 2001), the above problem can be reformulated as the following generalized eigenvalue problem:</Paragraph>
    <Paragraph position="9"> where, denoting again Rs and Rt the first m and last</Paragraph>
    <Paragraph position="11"> The standard approach to solve eq. 5 is to perform an incomplete Cholesky decomposition of a regularized form of D (Bach and Jordan, 2001).</Paragraph>
    <Paragraph position="12"> This yields pairs of source and target directions</Paragraph>
    <Paragraph position="14"> which to project words from each language. This sub-space plays the same role as the sub-space defined by translation pairs in the standard method, although with CCA, it is derived from the corpus via the context vectors of the translation pairs. Once projected, words from different languages can be compared through their dot-product or cosine. De-</Paragraph>
    <Paragraph position="16"> the similarity becomes (figure 3): S(v; w) = h s !v ; t !wi = !v&gt; &gt;s t !w (6) The number l of vectors retained in each language directly defines the dimensions of the final sub-space used for comparing words across languages. CCA and its kernelised version were used in (Vinokourov et al., 2002) as a way to build a cross-lingual information retrieval system from parallel corpora. We show here that it can be used to infer language-independent semantic representations from comparable corpora, which induce a similarity between words in the source and target languages.</Paragraph>
  </Section>
  <Section position="5" start_page="0" end_page="0" type="metho">
    <SectionTitle>
5 Multilingual probabilistic latent
</SectionTitle>
    <Paragraph position="0"> semantic analysis The matrix C described above encodes in each row k the context vectors of the source (first m columns) and target (last r columns) of each translation pair. Ideally, we would like to cluster this matrix such that translation pairs with synonymous words appear in the same cluster, while translation pairs with polysemous words appear in different clusters (soft clustering). Furthermore, because of the symmetry between the roles played by translation pairs and vocabulary words (synonymous and polysemous vocabulary words should also behave as described above), we want the clustering to behave symmetrically with respect to translation pairs and vocabulary words. One well-motivated method that fulfills all the above criteria is Probabilistic Latent Semantic Analysis (PLSA) (Hofmann, 1999).</Paragraph>
    <Paragraph position="1"> Assuming that C encodes the co-occurrences between vocabulary words w and translation pairs d, PLSA models the probability of co-occurrence w and d via latent classes :</Paragraph>
    <Paragraph position="3"> where, for a given class, words and translation pairs are assumed to be independently generated from class-conditional probabilities P(wj ) and P(dj ).</Paragraph>
    <Paragraph position="4"> Note here that the latter distribution is languageindependent, and that the same latent classes are used for the two languages. The parameters of the model are obtained by maximizing the likelihood of the observed data (matrix C) through Expectation-Maximisation algorithm (Dempster et al., 1977). In</Paragraph>
    <Paragraph position="6"> addition, in order to reduce the sensitivity to initial conditions, we use a deterministic annealing scheme (Ueda and Nakano, 1995). The update formulas for the EM algorithm are given in appendix B.</Paragraph>
    <Paragraph position="7"> This model can identify relevant bilingual latent classes, but does not directly define a similarity between words across languages. That may be done by using Fisher kernels as described below.</Paragraph>
    <Paragraph position="8"> Associated similarities: Fisher kernels Fisher kernels (Jaakkola and Haussler, 1999) derive a similarity measure from a probabilistic model. They are useful whenever a direct similarity between observed feature is hard to define or insufficient. Denoting '(w) = lnP(wj ) the log-likelihood for example w, the Fisher kernel is:</Paragraph>
    <Paragraph position="10"> The Fisher information matrix IF =</Paragraph>
    <Paragraph position="12"> keeps the kernel independent of reparameterisation. With a suitable parameterisation, we assume IF 1. For PLSA (Hofmann, 2000), the Fisher kernel between two words w1 and w2 becomes:</Paragraph>
    <Paragraph position="14"> where d ranges over the translation pairs. The Fisher kernel performs a dot-product in a vector space defined by the parameters of the model. With  only one class, the expression of the Fisher kernel (9) reduces to:</Paragraph>
    <Paragraph position="16"> Apart from the additional intercept ('1'), this is exactly the similarity provided by the standard method, with associations given by scaled empirical frequencies a(w; d) = bP(djw)=pP(d). Accordingly, we expect that the standard method and the Fisher kernel with one class should have similar behaviors. In addition to the above kernel, we consider two additional versions, obtained:through normalisation (NFK) and exponentiation (EFK):</Paragraph>
    <Paragraph position="18"> where K(w) stands for K(w; w).</Paragraph>
  </Section>
  <Section position="6" start_page="0" end_page="0" type="metho">
    <SectionTitle>
6 Experiments and results
</SectionTitle>
    <Paragraph position="0"> We conducted experiments on an English-French corpus derived from the data used in the multi-lingual track of CLEF2003, corresponding to the newswire of months May 1994 and December 1994 of the Los Angeles Times (1994, English) and Le Monde (1994, French). As our bilingual dictionary, we used the ELRA multilingual dictionary,4 which contains ca. 13,500 entries with at least one match in our corpus. In addition, the following linguistic preprocessing steps were performed on both the corpus and the dictionary: tokenisation, lemmatisation and POS-tagging. Only lexical words (nouns, verbs, adverbs, adjectives) were indexed and only single word entries in the dicitonary were retained.</Paragraph>
    <Paragraph position="1"> Infrequent words (occurring less than 5 times) were discarded when building the indexing terms and the dictionary entries. After these steps our corpus contains 34,966 distinct English words, and 21,140 distinct French words, leading to ca. 25,000 English and 13,000 French words not present in the dictionary. null To evaluate the performance of our extraction methods, we randomly split the dictionaries into a training set with 12,255 entries, and a test set with 1,245 entries. The split is designed in such a way that all pairs corresponding to the same source word are in the same set (training or test). All methods use the training set as the sole available resource and predict the most likely translations of the terms in the source language (English) belonging to the  test set. The context vectors were defined by computing the mutual information association measure between terms occurring in the same context window of size 5 (ie. by considering a neighborhood of +/- 2 words around the current word), and summing it over all contexts of the corpora. Different association measures and context sizes were assessed and the above settings turned out to give the best performance even if the optimum is relatively flat. For memory space and computational efficiency reasons, context vectors were pruned so that, for each term, the remaining components represented at least 90 percent of the total mutual information. After pruning, the context vectors were normalised so that their Euclidean norm is equal to 1. The PLSA-based methods used the raw co-occurrence counts as association measure, to be consistent with the underlying generative model. In addition, for the extended method, we retained only the N (N = 200 is the value which yielded the best results in our experiments) dictionary entries closest to source and target words when doing the projection with Q. As discussed below, this allows us to get rid of spurious relationships.</Paragraph>
    <Paragraph position="2"> The upper part of table 1 summarizes the results we obtained, measured in terms of F-1 score for different lengths of the candidate list, from 20 to 500. For each length, precision is based on the number of lists that contain an actual translation of the source word, whereas recall is based on the number of translations provided in the reference set and found in the list. Note that our results differ from the ones previously published, which can be explained by the fact that first our corpus is relatively small compared to others, second that our evaluation relies on a large number of candidates, which can occur as few as 5 times in the corpus, whereas previous evaluations were based on few, high frequent terms, and third that we do not use the same bilingual dictionary, the coverage of which being an important factor in the quality of the results obtained. Long candidate lists are justified by CLIR considerations, where longer lists might be preferred over shorter ones for query expansion purposes. For PLSA, the normalised Fisher kernels provided the best results, and increasing the number of latent classes did not lead in our case to improved results. We thus display here the results obtained with the normalised version of the Fisher kernel, using only one component. For CCA, we empirically optimised the number of dimensions to be used, and display the results obtained with the optimal value (l = 300).</Paragraph>
    <Paragraph position="3"> As one can note, the extended approach yields the best results in terms of F1-score. However, its performance for the first 20 candidates are below the standard approach and comparable to the PLSA-based method. Indeed, the standard approach leads to higher precision at the top of the list, but lower recall overall. This suggests that we could gain in performance by re-ranking the candidates of the extended approach with the standard and PLSA methods. The lower part of table 1 shows that this is indeed the case. The average precision goes up from 0.4 to 0.44 through this combination, and the F1-score is significantly improved for all the length ranges we considered (bold line in table 1).</Paragraph>
  </Section>
  <Section position="7" start_page="0" end_page="0" type="metho">
    <SectionTitle>
7 Discussion
</SectionTitle>
    <Paragraph position="0"> Extended method As one could expect, the extended approach improves the recall of our bilingual lexicon extraction system. Contrary to the standard approach, in the extended approach, all the dictionary words, present or not in the context vector of a given word, can be used to translate it. This leads to a noise problem since spurious relations are bound to be detected. The restriction we impose on the translation pairs to be used (N nearest neighbors) directly aims at selecting only the translation pairs which are in true relation with the word to be translated. null Multilingual PLSA Even though theoretically well-founded, PLSA does not lead to improved performance. When used alone, it performs slightly below the standard method, for different numbers of components, and performs similarly to the standard method when used in combination with the extended method. We believe the use of mere co-occurrence counts gives a disadvantage to PLSA over other methods, which can rely on more sophisticated measures. Furthermore, the complexity of the final vector space (several millions of dimensions) in which the comparison is done entails a longer processing time, which renders this method less attractive than the standard or extended ones.</Paragraph>
    <Paragraph position="1"> Canonical correlation analysis The results we obtain with CCA and its kernel version are disappointing. As already noted, CCA does not directly solve the problems we mentioned, and our results show that CCA does not provide a good alternative to the standard method. Here again, we may suffer from a noise problem, since each canonical direction is defined by a linear combination that can involve many different vocabulary words.</Paragraph>
    <Paragraph position="2"> Overall, starting with an average precision of 0.35 as provided by the standard approach, we were able to increase it to 0.44 with the methods we consider.</Paragraph>
    <Paragraph position="3"> Furthermore, we have shown here that such an improvement could be achieved with relatively simple  to the extended approach, whereas NFK stands for normalised Fisher kernel. methods. Nevertheless, there are still a number of issues that need be addressed. The most important one concerns the combination of the different methods, which could be optimised on a validation set. Such a combination could involve Fisher kernels with different latent classes in a first step, and a final combination of the different methods. However, the results we obtained so far suggest that the rank of the candidates is an important feature. It is thus not guaranteed that we can gain over the combination we used here.</Paragraph>
  </Section>
class="xml-element"></Paper>