ADVANCES IN MULTILINGUAL TEXT RETRIEVAL 
Mark Davis 
Computing Research Lab 
New Mexico State University 
Box 30001/3CRL 
Las Cruces, NM 88003 
madavis @ crl.nmsu, edu 
(505) 646-1148 
INTRODUCTION 
Multilingual text retrieval extends the basic mono- 
lingual detection task to include retrieving relevant doc- 
uments in languages other than the query language. The 
task therefore merges efforts in machine translation with 
efforts in text retrieval, but the machine translation com- 
ponent may be substantially simplified due to some 
basic assumptions about the design and implementation 
of high-performance text retrieval systems. A primary 
consideration is that most modem text retrieval systems 
regard queries and documents as unordered "bags" of 
words. The translation of an unordered set of terms is 
therefore approximately the translation of the terms 
themselves. Although a linearity assumption such as this 
breaks down when considering phrasal elements in most 
languages, it is reasonably accurate for many terms and 
becomes increasingly accurate at the sentence level and 
above. 
A second consideration in multilingual text retrieval 
is where the translation is done. It is possible to translate 
every document at index time, for example, but the 
resource costs are substantially higher than translating 
the query at retrieval time. An added benefit of translat- 
ing only the query is that queries can be prepared with 
no special weighting scheme applied to the terms. The 
queries are then available to any natural language text 
retrieval system. 
The range of translation techniques that are avail- 
able to a query translation system is greater than in stan- 
dard machine translation systems. Previously translated 
document corpora can be made available for exploiting 
domain-specific terminology by direct comparison of 
the retrieval results for the query and target document 
languages. No special heuristics are needed for using 
this "example-based" translation approach; the query 
can be optimized by adding or deleting terms until the 
target language retrieval results are approximately the 
same as the source language retrieval results. Lexical- 
transfer techniques can also be used in the same context, 
providing wide coverage of term senses. 
CRL evaluated five methods for query translation in 
Tipster II. The results were then evaluated in TREC by 
hand-translating the TREC Spanish monolingual que- 
ries into English and applying the automatic query 
translation methods to produce new Spanish queries. 
Ongoing work is focusing on improving the perfor- 
mance of query translation techniques while expanding 
the techniques to work with new languages and search 
engines, including WWW search services. 
MLTR IN TREC 
Starting with TREC- 3, Spanish corpora and query 
sets have been available for evaluating text retrieval 
engines. The queries and corpus are monolingual, how- 
ever, so testing a multilingual system is only possible if 
the query set or the corpus is translated into a different 
language. We chose to translate the queries since they 
were very short. With translated queries, a query transla- 
tion system that produces Spanish queries from hand- 
translated English versions of original Spanish queries 
can then be compared against the original queries. The 
differences between the two results are then a reason- 
able measure of the effectiveness of the translation pro- 
cess in preserving the characteristics of the original 
query that contribute to retrieval. Several of the Spanish 
TREC queries and their hand-translated versions are 
shown in Table 1, below. 
The query translation methods that we applied to 
produce new Spanish queries were of two major types: 
methods that used a prepared lexicon and methods that 
used a parallel training corpus. While a lexicon tends to 
produce translations that are shallow but comprehen- 
sive, covering all possible senses of a term but limited in 
the range of synonyms that are produced for each term, 
corpus methods tend to produce translations that are 
deep but narrow, with enormous repetition of domain- 
related senses of terminology. This justified an examina- 
tion of the comparative merits of both approaches. 
As is often the case, our parallel corpus was not 
precisely of the same domain as the TREC document 
185 
collection for the ultimate evaluation. The corpus itself 
was extremely large, however, which we hoped would 
offset the difficulties of using a distinctly different type 
of text. The corpus was 1.6 Gb of Spanish and English 
translations from the United Nations, containing pro- 
ceedings of meetings, policy documents and notes on 
UN activities in member countries. The documents were 
automatically aligned \[1\] at the sentence level using a 
procedure that is conservatively estimated to have an 
83% accuracy over grossly noisy document pairs (which 
the UN documents were not). This produced a parallel 
corpus of around 680,000 aligned sentence pairs. 
Lexical Transfer 
The first method was to perform term-by-term 
translation with the Collins English-Spanish bilingual 
dictionary. Individual terms in the English query were 
reduced to their morphological roots and lookup was 
performed. The resulting set of Spanish terms became 
the Spanish query. Some repetition of terms is apparent 
in the resulting queries because all senses of each term 
were used with no attempt to disambiguate the contex- 
tual usage of the English terms. For example, Query 28 
is transformed from 
Indicators of economic and business 
relations between Mexico and Asian 
countries, such as Japan, China and 
Korea. 
to 
indicador indicador ayuda expansi6n 
previsiones crecimiento comercio com- 
ercio narraci6n relaci6n parentesco 
M4xico Ciudad gripe patria campo 
regidn amor semejante parecido tanto 
el laca China Mar t4 porcelana vitrina 
coalln Corea Corea Corea mexicana mex- 
icano M~xico 
Note that "China" has been replaced with both 
"China" and "porcelana" as a result of this simple lexi- 
cal substitution scheme, and that "relations" has 
included the familial sense "parentesco". Lexicon-gen- 
erated Spanish Queries 
The lexical-transfer approach produced Spanish 
queries rapidly, requiring only a simple database lookup 
procedure. This process is shown in Figure 1 (a). 
High-Frequency Terms from Parallel Text 
In text, the terms that occur with the highest fre- 
quency are rarely of statistical significance, and are 
more often than not merely redundant. Yet the terms that 
occur with moderate frequency are sometimes signifi- 
cant. In order to evaluate other corpus-based methods, 
we wanted to establish a baseline for queries formed 
from these moderate frequency term sets. Using a vec- 
tor-based text retrieval system with no term spreading or 
other modifications, the English queries were translated 
by performing a lookup on the English side of the paral- 
lel corpus, collecting the Spanish sentences that were 
parallels to the top 100 retrieved documents, filtering the 
remaining terms to eliminate the top 500 most frequent 
Spanish terms, and collecting the next 100 most fre- 
quent Spanish terms to create the new query. This pro- 
cess is shown in Figure 1 (b): 
Several of the resulting queries are given in Table 2. 
Some formatting codes from the UN documents have 
been eliminated in some of the queries, reducing the 
count to below 100 terms in those queries. For brevity, 
only the first two queries are shown in Table 2. 
Statistically Significant Terms 
Whereas the high-frequency terms extracted in the 
previous method provide a baseline for examining 
improved methods, high-frequency terms are them- 
selves not necessarily the best terms for discriminating 
the significant features involved in text retrieval. A bet- 
ter approach is to extract the terms which are statisti- 
cally significant in the retrieved segments of parallel text 
in comparison to the corpus as a whole. Various meth- 
ods are possible for testing statistical significance, but 
the method we applied is based on a log-likelihood ratio 
test that assumes a X 2 distribution is an accurate model 
of the term distributions in text \[2\]. 
The method begins by extracting all of the terms 
from the sentences that are parallels to the top 100 
retrieved English sentences. The counts of the pooled 
terms are then compared with the counts for the entire 
UN training corpus to evaluate their statistical signifi- 
cance. The top 100 most-significant terms are then 
extracted and become the new Spanish query. Figure 1 
(c) diagrams the process. The resulting queries are in 
Table 3, below.s 
Evolutionary Optimization of Queries 
If we could make a set of derived Spanish queries 
retrieve documents in a manner that is similar to the 
English queries over a training corpus, then the Spanish 
query could conceivably produce similar results on a 
novel corpus. One way to change Spanish queries is to 
add and remove terms. The number of possible unique 
deletions that can be performed on a 70 word query is 
186 
quite large, however, making the direct examination of 
all possible modified queries effectively impossible. 
We applied an evolutionary programming (EP) \[3\] 
approach to modify a population of 50 queries. In an EP 
approach, an initial population of queries is needed 
along with a mutation strategy to modify queries. Opti- 
mization then proceeds by evaluating the comparative 
fitnesses of the queries, mutating a selected sub-popula- 
tion of the queries to produce "offspring" solutions and 
re-evaluating the queries iteratively until a suitable num- 
ber of generations have passed. Our EP approach con- 
sidered the comparative evaluation of document score 
vectors as an objective measure of the relative fitness of 
a query to the collection. This process is diagrammed in 
Figure 1 (d). 
The initial queries for this test were the queries 
from the high-frequency lookup strategy discussed 
above. Previously, we have used a lexicon to generate 
initial queries \[4\]. The mutation strategy applied 
between one and ten modification operations to each of 
the 50 queries per generation and collected only the best 
10% of the queries to propagate into the next generation. 
Optimization proceeded for 50 generations, resulting in 
a wide range of changes to each query. 
The types of queries produced by this system typi- 
cally showed the repetition of key terminology com- 
bined with the elimination of irrelevant terms. The 
fitness judgment for a query was based on comparative 
retrieval results using a training corpus of only 80,000 
aligned sentences. Table 4, below, shows two of the 
resulting queries from the EP method. 
Singular Value Decomposition and the 
Translation Matrix 
The final query translation method was a radical 
departure from the others, but is derived from earlier 
work by \[5\] and \[6\]. This method is at heart a numerical 
approach to derive a translation matrix from parallel 
texts. 
In this effort, we applied a QR-decomposition tech- 
nique to reduce the complexity of calculating the singu- 
lar value decomposition, resulting in query translation 
that took only a matter of seconds on a SPARC 10. Sev- 
eral of the generated queries are given in Table 6. Fig- 
ure 1 (e) diagrams the process. 
OVERVIEW OF RESULTS 
The resulting queries were given to University of 
Massachusetts, Amherst, who ran them against the 
Spanish TREC document collection using Spanish 
Inquery. The original Spanish TREC queries were also 
evaluated to establish a reference baseline. The results 
were as follows: 
1. On average, the dictionary-based queries produced 
performance which was about 50% worse than the 
reference queries. 
2. The EP-derived queries produced performance 
which was 60-70% worse than the reference que- 
ries, except at higher recall levels (.6-1.0), at which 
they performed better than the Method 1 queries. 
3. The other methods performed even more poorly. 
4. On at least two queries, performance of the lexical 
methods was as good or better than the reference 
queries. 
5. On two queries, performance of the EP approach 
was as good as the reference queries, although they 
tended to have better precision at higher recall. 
These modest results demonstrate that lexical and 
corpus methods can be applied to query translation in a 
large-scale multilingual text retrieval scenario, although 
at a fair penalty in performance. Each of these methods 
was purposely limited to as simple a scheme as possible, 
however, so there is plenty of room for improvement 
and further experimentation.The average precision- 
recall curve for all 25 queries is shown in Figure 2. 
RECENT AND ONGOING WORK 
Current work is focusing on improving the perfor- 
mance of MLTR methods, applying the methods to new 
languages and making use of new retrieval engines. 
An example of the latter is shown in Figure 3. Mun- 
dial is a query interface to Infoseek and Yahoo that takes 
queries in English, translates them to Spanish and sub- 
mits the resulting queries to the Infoseek and Yahoo 
search engines directly. Figure 4 shows the completed 
search for Spanish documents on Infoseek. The Mundial 
demo uses a bilingual dictionary combined with several 
heuristics to limit the terminological expansion of the 
input query. Limiting query size is important because 
most search engines, like Infoseek, restrict the size of a 
query to around 80 characters. Overgeneration in the 
translation process is handled by using the longest terms 
(in character count) in Mundial. Although in some cases 
this may be in error, the hope is that automatic stem- 
ming of query terms at the search engine will reduce 
long terms to stems common to many of the keywords 
that might have been substituted if the entire definition 
was transferred. The second motivation was that long 
187 
terms tend to be more precise than short terms, and con- 
tent words should be as precise as possible. 
Mundial may be accessed at: 
http://crl.nmsu.edu/ANG/ML/ml.html. 
REFERENCES 
[1] Davis, M. W., T. E. Dunning, and W. C. Ogden 
(1995) "Text Alignment in the Real World: Improving 
Alignments of Noisy Translations Using Common Lexi- 
cal Features, String Matching Strategies and N-Gram 
Comparisons," In Proceedings of the Conference of the 
European Chapter of the Association of Computational 
Linguistics. University College Dublin. March 1995. 
[2] Dunning, T. E. (1993), "Accurate Methods for the 
Statistics of Surprise and Coincidence," Computational 
Linguistics, 19, 1: 61-74. 
[3] Fogel, D. B. (1992), "A Brief History of Simulated 
Evolution," In Proc. of the First Annual Conference on 
Evolutionary Programming, ed. D.B. Fogel and J.W. 
Atmar, 1-16. San Diego: Evolutionary Programming 
Society. 
[4] Davis, M. W. and T.E. Dunning (1995) "Query 
Translation Using Evolutionary Programming for Multi- 
Lingual Information Retrieval," In Proceedings of the 
Fourth Annual Conference on Evolutionary Program- 
ming, San Diego, Evolutionary Programming Society, 
1995. 
[5] Dunning, T. E., and M. W. Davis (1993b), "Multi- 
Lingual Information Retrieval," Memoranda in Com- 
puter and Cognitive Science, MCCS-93-252, Comput- 
ing Research Laboratory, New Mexico State University. 
[6] Landauer, T. K. and M. L. Littman (1990). "Fully 
Automatic Cross-Language Document Retrieval Using 
Latent Semantic Indexing," In Proceedings of the 6th 
Conference of UW Centre for the New Oxford English 
Dictionary and Text Research, 31-38. Waterloo. 
188 
Q# 
26 
27 
Hand-translated English 
Indicators of economic and business rela- 
tions between Mexico and European con- 
tries. 
Indicators of economic and business rela- 
tions between Mexico and African con- 
tries. 
Corpus High-Frequency Spanish 
Checoslovaquia En Ghana Polonia nacional programa Australia Bajos Egiptu Espafia Filipi- 
nas La Palses Portugal igualdad Italia Paz recursos Austria Finiandia Acci6n Pide Venezuela 
Naeiones guberuarnentules Unidas como perlodo una Comisi6n Desarrollo regionales 
sesiones Mujer Mandial informaci6n nacionales informe Mtxico resoluci6n no proyecto un 
actividades palses Estados organizaciones desarrullo sus su E/CN mujer Secretario General 
pot Repfiblica al con se Conferencia sobre para del las que los el en la de 
Checoslovaquia Democrfitica Egipto Filipinas Francia Indonesia Irlanda Los Paises Secre- 
tario Uruguay aplicaci6n mils proyectus servicios Alemania Colombia La fuentes trabajo 
Asamblea haq Naciones Nigeria Pakistfin Unidos documento hart DE Unidas energia nuclear 
sus Brasil pnncipios siguientes utilizaci6n Argentina Chile En Venezuela como desarrollo 
espacio ultraterrestre El General una perlodo sesiones al palses su Estados sobre un para 
Repfiblica por con se Mtxico que las del los en el la de 
Table 1 Several Spanish TREC queries and their English translations 
Q# 
26 
27 
Hand-translated English 
Indicators of economic and business rela- 
tions between Mexico and European con- 
tries. 
Indicators of economic and business rela- 
tions between Mexico and African conaies. 
Statistically-significant Spanish Queries 
periodo un una Anguila CARICOM Dos ECCB En Este Oeste Europeo Guyana Jefes 
Magreb Oceidente Parlamento Principal T al ciencias con consentimiento consulares con- 
venciones correo cuantitativos de del diplom~tieos el empresarial en experiencias externas 
guias la las los para por que residente se sobre su sustituir tecnol6gica temporal tienden 
tomaron tono totalidad trabajan tradicionales transacci transacci6n transaceiones transici6n 
transparencia tratarh tratase trigtsimo trimestre tropiezan trueque ultimado un un Semi- 
nario una unificado university urbanas utilizarse vtanse vacantes validez vecindad vecinos 
venian vencimientos vende versi6n vigentes vinculadas vineulado vineulados voluntarios y 
Sud~frica y flnaneiaci6n y rechaz6 
@boles Anguila CARICOM ECCB EU Este Oeste Guyana Jefes Principal al ascanso aut6c- 
tonos ciencias con consentimiento consulates convenciones correo cuantitativos de del 
diplomfiticos el empresarial en experiencias cxtemas gufas la las litorales Ins mar nato occi- 
dental para pot que se semillas sobre su tRulos tecnol6gica temporal terremoto tienden tier- 
ras titular tomaron tono totalidad trabajan traditional tradicionales transacci6n 
transaeciones transiei6n transparencia tratar~ tratase trimestre tropieales tropiezan trueque 
un un Seminario una unas unificado urbanas utilizan vtanse victima vecindad veeinos 
verdan vencimientos vende verfin versi6n vigantes vineuladas vinculado vinculados volun- 
tarios vulnerables y Sudfifriea y finaneiaci6n y reehaz6 
Table 2 Examples of Statistically-significant Spanish Queries 
Q# 
26 
27 
Hand-translated English 
Indicators of economic and business 
relations between Mexico and European 
contries. 
Indicators of economic and business 
relations between Mexico and African 
contries. 
Evolutionary Optimized Spanish Queries 
Cbecoslovaquia En nacional Egipto Filipinas Portugal Finlandia gubernamentales Unidas una 
sesiones Mundial Mtxico resoluci6n no un palses organizacianes sus su Reptlblica al sobre 
queen la Egipto naeional Filipinas Confemncia parses Mtxico Checoslovaquia Mtxico Mtx- 
leo Egipto Mtxico Mtxico una Finlandia mujer Mtxico Egipto lass se Finlandia Egipto como 
Comisi6n informaei6n FE/CN sobre un Unidas General Unidas desarrollo palses Finlandia Fil- 
ipinas Mtxico actividades un nacional no Conferencia Filipinas Cbecoslovaquia Portugal 
nacionales Conferencia Mtxieo Repfbliea Egipto Mtxieo al nacional proyeeto Mtxico Secre- 
tario mujer que proyecto Filipinas que Mtxieo Filipinas Finlandia la Mtxico En Checoslova- 
quia mexicana mexicano Mtxieo 
Egipto Los servicios Colombia Asamblea Naciones Unidos documento sus Argentina En Gen- 
eral una al parses Estados sobre un Repfiblica con Mtxico del en ana Colombia Mtxico servi- 
cios una Mtxico que Estados Egipto Mtxieo en Mtxieo siguientes Argentina trabajo Egipto 
Mtxico Asamblea documento Egipto Argentina Repfiblica con de Secretario trabajo Mtxico 
pfincipios la aplicaci6n Colombia Argentina DE Egipto Colombia han las aplicaci6n General 
Colombia Argentina servieios Colombia un documentu han Mtxico los una en lass Mtxico 
Mtxico con mexicana mexicano Mtxico 
Table 3 Evolutionary-Optimized Spanish Queries 
189 
Q# Hand-translated English Corpus High-Frequency Spanish 
26 Indicators of economic and business relations Extedores Relaciones Guillermo Bedregal Culto loan Bolivia Ministro documento pffir- 
between Mexico and European contries, rafos M6xico con parte reproducido oficiosas ex Simone decisi6a "perfodo Voicu 
Rumania extemas Ayuda fitulado si Gufi6tcez asimismo decian mexicana mexicano 
M6xico 
27 Indicators of economic and business relations costeras Los constituir INTERES MUNDIAL pnncipales probablemente cambios bien 
between Mexico and African contries, curso profundamente posibles DE pobladas PROBLEMAS sf comprender particular 
configuas Ministro pr6ximo Lass verfin Culto donde pronosficado camino climfificos 
Zelandia causados mexicana mexicano M6xico 
30 Are there sports programs and exchanges Extefiores Relaciones Guillermo Bedregal Culto Finlandia Bolivia Ministro relaciona- 
between Mexico and the United States? dos programas Rumania sl sede conjunto distingue denominan Udi6n Sovi6ticas deter- 
minarse mofivos M6xico Voicu asociaci6n convenios iategrado Nam Gufi6rrez 
del SIDA entre mexicana mexicano M6xico 
Table 4 SVD generated queries 
190 
(a) 
(c~ ~el~, " 
(d) J F-Rag lish ~uery 
~o~ ~e'~ h 
English Query Lexicon I Spanish 
Query 
Parallel Corpus English Spanish 
~.Te.xt,~ Kill I-IExtractt_=d 7nTn~ea' ~,~,.~ ~°p'O0 1-'7 ~°~ lO0 I ~ 
Parallel Corpus English Spanish 
Text ~1 ~%Tgj-..~, k ~~ ~ 
Corpus Term 
Counts 
Z2 ~ Sj~anish k/uery 
Parallel Corpus 
S nish ery 
Parallel Corpus English Spanish 
ReTetri~tval ~~ ~~ SVD S~anish ~,/uery J 
S nish ery 
Figure 1 Diagrams of approaches to query translation: (a) lexical substitution, (b) corpus high-frequency, (c) corpus statistically-significant, (d) evolutionary optimization and (e) SVD approaches. 
191 
800.00 
750.00 
700.00 
650.00 
600.00 
550.00 
500.00 
x 450.00 
o 
• ~ 400.00 
350.00 
300.00 
250.00 
200.00 
150.00 
100.00 
50.00 
0.00 
Precision-Recall for CRL MLIR systems 
(Average for All Queries) 
........................ i .................. i .................. ~ .................. ~ ....................... 
0.00 0.20 0.40 0.60 0.80 1.00 
Recall 
Original -- 
Lexicon .... 
LQ ....... 
Chi-2 -- -- 
EP--" 
HF ......... 
Figure 2 Average precision-recall curves for MUIR methods over 25 Spanish queries. 
192 
MUNDIAL NET SEARCH 
I 
b1~_~_A\]is a demo of ~ krdemet se~ systemthat se~ches for docmumts bl Spanish trod English 
given a querym ~.~'~. 
M~_~_~\] does ~ by translating yore' se~ch tenfa and ~ cmda~ Infoseek c~ Yahoo ~md 
them to se~ch for 'the tr~s~lm'ed t~als. 
Mms~A\]\]mo'ws ho'wto trmmhde ~oud 35,000 te='ms, alUl \]alOWS \]iowto tile l~ra\]s irdo forms it 
cmt tr~_a!~e. 
Mmulial ce= easflybe ezteadedto tres,~lnte other hmgua~es, or to trmm~l~te queries bt other 
l~es to se~ch f~ EngI/sh docmnents 
Midial cml also be extendedto l~'esent re, l~ed terms to the user for evalm~ion befmfe sem~c, bing. 
That way the user c~ exmr~le bu:o~rect "senses" dtrmudafiom andveeed out the badte~ms. 
Mm~ial ¢mt be extended to prepeze smranm~ytrmudafi(ms of the retrieved docmaerds k, ~nty given \]~e, 
blmdial was mcittea~ by Mark Davis = Comlndbtg Rese~ch Laberat=y, New Mexico State 
Urdversity. The autlu~ cmtbe clmtacted at: 
matt,.~,,~is@erl~.edu/(SOS) ~46- ll= 
Figure 3 Mundial, a World Wide Web MLTR interface. English queries can be submitted to Infoseek or 
Yahoo after translating into Spanish or a mixture of English and Spanish for searches over both English and 
Spanish documents. 
193 
Search results for educ.qdmtal so~em p~ar~o 
Bookmark ~e acclaimed way to find what you want on the Net. 
RehdedtoldcS: 
Wofldweathea - Sganish goveJmment - Colleges & universities inPexa - Computer software 
companies - Software - Intennet software ere, hi.yes - Education software - O5/2 software 
vendors - Windows softwarevendors - Finance software for Windows - 
Results 1 - 18 (~the best 100), e~'tedby scoce: 
B/obras Ir~orm_~op~es sobre Diabetes 
-- hap ://www.b~obr~s.co~e_d~a6e~es.hr~d(Score 53, ,~ize 9K) 
Bianvanido a la pug.ha de las infoxraadones s obre diabetes de Biobras. En esta pagina usted va encontrar 
mfonuadones sobre:. Ardculos Asodacianes Dicta Oobiemo Navegadsn Ninos Pesquisa Pie ... (,Yee a/so 
Si.milax Pages) 
Ouipmet: Experiendas de Educacitm e~ Intecnetl hdemet Edutceaioa Experiences 
-- Imcp : ltvswvz d)x.upm, esl-es~sachez/ Proyee~o_ecr~ hmd (Score 53, Size 22,¢) 
Quipunet: Experiencias de Education an Interact/Intsrnet Education Experiences Here we gathea Interact 
educational experiences for QuipuNet volunteeas.. En esta pagina reeopilaremos ... (See ~o Similar Pages) 
EDUCObl - Sol~'e Educativo 
-- lhRo :/Iwww.dbv,~,.covp.coroJe~,covrd (.Score 5~ ,.,'ize 4K) 
Software Educativo. Educational Software. Estos l~rogramas asian disenados en esl~anol para ninos emxe :3 y 
13 anos.. Estan desartollados para PC compatible, ambiente Windows (3.1 en addante) .... (,Yee a/go Simil~ 
Pages) 
Ala~mtes p~a la Farm~a 
-- htrp ://www.psycl~med zun2cheduJweb/azrc~/opnzsFamlb'de.x.hzm(Seore 51, Size 5K) 
preparados per la Academia Americana de Psiquiauia de Ninos y Adolescentes. Esta hoja puede sea 
fotocopiada o xeimpres a. Otxos "Apuntes para la Familia" dispontbles:. # I. Los ninos ... (,Yea G,t~o Similax 
Pages) 
-- h.wp : / lww w. rauJtL corn. uy /pr esent~hxrd.ls o\]tw ew w. htm (Scor e 51, Site 2g ) 
pare DOS, WINDOWS 3.1 Y WINDOWS 95. Prog~amas Shareware para D~ de ~chivo s: 
Figure 4 Results of a Mundial search on Infoseek. 
194 
