File Information
File: 05-lr/acl_arc_1_sum/cleansed_text/xml_by_section/metho/88/c88-1016_metho.xml
Size: 28,353 bytes
Last Modified: 2025-10-06 14:12:05
<?xml version="1.0" standalone="yes"?> <Paper uid="C88-1016"> <Title>A STATISTICAL APPROACH TO LANGUAGE TRANSLATION</Title> <Section position="5" start_page="0" end_page="0" type="metho"> <SectionTitle> 2. A IIEURIST|C OUTLINE OF 'FILE BASIC PHII,OSOPttY </SectionTitle> <Paragraph position="0"> Figure I juxtaposes a rather typical pair of corresponding English mid \]:rench selltenees, as they appear in the Ih.nlsard corpus.</Paragraph> <Paragraph position="1"> They arc arranged graphically so as to make evident thai (a) the literal word order is on the whole preserved, (b) the chulsal (and perhaps phrasal) structure is preserved, and (c) the sentence pairs contain stretches of essentially literal correspondence interrupted by fixed locutions. In the latter category arc \[I rise on = ie souleve\], \]affecting = apropos\], and \[one which reflects o n = i/our mettre cn doutc\].</Paragraph> <Paragraph position="2"> It can thus be argued that translation ought to bc based on a complex glossary of correspondence el' fixed locutions. Inch~ded would be single words as well as phrases consisting el contiguous or tuna--contiguous words. E.g., I word = mot l, I word = proposl.</Paragraph> <Paragraph position="3"> \[not = ne ... pasl, \[no = ne ... pas\[, \[scat belt = ccmturc\[, late = a mangel and even (perhaps} lone which reflects Oil = \[)()ill&quot; mcttrc ell doute\], etc.</Paragraph> <Paragraph position="4"> Transhttion call he sotnewhat naively regarded as a tht'cc slagC/ process: ( 1 ) Partition the source text into a set of fixed locutions (2) Use the glossary plus contextual information to select the corresponding set of fixed locutions in the target language.</Paragraph> <Paragraph position="5"> (3) Arrange the words of the target fixed locutions into a sequence that forms the target sentence.</Paragraph> <Paragraph position="6"> This naive approach forms the basis of our work. In fact, we have developed statistical techniques facilitating the creation of the glossary, and the performance of the three translation steps. While the only way to refute the many weighty objections to our ideas woukl be to construct a machine that actually carries out satisfactory translation, some mitigating comments are ill order, We do not hope to partition uniquely the source sentence into locutions. In most cases, many partitions will be possible, each having a probability attached to it.</Paragraph> <Paragraph position="7"> Whether &quot;affccting&quot; is to be translated as &quot;apropos&quot; or &quot;cuncernant,&quot; or, as our dictionary has it, &quot;touchant&quot; or &quot;cmouvant,&quot; or in a variety of other ways, depends on the rest of the sentence. However, a statistical indication may be obtained from the presence or absence of particular guide words in that scntcncc. Tile statistical technique of decision trees \[9\] can be used to determine the guide word set, and to estimate the ln'obability to be attached to cach possible translate.</Paragraph> <Paragraph position="8"> The sequential arrangement of target words obtained from the glossary inay depend on an analysis of the source sentence* For instance, clause corrcspondence may be insisted upon, in which case only permutations of words which originate in the same source clause wotdd be possible. Furthermore, the character of the source clause may affect the probability of use of certain functioll words in the target clause. There is, of course, nothing to prcvent the use of more detailed information about the structure of the parse of the source sentence. However, preliminary experilnents presented below indicate that only a very crude grammar may be needed (see Section 6).</Paragraph> </Section> <Section position="6" start_page="0" end_page="72" type="metho"> <SectionTitle> 3. CREATING THE GLOSSARY,'FIRST ATTEMPT </SectionTitle> <Paragraph position="0"> We have already indicated in the previous section why creating a glossary is not just a matter of copying some currently available dictiouary into the computer, in fact, in the paired sentences of Figure 1, &quot;affecting&quot; was translated as &quot;apropos,&quot; a correspondence that is riot ordinarily available* Laying aside for the time being the desirability of (idiomatic) word cluster - to word cluster translation, what we are'after at first is to find for each word f in the (French) source language the list of words {e~, e2 ..... e,} of the (English) target language into which f can translate, and the probability P(e, If ) that such a translation takes place.</Paragraph> <Paragraph position="1"> A first approach to a solution that takes advantage of a large data basc of paired sentences (referred to as 'training text') may be as follows. Suppose for a moment that in every French / English sentence pair each French wordftranslates into one and only one English word e, and that this word is somehow revealed to the computer. Then we could proceed by!' I. Establish a counter C(e,,f) for each word e~ of the English w~cabulary. Initially set C(e~,f) = 0 for words et. Set J = 1.</Paragraph> <Paragraph position="2"> 2. Find the Jth occurrence of the word fin the French text. Let it take place in the Kth sentence, and let its translate be the qth word in the Kth English sentence E = e~,, e~ ..... e~,. Then increment by 1 the counter C(e,,C/f).</Paragraph> <Paragraph position="3"> 3. Increase J by 1 and repeat steps 2 and 3.</Paragraph> <Paragraph position="4"> Setting M(f ) equal to the sum of all the counters C(e,, f) at the conclusion of the above operation (in fact, it is easy to see that M(f) is the number of occurrences of fin the total French text), we could then estimate the probability P(e, J f ) of translating the word f by the word e, by the fraction C(e,,f)/M(f).</Paragraph> <Paragraph position="5"> The problem with the above approach is that it relies on correct identification of the translates of French words, i.e., on the solution of a significant part of tile translation problem. In the absence of such identification, the obvious recourse is to profess complete ignorance, beyond knowing that the translate is one of the words of the corresponding English sentence, each of its words being equally likely. Step 2 of the above algorithm then must be changed to 2'. Find the Jth occurrence of the word fin the French text. Let it take place in the Kth sentence, and let the Kth English sentence consist of words e,,, e,~, ..., e,deg. Then increment the counters C(e,,,f), C(e,,,f) ..... C(o,o,f) by tire fraction 1/n.</Paragraph> <Paragraph position="6"> This second approach is based on tile faith that in a large corpus, the frequency of occurrence of true translates of f in corresponding English sentences would overwhelm that of other candidates whose appearance in those sentences is accidental* This belief is obviously flawed. In particular, the article &quot;the&quot; would get the highest count since it would appear multiply in practically every English sentence, and similar problems would exist with other function words as well.</Paragraph> <Paragraph position="7"> What needs to bedone is to introduce some sort of normalization that would appropriately discount for the expected frequency of occurrence of words. Let P(e~) denote the probability (based on ttle above procedure) that the word e, is a translate of a randomly chosen French'word. P(e~) is given by</Paragraph> <Paragraph position="9"> where M is the total length of the French text, and M(f') is the number of occurrences off t in that text (as before). The fraction P(e, If) / P(e,) is an indicator of the strength of association of e, with f, since P(e, If) is normalized by the frequency P(e,) of associating e~ with an average word. Thus it is reasonable to consider e, a likely translate of f if P(e, I f ) is sufficiently large* The above normalization may seem arbitrary, but it has a sound underpinning from the field of Information Theory \[ 10\]. In fact, the quantity</Paragraph> <Paragraph position="11"> is the mutual information between the French word f and the English word e,.</Paragraph> <Paragraph position="12"> Unfortunately, while normalization yields ordered lists of likely English word translates of French words, it does not provide us with the desired probability values. Furthermore, we get no guidance as to the size of a threshold T such that e, would be a candidate translate of f if and only if l(~;f) > T (3.3) Various ad hoe modifications exist to circumvent the two problems* One might, for instance, find the pair e, f with the highest mutual information, criminate e~ and f from all corresponding sentences in which they occur (i.e. decide once and for all that in those sentences e, is tile translate of f !), then re-compute all the quantities over the shortened texts, determine the new maximizing pair e~,f ~ and continue the process until some arbitrary stopping rule is invoked* Before the next section introduces a better approach that yields probabilities, we present in Figure 2 a list of high mutual information English words for some selected French words. The reader will agree that even tire flawed technique is quite powerful.</Paragraph> </Section> <Section position="7" start_page="72" end_page="73" type="metho"> <SectionTitle> 4. A SIMPLE GI,OSSARY BASED ON A MODE\[, </SectionTitle> <Paragraph position="0"> O1&quot; TIlE TRANSI,ATION PROCESS We will now revert to our original ambition of deriving probabilities of translation, P(e,\[f). Let us start by observing that tlm algorithm of the previous section has the following flaw: Shonld it be &quot;decided&quot; that the qth word, e,, , of the English sentence is Ihc translate of the rth word, ~r, of the French sentence, that process makes no provision for removing e,. from eonskk ratiou as a candidate translate of any of tile remaining French words (those not in the rth position)! We need to find a mctho0 to decide (probabilistically !) which English word was general ed by which l.'rench one, and then estimate P(e, tf ) by the relative frequency with whiehfgave rise to e, as &quot;observed&quot; ira tire texts of paired French / English sentence transhttcs. Our procedure will be based on a model (an admittedly crude one) of how Ertgtish words are generated from their French counterparts.</Paragraph> <Paragraph position="1"> With a slight additional refinement to be specified in the next section (see the discussion on position distortion), the following model will do the trick. Augment the English vocabulary by the NULl, vcord eo that leaves no trace in tile English text. Then each French word f will prodnce exactly one 'primary' English word (which may be, however, invisible). Furthermore, primary English words can produce a number of secondary ones.</Paragraph> <Paragraph position="2"> The provisions for the null word and for tile production of secondary words will account for the unequal length of corresponding French and English sentences. It would be expected that some (but not all) French function words would be killed by producing null words, and that English ones would be crealed by secondary production. In particular, in the example of Figme l, one would expect that &quot;reflects&quot; woakl generate both &quot;which&quot; and &quot;on&quot; by secondary production, and &quot;rise&quot; would similarly generate &quot;on.&quot; On tbc other hand, the article 'T&quot; of 'TOrat( ur&quot; and the preposition &quot;a&quot; of &quot;apropos&quot; wotfld both be expected to generate a null word in the primary process.</Paragraph> <Paragraph position="3"> This model of generation of English words from French ones then requires the specification of the following quantities: 1. The probabilities P(e, lf) that the ith word of the English dictionary was generated by the French word f.</Paragraph> <Paragraph position="4"> 2. The probabilities Q(% l e,) that the jth English word is generated from tile ith one in a secondary generation process.</Paragraph> <Paragraph position="5"> 3. The probabilities R (k I e~) that the ith English word generates exactly k other words in the secondary process. By convention, we set R(0 \[ e0) = 1 to assure that the null word does not generate any other words.</Paragraph> <Paragraph position="6"> The lnollel probability that the word f generates e,, in tile primary process, and e~:,...,e~, in the secondary one, is equal to the product</Paragraph> <Paragraph position="8"> Given a pair of English and French sentences E and F, by the term generation pattern $ we understand the specification of which English words were generated from which French ones, and which~secondary words from which primary ones. Therefore, the probability P(E,$IF) of generating the words of E ira a pattern $ from those of F is given simply by a product of factors like (4.1), one for each French word. We can then think of estimating the probabilities P(e, l f), R(k l e,), and Q(e:lC/) by the following algorithm at tile start of which all counters are set to 0: 1. For a sentence pair E,F of the texts, find that pattern $ that gives the maximal value of P(E,$IF), and then make the (somewhat impulsive) decision that that pattern $ actually took place.</Paragraph> <Paragraph position="9"> 2. If in the pattern $, f gave rise to e,, augment counter CP(e,,f) by l; if e, gave rise to k sccoudary English words, augment counter CR(k, e,) by 1 ; if e~ is any (secondary) word that was given rise to by e,, augment counter CQ(e~, e,) by 1.</Paragraph> <Paragraph position="10"> 3. Carry out steps 1 and 2 for all sentence pairs of tile training text.</Paragraph> <Paragraph position="11"> 4. Estimate the model probabilities by nornmlizing the correspnndiug counters, i.e.,</Paragraph> <Paragraph position="13"> The problem with the above algorithm is that it is circular: ila order to evalnate P(E,$ \] F) one needs to know the probabilities P(e, I)c), R(kl e,), and Q(ejle,) in the first place! Forttmately, the difficulty can be alleviated by use of itcrative re-estimation, which is a technique that starts out by guessing the values of unknown quantities and gradually re-adjusts them so as to account better and better for given data \[ 11 \].</Paragraph> <Paragraph position="14"> More precisely, given any specification of the probabilitics</Paragraph> <Paragraph position="16"> the freshly obtained probabilities P(e, If), R(k \]e,), and Q(e, I e,) to repeat the process fi'om step I again, etc. We hah the computation when the obtained estimates stop changing from iteration to iteration.</Paragraph> <Paragraph position="17"> While it can be shown that tile probability estimates obtained in the above process will converge \[11,12\], it cannot be proven that the values obtained will be the desired ones. A heuristic argument can be formulated making it plausible that a more complex but computationally excessive version \[13\] will succceC Its truncated modification leads to a glossary that seems a very satisfactory one. We present some interesting examples of its P(e, If) entries in Figure 3.</Paragraph> <Paragraph position="18"> Two important aspects of this process have not yet been dealt with: the initial selection of values of P(e, lf), R(kle,) , and Q(51e,), and a method of finding the pattern $ maximizing</Paragraph> <Paragraph position="20"> C. To determine the initial distribution P(e, lf) proceed as I'ollows: (i) Estimate first P(< If ) by tile algorithm of Section 3. (ii) Compute the mutual information values l(e,; f) by formula (Y2), and for each f find the 20 words e, for which I(e,;f) is largest.</Paragraph> <Paragraph position="21"> (iii) I.ct P(<~I./') = P(< l f) = (I/21) - e for all words<on the list obtained in OiL whEre e is some small positive number. l)istributc the remaining probability e uniformly over all the I nglish words not on the list.</Paragraph> <Paragraph position="22"> I:inding tile maximizing pattern $ for a given sentence pair E, F is ~ well-studiEd technical problem with a variety of ,'{mHmtatiomdly feasible solutions that arc suboptimal in some practically uuimportant respects I 14\]. Not to interrupt the flow t,l imuiti\e ideas, we omit the discussion of the corresponding d 1~2',11 illnns.</Paragraph> </Section> <Section position="8" start_page="73" end_page="74" type="metho"> <SectionTitle> 5. TOWARD A COMPLEX G1,OfSSARY </SectionTitle> <Paragraph position="0"> In the previous section we have introduced a techniqne that derives a word - to - word translation glossary. We will now reline tile model to make the probabilities a better reflection of reality, and then outline an approach for including in tile glossary Ihe /ixEd locations discussed in Section 2.</Paragraph> <Paragraph position="1"> It should be noted that while English / French translation is quite k)cal (as illustrated by the alignment of Figure 1), the model leading to (4.1) did not take advantage of this affinity of the two languages: tile relative position of the word translate pairs ill their respective selltences was not taken into account. If m and n denote the respective lengths of corresponding French and l:.nglish sentences, then the probability that 6~ (the kth word in the English sentencE) is a primary translate of.f~0 (the hth word in the \[:rench sentence) shoukl more accurately be given by the probability P(e,,kl .f,,,h,m,n) that depends both on word positions and sentence lengths. '1'o keep the formulation as simple as possiblE, WE can restrict ourselves to tile functional form l'(ei ,k I /i,,,h,m,n) = PW(e,~ I fh) PD(k l h,m,n) (5.1) In (5.1) we make thc 'distortion' distribution PD(klh,m,n) indcpcndcnt o1' the identity of the words whose positional discrepancy it dcscribcs.</Paragraph> <Paragraph position="2"> As far as secondary generation is concerned, it is first clear that the production of preceding words differs from that of those that Iollow. So the R and Q probabilities should be split into left and right probabilities RL and QL, and RR and QR. Furthermore, \re shnuld provide the Q -probabilities with their own distortion components that would depend on the distance of the secondary word from its primary 'parent'. As a result of these cons!derations, the probability that f~, generates (for instance) the primary words e,~ and preceding and following secondary words</Paragraph> <Paragraph position="4"> Obviously, other distortion formulations are possible. The purpose of any is to sharpen the derivation process by restricting the choice of translates to the positinnally likely candidates in the corresponding sen tencc.</Paragraph> <Paragraph position="5"> To find fixed locutions in English, we can use the final probabilities QL and QR obtained by tile method of the previous section to compute mutual informations between primary and secondary word pairs,</Paragraph> <Paragraph position="7"> where P(e') = C(e')/N is the relative frequency of occurrence of the secondary word e' in the English text (C(e') denotes the number of occurrences of e ' in the text of size N), and QR and QL are the average secondary generation probabilities,</Paragraph> <Paragraph position="9"> WE can then establish an experimentally appropriate threshold 71, and iuchulc in the glossary all pairs (e, e') and (e', e) whose mutual information exceeds 7'.</Paragraph> <Paragraph position="10"> While tile process above results in two-word fixed locutions, longer locutions can be obtained iteratively in the next round after the two-word variety had been included in the glossary and in the formulation of its creation.</Paragraph> <Paragraph position="11"> To obtain French locutions, one must simply reverse the direction of the translation process, making English and French the source and target languages, respectively.</Paragraph> <Paragraph position="12"> With two-word locutions present in both the English and French parts of the glossary, it is necessary to reformulate the generation process (4.1). The change would be minimal if we could decide to treat the words of a locution (,/;f') as a single word f* = U, f') rather than as two separate words f and f' whenever both are found in a sentence. In such a case nothing more than a receding of the French text would be required. However, such a radical step would almost certainly be wrong: it could well connect auxiliaries and participles that were not part of a single past construction. Clearly then, the choice between separateness and unity should be statistical, with probabilities estimated in the overall glossary construction process and initialized according to the frequencies with which elements of the pair f,f~ were associated o1 not by secondary generation when they appeared in the same sentence.</Paragraph> <Paragraph position="13"> Since the approach of this section was not yet used to obtain any results, we will leave its complete mathematical specification to a future report.</Paragraph> </Section> <Section position="9" start_page="74" end_page="74" type="metho"> <SectionTitle> 6. GF, I~,IERATIION ()F TRANSLATED 'I'EXT </SectionTitle> <Paragraph position="0"> We have pointed out in Sectk)u 2 that translation can be somewhat xaively regarded as a lhrec stage process: ( I ) Partition the source text into a set of fixed locutions. (2) Use the glossary plus contextual information to select the corresponding set of fixed lomttious in the target language. (3) Atrange the words of thc target fixed locutkms into a seqtteltce forming the target sentence.</Paragraph> <Paragraph position="1"> We have just fitfished arguin b, itt Section 5 that the partilioning of sottrcc +ext ili1O locutions is SOIIIUWIIHt conlplex, and that it must be approached statistically. The basic idea of using cotltextttal iilfOl'l?lation tO select the correct 'sense' of a Iocutioll is to eonsh uct a contextual glossary based on a probability of the form P(el J; g'IFI ) where e and f are English anrl French locutions, ;tnd q, ilq denotes a 'lexical' equivalence class of the scalence F The tu,;t of class membership woukl typically depend on tilt pre~:ence of SOIIIC contbination of words in F. The choice of an app;opriate eqtfivalcncc chtssification schenlc would, of course, be .'+he subject of research based on yet another statistical formulation. The estimate of P(el ./'; ~11&quot;1 ) would be derived from courtts o1' locttlion alignments in sentmlce translate pai,s, the alignments being dstimated based on non-contextual glossary probabilitit+s of the form (5.2).</Paragraph> <Paragraph position="2"> The last stop in our translation scheme is the re-arrangement of the words o1&quot; the generated English locutions into an appropriate sequence. To see whc'ther this can be douc statistically, we explored what would happen in the ilnpossibly optimistic case where the words generated in (2) were exactly those of the l inglish sczttencc (only their order would be unknown): From a large f'+'uglish corpus we derived estimates of trigram probabilities, P(e3let, e:~), that the word el follows immediately the sequencc pair e~, % A model of 13,nglish sentence production based on a trigram estimate would conclude that a sentence e~, ca, ..J e,, is generated with probability P(el, e2) P(e3 Iet, e2) P(e41 e2, e3) ..- P(e,, I e,,+ 2, e, I) (6.1) We then rook other l:';nglish sentences (not included in the training COrlmS) and deterntined which of the n t different arrangements el + their n words was most likely, using the l'ormula (6.1). We found that in 63% of sentences of 10 words or less, the most likely arrangement was the original English sentence.</Paragraph> <Paragraph position="3"> I;urthermore, the most likely arrangement preserved the meaning of the original sentence in 79% of the cases.</Paragraph> <Paragraph position="4"> Figure, 4. shows examples hi' synonymous and non-.synonymous re-al'rangelnenL'~.</Paragraph> <Paragraph position="5"> We realize that very little hope exists of the glossary yielding the words and only the words of an English seutence translating the original French one, and that, furthermore, Euglish sentences arc typically longer than 10 words. Nevertheless, we feel that the abow: result is a hopeful one for fnture statistical translation methods incorporating the use of appropriate syntactic structure information.</Paragraph> <Paragraph position="6"> Mr. Speaker, I rise on a question of privilege Monsieur l'Orateur, je souleve la qoestion de privilege affecting the rights and prerogatives of pmliamentary committees a propos des droits et des prerogatives des eomites parlenmnlaires and o11o which reflects oii tile wol'd of two ininisters et i)otlr nlettre en d<mte les i)ro\])os tie detlX illhlistles of the Crown.</Paragraph> <Paragraph position="7"> tic la Cotlronne.</Paragraph> <Paragraph position="8"> 2. que 0.177 which 0.161 3. dont 0.082 that 0.084 4, de 0.060 0.038 5. d' 0.035 to 0.032 6. laquclle 0.(131 of 0.027 7. ou 0.027 the 0.026 8. ct 0.022 what 0.018 THEREFORE DONC 1. donc 0.514 therefore 0.322 2. consequent 0.075 so 0.147 3. pat&quot; 0.074 is 0.034 4. ce 0.066 then 0.024 5. pourquoi 0.064 thus 0.022 6. alors 0.025 the 0.018 7. il 0.025 that 0.013 8. aussi 0.015 us 0.012 STILL ENCORE 1. encore 0.435 still 0,181 2. toujours 0.230 again 0.174 3. reste 0.027 yet 0.148 4. *** 0.020 even 0.055 5. quand 0.018 more 0.046 6. meme 0.017 another 0,030 7. de 0.015 further 0.021 8. de 0.014 once 0.013</Paragraph> <Paragraph position="10"/> </Section> <Section position="10" start_page="74" end_page="74" type="metho"> <SectionTitle> EXAMPLES OF PARTIAL GLOSSARY LISTS OF MOST LIKELY WORD TRANSLATES AND THEIR PROBABILITIES </SectionTitle> <Paragraph position="0"> Note: *** denotes miscellaneous words not belonging to the lexicon.</Paragraph> </Section> <Section position="11" start_page="74" end_page="74" type="metho"> <SectionTitle> PEOPLE GENS </SectionTitle> <Paragraph position="0"> 1. les 0.267 people 0.781 2. gens 0.244 they 0.013 3. personnes 0.100 those 0.009 4. population 0.055 individuals 0.008 5. peuple 0.035 persons 0.005 6. canadiens 0.031 people's 0.004 7. habitants 0.024 men 0.004 8. ceux 0.023 person 0.003 OBTAIN OBTENIR l. obtenir 0.457 get 0.301 2. pour 0.050 obtain 0.108 3. les 0.033 have 0.036 4. de 0.031 getting 0.032 5. trouver 0.026 seeking 0.023 6. se 0.025 available 0.021 7. obtenu 0.020 obtaining 0.021 8. procurer 0.020 information 0.016 QUICKLY RAPIDEMENT 1. rapidement 0.508 quickly 0.389 2. vite 0.130 rapidly 0.147 3. tot 0.042 fast 0.052 4. rapide 0.021 quick 0.042 5. brievement 0.019 soon 0.036 6. aussitot 0.013 faster 0.035 7. plus 0.012 speedy 0.026 8. bientot 0.012 briefly 0.025</Paragraph> </Section> <Section position="12" start_page="74" end_page="74" type="metho"> <SectionTitle> FIGURE 3 (PART II) EXAMPLES OF PARTIAL GLOSSARY LISTS OF MOST LIKELY WORD TRANSLATES AND THEIR PROBABILITIES EXAMPLES OF RECONSTRUCTION TttAT PRESERVE MEANING: </SectionTitle> <Paragraph position="0"> would I report directly to you? I would report directly to you? now let me mention some of the disadvantages. let me mention some of the disadvantages now, he did this several hours later.</Paragraph> <Paragraph position="1"> this he did several hours later.</Paragraph> </Section> <Section position="13" start_page="74" end_page="74" type="metho"> <SectionTitle> EXAMPLES OF RECONSTRUCTION THAT DO NOT PRESERVE MEANING </SectionTitle> <Paragraph position="0"> these people have a fairly large rate of turnover.</Paragraph> <Paragraph position="1"> of these people have a fairly large turnover rate.</Paragraph> <Paragraph position="2"> in our organization research has two missions.</Paragraph> <Paragraph position="3"> in our missions research organization has two.</Paragraph> <Paragraph position="4"> exactly how this might be done is not clear.</Paragraph> <Paragraph position="5"> clear is not exactly how this might be done.</Paragraph> </Section> <Section position="14" start_page="74" end_page="76" type="metho"> <SectionTitle> FIGURE 4 STATISTICAL ARRANGEMENT OF WORDS BELONGING TO ENGLISH SENTENCES </SectionTitle> <Paragraph position="0"/> </Section> class="xml-element"></Paper>