<?xml version="1.0" standalone="yes"?>
<Paper uid="C00-1006">
  <Title>The Effects of Word Order and Segmentation on Translation Retrieval Performance</Title>
  <Section position="6" start_page="38" end_page="40" type="evalu">
    <SectionTitle>
4.2 Results
</SectionTitle>
    <Paragraph position="0"> The results for the different similarity metrics with character-based and word-based indexing are given in Tal)le 1, with the two bag-of-words al)t)roaches partitioned off from the three word order-s(msitive al)I)roaches tor ea(:h indexing paradigm. &amp;quot;Accuracy&amp;quot; is an indication of the prol)ortion of intmts fbr whi(:h  an optimal translation was produced; character-based indexing accuracies in bold indicate a significant ~ advantage over the corresponding wprd-based indexing accuracy, and figures in brackets for word-based indexing indicate the relative pert'ormaime gain over the corresponding character-based indexing configuration. &amp;quot;Edit discrep.&amp;quot; refers to the mean minimum edit distance discrepancy between translation candidate(s) and optimal translation(s) in the case of the translation candidate set containiug uo optimal translations. &amp;quot;Ave. outputs&amp;quot; describes the average number of translation candidates output by the system, with the figure in brackets being the proportion of int)uts for which a unique translation candidate was produced. &amp;quot;Ave. time&amp;quot; describes the average time taken to deterlnine the translation era&gt; didate(s) for a single output, relative to the time taken tbr word-based edit distance retrieval.</Paragraph>
    <Paragraph position="1"> Perhaps the most striking result is ttmt character-based indexing produces a superior match accuracy to word-based indexing tbr all similarity metrics, at; a significant margin tbr all three word order-based methods. This is the complete opposite of what we had expected, although it does fit in with the findings of Fujii and Croft (1993) that character-based indexing performs comparably with word-based indexing in Japanese information retrieval.</Paragraph>
    <Paragraph position="2"> Looking to word order, we see that edit distance outperforms all other methods for t)oth characterand word-based indexing, peaking at just over 50% for character-based indexing. Tile relative performance of the remaining methods is variable, with the two bag-of-words methods being superior to or roughly equivalent to sequential correspondence and weighted sequential correspondence tbr word-based indexing, but tile word order-based methods having a cleat' advantage over the bag-of-words methods for character-based indexing. It is thus difticult to draw any hard and fast conclusion as to the relative merits of word order-based versus bag-of words methods, other than to say that edist distance would appear to have a clear advantage over other methods.</Paragraph>
    <Paragraph position="3"> The figures for edit discrepancy in the case of non-optimal translation candidate(s) are equally interesting, and suggest that on the whole, the various methods err more conservatively for character-based than word-based indexing. The most robust method is (source language) edit distance, at all edit discrepancy of 1.82 and 2.O3 for character-based and word-based indexing, respectively.</Paragraph>
    <Paragraph position="4"> All methods were able to produce just over one translation candidate on average, with all other than edit distance returning a unique translation candidate over 90% of the time. The greater number of outtmts for the edit distance method can certainly be viewed as one reason for its inflated performance, although the lower level of mnbiguity for character-based indexing but higher accuracy, would tend to suggest otherwise.</Paragraph>
    <Paragraph position="5"> Lastly, word-based indexing was found to be faster than character-based indexing across the board, for the simple reason that the immber of character seg~As determined by the paired t test (p &lt; 0.05).</Paragraph>
    <Paragraph position="6"> ments is always going to be greater than or equal to the number of word segments. The average segment lengths quoted above (26.1 characters vs. 13.4 words) indicate that we generally have twice as many characters as words in a given striug. Additionally, tile acceleration technique described in SS 3.2 of sequentially working through the segment component of the input string in increasing order of global frequency, has a greater ett&gt;ct for word-tmsed indexing than character-based indexing, accentuating any speed disparity.</Paragraph>
    <Section position="1" start_page="39" end_page="40" type="sub_section">
      <SectionTitle>
4.3 Reflections on the results
</SectionTitle>
      <Paragraph position="0"> An immediate exlflanation tbr character-based indexing's empirical edge over word-based iudexing is the semantic smoothing effects of individual kanji characters, alluded to above (SS 2). To take an example, the single-segment nouns A': n \[s6sa\] and : ng0 \[sadS\] both mean &amp;quot;operation&amp;quot;, but would not match under word-based indexing. Character-based indexing, on the other hand, would recogifise the overlap in character content, and in the process pick up on the semantic corresi)ondenee between the two words.</Paragraph>
      <Paragraph position="1"> To take tile opposite tack, one reason wily word-based indexing may have been disadvantaged is the we did not stem or lemmatise words in word-based indexing. Having said this, the. output fl'om ChaSen is such that stems of inflecting words are given as a single segment, with inflectional morphemes each presented as sel)arate segments. In this sense, stemruing would only act to delete the inflectional morphemes, and not add allything new.</Paragraph>
      <Paragraph position="2"> Another way in which the outlmt of ChaSen could conceivably have atlbcted retrieval perforiilance is that technical terms tended to be oversegmented. Experilnentally combining recognised technical terms into a single segment (particularly in the case of contiguous katakana segments in the manner of Nljii and Croft (1993)), however, degraded rather than lint)roved retrieval performance for both character-based and word-based indexing.</Paragraph>
      <Paragraph position="3"> As such, this side-etfect of ChaSen would not appear to have impinged on retriewfl accuracy.</Paragraph>
      <Paragraph position="4"> One other plausible reason for tile unexpected results is that the test data could have been ill some way inherently better suited to character-based indexing than word-based indexing, although the fact that the results were cross-wtlidatcd would tend to rule out this possibility.</Paragraph>
      <Paragraph position="5"> A surprising result was the lacklustre performance of the weighted sequential correspondence method as compared to simple sequential correspondence. We have no explanation for the drop in accuracy, other than to speculate that either the proposed formulation is in some way flawed or contiguity of match does not impinge on translation similarity to the degree we had expected.</Paragraph>
      <Paragraph position="6"> To return to the original question posed above of retrieval speed vs. accuracy, the word order-sensitive edit distance approach would seem to hold a genuine edge over the other methods, to an order that would suggest the extra computational overhead is warranted, ill both accuracy and translation discrepancy. It must be said that the TM used in evalua- null tion was too small to get a gemfine f(;el for the comt)ul;ational overhead that would 1)e cxp(,,ri(;ncc, d in ~ real-world TM system context of t)ot;entially millions rath(;r than thousands of translation records. A C the saint', (tim(;, however, coding Ul) the c(lit distan(:(; l)roc(',dure in a language fasto, r than Perl using chara(;l;(?r r~d;h(~,r \[;\]lall SI;t'illg COIlq)arisol~ 1)roc(?(hlrcs mid ai)l)lying (lynami(&amp;quot; 1)rogl'amming t(whni(lu(,,s or similar, may well oIl~set th('. large \]nero.as(; in number of comparisons dcmand(',d of the system.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>