<?xml version="1.0" standalone="yes"?>
<Paper uid="I05-2021">
  <Title>Evaluating the Word Sense Disambiguation Performance of Statistical Machine Translation</Title>
  <Section position="9" start_page="124" end_page="124" type="concl">
    <SectionTitle>
8 Conclusion
</SectionTitle>
    <Paragraph position="0"> We presented empirical results casting doubt on the increasingly common assumption that SMT models are very good at WSD, even though they do not explicitly address WSD as an independent task.</Paragraph>
    <Paragraph position="1"> Using the Senseval-3 Chinese lexical sample task as a testbed, we directly compared the performance of a typical Chinese-to-English SMT model, built from off-the-shelf toolkits, with that of state-of-the-art Senseval models and found that the SMT model does not achieve the same high accuracies as any the dedicated WSD models considered. Even after attempting to compensate for the difference between training and evaluation data in favor of the SMT model, the accuracy of the SMT model is still significantly lower than that of the dedicated WSD systems.</Paragraph>
    <Paragraph position="2"> Error analysis confirms the weaknesses of the SMT models for the WSD task. Unlike dedicated WSD models, SMT models only rely on the local context to make translation choices, and tend to prefer phrasal cohesion in the target language and fluency, rather than adequacy of the translation of each source word.</Paragraph>
    <Paragraph position="3"> These results cast doubt on the speculative claim that SMT systems do not need sophisticated WSD models, and suggest on the contrary that the predictions of the dedicated models should be useful. Puzzlingly, in converse experiments, using a state-of-the-art WSD model to choose translation candidates for a typical IBM SMT system, we find that WSD does not yield significantly better translation quality than the SMT system alone (Carpuat and Wu, 2005). Taken together, these results suggest that another SMT formulation might be needed. In particular, more grammatically structured statistical MT models that are better equipped to handle long distance dependencies, such as the ITG based &amp;quot;grammatical channel&amp;quot; translation model (Wu and Wong, 1998), might make better use of the WSD predictions.</Paragraph>
  </Section>
class="xml-element"></Paper>