<?xml version="1.0" standalone="yes"?>
<Paper uid="W03-0413">
  <Title>Confidence Estimation for Translation Prediction</Title>
  <Section position="10" start_page="2" end_page="2" type="concl">
    <SectionTitle>
9 Conclusion
</SectionTitle>
    <Paragraph position="0"> The results obtained in this paper can be summarized in the following set of questions and answers: + Can the probabilities of correctness estimated by the CE layer exceed the native probablities in discrimination capacity? Depending on the underlying SMT model, we obtained a relative improvement in correct rejection rate (CR) ranging from 3:90% to 33:09% at a fixed 0:80 (CA) correct acceptance rate for prediction lengths varying between 1 and 4.</Paragraph>
    <Paragraph position="1"> + Can we improve the overall performance of the underlying SMT application using confidence estimation? In simulated results, we found a significant gain (10% relative) in benefit to a translator due to the use of a CE layer in two of three translation models tested.</Paragraph>
    <Paragraph position="2"> + Can prediction accuracy of the application be improved using prediction model combinations? A maximum CE voting scheme yields a 29:31% accuracy improvement of the maximum possible accuracy gain. A similar voting scheme using native probabilies significantly decreases the accuracy of the model combination.</Paragraph>
    <Paragraph position="3"> + How does the prediction accuracy of the native models influence the CE accuracy? Prediction accuracy didn't prove to be a significant factor in determining the discrimination capacity of the confidence estimate. null + How does CE accuracy change with various ML approches? A multi-layer perceptron (MLP) with 20 hidden units significantly outperformed one with 0 hidden units (equivalent to a maxent model for this application).</Paragraph>
    <Paragraph position="4"> + Confidence feature selection: which confidence features are more useful and how does their discrimination capacity vary with different contexts and different native SMT models? Confidence features based on the original model and the n-best prediction turned out to be the most relevant group of featured, folowed by features that capture the intrinsic difficulty of the source text and finally translationdifficulty-specific features. We also observed interesting variations in relevance as the original models changed.</Paragraph>
    <Paragraph position="5"> Future work will include the search for more relevant confidence features, such as features based on consenus over word-lattices ((Mangu et al., 2000)), past performance, the use of more appropriate correct/false tagging methods and experiments with different machine learning techniques. Finally, we would like to investigate whether confidence estimation can be used to improve the model prediction accuray, either by using re-scoring techniques or using the confidence estimates during search (decoding). null</Paragraph>
  </Section>
class="xml-element"></Paper>