<?xml version="1.0" standalone="yes"?>
<Paper uid="C02-1057">
  <Title>Tiejun Zhao +</Title>
  <Section position="4" start_page="40" end_page="40" type="metho">
    <SectionTitle>
3 Result Analysis
</SectionTitle>
    <Paragraph position="0"> The experimental results and the charts have shown some intuitionistic relationship among the automatic criteria of Dice coefficient, cosine value, edit distance and the human evaluation result. A more solid analysis is made in this section to verify this relationship. Statistical analysis is a useful tool to 1) find the relationship between data sets and 2) decide whether the relationship is significant enough or just for random errors.</Paragraph>
    <Paragraph position="1"> The measure of linear correlation is a way of assessing the degree to which a linear relationship between two variables is implied by observed data. The correlation coefficient between variable X and Y is defined as  The symbol meanings are as follows: sX: sample standard deviation of variable X sY: sample standard deviation of variable Y n: sample size</Paragraph>
    <Paragraph position="3"> ): the sample mean of variable X (Y) From its definition, we know that the correlation coefficient is scale-independent and</Paragraph>
  </Section>
  <Section position="5" start_page="40" end_page="40" type="metho">
    <SectionTitle>
11 [?][?][?] r
</SectionTitle>
    <Paragraph position="0"> .</Paragraph>
    <Paragraph position="1"> After we get the correlation coefficient r, a significance test at the level 01.0=a is made to verify whether the correlation is real or just due to random errors. Linear regression is used to construct a model that specifies the linear relationship between the variables X and Y. A scatter diagram and regression line will be presented for an intuitionistic view of the relationship. The results are presented in the graphs below. In the graphs, the human evaluation results are placed on the X axis, while the automatic results are on the Y axis. Correlation coefficient and the linear regression equation are shown below the graphs. Taking into the sample size and the correlation coefficient, the significance level is also calculated for the statistical analysis.</Paragraph>
    <Paragraph position="3"> It is a property of r that it has a value domain of [-1,+1]. A positive r implies that the X and Y tend to increase/decrease together. A minus r implies a tendency for Y to decrease as X increases and vice versa. When there is no particular relation between X and Y, r tends to have a value close to zero. From the above analysis, we can see that the Dice coefficient, cosine, and average of the automatic values are highly correlated with the human evaluation results with r=0.96. P &lt; 0.01 shows the two variables are strongly correlated with a significance level beyond the 99%. While P &lt; 0.01 for the linear regression equation has the same meaning.</Paragraph>
    <Paragraph position="4"> Conclusion Our evaluation method is designed for the localization oriented EBMT system. This is why we take string similarity criteria as basis of the evaluation. In our approach, we take edit distance, dice coefficient and cosine correlation between the machine translation results and the standard translation as evaluation criteria. A theoretical analysis is first made so that we can know clearly the goodness and shortcomings of the three factors. The evaluation has been used in our development to distinguish bad translations from good ones. Significance test at 0.01 level is made to ensure the reliability of the results. Linear regression and correlation coefficient are calculated to map the automatic scoring results to human scorings.</Paragraph>
  </Section>
class="xml-element"></Paper>