<?xml version="1.0" standalone="yes"?>
<Paper uid="W05-1628">
  <Title>2Information and Communication Technologies</Title>
  <Section position="7" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
6 Evaluation
</SectionTitle>
    <Paragraph position="0"> In this section, we outline our preliminary evaluation of grammaticality in which we compare our dependency based generation method against a baseline. To study any improvements in grammaticality, we compare our dependency based generation method against a baseline consisting of sentences generated using bigram model.</Paragraph>
    <Paragraph position="1"> In the evaluation, we do not use any smoothing algorithms for dependency counts. For both our approach and the baseline, Katz's Back-off smoothing algorithm is used for bigram probabilities.</Paragraph>
    <Paragraph position="2"> For our evaluation cases, we use the Information Fusion data collected by [Barzilay et al., 1999]. This data is made up of news articles that have been rst grouped by topic, and then component sentences further clustered by similarity of events. There are 100 sentence clusters and on average there are 4 sentences per cluster. Each sentence in the cluster is initially passed through the Connexor dependency parser (www.connexor.com) to obtain dependency relations. Each sentence cluster forms an evaluation case in which we generate a single sentence. Example output and the original text of the cluster is presented in Figure 4.</Paragraph>
    <Paragraph position="3"> To give both our approach and the baseline the greatest chance of generating a sentence, we obtain our bigrams from our evaluation cases.5 Aside from this preprocessing to collect input sentence bigrams and dependencies, there is no training as such. For each evaluation case, both our system and the baseline method generates a set of answer strings, from 3 to 40 words in length.</Paragraph>
    <Paragraph position="4"> For each generated output of a given sentence length, we count the number of times the Connexor parser resorts to returning partial parses. This count, albeit a noisy one, is used as our measure of ungrammaticality. We calculate the average ungrammaticality score across evaluation cases for each sentence length.</Paragraph>
    <Paragraph position="5"> 5Note that this is permissible in this case because we are not making any claims about the coverage of our model.</Paragraph>
    <Paragraph position="6"> Original Text A military transporter was scheduled to take off in the afternoon from Yokota air base on the outskirts of Tokyo and y to Osaka with 37,000 blankets . Mondale said the United States, which has been ying in blankets and is sending a team of quake relief experts, was prepared to do more if Japan requested . United States forces based in Japan will take blankets to help earthquake survivors Thursday, in the U.S. military's rst disaster relief operation in Japan since it set up bases here.</Paragraph>
    <Paragraph position="7"> Our approach with Dependencies and End of Sentence Check 6: united states forces based in blankets 8: united states which has been ying in blankets 11: a military transporter was prepared to osaka with 37,000 blankets 18: mondale said the afternoon from yokota air base on the united states which has been ying in blankets 20: mondale said the outskirts of tokyo and is sending a military transporter was prepared to osaka with 37,000 blankets 23: united states forces based in the afternoon from yokota air base on the outskirts of tokyo and y to osaka with 37,000 blankets 27: mondale said the afternoon from yokota air base on the outskirts of tokyo and is sending a military transporter was prepared to osaka with 37,000 blankets 29: united states which has been ying in the afternoon from yokota air base on the outskirts of tokyo and is sending a team of quake relief operation in blankets 31: united states which has been ying in the afternoon from yokota air base on the outskirts of tokyo and is sending a military transporter was prepared to osaka with 37,000 blankets 34: mondale said the afternoon from yokota air base on the united states which has been ying in the outskirts of tokyo and is sending a military transporter was prepared to osaka with 37,000 blankets 36: united states which has been ying in japan will take off in the afternoon from yokota air base on the outskirts of tokyo and is sending a military transporter was prepared to osaka with 37,000 blankets  Higher scores indicates worse performance.</Paragraph>
    <Paragraph position="8"> The results are presented in Figure 5. Our approach almost always performs better than the baseline, producing less errors per sentence length. Using the Wilcoxon Signed Rank Text (alpha = 0.5), we found that for sentences of length greater than 12, the differences were usually signi cant.</Paragraph>
  </Section>
class="xml-element"></Paper>