<?xml version="1.0" standalone="yes"?>
<Paper uid="W97-0303">
  <Title>An Efficient Distribution of Labor in a Two Stage Robust Interpretation Process</Title>
  <Section position="8" start_page="32" end_page="33" type="evalu">
    <SectionTitle>
7 Evaluation
</SectionTitle>
    <Paragraph position="0"> The LR MDP parser was run over the corpus at three different flexibility settings. The first setting, MDP 1, is Minimum Distance Parsing with maximum deviation penalty of 1. Similarly, MDP 3 and MDP 5 are MDP with maximum devaition penalty of 3 and 5 respectively. We also ran the version of GLR* where only initial segments can be skipped which we refer to as GLR* with Restarts.</Paragraph>
    <Paragraph position="1"> Thus, while the parser can restart from each word in the sentence, analyses produced are always for contiguous segments of the sentence. We ran GLR* with Restarts both with and without repair. Timings for all five of these iterations over the corpus are displayed in Figure 5. Notice that GLR* with Restarts is significantly faster than even MDP 1.</Paragraph>
    <Paragraph position="2"> And since the repair stage is run only for sentences that the repair module determines need repair, and since the repair process takes only seconds on average to run, no significant difference in time can be seen in this graph between the case with repair and the case without repair.</Paragraph>
    <Paragraph position="3"> The translation quality ratings for the five different iterations over the corpus are found in Figure 6. Predictably, MDP 5 is an improvement over MDP 1 and MDP 3, with an associated significant cost in run time. Also, not surprisingly, the very restricted GLR* with Restarts, while faster than either of the other two, has a correspondingly lower associated translation quality. However, GLR* with Restarts --{- Repair outperforms the other methods in terms of total number of acceptable translations, while not being significantly slower than GLR*  with Restarts without repair. Though these results display certain trends in the performance of these alternative approaches, the differences in general are very small. For example, the difference in number of acceptable translations between MDP 5 and GLR* with restarts + repair is only about 1%. The largest difference between the two is that GLR* with restarts + repair has about 7% more sentences with translation quality of Partial or better, indicating that GLR* with restarts + repair produces analyses that are useful for furthering the conversation between the two speakers using the system 7% more often than MDP 5.</Paragraph>
    <Paragraph position="4"> While we have no doubt that increasingly more flexible versions of MDP would perform better than MDP 5, we have already demonstrated that even MDP 5 is impractical in terms of its run-time performance. Thus we conclude that the two stage ROSE approach, even with a very limited flexibility parser, is a superior choice. We believe that by increasing the flexibility of the parser to include very limited skipping in addition to restarts would increase the performance of this two stage approach without incurring a significant increase in run time performance. Determining exactly how much skipping is ideal is a direction for future research.</Paragraph>
  </Section>
class="xml-element"></Paper>