<?xml version="1.0" standalone="yes"?>
<Paper uid="J00-2005">
  <Title>Squibs and Discussions Pipelines and Size Constraints</Title>
  <Section position="6" start_page="256" end_page="257" type="concl">
    <SectionTitle>
6. Implications
</SectionTitle>
    <Paragraph position="0"> In STOP, the single-solution pipeline does a poor job at meeting the size constraint while utilizing as much of the available space as possible. No doubt the performance of the single-solution pipeline could be enhanced by adding more complexity to the  Computational Linguistics Volume 26, Number 2 size estimator; but such a system still would not give 100% accurate estimates on 100% of the generated documents. Furthermore additional complexity would make the estimator harder to maintain as changes were made to the code being estimated. Both the multiple-solution pipeline and revision mode do a much better job of utilizing the available space while observing the size constraint. Revision mode does better than the multiple-solution pipeline, but only slightly. However, revision mode is robust in the face of increased data set size and changes to the code.</Paragraph>
    <Paragraph position="1"> The effectiveness of multiple-solution pipelines should perhaps not be surprising, given the popularity of such pipelines in other areas of speech and language processing. For example, in a speech system a word-level analysis component may pass several word hypotheses to a language model; and in a natural language analysis system, a morphology system may pass several possible analyses of a surface form word to a parser. However, multiple-solution pipelines have not received a great deal of attention in the NLG community. I am not aware of any previous NLG papers that presented experimental data comparing single-solution to multiple-solution pipelines, and many NLG pipeline critics (including Danlos) assume that pipeline modules only produce one solution.</Paragraph>
    <Paragraph position="2"> Do these results generalize to other constraints and optimizations? In principle, it seems that similar findings should apply to other constraints and optimizations that depend on decisions or measurements made in more than one module. However, a big caveat is that many of the constraints and optimizations important to NLG systems are difficult to measure, which may lessen the benefits of complex architectures. For example, an important constraint in STOP is that texts should be easy to read for poor readers. However, the only computational mechanism we are aware of for measuring reading difficulty is reading-level formulas (such as Flesch Reading Ease), whose accuracy is doubtful (Kintsch and Vipond 1979). Without reliable global measures of readability, perhaps the best we can do (and the approach adopted in STOP) is to design messages that readability experts think are appropriate for poor readers; this is something that can be done in a single-solution pipeline architecture.</Paragraph>
    <Paragraph position="3"> In other words, if we cannot properly measure the thing we are trying to optimize or satisfy (which may be the case with the majority of constraints and optimizations that today's NLG systems builders are concerned with), then there may be little value in shifting to a complex architecture that supports more sophisticated search (which is perhaps the main benefit of revision and multiple-solution pipelines). This may explain the continuing popularity of single-solution pipeline architectures in applied NLG systems.</Paragraph>
  </Section>
class="xml-element"></Paper>