<?xml version="1.0" standalone="yes"?>
<Paper uid="J00-2005">
  <Title>Squibs and Discussions Pipelines and Size Constraints</Title>
  <Section position="4" start_page="254" end_page="255" type="intro">
    <SectionTitle>
4. Experimental Results
</SectionTitle>
    <Paragraph position="0"> STOP is currently being tested in a clinical trial, in order to determine its effectiveness in helping people stop smoking. The version of STOP used in the clinical trial had a single-solution pipeline architecture as described above. Its trimmer used a size estimator that was tuned to be conservative (and hence often produced leaflets that were smaller than they could have been), but still in a few cases underestimated true length and hence resulted in leaflets that were five A5 pages instead of four. Such leaflets were manually fixed by the researchers running the trial, usually by adjusting the formatting of the leaflet (for example, margins or interparagraph separation). We felt this was not acceptable for a production version of STOP, however; such a system should guarantee conformance to the length constraint without needing manual intervention.</Paragraph>
    <Paragraph position="1"> Also, conformance should be achieved by adjusting content, not formatting. The formatting of STOP leaflets was designed by an expert graphic designer with the goal of enhancing readability, and we believed it should be treated as fixed, not variable. 2 In order to explore what should be done in a production version of STOP, we conducted some experiments (after the STOP clinical trial had started) on the impact of different architectures on satisfying the size constraint while utilizing as much as possible of the available space. For these experiments, we took the version of the system used in the clinical trial (including accumulated bug fixes and enhancements), and retuned the size estimator to take into account these accumulated changes. After retuning, STOP produced leaflets that fit the size constraint for all members of a &amp;quot;tuning set&amp;quot; of 150 questionnaires. Then we made the following changes: A delta parameter was added to the size estimator; essentially, a delta of N makes the estimator think that a page can hold N more lines of text  than it can in reality contain.</Paragraph>
    <Paragraph position="2"> 2 Similarly, I believe the editors of Computational Linguistics would not be pleased if I submitted a squib that conformed to the eight-page size limit by using nonstandard margins or line spacing.  Computational Linguistics Volume 26, Number 2 A multiple-solution mode was added to the system. In this mode, the trimmer is run several times, at different delta values. The resultant document plans are processed by the rest of the system and by Word, and a choice module picks the resulting document that has the highest word count while still satisfying the size constraint.</Paragraph>
    <Paragraph position="3"> A revision mode was added to the system. In this mode, the system generates an initial document using a fixed delta. Then, a revision module obtains the actual size of the document from Word, and either deletes an additional message (if the document is too large) or restores the last deleted message (if the document meets the size constraint). This process continues until the system finds the largest document that meets the size constraint. 3 The modified system was run on a set of 1,000 questionnaires from the clinical trial, in the original single-solution pipeline mode, in the multiple-solution pipeline mode, and in revision mode. For the pipeline modes, the system was run with the deltas -2, -1, 0, 1, 2, 3, 4, 5, and 6. Measurements were made of: * The percentage of leaflets that exceeded the size constraint.</Paragraph>
    <Paragraph position="4"> * For leaflets satisfying the size constraint, the average number of words in the two inside pages, both as an absolute number and as a percentage of the number of words in the inside pages when processed under revision mode.</Paragraph>
    <Paragraph position="5"> * The average processing time (total elapsed time, not just computation time) required per document, on a Pentium 266MHz with 128MB of memory. 4 These results are shown in Tables 1 and 2. For multiple-solution pipelines, we tried all pairs, triples, and quadruples of deltas between -2 and 6, and in Table 2 only show the results for the pair, triple, and quadruple that led to the highest average word count while always satisfying the size constraint. We also ran STOP on the full set of 2,582 clinical-trial questionnaires in single-delta mode with deltas of -1, 0, and 1, in order to get a more accurate estimate of the number of constraint violations under these deltas.</Paragraph>
  </Section>
class="xml-element"></Paper>