<?xml version="1.0" standalone="yes"?>
<Paper uid="P03-1034">
  <Title>Integrating Discourse Markers into a Pipelined Natural Language Generation Architecture</Title>
  <Section position="3" start_page="0" end_page="0" type="metho">
    <SectionTitle>
2 Discourse Markers in NLG
</SectionTitle>
    <Paragraph position="0"> Discourse markers, or cue words, are single words or small phrases which mark specific semantic relations between adjacent sentences or small groups of sentences in a text. Typical examples include words like however, next,andbecause. Discourse markers pose a problem for both the parsing and generation of clauses in a way similar to the problems that referring expressions pose to noun phrases: changing the lexicalization of a discourse marker can change the semantic interpretation of the clauses affected.</Paragraph>
    <Paragraph position="1"> Recent work in the analysis of both the distribution and role of discourse markers has greatly extended our knowledge over even the most expansive previous accounts of discourse connectives (Quirk et al., 1985) from previous decades. For example, using a large scale corpus analysis and human subjects employing a substitution test over the corpus sentences containing discourse markers, Knott and Mellish (1996) distilled a taxonomy of individual lexical discourse markers and 8 binary-valued features that could be used to drive a discourse marker selection algorithm.</Paragraph>
    <Paragraph position="2"> Other work often focuses on particular semantic categories, such as temporal discourse markers. For instance, Grote (1998) attempted to create declarative lexicons that contain applicability conditions and other constraints to aid in the process of discourse marker selection. Other theoretical research consists, for example, of adapting existing grammatical formalisms such as TAGs (Webber and Joshi, 1998) for discourse-level phenomena.</Paragraph>
    <Paragraph position="3"> Alternatively, there are several implemented systems that automatically insert discourse markers into multi-sentential text. In an early instance, Elhadad and McKeown (1990) followed Quirk's pre-existing non-computational account of discourse connectives to produce single argumentative discourse markers inside a functional unification surface realizer (and thereby postponing lexicalization till the last possible moment).</Paragraph>
    <Paragraph position="4"> More recent approaches have tended to move the decision time for marker lexicalization higher up the pipelined architecture. For example, the MOOSE system (Stede and Umbach, 1998; Grote and Stede, 1999) lexicalized discourse markers at the sentence planning level by pushing them directly into the lexicon. Similarly, Power et al. (1999) produce multiple discourse markers for Patient Information Leaflets using a constraint-based method applied to RST trees during sentence planning.</Paragraph>
    <Paragraph position="5"> Finally, in the CIRC-SIM intelligent tutoring system (Yang et al., 2000) that generates connected dialogues for students studying heart ailments, discourse marker lexicalization has been pushed all the way up to the discourse planning level. In this case, CIRC-SIM lexicalizes discourse markers inside of the discourse schema templates themselves.</Paragraph>
    <Paragraph position="6"> Given that these different implemented discourse marker insertion algorithms lexicalize their markers at three distinct places in a pipelined NLG architecture, it is not clear if lexicalization can occur at any point without restriction, or if it is in fact tied to the particular architectural modules that a system designer chooses to include.</Paragraph>
    <Paragraph position="7"> The answer becomes clearer after noting that none of the implemented discourse marker algorithms described above have been incorporated into a comprehensive NLG architecture containing additional significant components such as revision (with the exception of MOOSE's lexical choice component, which Stede considers to be a submodule of the sentence planner).</Paragraph>
  </Section>
  <Section position="4" start_page="0" end_page="0" type="metho">
    <SectionTitle>
3 Current Implemented Revision Systems
</SectionTitle>
    <Paragraph position="0"> Revision (or clause aggregation) is principally concerned with taking sets of small, single-proposition sentences and finding ways to combine them into more fluent, multiple-proposition sentences. Sentences can be combined using a wide range of different syntactic forms, such as conjunction with &amp;quot;and&amp;quot;, making relative clauses with noun phrases common to both sentences, and introducing ellipsis.</Paragraph>
    <Paragraph position="1"> Typically, revision modules arise because of dissatisfaction with the quality of text produced by a simple pipelined NLG system. As noted by Reape and Mellish (1999), there is a wide variety in revision definitions, objectives, operating level, and type. Similarly, Dalianis and Hovy (1993) tried to distinguish between different revision parameters by having users perform revision thought experiments and proposing rules in RST form which mimic the behavior they observed.</Paragraph>
    <Paragraph position="2"> While neither of these were implemented revision systems, there have been several attempts to improve the quality of text from existing NLG systems.</Paragraph>
    <Paragraph position="3"> There are two approaches to the architectural position of revision systems: those that operate on semantic representations before the sentence planning level, of which a prototypical example is (Horacek, 2002), and those placed after the sentence planner, operating on syntactic/linguistic data. Here we treat mainly the second type, which have typically been conceived of as &amp;quot;add-on&amp;quot; components to existing pipelined architectures. An important implication of this architectural order is that the revision components expect to receive lexicalized sentence plans. Of these systems, Robin's STREAK system (Robin, 1994) is the only one that accepts both lexicalized and non-lexicalized data. After a sentence planner produces the required lexicalized information that can form a complete and grammatical sentence, STREAK attempts to gradually aggregate that data. It then proceeds to try to opportunistically include additional optional information from a data set of statistics, performing aggregation operations at various syntactic levels. Because STREAK only produces single sentences, it does not attempt to add discourse markers. In addition, there is no apriori way to determine whether adjacent propositions in the input will remain adjacent in the final sentence.</Paragraph>
    <Paragraph position="4"> The REVISOR system (Callaway and Lester, 1997) takes an entire sentence plan at once and iterates through it in paragraph-sized chunks, employing clause- and phrase-level aggregation and re-ordering operations before passing a revised sentence plan to the surface realizer. However, at no point does it add information that previously did not exist in the sentence plan. The RTPI system (Harvey and Carberry, 1998) takes in sets of multiple, lexicalized sentential plans over a number of medical diagnoses from different critiquing systems and produces a single, unified sentence plan which is both coherent and cohesive.</Paragraph>
    <Paragraph position="5"> Like STREAK,Shaw'sCASPER system (Shaw, 1998) produces single sentences from sets of sentences and doesn't attempt to deal with discourse markers. CASPER also delays lexicalization when aggregating by looking at the lexicon twice during the revision process. This is due mainly to the efficiency costs of the unification procedure. However, CASPER's sentence planner essentially uses the first lexicon lookup to find a &amp;quot;set of lexicalizations&amp;quot; before eventually selecting a particular one.</Paragraph>
    <Paragraph position="6"> An important similarity of these pipelined revision systems is that they all manipulate lexicalized representations at the clause level. Given that both aggregation and reordering operators may separate clauses that were previously adjacent upon leaving the sentence planner, the inclusion of a revision component has important implications for any upstream architectural module which assumed that initially adjacent clauses would remain adjacent throughout the generation process.</Paragraph>
  </Section>
  <Section position="5" start_page="0" end_page="0" type="metho">
    <SectionTitle>
4 Architectural Implications
</SectionTitle>
    <Paragraph position="0"> The current state of the art in NLG can be described as small pipelined generation systems that incorporate some, but not all, of the available pipelined NLG modules. Specifically, there is no system todate which both revises its output and inserts appropriate discourse markers. Additionally, there are no systems which utilize the latest theoretical work in discourse markers described in Section 2. But as NLG systems begin to reach toward multi-page text, combining both modules into a single architecture will quickly become a necessity if such systems are to achieve the quality of prose that is routinely achieved by human authors.</Paragraph>
    <Paragraph position="1"> This integration will not come without constraints. For instance, discourse marker insertion algorithms assume that sentence plans are static objects. Thus any change to the static nature of sentence plans will inevitably disrupt them. On the other hand, revision systems currently do not add information not specified by the discourse planner, and do not perform true lexicalization: any new lexemes not present in the sentence plan are merely delayed lexicon entry lookups. Finally, because revision is potentially destructive, the sentence elements that lead to a particular discourse marker being chosen may be significantly altered or may not even exist in a post-revision sentence plan.</Paragraph>
    <Paragraph position="2"> These factors lead to two partial order constraints on a system that both inserts discourse markers and revises at the clause level after sentence planning: AF Discourse marker lexicalization cannot precede revision AF Revision cannot precede discourse marker lexicalization In the first case, assume that a sentence plan arrives at the revision module with discourse markers already lexicalized. Then the original discourse marker may not be appropraite in the revised sentence plan. For example, consider how the application of the following revision types requires different lexicalizations for the initial discourse markers:</Paragraph>
  </Section>
  <Section position="6" start_page="0" end_page="0" type="metho">
    <SectionTitle>
AF Clause Aggregation:Themergingoftwo
</SectionTitle>
    <Paragraph position="0"> main clauses into one main clause and one subordinate clause: John had always liked to ride motorbikes. A8 On account of this, his wife passionately hated motorbikes. B5 John had always liked to ride motorbikes, which his wife CU* on account of this CY thusCV passionately hated.</Paragraph>
    <Paragraph position="1"> AF Reordering: Two originally adjacent main clauses no longer have the same fixed position relative to each other: Diesel motors are well known for emitting excessive pollutants. A8 Furthermore, diesel is often transported unsafely. A8 However, diesel motors are becoming cleaner. B5 Diesel motors are well known for emitting excessive pollutants, CU*howeverCY althoughCV they are becoming cleaner. Furthermore, diesel is often transported unsafely.</Paragraph>
    <Paragraph position="2"> AF Clause Demotion: Two main clauses are merged where one of them no longer has a clause structure: The happy man went home. A8 However,the man was poor. B5 The happy CU*howeverCY butCV poor man went home.</Paragraph>
    <Paragraph position="3"> These examples show that if discourse marker lexicalization occurs before clause revision, the changes that the revision module makes can render those discourse markers undesirable or even grammatically incorrect. Furthermore, these effects span a wide range of potential revision types.</Paragraph>
    <Paragraph position="4"> In the second case, assume that a sentence plan is passed to the revision component, which performs various revision operations before discourse markers are considered. In order to insert appropriate discourse markers, the insertion algorithm must access the appropriate rhetorical structure produced by the discourse planner. However, there is no guarantee that the revision module has not altered the initial organization imposed by the discourse planner. In such a case, the underlying data used for discourse marker selection may no longer be valid.</Paragraph>
    <Paragraph position="5"> For example, consider the following generically represented discourse plan: C1: &amp;quot;John and his friends went to the party.&amp;quot; CJtemporal &amp;quot;before&amp;quot; relation, time(C1, C2)CL C2: &amp;quot;John and his friends gathered at the mall.&amp;quot; CJcausal relation, cause(C2, C3)CL C3: &amp;quot;John had been grounded.&amp;quot; One possible revision that preserved the discourse plan might be: &amp;quot;Before John and his friends went to the party, they gathered at the mall since he had been grounded.&amp;quot; In this case, the discourse marker algorithm has selected &amp;quot;before&amp;quot; and &amp;quot;since&amp;quot; as lexicalized discourse markers prior to revision. But there are other possible revisions that would destroy the ordering established by the discourse plan and make the selected discourse markers unwieldy: &amp;quot;John, CU*sinceCY AFCV who had been grounded, gathered with his friends at the mall before going to the party.&amp;quot; &amp;quot;CU*SinceCY BecauseCV he had been grounded, John and his friends gathered at the mall and CU*beforeCY thenCV went to the party.&amp;quot; Reordering sentences without updating the discourse relations in the discourse plan itself would result in many wrong or misplaced discourse marker lexicalizations. Given that discourse markers cannot be lexicalized before clause revision is enacted, and that clause revision may alter the original discourse plan upon which a later discourse marker insertion algorithm may rely, it follows that the revision algorithm should update the discourse plan as it progresses, and the discourse marker insertion algorithm should be responsive to these changes, thus delaying discourse marker lexicalization.</Paragraph>
  </Section>
  <Section position="7" start_page="0" end_page="0" type="metho">
    <SectionTitle>
5 Implementation
</SectionTitle>
    <Paragraph position="0"> To demonstrate the application of this problem to real world discourse, we took the STORYBOOK (Callaway and Lester, 2001; Callaway and Lester, 2002) NLG system that generates multi-page text in the form of Little Red Riding Hood stories and New York Times articles, using a pipelined architecture with a large number of modules such as revision (Callaway and Lester, 1997). But although it was capable of inserting discourse markers, it did so in an ad-hoc way, and required that the document author notice possible interferences between revision and discourse marker insertion and hard-wire the document representation accordingly.</Paragraph>
    <Paragraph position="1"> Upon adding a principled discourse marker selection algorithm to the system, we soon noticed various unwanted interactions between revision and discourse markers of the type described in Section 4 above. Thus, in addition to the other constraints already considered during clause aggregation, we altered the revision module to also take into account the information available to our discourse marker insertion algorithm (in our case, intention and rhetorical predicates). We were thus able to incorporate the discourse marker selection algorithm into the revision module itself.</Paragraph>
    <Paragraph position="2"> This is contrary to most NLG systems where discourse marker lexicalization is performed as late as possible using the modified discourse plan leaves after the revision rules have reorganized all the original clauses. In an architecture that doesn't consider discourse markers, a generic revision rule without access to the original discourse plan might appear likethis(wheretype refers to the main clause syntax, and rhetorical type refers to its intention):</Paragraph>
    <Paragraph position="4"> But by making available the intentional and rhetorical information from the discourse plan, our modified revision rules instead have this form:</Paragraph>
    <Paragraph position="6"> where the function lexicalize-discourse-marker determines the appropriate discourse marker lexicalization given a set of features such as those described in (Knott and Mellish, 1996) or (Grote and Stede, 1999), and update-rhetorical-relation causes the appropriate changes to be made to the running discourse plan so that future revision rules can take those alterations into account.</Paragraph>
    <Paragraph position="7"> STORYBOOK takes a discourse plan augmented with appropriate low-level (i.e., unlexicalized, or conceptual) rhetorical features and produces a sentence plan without discarding rhetorical information. It then revises and lexicalizes discourse markers concurrently before passing the results to the surface realization module for production of the surface text.</Paragraph>
    <Paragraph position="8"> Consider the following sentences in a short text plan produced by the generation system:  1. &amp;quot;In this case, Mr. Curtis could no longer be tried for the shooting of his former girlfriend's companion.&amp;quot; BOagent-actionBQ CJcausal relationCL 2. &amp;quot;There is a five-year statute of limitations on that crime.&amp;quot; BOexistentialBQ CJopposition relationCL 3. &amp;quot;There is no statute of limitations in murder cases.&amp;quot; BOexistentialBQ  Without revision, a discourse marker insertion algorithm is only capable of adding discourse markers before or after a clause boundary: &amp;quot;In this case, Mr. Curtis could no longer be tried for the shooting of his former girlfriend's companion. This is because there is a five-year statute of limitations on that crime. However, there is no statute of limitations in murder cases.&amp;quot; But a revised version with access to the discourse plan and integrating discourse markers that our system generates is: &amp;quot;In this case, Mr. Curtis could no longer be tried for the shooting of his former girlfriend's companion, because there is a five-year statute of limitations on that crime even though there is no statue of limitations in murder cases.&amp;quot; A revision module without access to the discourse plan and a method for lexicalizing discourse markers will be unable to generate the second, improved version. Furthermore, a discourse marker insertion algorithm that lexicalizes before the revision algorithm begins will not have enough basis to decide and frequently produce wrong lexicalizations. The actual implemented rules in our system (which generate the example above) are consistent with the abstract rule presented earlier.</Paragraph>
    <Paragraph position="9">  update-rhetorical-relation(clause1, clause2, existential, existential, current-relations) Given these parameters, the discourse markers will be lexicalized as because and even though respectively, and the revision component will be able to combine all three base sentences plus the discourse markers into the single sentence shown above.</Paragraph>
  </Section>
class="xml-element"></Paper>