<?xml version="1.0" standalone="yes"?>
<Paper uid="P98-1052">
  <Title>An Empirical Investigation of Proposals in Collaborative Dialogues</Title>
  <Section position="3" start_page="0" end_page="325" type="metho">
    <SectionTitle>
2 Tracking Agreement
</SectionTitle>
    <Paragraph position="0"> Our corpus consists of 24 computer-mediated dialogues 1 in which two participants collaborate on a simple task of buying furniture for the living and dining rooms of a house (a variant of the task in (Walker, 1993)). The participants' main goal is to negotiate purchases; the items of highest priority are a sofa for the living room and a table and four chairs for the dining room. The problem solving task is complicated by several secondary goals: 1) Match colors within a room, 2) Buy as much furniture as you can, 3) Spend all your money. A point system is used to motivate participants to try to achieve as many goals as possible. Each subject has a budget and inventory of furniture that lists the quantities, colors, and prices for each available item. By sharing this initially private information, the participants can combine budgets and select furniture from either's inventory. The problem is collaborative in that all decisions have to be consensual; funds are shared and purchasing decisions are joint.</Paragraph>
    <Paragraph position="1"> In this context, we characterize an agreement as accepting a partner's suggestion to include a specific furniture item in the solution. In this paper we will focus on the issue of recognizing that a suggestion has been made (i.e. a proposal). The problem is not easy, since, as speech act theory points out (Austin, 1962; Searle, 1975), surface form is not a clear indicator of speaker intentions. Consider excerpt (1): 2  (1) A: \[35\]: i have a blue sofa for 300.</Paragraph>
    <Paragraph position="2"> \[36\]: it's my cheapest one.</Paragraph>
    <Paragraph position="3"> B: \[37\]: I have 1 sofa for 350  tion and thus cannot count as a proposal to include it in the solution. The sofa A offers for consideration, is effectively proposed only after the exchange of information in \[37\]--\[39\].</Paragraph>
    <Paragraph position="4"> However, if the dialogue had proceeded as below, \[35'\] would count as a proposal: (2) B: \[32'\]: I have 1 sofa for 350 \[33'\]: that is yellow \[34'\]: which is my cheapest.</Paragraph>
    <Paragraph position="5"> A: \[35'\]: i have a blue sofa for 300.</Paragraph>
    <Paragraph position="6"> Since context changes the interpretation of \[35\], our goal is to adequately characterize the context. For this, we look for guidance from corpus and domain features. Our working hypothesis is that for both participants context is partly determined by the domain reasoning situation. Specifically, if the suitable courses of action are highly limited, this will make an utterance more likely to be treated as a proposal; this correlation is supported by our corpus analysis, as we will discuss in Section 5.</Paragraph>
  </Section>
  <Section position="4" start_page="325" end_page="327" type="metho">
    <SectionTitle>
3 Coding Scheme
</SectionTitle>
    <Paragraph position="0"> We will present our coding scheme by first describing the core DR/ scheme, followed by the adaptations for our corpus and research issues. For details about our scheme, see (Di Eugenio et al., 1997); for details about features we added to DR/, but that are not relevant for this paper, see (Di Eugenio et al., 1998).</Paragraph>
    <Section position="1" start_page="325" end_page="325" type="sub_section">
      <SectionTitle>
3.1 The DRI Coding Scheme
</SectionTitle>
      <Paragraph position="0"> The aspects of the core DR/scheme that apply to our corpus are a subset of the dimensions under Forward- and Backward-Looking Functions.</Paragraph>
      <Paragraph position="1">  This dimension characterizes the potential effect that an utterance Ui has on the subsequent dialogue, and roughly corresponds to the classical notion of an illocutionary act (Austin, 1962; Searle, 1975). As each Ui may simultaneously achieve multiple effects, it can be coded for three different aspects: Statement, Influence-on-Hearer, Influence-on-Speaker. Statement. The primary purpose of Statements is to make claims about the world. Statements are sub-categorized as an Assert when Speaker S is trying to change Hearer H's beliefs, and as a Reassert if the claim has already been made in the dialogue.</Paragraph>
      <Paragraph position="2"> Influence-on-Hearer (I-on-H). A Ui tagged with this dimension influences H's future action. DR/distinguishes between S merely laying out options for H's future action (Open-Option), and S trying to get H to perform a certain action (see Figure 1). Infe-Rdegquest includes all actions that request information, in both explicit and implicit forms. All other actions 3 are Action-Directives.</Paragraph>
      <Paragraph position="3">  .......................</Paragraph>
      <Paragraph position="4"> i' Is S discussing potential actions of H? ',--Is S ~'-th-g-to get H to d ..... thing? : Open-Op.on ....... ;;-/ ....... -%.o..o.</Paragraph>
      <Paragraph position="5"> Is 14 supposed to provide information'? \[ .... 3 ( ^otio..Diroo.vo  this dimension potentially commits S (in varying degrees of strength) to some future course of action. The only distinction is whether the commitment is conditional on H's agreement (Offer) or not (Commit). With an Offer, S indicates willingness to commit to an action if H accepts it. Commits include promises and other weaker forms.</Paragraph>
      <Paragraph position="6">  This dimension indicates whether Ui is unsolicited, or responds to a previous Uj or segment. 4 The tags  of interest for our corpus are: * Answer: Ui answers a question.</Paragraph>
      <Paragraph position="7"> * Agreement: 1. Ui Accept/Rejects if it indicates S's attitude towards a belief or proposal embodied in its antecedent. null 2. Ui Holds if it leaves the decision about the proposal embodied in its antecedent open pending further discussion.</Paragraph>
    </Section>
    <Section position="2" start_page="325" end_page="326" type="sub_section">
      <SectionTitle>
3.2 Refinements to Core Features
</SectionTitle>
      <Paragraph position="0"> The core DRI manual often does not operationalize the tests associated with the different dimensions, such as the two dashed nodes in Figure 1 (the shaded node is an addition that we discuss below). This resulted in strong disagreements regarding Forward Functions (but not Backward Functions) during our initial trials involving three coders.</Paragraph>
      <Paragraph position="1"> Statement, In the current DR/manual, the test for Statement is whether Ui can be followed by &amp;quot;That's not true.&amp;quot;. For our corpus, only syntactic imperatives or interrogatives were consistently filtered out by this purely semantic test. Thus, we refined it by appealing to syntax, semantics, and domain knowledge: Ui is a Statement if it is declarative  and it is 1) past; or 2) non past, and contains a stative verb; or 3) non past, and contains a non-stative verb in which the implied action: * does not require agreement in the domain; * or is supplying agreement.</Paragraph>
      <Paragraph position="2"> For example, We could start in the living room is not tagged as a statement if meant as a suggestion, i.e. if it requires agreement.</Paragraph>
      <Paragraph position="3"> I-on-H and I-on-S. These two dimensions depend on the potential action underlying U~ (see the root node in Figure 1 for I-on-H). The initial disagreements with respect to these functions were due to the coders not being able to consistently identify such actions; thus, we provide a definition for actions in our domain, s and heuristics that correlate types of actions with I-on-H/I-on-S.</Paragraph>
      <Paragraph position="4"> We have two types of potential actions: put furniture item X in room Y and remove furniture item X from room Y. We subcategorize them as specific and general. A specific action has all necessary parameters specified (type, price and color of item, and room). General actions arise because all necessary parameters are not set, as in I have a blue sofa uttered in a null context.</Paragraph>
      <Paragraph position="5"> Heuristic for I-on-H (the shaded node in Figure 1). If H's potential action described by Ui is specific, Ui is tagged as Action-Directive, otherwise as Open-Option.</Paragraph>
      <Paragraph position="6"> Heuristic for I-on-S. Only a Ui that describes S's specific actions is tagged with an 1-on-S tag.</Paragraph>
      <Paragraph position="7"> Finally, it is hard to offer comprehensive guidance for the test is S trying to get H to do something? in Figure 1, but some special cases can be isolated. For instance, when S refers to one action that the participants could undertake, but in the same turn makes it clear the action is not to be performed, then S is not trying to get H to do something. This happens in excerpt (1) in Section 2. A specific action (get B's $350 yellow sofa) underlies \[38\], which qualifies as an Action-Directive just like \[35\]. However, because of \[40\], it is clear that B is not trying to get A to use B's sofa. Thus, \[38\] is tagged as an Open-Option.</Paragraph>
    </Section>
    <Section position="3" start_page="326" end_page="327" type="sub_section">
      <SectionTitle>
3.3 Coding for problem solving features
</SectionTitle>
      <Paragraph position="0"> In order to investigate our working hypothesis about the relationship between context and limits on the courses of action, we coded each utterance for features of the problem space. Since we view the problem space as a set of constraint equations, we decided to code for the variables in these equations and the number of possible solutions given all the possible assignments of values to these variables.</Paragraph>
      <Paragraph position="1"> The variables of interest for our corpus are the objects of type t in the goal to put an object in a room (e.g. varsola, vartabte or varchairs). For a solution to 5Our definition of actions does not apply to Into-Requests, as the latter are easy to recognize.</Paragraph>
      <Paragraph position="2">  exist to the set of constraint equations, each varl in the set of equations must have a solution. For example, if 5 instances of sofas are known for varsola, but every assignment of a value to varsoIa violates the budget constraint, then varsola and the constraint equations are unsolvable.</Paragraph>
      <Paragraph position="3"> We characterize the solution size for the problem as determinate if there is one or more solutions and indeterminate otherwise. It is important to note that the set of possible values for each vari is not known at the outset since this information must be exchanged during the interaction. If S supplies appropriate values for vari but does not know what H has available for it then we say that no solution is possible at this time. It is also important to point out that during a dialogue, the solution size for a set of constraint equations may revert from determinate to indeterminate (e.g. when S asks what else H has available for a vari).</Paragraph>
      <Paragraph position="4"> 4 Analysis of the Coding Results Two coders each coded 482 utterances with the adapted DRI features (44% of our corpus). Table 1 reports values for the Kappa (K) coefficient of agreement (Carletta, 1996) for Forward and Backward Functions .6 The columns in the tables read as follows: if utterance Ui has tag X, do coders agree on the subtag? For example, the possible set of values for I-on-H are: NIL (Ui is not tagged with this dimension), Action-Directive, Open-Option, and Info-Request.</Paragraph>
      <Paragraph position="5"> The last two columns probe the subtypes of Backward Functions: was Ui tagged as an answer to the same antecedent? was Ui tagged as accepting, re.</Paragraph>
      <Paragraph position="6"> jecting, or holding the same antecedent? T K factors out chance agreement between coders; K=0 means agreement is not different from chance, and K=I means perfect agreement. To assess the import of the values 0 &lt;: K &lt; 1 beyond K's statistical significance (all of our K values are significant at p=0.000005), the discourse processing community uses Krippendorf's scale (1980) 8, which diseFor problem solving features, K for two doubly coded dialogues was &gt; .8. Since reliability was good and time was short, we used one coder for the remaining dialogues.</Paragraph>
      <Paragraph position="7"> 7In general, we consider 2 non-identical antecedents as equivalent if one is a subset of the other, e.g. if one is an utterance Uj and the other a segment containing Uj.</Paragraph>
      <Paragraph position="8"> SMore forgiving scales exist but have not yet been discussed by the discourse processing community, e.g. the one in (Rietveld and van Hour, 1993).</Paragraph>
      <Paragraph position="9">  counts any variable with K &lt; .67, and allows tentative conclusions when .67 &lt; K &lt; .8 K, and definite conclusions when K&gt;.8. Using this scale, Table 1 suggests that Forward Functions and Answer can be recognized far more reliably than Agreement.</Paragraph>
      <Paragraph position="10"> To assess the DRI effort, clearly more experiments are needed. However, we believe our results show that the goal of an adaptable core coding scheme is reasonable. We think we achieved good results on Forward Functions because, as the DRI enterprise intended, we adapted the high level definitions to our domain. However, we have not yet done so for Agreement since our initial trial codings did not reveal strong disagreements; now given our K results, refinement is clearly needed. Another possible contributing factor for the low K on Agreement is that these tags are much rarer than the Forward Function tags. The highest possible value for K may be smaller for low frequency tags (Grove et al., 1981).</Paragraph>
      <Paragraph position="11"> Our assessment is supported by comparing our results to those of Core and Allen (1997) who used the unadapted DRI manual -- see Table 2. Overall, our Forward Function results are better than theirs (the non significant K for I-on-S in Table 2 reveals problems with coding for that tag), while the Backward Function results are compatible. Finally, our assessment may only hold for task-oriented collaborative dialogues. One research group tried to use the DRI core scheme on free-flow conversations, and had to radically modify it in order to achieve reliable coding (Stolcke et al., 1998).</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="327" end_page="328" type="metho">
    <SectionTitle>
5 Tracking Propose and Commit
</SectionTitle>
    <Paragraph position="0"> It appears we have reached an impasse; if human coders cannot reliably recognize when two participants achieve agreement, the prospect of automating this process is grim. Note that this calls into question analyses of agreements based on a single coder's tagging effort, e.g. (Walker, 1996). We think we can overcome this impasse by exploiting the reliability of Forward Functions. Intuitively, a U~ tagged as Action-Directive + Offer should correlate with a proposal -- given that all actions in our domain are joint, an Action-Directive tag always co-occurs with either Offer (AD+O) or Commit (AD/C). Further, analyzing the antecedents of Commits should shed light on what was treated as a proposal in the dialogue. Clearly, we cannot just analyze the antecedents of Commit to characterize proposals, as a  proposal may be discarded for an alternative.</Paragraph>
    <Paragraph position="1"> To complete our intuitive characterization of a proposal, we will assume that for a Ui to count as a well-formed proposal (WFP), the context must be such that enough information has already been exchanged for a decision to be made. The feature solution size represents such a context. Thus our first testable characterization of a WFP is:</Paragraph>
    <Section position="1" start_page="327" end_page="328" type="sub_section">
      <SectionTitle>
1.1 Ui counts as a WFP if it is tagged as Action-
</SectionTitle>
      <Paragraph position="0"> Directive + Offer and if the associated solution size is determinate.</Paragraph>
      <Paragraph position="1"> To gain some evidence in support of 1.1, we checked whether the hypothesized WFPs appear as antecedents of Commits? Of the 32 AD/Os in Table 3, 25 have determinate solution size; thus, WFPs are the largest class among the antecedents of Commit, even if they only account for 43% of such antecedents. Another indirect source of evidence for hypothesis 1.1 arises by exploring the following questions: are there any WFPs that are not committed to? if yes, how are they dealt with in the dialogue? If hypothesis 1.1 is correct, then we expect that each such Ui should be responded to in some fashion. In a collaborative setting such as ours, a partner cannot just ignore a WFP as if it had not occurred.</Paragraph>
      <Paragraph position="2"> We found that there are 15 AD+Os with determinate solution size in our data that are not committed to. On closer inspection, it turns out that 9 out of these 15 are actually indirectly committed to.</Paragraph>
      <Paragraph position="3"> Of the remaining 6, four are responded to with a counterproposal (another AD+O with determinate solution size). Thus only two are not responded to in any fashion. Given that these 2 occur in a dialogue where the participants have a distinctively non-collaborative style, it appears hypothesis 1.1 is supported.</Paragraph>
      <Paragraph position="4"> Going back to the antecedents of Commit (Table 3), let's now consider the 7 indeterminate AD/Os. They can be considered as tentative proposals that need to be negotiated. 1deg To further refine our characterization of proposals, we explore the hypothesis: 9Antecedents of Commits are not tagged. We reconstructed them from either variable tags or when Ui has both Commit and Accept tags, the antecedent of the Accept.</Paragraph>
      <Paragraph position="5"> 1degBecanse of our heuristics of tagging specific actions as ActionDirectives, these utterances are not Open-Options.  1.2 When the antecedent of a Commit is an AD+O and indeterminate, the intervening dialogue renders the solution size determinate.</Paragraph>
      <Paragraph position="6"> In 6 out of the 7 indeterminate antecedent AD+Os, our hypothesis is verified (see excerpt (1), where \[35\] is an AD+ 0 with indeterminate solution size, and the antecedent to the Commit in \[40\]).</Paragraph>
      <Paragraph position="7"> As for the other antecedents of Commit in Table 3, it is not surprising that only 4 Open-Options occur given the circumstances in which this tag is used (see Figure 1). These Open-Options appear to function as tentative proposals like indeterminate AD+ Os, as the dialogue between the Open-Option and the Commit develops according to hypothesis 1.2. We were instead surprised that AD+Cs are a very common category among the antecedents of Commit (20%); the second commit appears to simply reconfirm the commitment expressed by the first (Walker, 1993; Walker, 1996), and does not appear to count as a proposal. Finally, the Other column is a collection of miscellaneous antecedents, such as Info-Requests and cases where the antecedent is unclear, that need further analysis. For further details, see (Di Eugenio et al., 1998).</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>