<?xml version="1.0" standalone="yes"?>
<Paper uid="W90-0119">
  <Title>Resolving Plan Ambiguity for Response Generation</Title>
  <Section position="1" start_page="0" end_page="147" type="abstr">
    <SectionTitle>
Abstract
</SectionTitle>
    <Paragraph position="0"> Recognizing the plan underlying a query aids in the generation of an appropriate response. In this paper, we address the problem of how to generate cooperative responses when the user's plan is ambiguous. We show that it is not always necessary to resolve the ambiguity, and provide a procedure that estimates whether the ambiguity matters to the task of formulating a response. If the ambiguity does matter, we propose to resolve the ambiguity by entering into a clarification dialogue with the user and provide a procedure that performs this task. Together, these procedures allow a question-answering system to take advantage of the interactive and collaborative nature of dialogue in recognizing plans and resolving ambiguity. null Introduction Somewhat obviously, plan recognition is the process of inferring an agent's plan from observation of the agent's actions. The agent's actions can be physical actions or speech actions. Four principal methods for plan recognition have been proposed in the literature. The methods are plausible inference (Allen \[1\], Carberry \[2\], Litman \[15\], Sidner \[25\]), parsing (Huff and Lesser \[9\]), circumscribing a hierarchical representation of plans and using deduction (Kautz \[12, 13\]), and abduction (Charniak and McDermott \[6\], Konolige and Pollack \[14\], Poole \[24\]). Our particular interest is in the use of plan recognition in question-answering systems, where recognizing the plan underlying a user's queries aids in the generation of an appropriate response. Here, the plans and goals of the user, once recognized, have been used to: supply more information than is explicitly requested (Allen \[1\], Luria \[16\]), handle pragmatically ill-formed queries and resolve some inter-sentential ellipses (Carberry \[2, 3, 4\]), provide an explanation from the appropriate perspective (McKeown et hi. \[17\]), respond to queries that result from an invalid plan (Pollack \[20, 21, 22\]), and avoid misleading responses and produce user-specific cooperative responses (Joshi et a\]. \[10, 11\], van Beck and Cohen \[26, 27\], Cohen et al. \[7\]).</Paragraph>
    <Paragraph position="1"> Example 1 (Joshi et al. \[11\]). As an example of a cooperative response consider the following exchange between student and student-advisor system. The plan of the student is to avoid failing the course by dropping it.</Paragraph>
    <Paragraph position="2"> User: Can I drop numerical analysis? System: Yes, however you will still fail the course since your mark will be recorded as withdrawal while failing.</Paragraph>
    <Paragraph position="3"> If the system just gives the direct answer, &amp;quot;Yes&amp;quot; the student will remain unaware that the plan is faulty.</Paragraph>
    <Paragraph position="4"> The more cooperative answer warns the student.</Paragraph>
    <Paragraph position="5"> An important weakness of this work in response generation, however, is the reliance on a plan recognition component being able to uniquely determine the plan of the user. This is clearly too strong an assumption as the user's actions often will be consistent with more than one plan, especially after only one or a few utterances when there is insufficient context to help decide the plan of the user. In Example 1 there are many reasons why a student may want to drop a course, such as resolving a scheduling conflict, avoiding failing the course, or finding the material uninteresting. There may be no reason to prefer one alternative over the other, yet we may still want to generate a response that does more than just give a direct answer to the user's query.</Paragraph>
    <Paragraph position="6"> In this paper, we address the problem of what the system should do when the user's actions are ambiguous as they are consistent with more than one plan. To the extent that this problem has been considered by researchers in plan recognition, it is generally assumed that the plan recognition system overcomes the ambiguity problem by inferring the most likely interpretation, given its assessment of the context and dialogue so far and knowledge of typical plans of action. Thus there is a dependence on salience heuristics to solve the ambiguity problem \[e.g. 1, 2, 17, and see the final section\]. Existing proposals for resolving ambiguity beyond heuristics are underspecified and what usually underlies these proposals is the assumption that we always want to determine one unique plan \[2, 15, 19, 25\].</Paragraph>
    <Paragraph position="7"> We show how to relax the assumption that the plan recognition component returns a single plan.</Paragraph>
    <Paragraph position="8"> That is, given that the result of the plan recognition phase will usually be a disjunction of possible plans, we show how to design a response component to generate cooperative responses given the disjunction.</Paragraph>
    <Paragraph position="9"> We show that it is not always necessary to resolve ambiguity, and provide a procedure that allows the  response component to estimate whether the ambiguity matters to the task of formulating a response. If the ambiguity does not matter, the response component can continue to answer the user's queries and ignore the ambiguity in the underlying goals and plan until further queries help clarify which plan the user is pursuing. If the ambiguity does matter, the system should take advantage of the interactive and collaborative nature of dialogue in recognizing plans and resolving ambiguity. A key contribution of this work therefore is providing a clear criterion for when to respond to a question with a question that will differentiate between some of the possibilities.</Paragraph>
    <Paragraph position="10"> We also propose a specific solution to what questions should then be asked of the user. Moreover, these questions are asked only to resolve the ambiguity to the point where it no longer matters (this is not necessarily to a unique plan).</Paragraph>
    <Paragraph position="11"> Example 2. Here are two different examples to give a flavor of what we are proposing. There are two agents: a cook and an expert who is cooperative, helpful, and adept at recognizing plans. The expert observes the cook making marinara sauce and recognizes the cook could be pursuing three possible plans, all with the same top level goal of preparing a meal: make fettucini marinara or spaghetti marinara (both a pasta dish) or chicken marinara (a meat dish).</Paragraph>
    <Paragraph position="12"> a. Suppose the cook then asks the expert: &amp;quot;Is a red wine a good choice?&amp;quot; The expert has the criteria for wine selection that red wine should be served if the meal is chicken, fettucini marinara, or spaghetti marinara and white if fettucini alfredo. There is enough information for the expert to decide that red wine should be bought and the ambiguity does not need to be resolved to cooperatively answer the question.</Paragraph>
    <Paragraph position="13"> b. Now suppose the expert also knows that the guest of honor is allergic to gluten and so would not be able to eat if a pasta dish was served. Here the ambiguity is important as the expert has recognized that the cook's prepare-a-meal goal may conflict with the cook's entertain-important-guest goal. The expert will want to resolve the ambiguity enough to he assured that the proposed meal does not include a pasta dish and so clarifies this with the cook.</Paragraph>
    <Section position="1" start_page="144" end_page="144" type="sub_section">
      <SectionTitle>
Estimating Whether the
Ambiguity Matters
</SectionTitle>
      <Paragraph position="0"> Example 2, above, showed that sometimes it is necessary to resolve ambiguity and sometimes it is not. Here we give criteria for judging which is the case. The result is a procedure that allows the response component to estimate whether the ambiguity matters to the task of formulating a response. Assuming we can answer the user's query, deciding when we want to give more than just a direct answer to the user's query depends on the plan of the user. Previous work has identified several kinds of faults in a plan that a cooperative response should warn a user about. We generalize this work in identifying faults and call it plan critiquing. Our proposal therefore is to first apply a plan critiquing procedure to determine possible faults.</Paragraph>
    </Section>
    <Section position="2" start_page="144" end_page="147" type="sub_section">
      <SectionTitle>
Plan Critiquing
</SectionTitle>
      <Paragraph position="0"> A plan may be labeled as faulty if it will fail to achieve its high level goal (e.g. Allen \[1\]) or if there are simply better alternatives for reaching that goal (e.g. Pollack \[20\]). Joshi et al. \[10, 11\] formalize the above and identify additional cases (such as warning the user that a certain plan is the only way to achieve a goal).</Paragraph>
      <Paragraph position="1"> In \[26, 27\], we make some extensions to Joshi et al. \[10, 11\] and give a procedure to determine faults in plans and to address these faults through cooperative responses. Faults include both plans which fail and plans which can be replaced by better alternatives. In \[26, 27\], we also include the critical extension of domain goals. To explain, the system needs to not only respond to goals inferred from the current discourse but also to the domain goals a user is likely to have or known to have even though they are not stated in, or derivable from, the discourse.</Paragraph>
      <Paragraph position="2"> Example 3.</Paragraph>
      <Paragraph position="3"> User: I'm not interested in the material and so would like to drop the course. Can I? The ideal response should say more than just &amp;quot;Yes&amp;quot;, but also warn the student that the domain goal of achieving a degree conflicts with the immediate goal of dropping the course. This example shows how competing goals may exist and need to be addressed in a response.</Paragraph>
      <Paragraph position="4"> To determine whether the ambiguity matters, we propose to apply an extension of the procedure of \[26, 27\], which will be sensitive to complementary goals as well. The standard assumption in cooperative response work is that the user is pursuing only a single chain of goals; we allow actions to be used to achieve more than one complementary goal. For example, in the course-advising domain we can assume that all users have the goal to avoid failing a course. If a user then asks &amp;quot;I'm not interested in the course and so would like to drop it. Can I?&amp;quot; the response should address not just the goal of avoiding uninteresting courses but also the complementary goal of avoiding failing courses. For example, the response should warn the user if he will fail the course because of withdrawal while failing (as in Example I).</Paragraph>
      <Paragraph position="5"> The algorithm to estimate whether the ambiguity matters is below. The deciding criterion is whether the critiques for all plans are the same. By same critique or same fault (Case 1. b. in the algorithm) we mean, for example, same better way or same conflict with competing domain goal.</Paragraph>
      <Paragraph position="6">  Input: A set of possible plans (the output from a plan recognition algorithm).</Paragraph>
      <Paragraph position="7"> Output: The set of possible plans with critiques attached to each plan.</Paragraph>
      <Paragraph position="8"> Algorithm Critique: begin for each plan in the set of possible plans do critique the plan and, if there is a fault, annotate the plan with the fault Input: A set of critiqued possible plans.</Paragraph>
      <Paragraph position="9"> Output: &amp;quot;Yes&amp;quot;, the ambiguity matters, or &amp;quot;No&amp;quot; the ambiguity does not matter.</Paragraph>
      <Paragraph position="10"> Algorithm Ambiguity_matters: We are in one of the following two cases: Case 1. &amp;quot;No&amp;quot;, the ambiguity does not matter. The critiques are the same for all the plans. That is, either a. every plan is faultless, or b. every plan is annotated with the same fault.</Paragraph>
      <Paragraph position="11"> Case 2. &amp;quot;Yes&amp;quot;, the ambiguity does matter. The critiques are different for some or all of the plans. That is, either a. some, but not all, of the plans are faultless (the faults may or may not be the same), or b. every plan is annotated with a fault and the faults are not all the same.</Paragraph>
      <Paragraph position="12"> end An Example to Illustrate the Proposal Suppose the user asks &amp;quot;Can I drop numerical analysis?&amp;quot;. First, a plan recognition algorithm is called to determine the possible plans of the user. They are found to be: end ~ resolve schedule conflict --~ drop course end ~ avoid uninteresting prof --~ drop course Second, algorithm Critique is called to critique the plans. As a result, both plans are labeled with the same fault that there is a better plan for achieving the goal. Third, Mgorithm Ambiguity_matter8 is called to determine whether the ambiguity regarding the plan of the user matters to the task of formulating a response. It is found that the ambiguity does not matter as both plans are annotated with the same fault (Case 1.b of the algorithm). Finally, the critiqued plans are passed to a response generation procedure. The answer then given in this example is, &amp;quot;Yes, but a better way is to switch to another section.&amp;quot; In general, in (Case 1.a) a response generation procedure can just give a direct answer to the user's query, and in (Case 1.b) can give a direct answer plus any warranted additional information, such as telling the user about the fault.</Paragraph>
      <Paragraph position="13"> In the above example it was found that the ambiguity did not matter as there was enough information to generate a cooperative response. If instead it was found that the ambiguity did matter (Case 2 of the algorithm) we propose that we we enter into a clarification dialogue with the user to resolve the ambiguity to the point where it no longer does matter. That is, until we are in Case 1. A response generation procedure would then be called.</Paragraph>
      <Paragraph position="14"> Clarification Dialogues: Questions in Response to Questions What should we ask the user when a clarification is necessary? Clearly, we do not want to simply list the set of possible plans and ask which is being pursued. Below is an algorithm that determines what to say. Our algorithm for estimating whether the ambiguity matters is not dependent on the method of plan recognition used. Now our proposal for clarification dialogues is tied to a hierarchical plan library in the style of Kautz \[12\]. The input to the algorithm is a set of possible plans where the user's action is related to top-level or end goals through chMns of goals. Each plan in the set is annotated with a a critique. The key idea is to ask about the highest level possible, check whether the ambiguity still needs to be further resolved, and if so, ask at the next level down, iteratively, through the hierarchy of goals.</Paragraph>
      <Paragraph position="15"> Input: A set of critiqued possible plans (the output from Mgorithm Critique).</Paragraph>
      <Paragraph position="16"> Output: The pruned set of possible plans such that the ambiguity no longer matters.</Paragraph>
      <Paragraph position="17"> Algorithm Clarify: begin initiMize the current level to be the first branch point from the top in the set of possible plans while Ambiguity_matter8 = &amp;quot;Yes&amp;quot; do separate out the distinct goals in the set of remaining possible plans that are one level below the current level and are annotated with a fault list the goals (perhaps with their accompanying annotations as justification for why we are asking) and ask the user whether one of them is part of the plan being pursued according to the answer, remove the plans that are not being pursued from the set of possible plans and update the current level in the hierarchy that is being looked at to be the next  branch point end while end  next query ..... ~ ........ -I f&amp;quot; .... 1 I I I 'l' H resdeglve H4&amp;quot; generate ' i 1. generate 2. resolve i i candidate ambiguity w. ' I I ambiguity response I &amp;quot;i---'i with user i plans heuristics i  Example 2b (Revisited). In our story in the introduction about cooking for our allergic guest the expert has recognized the following three plans:  1. end ~ prepare meal ~ make meat dish ~ make chicken marinara ~ make marinara 2. end ~ prepare meal ~ make pasta dish ~ make fettucini marinara ~ make marinara 3. end ~ prepare meal ~ make pasta dish ~ make  spaghetti marinara ~ make marinara Using the algorithm of the previous section, the three plans are critiqued and it is found that the ambiguity matters. The plan involving a meat dish is found to be faultless but the two plans involving a pasta dish are found to conflict with another goal of the cook: to entertain the guest. Using the algorithm of this section, the question asked to resolve the ambiguity would be &amp;quot;Are you making a pasta dish (perhaps with justification of why we are asking)?&amp;quot; After either answer of yes or no we know enough that the ambiguity no longer matters. Note that if we just ask the more general &amp;quot;What are you making?&amp;quot; this allows such uninformative responses as &amp;quot;dinner&amp;quot; or just &amp;quot;you'll see&amp;quot;. When asking a question we propose to ask about as high a level of goal as possible that still helps to resolve the ambiguity and to work top down. A top down approach is better as it provides a useful context for any later queries and makes users give us an answer at a higher level than they may otherwise do. Moreover, the user may be mistaken about decompositions or have some other wrong view of the domain and by stepping downward through the paths of possible plans these misconceptions may be detected. Here is an example. Suppose the user asks &amp;quot;Can I take numerical analysis?&amp;quot;. The system recognizes  two plans.</Paragraph>
      <Paragraph position="18"> end ~ get_degree ~ B.A. ~ electives ~ course end ~ get_degree ~ B.Sc. ~ required ~ course a. System: Are you pursuing a B.Sc.? b. System: Are you trying to fulfill your elected or required courses? Question (a) is what our algorithm would ask. Ques null tion (b) is what a procedure which uses a bottom up approach would ask. But (b) allows potential confusion to go undetected. The user could answer &amp;quot;required&amp;quot;, believing that the course is required for a B.A., for example. Thus, we advocate Kautz style plan recognition \[12\], as other plan recognition methods \[2, 15, 25\] would stop after branch points points and thus could only propose electives and required as the two possible plans. Question (b) is the only question this previous work could ask the user.</Paragraph>
      <Paragraph position="19"> Starting with the top most goals and working down may sometimes give as many questions as bottom up approaches. However, others have noted that bottom up dialogues are difficult to assimilate without misinterpretation \[2, p. 54\]. Therefore, we maintain that the top down approach is more desirable. Moreover, some higher level questions, such as question (a), above, can be eliminated using goals known from the previous discourse or background knowledge about the user.</Paragraph>
      <Paragraph position="20"> Current extensions we are examining include allowing the user to have multiple goals so that more than one path from top level goals to action may be correct. This requires resolving ambiguity in multiple paths through the set of possible plans. This could be done in a depth-first manner, using clue or redirection words to guide the user when we return to resolve the ambiguity in the other branches. We are also investigating selecting the minimal subset of the possible plans from those with faults and those without faults (at the moment the algorithm always takes the subset with faults).</Paragraph>
      <Paragraph position="21">  In this section we summarize our proposals and defend our position that this straightforward way of doing things is a good way. With reference to Fig. 1, we discuss the design of boxes 2, 3, and 4 and the tradeoffs involved between boxes 2 and 3.</Paragraph>
      <Paragraph position="22"> Box 2: Resolve the ambiguity with heuristics. As mentioned earlier, many researchers have proposed heuristics to prefer one plan over another \[1, 2, 17, 8, 18, 12, 23, 14\]. Some of these heuristics can be incompatible with cooperative response  generation. For example, Allen's \[1\] preference heuristics are generally incompatible with recognizing and responding to faulty plans (such as the response in Example 1). Because we are using plan recognition for response generation, this should affect the design of Box 2 and therefore what gets passed to Box 3.</Paragraph>
      <Paragraph position="23"> Box 3: Resolve the ambiguity with the user.</Paragraph>
      <Paragraph position="24"> Previous work in response generation makes the assumption that what gets passed to the RG component is a single plan the PR 'component proposes the user is pursuing. We argue that, unless we are willing to sometimes arbitrarily commit to one plan instead of another, there will be times when one plan cannot be chosen over another and therefore there will be ambiguity about which plan the user is pursuing. Result: we need a method to resolve the ambiguity. In plan recognition in a discourse setting (as opposed to key-hole recognition), the goals and plan the user holds are knowable simply by asking the user. But we do not want to always just ask if it is not necessary so we need to know when to start a clarification dialogue and what to say. And when we do ask, we want to ask the minimal number of questions necessary to resolve the ambiguity until it no longer matters. To this end, box 3 contains a procedure that estimates by plan critiquing whether the ambiguity matters to the task of formulating a response. If the ambiguity does not matter the result is passed to box 4. If the ambiguity does matter, a procedure is called that starts a clarification dialogue, responding to the user's question with questions that iteratively differentiate between the possibilities.</Paragraph>
      <Paragraph position="25"> Box 2 vs. Box 3: The tradeoffs. Much previous work in plan recognition makes the assumption that we want the PR component to commit to and return a single plan. Carberry and McKeown, for example, use a strong heuristic to commit to a single plan \[2, 17\]. However, this means the system will at times commit to the wrong plan. Doing it this way requires the ability to handle natural language debugging dialogues. Why we do not want to commit to a single plan and then, if we are wrong, repair using a debugging dialogue? Carberry \[5, p.4\] argues that a system will appear &amp;quot;unintelligent, obtuse, and uncooperative&amp;quot; if it engages in lengthy clarification dialogues. However, a procedure to perform a debugging dialogue is not specified and is, we speculate, a difficult problem. We argue for not committing early. Our hypothesis is that a clarification dialogue is better than a debugging dialogue. The questions in the clarification dialogues are simple to answer, whereas determining that the system has misunderstood your goals and plan requires users to engage in plan recognition. That is, users must recognize the plan the RG component is using from its responses and note that it differs from their plans. Moreover, the user may not recognize we are wrong and be mislead. Finally, we argue that, if the questions are carefully chosen, the clarification dialogues need not be lengthy or too frequent. Note that preference heuristics can still be used in our approach. These would best be applied when too many top level goals give an unwieldy clarification question.</Paragraph>
      <Paragraph position="26"> Box 4: Generate the response. Once Box 3 has estimated that any remaining ambiguity does not matter to generating a cooperative response, the disjunction of possible plans is passed to Box 4. There are two cases; both can now be handled as in previous work except that there is now the additional complication that we allow one action to contribute to more than one chain of goals. The response component must then generate a conjunctive response that addresses each of the goals.</Paragraph>
      <Paragraph position="27"> 1.Every plan is faultless, so we just give a direct answer to the user's query but ignore the underlying goals and plan until further queries help clarify which plan the user is pursuing, and 2. Every plan has the same fault, so we give a direct answer plus some additional information that warns the user about the deficiency and perhaps suggests some alternatives (see \[10\], \[26\]).</Paragraph>
      <Paragraph position="28"> Soap Box: This paper offers an important contribution to natural language generation. It discusses a clear criterion for when to initiate a clarification dialogue and proposes a specific solution to what questions should be asked of the user to achieve clarification. We believe that natural language response generation systems should be designed to involve the user more directly and this is sometimes achieved quite successfully with our proposals. null There may be tradeoffs between overcommitting in the plan recognition process and engaging in lengthy clarification dialogue, particularly with a large set of complex candidate plans. This may suggest applying pruning heuristics more actively in the plan recognition process (Box 2) to reduce the number of questions asked in the clarification dialogue (Box 3). For future work, these tradeoffs will be examined more closely as we test the algorithms more extensively.</Paragraph>
      <Paragraph position="29"> Acknowledgements. We thank Fei Song and Bruce Spencer for comments on an earlier version of this paper and for many discussions about plan recognition. Financial assistance was received from the Natural Sciences and Engineering Research Council of Canada (NSERC).</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>