<?xml version="1.0" standalone="yes"?>
<Paper uid="H05-1029">
  <Title>Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 225-232, Vancouver, October 2005. c(c)2005 Association for Computational Linguistics Error Handling in the RavenClaw Dialog Management Framework</Title>
  <Section position="6" start_page="230" end_page="231" type="evalu">
    <SectionTitle>
4 Deployment and Current Research
</SectionTitle>
    <Paragraph position="0"> While a quantitative evaluation of design characteristics such as task-independence, scalability, and ease-of-use is hard to perform, a first-order empirical evaluation of the proposed error handling architecture can be accomplished by using it in different systems and monitoring the system authoring process and the system's operation.</Paragraph>
    <Paragraph position="1"> To date, the architecture has been successfully deployed in three different spoken dialog systems.</Paragraph>
    <Paragraph position="2"> A first system, RoomLine (2003), is a phone-based mixed-initiative system that assists users in making conference room reservations on campus. A second system, the Let's Go! Bus Information System (Raux et al, 2005), provides information about bus routes and schedules in the greater Pittsburgh area (the system is available to the larger public). Finally, Vera is a phone-based taskable agent that can be instructed to deliver messages to a third party, make wake-up calls, etc. Vera actually consists of two dialog systems, one which handles incoming requests (Vera In) and one which performs message delivery (Vera Out). In each of these systems, the authoring effort with respect to error handling consisted of: (1) specifying which models and policies should be used for the concepts and request-agents in the dialog task tree, and (2) writing the language generation prompts for explicit and implicit confirmations for each concept.</Paragraph>
    <Paragraph position="3"> Even though the first two systems operate in similar domains (information access), they have very different user populations: students and faculty on campus in the first case versus the entire Pittsburgh community in the second case. As a result, the two systems were configured with different error handling strategies and policies (see Table 1). RoomLine uses explicit and implicit confirmations with an optimistic policy to handle potential misunderstandings. In contrast, the Let's Go Public Bus Information System always uses explicit confirmations, in an effort to increase robustness (at the expense of potentially longer dialogs). For non-understandings, RoomLine uses the full set of non-understanding recovery strategies presented in section 3.1. The Let's Go Bus Information system uses the YouCanSay and FullHelp strategies. Additionally a new GoToAQuieterPlace strategy was developed for this system (and is now available for use into any other RavenClaw-based system). This last strategy asks the user to move to a quieter place, and was prompted by the observation that a large number of users were calling the system from noisy environments.</Paragraph>
    <Paragraph position="4"> While the first two systems were developed by authors who had good knowledge of the RavenClaw dialog management framework, the third system, Vera, was developed as part of a class project, by a team of six students who had no prior experience with RavenClaw. Modulo an initial lack of documentation, no major problems were encountered in configuring the system for automatic error handling. Overall, the proposed error handling architecture adapted easily and provided the desired functionality in each of these domains: while new strategies and recovery policies were developed for some of the systems, no structural changes were required in the error handling architecture.</Paragraph>
  </Section>
class="xml-element"></Paper>