<?xml version="1.0" standalone="yes"?>
<Paper uid="W00-0312">
  <Title>Building a Robust Dialogue System with Limited Data *</Title>
  <Section position="7" start_page="62" end_page="64" type="concl">
    <SectionTitle>
6 Discussion and Conclusions
</SectionTitle>
    <Paragraph position="0"> CommandTalk is an example of a successful and robust dialogue system in a domain with limited ac- null cess to both data and subjects. The pre-dialogue version of CommandTalk was used in the STOW (Synthetic Theater of War) '97 ACTD (Advanced Concept Technology Demonstration) exercise, an intensive 48-hour continuous military simulation by all four U.S. military services, and received high praise. The dialogue portion of the system has increased CommandTalk's usefulness and robustness. Nevertheless, several questions remain, not the least of which is whether the robustness techniques used for CommandTalk can be successfully transferred to other domains.</Paragraph>
    <Paragraph position="1"> We have no doubt that our methods for adding robustness at the dialogue level can and should be implemented in other domains, but this is not as clear for our parsing a-nd recognition robustness methods. The one-grammar approach is key to our eliminating the necessity for robust parsing, renders a large corpus for generating a recognition model unnecessary, and has other advantages as well. Yet our experience in the ATIS domain suggests that further research into this approach is needed. Our ATIS grammar is based on a grammar of general English and has a very different structure from that of CommandTalk's semantic grammar, but we were unable to isolate the factor or factors responsible for its poor recognition performance. Recent research (Rayner et al., 2000) suggests that it may be possible to compile a useful recognition model from a general English unification grammar if the grammar is constructed carefully and a few compromises are made. We also believe that using an appropriate grammar approximation algorithm to reduce the complexity of the recognition model may prove fruitful. This would reintroduce some discrepancy between the recognition and understanding language models, but maintain the other advantages of the one-grammar approach.</Paragraph>
    <Paragraph position="2"> In either case, the effectiveness of our recognition robustness techniques remains an open question. We know they have no significant negative impact on in-grammar recognition, but whether they are helpful in recognizing and~ more importantly, interpreting out-of-grammar utterances is unknown. We have been unable to evaluate them so far in the CommandTalk or any other domain, although we hope to do so in the future.</Paragraph>
    <Paragraph position="3"> Another possible solution to the problem of producing a workable robust recognition grammar would return to a statistical approach rather than using word insertions and deletions. Stolcke and Segal (1994) describe a method for combining a context-free grammar with an n-gram model generated from a small corpus of a few hundred utterances to create a more accurate n-gram model. This method would provide a robust recognition model based on the context-free grammar compiled from  our unification grammar. We would'still have to write only one grammar for the system, it would still influence the recognition model, and we could still be sure that the system would never say anything it couldn't recognize. This approach Would require using robust parsing methods, but might be the best solution for other domains if compiling a practical recognition grammar proves too difficult.</Paragraph>
    <Paragraph position="4"> Despite the success of the CommandTalk system, it is clear that more investigation is called for to determine how best to develop dialogue systems in domains with limited data. Researchers must determine which types of unification grammars can be compiled into practical recognition grammars using existing technology, whether grammar approximations or other techniques can produce good results for a broader range of grammars, whether allowing word insertions and deletions is an effective robustness technique, orwhether we should use other methods altogether.</Paragraph>
  </Section>
class="xml-element"></Paper>