<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-3003">
  <Title>Interactive Machine Learning Techniques for Improving SLU Models</Title>
  <Section position="2" start_page="0" end_page="0" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> The use of spoken dialogue systems to automate services in call centers is continually expanding. In one such system, unconstrained speech recognition is used in a limited domain to direct call traffic in customer call centers (Gorin et al, 1997). The challenge in this environment is not only the accuracy of the speech recognition but more importantly, the knowledge and understanding of how the customer request is mapped to the business requirement.</Paragraph>
    <Paragraph position="1"> The first step of the process is to collect utterances from customers, which are transcribed. This gives us a baseline for the types of requests (namely, the user intents) that customers make when they call a client. A UE expert working with the business customer uses either a spreadsheet or a text document to classify these calls into call types. For example, * &amp;quot;I want a refund&amp;quot; REFUND * &amp;quot;May I speak with an operator&amp;quot; GET_CUSTOMER_REP The end result of this process is a document, the annotation guide, that describes the types of calls that may be received and how to classify them. This guide is then given to a group of &amp;quot;labelers&amp;quot; who are trained and given thousands of utterances to label. The utterances and labels are then used to create the SLU model for the application. The call flow which maps the call types to routing destinations (dialog trajectory) is finalized and the development of the dialogue application begins.</Paragraph>
    <Paragraph position="2"> After the field tests, the results are given to the UE expert, who then will refine the call types, create a new annotation guide, retrain the labelers, redo the labels and create new ones from new data and rebuild the SLU model.</Paragraph>
    <Paragraph position="3"> Previously, this knowledge was only captured in a document and was not formalized until the SLU model was generated. Our goal in creating our system is not only to give the UE expert tools to classify the calls, but to capture and formalize the knowledge that is gained in the process and to pass it on to the labelers. We can thus automatically generate training instances and testing scenarios for the labelers, thereby creating more consistent results. Additionally, we can use the SLU model generated by our system to &amp;quot;pre-label&amp;quot; the utterances. The labelers can then view these &amp;quot;pre-labeled&amp;quot; utterances and either agree or disagree with the generated labels. This should speed up the overall labeling process.</Paragraph>
    <Paragraph position="4"> More importantly, this knowledge capture will enable the UE expert to generate and test a SLU model as part of the process of creating the call types for the speech data. The feedback from this initial SLU test allows the UE expert to refine the call types and to improve them without having to train a group of labelers and to run a live test with customers. This results in an improved SLU model and makes it easier to find problems before deployment, thus saving time and money.</Paragraph>
    <Paragraph position="5">  At the same time, the process is more efficient due to the increased uniformity in the way different UE experts classify calls into call type labels.</Paragraph>
    <Paragraph position="6"> We will describe Annomate, an interactive system for speech data mining. In this system, we employ several machine learning techniques such as clustering and relevance feedback in concert with standard text searching methods. We focus on interactive dynamic techniques and visualization of the data in the context of the application.</Paragraph>
    <Paragraph position="7"> The paper is organized as follows. The overview of the system is presented in Section 2. Section 3 briefly discusses the different components of the system. Some results are given in Section 4. Finally, in Sections 5 and 6, we give conclusions and point to some future directions. null</Paragraph>
  </Section>
class="xml-element"></Paper>