<?xml version="1.0" standalone="yes"?>
<Paper uid="H01-1016">
  <Title>Development of the HRL Route Navigation Dialogue System</Title>
  <Section position="6" start_page="1" end_page="1" type="evalu">
    <SectionTitle>
4. EVALUATION
</SectionTitle>
    <Paragraph position="0"> We are have implemented an initial system and are conducting ongoing evaluations and iterative enhancements as part of a second phase of effort. We are in advanced development on the second phase. We report here on some results of the first phase of our project.</Paragraph>
    <Paragraph position="1"> Evaluation of a route guidance system is difficult because the majority of time is spent driving with only a periodic need for instructions. Therefore, for the purposes of developing the language and dialogue models we tried to expedite data collection by having dialogues in which the user simulated a trip by means of a more or less continuous conversation with the system. The position of the vehicle along the route was determined by the user providing odometer readings relative to the start of the trip. At each point the user would query the system and input a new odometer location along the route and continue the dialogue.</Paragraph>
    <Paragraph position="2"> While certainly not as meaningful as queries under normal driving condition, we did obtain good data for our recognizer language model and grammar coverage. In addition we could debug and tune our turn manager functions to make sure we were properly accessing the database and providing correct responses.</Paragraph>
    <Paragraph position="3"> Each query essentially represents a single task and the most meaningful metric for this type of system seems to be the number of dialogue turns per correct response. By correct response we mean that the system provides the final answer versus providing a request to repeat or disambiguate the user query. We have accumulated several thousand utterances during dialogues that run around fifteen to twenty turns per session for a simple route like the one in Figure 2. About a third of the utterances are used to set the vehicle position via inputting odometer data.</Paragraph>
    <Paragraph position="4"> We can also divide the dialogues into task oriented dialogues, where the user is trying to get helpful answers, and dialogues where the user is exploring the limits of the system. We find with the task oriented dialogues that the number of dialogue turns are about 15-20% greater that the number of correct responses and that the inital implementations even without street name recognition is a useful system.</Paragraph>
  </Section>
class="xml-element"></Paper>