<?xml version="1.0" standalone="yes"?>
<Paper uid="N04-4019">
  <Title>Speech Graffiti vs. Natural Language: Assessing the User Experience</Title>
  <Section position="6" start_page="7" end_page="7" type="concl">
    <SectionTitle>
4 Discussion
</SectionTitle>
    <Paragraph position="0"> Overall, we found that Speech Graffiti performed favorably compared to the natural language interface.</Paragraph>
    <Paragraph position="1"> Speech Graffiti generated significantly higher user satisfaction scores, and task completion rates and times were similar.</Paragraph>
    <Paragraph position="2"> The higher turns-to-completion rate for Speech Graffiti is not necessarily problematic. The phrasal nature of Speech Graffiti syntax seems to encourage users to input single phrases; we suspect that in a longitudinal study, we would find single-utterance command use in SG-ML increasing as users became more familiar with the system. Furthermore, because the SG-ML splits long output lists into smaller chunks, a user often has to explicitly issue a request to hear more items in a list, adding at least one more turn to the interaction. Thus there exists a trade-off between turn-wise efficiency and reduced cognitive load. Because of the reasonable results shown for the SG-ML in user satisfaction and completion time, we view this as a reasonable trade-off.</Paragraph>
    <Paragraph position="3"> It is possible that if lower word-error rates can be achieved, Speech Graffiti would become unnecessary.</Paragraph>
    <Paragraph position="4"> This may be true for consistent, extremely low word-error rates, but such rates do not appear to be attainable in the near term. Furthermore, the correlations in Fig. 3 suggest that as WER decreases, users become more satisfied with the SG interface but that this is not necessarily true for the NL interface. Consider also the effect of understanding error. UER is the key to good system performance since even if the system has correctly decoded a word string, it must still match that string with the appropriate concepts in order to perform the desired action. Although WER may be reduced via improved language and acoustic models, matching input to understanding in NL systems is usually a labor-intensive and domain-specific task. In contrast, the structured nature of Speech Graffiti significantly reduces the need for such intensive concept mapping.</Paragraph>
    <Paragraph position="5"> Future work. For Speech Graffiti, scores for habitability (represented by statements like &amp;quot;I always knew what to say to the system&amp;quot;) were typically the lowest of any of the six user satisfaction factors, suggesting that this is a prime area for further work.</Paragraph>
    <Paragraph position="6"> In this regard, it is instructive to consider the experience of the six users who preferred the NL interface. Overall, they accounted for the six highest SG-ML word- and understanding-error rates and the six lowest SG-ML task completion rates: clearly not a positive experience. An additional measure of habitability is grammaticality: how often do users speak within the Speech Graffiti grammar? The six NL-ML-preferring users also had low grammaticality rates (Tomko &amp; Rosenfeld, 2004). These users have become a motivator of future work: what can be done to make the interface work for them and others like them? (Future studies will focus on a broader population of adults.) How can we help users who are having severe difficulties with an interface learn how to use it better and faster? To improve the habitability of Speech Graffiti, we plan to explore allowing more natural language-esque interaction while retaining an application-portable structure. We also plan to refine Speech Graffiti's runtime help facilities in order to assist users more effectively in saying the right thing at the right time.</Paragraph>
    <Paragraph position="7"> In addition to these core interface goals, we plan to extend the functionality of Speech Graffiti beyond information access to support the creation, deletion and modification of information in a database.</Paragraph>
  </Section>
class="xml-element"></Paper>