<?xml version="1.0" standalone="yes"?>
<Paper uid="W02-0219">
  <Title>A New Taxonomy for the Quality of Telephone Services Based on Spoken Dialogue Systems</Title>
  <Section position="2" start_page="0" end_page="0" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> Telephone services which rely on spoken dialogue systems (SDSs) have now been introduced at a large scale. For the human user, when dialing the number it is often not completely clear that the agent on the other side will be a machine, and not a human operator. Because of this fact, and because the interaction with the SDS is performed through the same type of user interface (e.g. the handset telephone), comparisons will automatically be drawn to the quality of human-human communication over the same channel, and sometimes with the same purpose. Thus, while acknowledging the differences in behaviors from both -- human and machine - sides, it seems justified to take the human telephone interaction (HHI) as one reference for telephone-based human-machine interaction (HMI).</Paragraph>
    <Paragraph position="1"> The quality of interactions with spoken dialogue systems is difficult to determine. Whereas structured approaches have been documented on how to design spoken dialogue systems so that they adequately meet the requirements of their users (e.g. by Bernsen et al., 1998), the quality which is perceived when interacting with SDSs is often addressed in an intuitive way. Hone and Graham (2001) describe efforts to determine the underlying dimensions in user quality judgments, by performing a multidimensional analysis on subjective ratings obtained on a large number of different scales. The problem obviously turned out to be multi-dimensional. Nevertheless, many other researchers still try to estimate &amp;quot;overall system quality&amp;quot;, &amp;quot;usability&amp;quot; or &amp;quot;user satisfaction&amp;quot; by simply calculating the arithmetic mean over several user ratings on topics as different as perceived TTS quality, perceived system understanding, and expected future use of the system. The reason is the lack of an adequate description of quality dimensions, both with respect to the system design and to the perception of the user.</Paragraph>
    <Paragraph position="2"> In this paper, an attempt is made to close this gap. A taxonomy is developed which allows quality dimensions to be classified, and methods for their measurement to be developed. The starting point for this taxonomy was a similar one which has fruitfully been used for the description of human-to-human services in telecommunication networks (e.g. traditional telephony, mobile telephony, or voice over IP), see M&amp;quot;oller (2000). Such a taxonomy can be helpful in three respects: (1) system elements which are in the hands of developers, and responsible for specific user perceptions, can be identified, (2) the Philadelphia, July 2002, pp. 142-153. Association for Computational Linguistics. Proceedings of the Third SIGdial Workshop on Discourse and Dialogue, dimensions underlying the overall impression of the user can be described, together with adequate (subjective) measurement methods, and (3) prediction models can be developed to estimate quality - as it would be perceived by the user - from purely instrumental measurements. While we are still far from the last point in HMI, examples will be presented of the first two issues.</Paragraph>
    <Paragraph position="3"> The next section will discuss what is understood by the term &amp;quot;quality&amp;quot;, and will present the taxonomy for HMI. In Section 3, quality features underlying the aspects of the taxonomy are identified, and dialogue- and system-related measures for each aspect are presented in Section 4, based on measures which are commonly documented in literature. Section 5 shows the parallels to the original taxonomy for HHI. The outlook gives implications for the development of evaluation and prediction models, such as the PARADISE framework.</Paragraph>
  </Section>
class="xml-element"></Paper>