<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-2802">
  <Title>Towards Measuring Scalability in Natural Language Understanding Tasks</Title>
  <Section position="2" start_page="0" end_page="0" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> Current evaluation frameworks for uni- or multi-modal dialogue systems (Walker et al., 2000; Beringer et al., 2002) that allow for spoken language input do not include metrics for measuring the accuracy of the involved intention recognition systems, simply because such information is hard to extract automatically from log files. Furthermore no general computational method or framework for measuring the difficulty of natural language understanding tasks have been proposed so far. We are, therefore, faced with a lack of methods for measuring the difficulties of the individual tasks involved in the language understanding process. Such generally applicable methods, however, are needed for measuring the scalability of natural language understanding systems and components.</Paragraph>
    <Paragraph position="1"> In this paper we first discuss existing metrics for measuring task-specific performances and the corresponding baseline metrics in natural language understanding in Section 2. We, then, propose a generalized baseline-based metric in Section 4.1 as well as a general entropy-based metric in Section 4.2. Both methods can be employed for measuring the difficulties of various understanding tasks and, consequently, for evaluating natural language understanding components involved in the intention recognition process. Section 5 provides a case study evaluation of the proposed methods. In Section 6 we discuss how an analysis of a specific system on tasks differing in their difficulty can yield a first approach for measuring the scalability of a natural language understanding systems and its components.</Paragraph>
  </Section>
class="xml-element"></Paper>