<?xml version="1.0" standalone="yes"?>
<Paper uid="W01-1614">
  <Title>Empirical Methods for Evaluating Dialog Systems</Title>
  <Section position="7" start_page="2" end_page="2" type="concl">
    <SectionTitle>
DARPA Communicator Project where
</SectionTitle>
    <Paragraph position="0"> participants work within the same domain task, a funding agency may wish to conduct the WOZ studies on behalf of the participants. To defray the cost of running the studies, the agency may wish to determine its own cost function with respect to a given performance metric and utilize average marginal cost to decide which dialog systems to continue sponsoring.</Paragraph>
    <Paragraph position="1"> Because the focus of this paper has been on how to apply the empirical methods, hypothetical examples were considered. Work is currently underway to collect data for evaluating implemented dialog systems. We maintain that without these empirical methods, readers of reported dialog metrics cannot really make sense of the numbers.</Paragraph>
  </Section>
class="xml-element"></Paper>