<?xml version="1.0" standalone="yes"?>
<Paper uid="H92-1003">
  <Title>Multi-Site Data Collection for a Spoken Language Corpus MADCOW *</Title>
  <Section position="9" start_page="2301" end_page="2301" type="concl">
    <SectionTitle>
7. Conclusion
</SectionTitle>
    <Paragraph position="0"> The MADC0W experiment in multi-site data collection and evaluation has been successful. The participating sites have collected a rich corpus of training data, have put in place methods for distributing the data, and have devised test procedures to evaluate speech, natural language, and spoken language results on a test corpus. The resources made available by the multi-site paradigm have allowed us to collect more data and to learn more about data collection than would have been possible with only one or two sites collecting data under a special contract.</Paragraph>
    <Paragraph position="1"> Some difficult problems still remain. Our shared goal is to build interactive spoken language systems; however, our evaluation methods rely on a canned corpus and evaluate a system's recognition performance under static conditions that are not representative of the interactive environments in which these systems will eventually be used. In addition, the ATIS task has been limited so far to a small, static subset of the air travel domain. These difficulties will increase as research sites develop different approaches to actively managing interaction with the user: different processing strategies will generate divergent behaviors on the part of users, but this divergence will lessen the validity of tests that assume comparable responses to a sequence of queries.</Paragraph>
    <Paragraph position="2"> The MADCOW collection and evaluation procedures have provided effective tools for assessing the current capabilities of interactive spoken language systems. However, we must continue to improve our methods of data collection and evaluation. For example, we have  only just begun to explore the use of real-time spoken language systems for data collection and evaluation. We also need to more towards a larger, more realistic, database. As our spoken language systems evolve, data collection and evaluation methods must evolve with them.</Paragraph>
  </Section>
class="xml-element"></Paper>