<?xml version="1.0" standalone="yes"?>
<Paper uid="W05-0208">
  <Title>Towards a Prototyping Tool for Behavior Oriented Authoring of Conversational Agents for Educational Applications</Title>
  <Section position="4" start_page="45" end_page="46" type="intro">
    <SectionTitle>
2 A Historical Perspective
</SectionTitle>
    <Paragraph position="0"> While a focus on design based on standards and practices from human-computer interaction community have not received a great deal of attention in previously published tool development efforts known to the computational linguistics community, our experience tells us that insufficient attention to these details leads to the development of tools that are unusable, particularly to the user population that we target with our work.</Paragraph>
    <Paragraph position="1"> Some desiderata related to the design of our system are obvious based on our target user population. Currently, many educational technology oriented research groups do not have computational linguists on their staff with the expertise required to author domain specific knowledge sources for use with sophisticated state-of-the-art understanding systems, such as CARMEL (Rose, 2000) or TRIPS (Allen et al., 2001). However, previous studies have shown that, while scaffolding and guidance is required to support the authoring process, non-computational linguists possess many of the basic skills required to author conversational interfaces (Rose, Pai, &amp; Arguello, 2005). Because the main barrier of entry to such sophisticated tools are expertise in understanding the underlying data structures and linguistically motivated representation, our tools should have an interface that masks the unnecessary details and provides intuitive widgets that manipulate the data in ways that are consistent with the mental models the users bring with them to the authoring process. In order to be maximally accessible to developers of educational technology, the system should involve minimal programming.</Paragraph>
    <Paragraph position="2"> The design of Carmel-Tools (Rose et al., 2003; Rose &amp; Hall, 2004), the first generation of our authoring tools, was based on these obvious desiderata and not on any in-depth analysis of data collected from our target user population.</Paragraph>
    <Paragraph position="3"> While an evaluation of the underlying computational linguistics technology showed promise (Rose &amp; Hall, 2004), the results from actual authoring use were tremendously disappointing.</Paragraph>
    <Paragraph position="4"> A formal study reported in (Rose, et al., 2005) demonstrates that even individuals with expertise in computational linguistics have difficulty predicting the coverage of knowledge sources that would be generated automatically from example texts annotated with desired representations. Informal user studies involving actual use of Carmel-Tools then showed that a consequence of this lack of ability is that authors were left without a clear strategy for moving through their corpus. As a result, time was lost from annotating examples that did not yield the maximum amount of new knowledge in the generated knowledge sources. Furthermore, since authors tended not to test the generated knowledge sources as they were annotating examples, errors were difficult for them to track later, despite facilities designed to help them with that task.</Paragraph>
    <Paragraph position="5"> Another finding from our user studies was that although the interface prevented authors from violating the constraints they designed into their predicate language, it did not keep authors from annotating similar texts with very different representations, thus introducing a great deal of spurious ambiguity. Thus, they did not naturally maintain consistency in their application of their own designed meaning representation languages across example texts. An additional problem was that authors sometimes decomposed examples in ways that lead to overly general rules, which then lead to incorrect analyses when these rules matched inappropriate examples.</Paragraph>
    <Paragraph position="6"> These disappointing results convinced us of the importance of taking a user-centered design approach to our authoring interface redesign process.</Paragraph>
  </Section>
class="xml-element"></Paper>