<?xml version="1.0" standalone="yes"?>
<Paper uid="J97-1004">
  <Title>Developing and Empirically Evaluating Robust Explanation Generators: The KNIGHT Experiments</Title>
  <Section position="3" start_page="0" end_page="66" type="intro">
    <SectionTitle>
* Discourse-Knowledge Engineering: Discourse-knowledge engineers, i.e.,
</SectionTitle>
    <Paragraph position="0"> knowledge engineers who encode discourse knowledge, should be able to inspect and easily modify discourse-planning specifications for rapid iterative refinement. The Explanation Design Package (EDP) formalism is a convenient, schema-like (McKeown 1985; Paris 1988) programming language for text planning. Because the EDP formalism is a hybrid of the declarative and procedural paradigms, discourse-knowledge engineers can easily understand EDPs, modify them, and use them to represent new discourse knowledge. EDPS have been used by KNIGHT to generate hundreds of expository explanations of biological objects and processes.</Paragraph>
    <Paragraph position="1"> * Explanation Planning: KNIGHT employs a robust explanation planner that selects EDPS and applies them to invoke knowledge-base accessors. The explanation planner considers the desired length of explanations and the relative importance of subtopics as it constructs explanation plans encoding content and organization.</Paragraph>
    <Paragraph position="2"> * Functional Realization: KNIGHT's functional realization system (Callaway  Lester and Porter Robust Explanation Generators and Lester 1995) is built on top of a unification-based surface generator with a large systemic grammar (Elhadad 1992).</Paragraph>
    <Paragraph position="3"> To assess KNIGHT'S performance, we developed the Two-Panel evaluation methodology for natural language generation and employed it in the most extensive and rigorous empirical evaluation ever conducted on an explanation system. In this study, KNIGHT constructed explanations on randomly chosen topics from the Biology Knowledge Base. A panel of domain experts was instructed to produce explanations on these same topics, and both KNIGHT'S explanations and the explanations produced by this panel were submitted to a second panel of domain experts. The second panel then graded all of the explanations on several dimensions with an A-F scale. KNIGHT scored within approximately half a grade of the domain experts, and its performance exceeded that of one of the domain experts.</Paragraph>
    <Paragraph position="4"> This paper is structured as follows: The task of explanation generation is characterized and the Biology Knowledge Base is described. A brief description of KNIGHT's knowledge-base access methods is followed by (1) a description of the EDP language, (2) KNIGHT'S explanation planner, and (3) an overview of the realization techniques. The empirical evaluation is then discussed in some detail. The paper concludes with discussions of related work and future research directions.</Paragraph>
  </Section>
class="xml-element"></Paper>