<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-1611">
  <Title>Exploiting Discourse Structure for Spoken Dialogue Performance Analysis</Title>
  <Section position="4" start_page="85" end_page="87" type="intro">
    <SectionTitle>
2 Annotation
</SectionTitle>
    <Paragraph position="0"> Our annotation for discourse structure and student state has been performed on a corpus of 95 experimentally obtained spoken tutoring dialogues between 20 students and our system ITSPOKE (Litman and Silliman, 2004).</Paragraph>
    <Paragraph position="1"> ITSPOKE is a speech-enabled version of the text-based Why2-Atlas conceptual physics tutoring system (VanLehn et al., 2002). When interacting with ITSPOKE, students first type an essay answering a qualitative physics problem using a graphical user interface. ITSPOKE then engages the student in spoken dialogue (using head-mounted microphone input and speech output) to correct misconceptions and elicit more complete explanations, after which the student revises the essay, thereby ending the tutoring or causing another round of tutoring/essay revision.</Paragraph>
    <Paragraph position="2"> Each student went through the same procedure: 1) read a short introductory material, 2) took a pretest to measure the initial physics knowledge, 3) work through a set of 5 problems with ITSPOKE, and 4) took a posttest similar to the pretest. The resulting corpus had 2334 student turns and a comparable number of system turns.</Paragraph>
    <Section position="1" start_page="85" end_page="87" type="sub_section">
      <SectionTitle>
2.1 Discourse structure
</SectionTitle>
      <Paragraph position="0"> We base our annotation of discourse structure on the Grosz &amp; Sidner theory of discourse structure (Grosz and Sidner, 1986). A critical ingredient of this theory is the intentional structure. According to the theory, each discourse has a discourse purpose/intention. Satisfying the main discourse purpose is achieved by satisfying several smaller purposes/intentions organized in a hierarchical structure. As a result, the discourse is segmented in discourse segments each with an associated discourse segment purpose/intention. This theory has inspired several generic dialogue managers for spoken dialogue systems (Bohus and Rud- null tation We automate our annotation of the discourse structure by taking advantage of the structure of the tutored information. A dialogue with ITSPOKE follows a question-answer format (i.e.</Paragraph>
      <Paragraph position="1"> system initiative): ITSPOKE asks a question, the student provides the answer and then the process is repeated. Deciding what question to ask, in what order and when to stop is hand-authored beforehand in a hierarchical structure that resembles the discourse segment structure (see Figure 1). Tutor questions are grouped in segments which correspond roughly to the discourse segments. Similarly to the discourse segment purpose, each question segment has an associated tutoring goal or purpose. For example, in  ITSPOKE there are question segments discussing about forces acting on the objects, others discussing about objects' acceleration, etc.</Paragraph>
      <Paragraph position="2"> In Figure 1 we illustrate ITSPOKE's behavior and our discourse structure annotation. First, based on the analysis of the student essay, ITSPOKE selects a question segment to correct misconceptions or to elicit more complete explanations. This question segment will correspond to the top level discourse segment (e.g. DS1).</Paragraph>
      <Paragraph position="3"> Next, ITSPOKE asks the student each question in DS1. If the student answer is correct, the system moves on to the next question (e.g. Tu- null ). If the student answer is incorrect, there are two alternatives. For simple questions, the system will simply give out the correct answer and move on to the next question (e.g. Tutor null  ). For complex questions (e.g. applying physics laws), ITSPOKE will engage into a remediation subdialogue that attempts to remediate the student's lack of knowledge or skills. The remediation subdialogue is specified in another question segment and corresponds to a new discourse segment (e.g DS2). The new discourse segment is dominated by the current discourse segment (e.g. DS2 dominated by DS1). Tutor  system turn is a typical example; if the student answers it incorrectly, ITSPOKE will enter discourse segment DS2 and go through its questions (Tutor  and Tutor  ). Once all the questions in DS2 have been answered, a heuristic determines whether ITSPOKE should ask the original question again (Tutor  ) or simply move on to the next question (Tutor  ).</Paragraph>
      <Paragraph position="4"> To compute interaction parameters from the discourse structure, we focus on the transitions in the discourse structure hierarchy. For each system turn we define a transition feature. This feature captures the position in the discourse structure of the current system turn relative to the previous system turn. We define six labels (see Table 1). NewTopLevel label is used for the first question after an essay submission (e.g. Tutor  ).</Paragraph>
      <Paragraph position="5"> If the previous question is at the same level with the current question we label the current question as Advance (e.g. Tutor  ). The first question in a remediation subdialogue is labeled as Push (e.g. Tutor  ). After a remediation subdialogue is completed, ITSPOKE will pop up and it will either ask the original question again or move on to the next question. In the first case, we label the system turn as PopUp. Please note that Tutor  will not be labeled with PopUp because, in such cases, an extra system turn will be created be- null . In addition, variations of &amp;quot;Ok, back to the original question&amp;quot; are also included in the new system turn to mark the discourse segment boundary transition. If the system moves on to the next question after finishing the remediation subdialogue, we label the system turn as PopUpAdv (e.g. Tutor  ). Note that while the sum of PopUp and PopUpAdv should be equal with Push, it is smaller in our corpus because in some cases ITSPOKE popped up more than one level in the discourse structure hierarchy. In case of rejections, the system question is repeated using variations of &amp;quot;Could you please repeat that?&amp;quot;. We label such cases as SameGoal (e.g. Tutor  Please note that each student dialogue has a specific discourse structure based on the dialogue that dynamically emerges based on the correctness of her answers. For this reason, the same system question in terms of content may get a different transition label for different students. For example, in Figure 1, if the student would have answered Tutor  correctly, the next tutor turn would have had the same content as Tutor  but the Advance label. Also, while a human annotation of the discourse structure will be more complex but more time consuming (Hirschberg and Nakatani, 1996; Levow, 2004), its advantages are outweighed by the automatic nature of our discourse structure annotation.</Paragraph>
      <Paragraph position="6"> We would like to highlight that our transition annotation is domain independent and automatic. Our transition labels capture behavior like starting a new dialogue (NewTopLevel), crossing discourse segment boundaries (Push, PopUp, PopUpAdv) and local phenomena inside a discourse segment (Advance, SameGoal). If the discourse structure information is available, the  transition information can be automatically computed using the procedure described above.</Paragraph>
    </Section>
    <Section position="2" start_page="87" end_page="87" type="sub_section">
      <SectionTitle>
2.2 Student state
</SectionTitle>
      <Paragraph position="0"> Because for our tutoring system student learning is the relevant performance metric, we hypothesize that information about student state in each student turn, in terms of correctness and certainty, will be an important indicator. For example, a student being more correct and certain during her interaction with ITSPOKE might be indicative of a higher learning gain. Also, previous studies have shown that tutoring specific parameters can improve the quality of SDS performance models that model the learning gain (Forbes-Riley and Litman, 2006).</Paragraph>
      <Paragraph position="1"> In our corpus, each student turn was manually labeled for correctness and certainty (Table 1).</Paragraph>
      <Paragraph position="2"> While our system assigns a correctness label to each student turn to plan its next move, we choose to use a manual annotation of correctness to eliminate the noise introduced by the automatic speech recognition component and the natural language understanding component. A human annotator used the human transcripts and his physics knowledge to label each student turn for various degrees of correctness: correct, partially correct, incorrect and unable to answer.</Paragraph>
      <Paragraph position="3"> &amp;quot;Unable to Answer&amp;quot; label was used for turns where the student did not answer the system question or used variants of &amp;quot;I don't know&amp;quot;. Previous work has shown that certainty plays an important role in the learning and tutoring process (Pon-Barry et al., 2006; VanLehn et al., 2003). A human annotator listened to the dialogues between students and ITSPOKE and labeled each student turn for its perceived degree of certainness. Four labels were used: certain, uncertain, neutral and mixed (both certain and uncertain). To date, one annotator has labeled all student turns in our corpus</Paragraph>
      <Paragraph position="5"/>
    </Section>
  </Section>
class="xml-element"></Paper>