<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-1303">
  <Title>Building Effective Question Answering Characters</Title>
  <Section position="3" start_page="0" end_page="18" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> In the recent Hollywood movie &amp;quot;iRobot&amp;quot; set in 2035 the main character played by Will Smith is running an investigation into the death of an old friend. The detective finds a small device that projects a holographic image of the deceased. The device delivers a recorded message and responds to questions by playing back prerecorded answers.</Paragraph>
    <Paragraph position="1"> We are developing virtual characters with similar capabilities.</Paragraph>
    <Paragraph position="2"> Our target applications for these virtual characters are training, education, and entertainment. For use in education, such a character should be able to deliver a message to the student on a specific topic. It also should be able to support a basic spoken dialog on the subject of the message, e.g., answer questions about the message topic and give additional explanations. For example, consider a student learning about an event in a virtual world.</Paragraph>
    <Paragraph position="3"> Lets say there is a small circus in a small town and someone has released all the animals from circus.</Paragraph>
    <Paragraph position="4"> A young student plays a role of a reporter to find out who caused this local havoc. She is out to interrogate a number of witnesses represented by the virtual characters. It is reasonable to expect that each conversation is going to be focused solely on the event of interest and the characters may refuse to talk about anything else. Each witness may have a particular and very narrow view into an aspect of the event, and the student's success would depend on what sort of questions she asks and to which character she addresses them.</Paragraph>
    <Paragraph position="5"> Automatic question answering (QA) has been studied extensively in recent years. For example, there is a significant body of research done in the context of the QA track at the Text REtrieval Conference (TREC) (Voorhees, 2003). In contrast to the TREC scenario where both questions and answers are based on facts and the goal is to provide the most relevant answer, we focus the answer's appropriateness. In our example about an investigation, an evasive, misleading, or an &amp;quot;honestly&amp;quot; wrong answer from a witness character would be appropriate but might not be relevant. We try to highlight that distinction by talking about QA characters as opposed to QA systems or agents.</Paragraph>
    <Paragraph position="6"> We expect that a typical simulation would contain quite a few QA characters. We also expect those characters to have a natural spoken language interaction with the student. Our technical requirements for such a QA character is that it should be able to understand spoken language. It should be robust to disfluencies in conversational English. It should be relatively fast, easy, and inexpensive to construct without the need for extensive domain knowledge and dialog management design expertise. null In this paper we describe a QA character by the name of SGT Blackwell who was originally designed to serve as an information kiosk at an army  conference (see Appendix C for a photograph of the system) (?). We have used SGT Blackwell to develop our technology for automatic answer selection, conversation management, and system integration. We are presently using this technology to create other QA characters.</Paragraph>
    <Paragraph position="7"> In the next section we outline the SGT Blackwell system setup. In Section 3 we discuss the answer selection problem and consider three different algorithms: Support Vector Machines classifier (SVM), Language Model retrieval (LM), and Cross-lingual Language Model (CLM) retrieval.</Paragraph>
    <Paragraph position="8"> We present the results of off-line experiments showing that the CLM method performs significantly better than the other two techniques in Section 4. Section 5 describes a user study of the system that uses the CLM approach for answer selection. Our results show that the approach is very robust to deviations in wording from expected answers, and speech recognition errors. Finally, we summarize our results and outline some directions for future work in Section 6.</Paragraph>
  </Section>
class="xml-element"></Paper>