<?xml version="1.0" standalone="yes"?>
<Paper uid="H01-1012">
  <Title>A Conversational Interface for Online Shopping</Title>
  <Section position="3" start_page="0" end_page="2" type="metho">
    <SectionTitle>
2. NATURAL LANGUAGE ASSISTANT
</SectionTitle>
    <Paragraph position="0"> NLA assists users in finding notebooks that satisfy their needs by engaging them in a dialog. At each turn of the dialog, NLA provides incremental feedback about its understanding of the user's constraints and shows products that match these constraints. By encouraging iterative refinement of the user's query, the system finds more user constraints and, ultimately, recommends a product that best matches the user's criteria.</Paragraph>
    <Paragraph position="1"> The system consists of three major modules (cf. Figure 1): Presentation Manager, Dialog Manager, and Action Manager. The Presentation Manager interprets user input and generates system responses. It embodies the user interface and contains a shallow semantic parser and a response generator. The semantic parser identifies concepts (e.g., MULTIMEDIA) and constraints on product attributes (e.g., hard disk size more than 20GB)fromthe textual user input. The concepts mediate the mapping between user input and available products through product specifications. They implement the business logic.</Paragraph>
    <Paragraph position="2"> The Dialog Manager uses the current requirements and formulates action plans for the Action Manager to perform back-end operations (e.g., database access  ). The Dialog Manager constructs a response to the user based on the results from the Action Manager and the discourse history and sends the system response to the Presentation Manager that displays it to the user. The system prompts for features relevant in the current context. In our mixed initiative dialog system, the user can always answer the specific question put to him/her or provide any constraints.</Paragraph>
    <Paragraph position="3"> The system has been recently deployed on an external website.</Paragraph>
    <Paragraph position="4"> Figure 2 shows the start of a dialog.</Paragraph>
    <Paragraph position="5">  See [1] for a survey of natural language interfaces to databases.  We are demonstrating the system at HLT'2001 [2].</Paragraph>
  </Section>
  <Section position="4" start_page="2" end_page="4" type="metho">
    <SectionTitle>
3. USER STUDIES
</SectionTitle>
    <Paragraph position="0"> We conducted a preliminary market survey and two user studies described in subsections 3.1 and 3.2 respectively.</Paragraph>
    <Section position="1" start_page="2" end_page="2" type="sub_section">
      <SectionTitle>
3.1 Market Survey
</SectionTitle>
      <Paragraph position="0"> For understanding specific user needs and user vocabulary, we conducted a user survey. Users were given three sets of questions.</Paragraph>
      <Paragraph position="1"> The first set, in turn, contained three questions: &amp;quot;What kind of notebook computer are you looking for?&amp;quot;, &amp;quot;What features are important to you?&amp;quot;, and &amp;quot;What do you plan to use this notebook computer for?&amp;quot;. By applying statistical n-gram models and a shallow noun phrase grammar to the user responses, we extracted keywords and phrases expressing user's needs and interests. In the second set of questions, users were asked to rank 10 randomly selected terms from 90 notebook related terms in order of familiarity to them. The third set of questions asked for demographical information about users such as their gender, years of experience with notebook computers, native language, etc. We computed correlations between vocabulary/terms and user demographic information. Over a 30-day period, we received 705 survey responses. From these responses, we learned 195 keywords and phrases that were included in NLA.</Paragraph>
    </Section>
    <Section position="2" start_page="2" end_page="4" type="sub_section">
      <SectionTitle>
3.2 Usability Testing
</SectionTitle>
      <Paragraph position="0"> We conducted two user studies to evaluate usability of the system, focusing on: dialog flow, ease of use, system responses, and user vocabulary. The first user study focused on the functionality of NLA and the second user study compared the functionality of NLA with that of a directed dialog system and a menu driven navigation system.</Paragraph>
      <Paragraph position="1"> The moderators interviewed 52 users in the user studies: 18 and 34 in the two studies, respectively. All participants were consumers or small business users with &amp;quot;beginner&amp;quot; or &amp;quot;intermediate&amp;quot; computer skills. Each participant was asked to find laptops for a variety of scenarios using three different systems (the NLA, a directed dialog system and a menu driven navigation system). Participants were asked to rate each system for each task on a 1 to 10 scale (10 - easiest) with respect to the ease of navigation, clarity of terminology and their confidence in the system responses. The test subjects were also asked whether the system had found relevant products and were prompted to share their impressions as to how well the system understood them and responded to their requests.</Paragraph>
      <Paragraph position="2">  In both studies, participants were very receptive to using natural language dialog-based search. The users clearly preferred dialog-based searches to non-dialog based searches  (79% to 21% users).</Paragraph>
      <Paragraph position="3"> Furthermore, they liked the narrowing down of a product list based on identified constraints as the interaction proceeded. In the first user study, comparing NLA with a menu driven system, we found that using NLA reduced the average number of clicks by 63% and the average interaction time by 33%.</Paragraph>
      <Paragraph position="4"> In the second user study, we compared NLA with a directed dialog system and a menu driven search system for finding computers. One goal of the comparative study was to find out if there were any statistical differences in confidence, terminology and navigation ratings across the three systems and whether they were correlated with different categories of users. The ANOVA analysis reveals statistical differences in terminology ratings among the three systems for the category of beginner users only. There were no statistical differences found in the other ratings of navigation and confidence across the three sites for different categories of users. Sandler's A test confirmed that the terminology rating was significantly different for the categories of consumers, small business owners, beginners and intermediates.</Paragraph>
      <Paragraph position="5"> These comparative results suggest that asking questions relative to the right level of end user experience is crucial. Asking users questions about their lifestyle and how they were going to use a computer accounted for a slight preference of the directed dialog system over the NLA that uses questions presented on the basis of understanding features and functions of computer terms.</Paragraph>
      <Paragraph position="6"> 3.2.3 Lessons from the user studies Both user studies revealed several dimensions along which NLA can be improved. The first user study highlighted a definite need for system acknowledgement and feedback. The users wanted to know whether the system had understood them. User comments also revealed that a comparison of features across the whole pool of products was important for them.</Paragraph>
      <Paragraph position="7"> The focus of the second study, incorporating 34 subjects, was to compare systems of similar functionality and to draw conclusions about the functionality of NLA. Both the ANOVA and the Sandler's test point out that terminology was a statistically significant factor differentiating among the systems. We believe that using terminology that is not overly technical would contribute to the success of the dialog search. While the questions asked by NLA were based on features and functionality of notebook computers, the users preferred describing usage patterns and life style issues rather than technical details of computers. We also found that users' confidence in NLA decreased when the system responses were inconsistent i.e., were not relevant to their input. Lack of consistent visual focus on the dialog box was also a serious drawback since it forced users to scroll in search of the dialog box on each interaction page.</Paragraph>
      <Paragraph position="8">  We define a dialog-based search as one comprising of a sequence of interactions with a system where the system keeps track of contextual (discourse) information.</Paragraph>
      <Paragraph position="9">  Based on the results of the user studies, we are currently focused on: developing a dynamic and adaptive dialog management strategy, improving the robustness of the natural language processing (NLP), and improving the user interface. Some of issues mentioned here have been implemented in the next version of NLA.</Paragraph>
      <Paragraph position="10"> We are currently re-designing the questions that NLA asks users to be simpler, and to focus on usage patterns rather than technical features. We are also implementing a new dialog management strategy in NLA that is more adaptive to the user's input, and implements a mapping from high-level usage patterns to constraints on low-level technical features.</Paragraph>
      <Paragraph position="11"> We are integrating a statistical parser with NLA to more robustly handle varied user input. The statistical parser should enable NLA to scale to multiple languages and multiple domains in a more robust and reliable fashion. We are aiming at an architecture that separates the NLP processing from the business logic that will make maintenance of the system easier.</Paragraph>
      <Paragraph position="12">  Improvements to the GUI include better acknowledgement and feedback mechanisms as well as graphical UI issues. We now reiterate the user's last query at the beginning of each interaction page and also convey to the user an explanation of features incrementally accumulated in the course of the interaction. We have designed a more uniform, more compact and consistent UI.</Paragraph>
      <Paragraph position="13"> In the welcome page, we have abandoned a three-step initiation (typed input, experience level and preferences for major specifications) keeping the emphasis on the dialog box. The user preferences contributed to creating confusion as to the main means of interaction (many users just clicked on the radial buttons and did not use the full dialog functionality). We now infer the technical specifications based the user's stated needs and usage patterns. Our UI now has a no scrolling policy and we allow for larger matching set of products to be visualized over a number of pages.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>