<?xml version="1.0" standalone="yes"?>
<Paper uid="H89-2017">
  <Title>DATA COLLECTION AND ANALYSIS IN THE AIR TRAVEL PLANNING DOMAIN</Title>
  <Section position="4" start_page="119" end_page="121" type="metho">
    <SectionTitle>
METHOD
</SectionTitle>
    <Paragraph position="0"> Before collecting data from human-machine interactions, we observed problem solving in human-human dialogues. Human-human dialogues provide some knowledge of subjects' expectations of the system, the problems which could arise, and solution paths subjects might choose.</Paragraph>
    <Paragraph position="1"> Human-human data collection We collected more than 12 hours (over 100 conversations) of on-site tape recordings of 6 travel agents at a travel agency interacting with clients and with airline agents via telephone. Tape recording equipment was out of the sight of the agent. Both parties knew their voices were being recorded. However, after a few brief interchanges, conversations proceeded as usual. Data collection occurred at the busiest time of day. The tape recorder stayed on for 45-minute durations, except when personal calls interrupted. For each reservation a client makes, agents estimated that the client calls an average of three times: to ask information, to book a flight, and to ticket the flight or make slight changes. We were most interested in first-time calls in which clients booked a flight, although we included data from all three types of calls in our analysis.</Paragraph>
    <Paragraph position="2"> Human-machine data collection To simulate an air travel planning spoken language system, we combined a database retrieval program and a human speech-recognizer/database-accessor, the &amp;quot;wizard.&amp;quot; The experiments involved two computer consoles. One Sun 4 graphics console displayed three windows for the subject: a template window of shared information (fields for departure city, arrival city, date, earliest departure time, latest departure time, earliest arrival time, and latest arrival time), a flights schedule window, and a fare window. The wizard could also send a limited number of messages to the subject: &amp;quot;Cannot handle that request&amp;quot;, &amp;quot;Would you please repeat that?&amp;quot;, and &amp;quot;Ready for more speech input.&amp;quot; The subject's console was controlled by the wizard's Sun 3 console, in another office down the hallway. The wizard entered data into the database retrieval program by clicking the mouse.</Paragraph>
    <Paragraph position="3"> The user wore a Sennheiser headset microphone, connected to a tape recorder, and spoke to the system via an unobtrusive speakerphone. The system's only means of response was through graphic display. A two-pitch tone coming from the telephone before and after each condition indicated that the experimental system was turned either on or off.</Paragraph>
    <Paragraph position="4">  A current total of 44 subjects (26 men, 18 women) participated in the simulated human-machine interactive experiment. Electronic failure caused the loss of data from one (male) subject, leaving 43 who successfully completed their tasks. Two travel planning tasks (one more constrained by fare and the other by schedule, described further below) were assigned each subject counter-balanced with two interaction conditions (database queries only or &amp;quot;regular&amp;quot; -- expressing constraints such as &amp;quot;1 can't leave till 3 pm&amp;quot; allowed). The order cycled every four subjects. One quarter of the subjects participated in each of the following test orders:  1. fare task in database query condition, schedule task in regular condition, 2. schedule task in regular condition, fare task in database query condition, 3. fare task in regular condition, schedule task in database query condition, and 4. schedule in database query condition, fare task in regular condition.</Paragraph>
    <Paragraph position="5">  Subjects were presented with general written instructions indicating that they were going to help assess and debug an experimental computer-aided travel planner using voice input. Whether the system was completely automated or not was purposefully left ambiguous. The experimenter, the same person as the wizard (author JK), always referred to the experimental system as &amp;quot;the system&amp;quot; or &amp;quot;it.&amp;quot; The subject was asked to make a simple flight reservation, interacting with the system to find an optimal flight for the assigned task. General examples of acceptable and unacceptable utterances were provided. The subject was requested to end the session by saying, &amp;quot;ok, book that one.&amp;quot; The subject was also told that as the system received information, it would begin to display pieces of information in the template display window. The experimenter then read instructions describing the assigned travel-planning task to the subject, allowing the subject to take notes. This was to avoid any poisoning of the data that might be induced if the subjects simply read the task description. The experimenter then explained the condition to the subject (database query only or regular). Examples of acceptable and unacceptable database queries were given for the relevant condition, and the idea that a database query is a sentence that results in a database retrieval was explained. The subject was also told what types of information the system could provide. The tasks, which each took about 5 minutes to complete, are described below : A. Book a one-way flight from San Francisco to Los Angeles, for &lt;date&gt;, leaving after &lt;time&gt;, arriving before &lt;time&gt;, subject to the following ordered constraints:  1. cost under $200 2. arrive as early as possible (after &lt;time&gt;) 3. prefer SFO airport to OAK or SJC, and prefer LAX to Burbank B. Book a one-way flight from San Francisco to Los Angeles, for &lt;date&gt;, arriving before &lt;time&gt;, leaving after &lt;time&gt;, subject to the following ordered constraints: 1. arrive as close as possible to &lt;time&gt; 2. spend as little time in transit as possible 3. prefer SJC airport departure to SFO or OAK 4. price under $400  The flight information database used is a subset of the Official Airline Guide (OAG) database obtained from the OAG in May 1989. The data was reformatted to allow for easier access and to avoid infringing on OAG's proprietary rights in any later distribution of the data. The data was accessed via a wizard's interface. Developing tools for the wizard is an important task. The wizard takes complete control of the speech and natural language functions of the system and needs a swift means of retrieving data for the user. Being the wizard is difficult because the human must simulate the consistent and more limited response of a computer. By accepting an utterance or producing an error message, the wizard has a large influence over the user's expectation of the system's capabilities.</Paragraph>
    <Paragraph position="6">  The wizard accessed the database upon request from the user and controlled the screen of the user by showing tables of fares and schedules, displaying an error message, or requesting that the user ask another question or repeat the previous question. The wizard's screen displayed the same three windows as the subjects' and had additional windows for inputting information with the mouse. The mouse was used to select a category such as departure city and then select the proper value from a pop-up window. The wizard's screen always showed a superset of the information displayed on the user's screen.</Paragraph>
  </Section>
class="xml-element"></Paper>