<?xml version="1.0" standalone="yes"?>
<Paper uid="E89-1016">
  <Title>User studies and the design of Natural Language Systems</Title>
  <Section position="5" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
3 Results
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
3.1 Preliminary analysis and filtering
</SectionTitle>
      <Paragraph position="0"> This analysis is concerned with user input and so the Wizard's responses are not considered here. We began by taking all the 384 subject utterances, entering them into the NL prototype and observing what analysis the system produced. We found that by far the largest category of errors was unknown words, so we began by analysing the total of 401 instances of 104 unknown words.</Paragraph>
      <Paragraph position="1"> Our interest here lay in the influence of the task on language use so we focus on 3 classes of unknown words which demonstrate this in different ways: these were operators and explicit reference to set properties; references to context; and references to the information source.</Paragraph>
      <Paragraph position="2"> Our interest was in the target set of queries input by people who wanted to use the system for database access. We therefore gave the Wizard instructions to answer all queries regardless of linguistic complexity. There was however one exception to this rule: each task was expressed as a series of requirements and one possible strategy for the task was to enter all these requirements as one long query. If the Wizard had answered this query then the dialogue would have been extremely short, i.e it would have been one query and a response which was the answer to the whole task.</Paragraph>
      <Paragraph position="3"> To prevent this, the .Wizard was told to reply to such long queries by saying Too much information to process. There were no other constraints on the type of input that the Wizard could process and answers were given to all other types of query.</Paragraph>
      <Paragraph position="4"> Subject and Wizard both used HP-Unix Workstations and communicated by writing in networked X windows. The inputs of both subject and Wizard were displayed in a single window on each of the machines with the subject's entries presented in lower case and the Wizard's in upper case, so the contents of the display windows on both machines were identical. To avoid teaching the subjects skills like scrolling, we also provided them with hard copy output of the whole of the interaction by printing the contents of the windows to a printer next to the subjeet's machine. If they wanted to refer back to much earlier in the dialogue, the subjects could consult the  of set properties The task of database access involves the construction and manipulation of answer sets with various properties. null The unknown words that were used for set construction and manipulation were mainly verbs. These we called operators. They can be further subclassifled into verbs which were used to select sets, those which were used to permute already constructed sets and those which operate over a set of queries.</Paragraph>
      <Paragraph position="5"> The majority of operators invoked simple set selection: these included for example, state and tell. There were also instances of indirect requests for selection, e.g. need and want. Subjects tried to permute the presentation of sets by using words like arrange. Finally queries such as All the conditions from now on will apply to ... show there were verbs which operated over sets of queries.</Paragraph>
      <Paragraph position="6"> A second way in which these set manipulation operations appeared was in the subjects' explicit reference to the fact that they were constructing sets with specific properties. Find paintings that satisfy the following criteria ... was an example of this.</Paragraph>
      <Paragraph position="7"> Altogether operators and explicit reference to set - 119properties occurred on 102 occasions which accounted for 25% of the unknown words.</Paragraph>
      <Paragraph position="8">  The task could not be accomplished in one query so we expected that this would necessitate our subjects making reference to previous queries. We therefore went on to analyse those unknown words that required information from outside the current query for their interpretation. Among the unknown words which relied upon context, we distinguished between what we called pointers (N = 42 instances) and exclusion operators (N = 21 instances). Together they accounted for 16% of unknown words.</Paragraph>
      <Paragraph position="9"> Pointers signalled to the listener that the reference set lay outside the current utterance. These could be further subdivided according to whether or not they pointed forwards, e.g. Give me the dates of the following paintings ... or backwards in the dialogue, e.g. previous and above. There were two instances of forwards pointers following and now on.</Paragraph>
      <Paragraph position="10"> The backwards pointers could be subclassified according to how many previous answer sets they referred to. The majority referred to a single answer set and this was most often the one generated by the immediately prior query. Other pointers referred to a number of prior answer sets, which could scope as far back as the beginning of the current subdialogue, or even the beginning of the whole dialogue.</Paragraph>
      <Paragraph position="11"> Exclusion operators applied to sets created earlier in the dialogue. They served to exclude elements of these sets from the current query. The simplest examples of this occurred when people had (a) identified a set previously; (b) they had then selected a subset of this original set; and (c) they wanted all or part of the set of the original set which had not been selected by the second opergtion. These included words like another and more, as in Give me I0 more Van Gogh paintings.</Paragraph>
      <Paragraph position="12"> A more complex instance of this type of exclusion was when the word was used, not to exclude sub-sets from sets already identified, but to exclude the attributes of the items in the excluded subsets, e.g.</Paragraph>
      <Paragraph position="13"> Find me a painting with a theme that is different from those already mentioned. Here the system has first to generate the set of paintings already mentioned, then it has to generate their themes and then finally it has to find a painting whose theme is different from the set of themes already identified.</Paragraph>
      <Paragraph position="14">  Our subjects believed that they were interacting with a real information source, in this case a database, also seemed to affect their language use. We found 19 (5% of all unknown words) which seemed to refer to the database and its structure directly.</Paragraph>
      <Paragraph position="15"> There were words which seemed to refer to field names in the database, e.g. categories and information, e.g. What information on each painting is there? There were also words which seemed to refer to values within a field, e.g. types as in List the media types. In addition, there were references to the ordering of entities, e.g. first or second, as in What is the first painting in your list?. Finally, there were words which referred to the general scope or properties of the database: e.g. database and represented, e.g. What different paint media are represented?.</Paragraph>
      <Paragraph position="16"> There were also 3 occasions on which reference is made both to database structure and to context.</Paragraph>
      <Paragraph position="17"> These are the instances of next being used to access entities in a column but also referring to context. The utterance List next 10 paintings, references 10 items in the sequence that they appear in the database, but excludes the 10 items already chosen. Finally there was one instance of a question which would have required inferencing based on the structure of the information source, Is a portrait the same as a self portrait?. Here the question was about the type relation.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>