<?xml version="1.0" standalone="yes"?>
<Paper uid="P98-1096">
  <Title>Robust Interaction through Partial Interpretation and Dialogue Management</Title>
  <Section position="3" start_page="0" end_page="590" type="metho">
    <SectionTitle>
2 Dialogue management
</SectionTitle>
    <Paragraph position="0"> Partial interpretation is particularly well-suited for dialogue systems, as we can utilize information from a dialogue manager on what is expected and use this to guide the analysis. Furthermore, dialogue management allows for focus tracking as well as clarification subdialogues to further improve the interaction (JSnsson, 1997).</Paragraph>
    <Paragraph position="1"> In information retrieval systems a common user initiative is a request for domain concept information from the database; users specify a database object, or a set of objects, and ask for the value of a property of that object or set of objects. In the dialogue model this can be modeled in two focal parameters: Objects related to database objects and Properties modeling the domain concept information. The Properties parameter models the domain concept in a sub-parameter termed Aspect which can be specified in another sub-parameter termed Value. The specification of these parameters in turn depends on information from the user initiative together with context information and the answer from the database system. The action to be carried out by the interface for task-related questions depends on the specification of values passed to the Objects and Properties parameters (JSnsson, 1997).</Paragraph>
    <Paragraph position="2"> We can also distinguish two types of information sources utilized by the dialogue manager; the database with task information, T, or system-related information about the application, S.</Paragraph>
  </Section>
  <Section position="4" start_page="590" end_page="590" type="metho">
    <SectionTitle>
3 Types of information
</SectionTitle>
    <Paragraph position="0"> We can identify different types of information utilized when interpreting an utterance in a natural language interface to a database system. This information corresponds to the information that needs to be analyzed in userutterances. null Domain concepts are concepts about which the system has information, mainly concepts from the database, T, but also synonyms to such concepts acquired, for instance, from the information base describing the system, S.</Paragraph>
    <Paragraph position="1"> In a database query system users also often request information by relating concepts and objects, e.g. which one is the cheapest. We call this type of language constructions relational e~pressions. The relational expressions can be identified from the corpus.</Paragraph>
    <Paragraph position="2"> Another common type of expressions are numbers. Numbers can occur in various forms, such as dates, object and property values.</Paragraph>
    <Paragraph position="3"> Set operations. It is necessary to distinguish utterances such as: show all cars costing less than 70 000 from which of these costs less than 70 000. The former should get all cars costing less than 70 000 whereas the latter should utilize the set of cars recorded as Objects by the dialogue manager. In some cases the user uses expressions such as remove all cars more expensire than 70 000, and thus is restricting a set by mentioning the objects that should be removed.</Paragraph>
    <Paragraph position="4"> Interactional concepts. This class of concepts consists of words and phrases that concern the interaction such as Yes, No, etc (cf. (Byron and Heeman, 1997)).</Paragraph>
    <Paragraph position="5"> Task/System expressions. Users can use domain concepts such as explain, indicating that the domain concept is not referring to a request for information from the database, T, but instead from the system description, S.</Paragraph>
    <Paragraph position="6"> When acquiring information for the interpreter, three different sources of information can be utilized: 1) background system information, i.e.</Paragraph>
    <Paragraph position="7"> the database, T, and the information describing the background system's capabilities, S, 2) information from dialogues collected with users of the system, and 3) common sense and prior knowledge on human-computer interaction and natural language dialogue. The various information sources can be used for different purposes (JSnsson, 1993).</Paragraph>
  </Section>
  <Section position="5" start_page="590" end_page="591" type="metho">
    <SectionTitle>
4 The interpretation module
</SectionTitle>
    <Paragraph position="0"> The approach we are investigating relies on analyzing as small and crucial parts of the utterances as possible. One of the key issues is to find these parts. In some cases an analysis could consist of one single domain or interactional concept, but for most cases we need to analyze small sub-phrases of an utterance to get a more reliable analysis. This requires flexibility in processing of the utterances and is a further development of the ideas described in StrSmb~ick (1994). In this work we have chosen to use PATR-II but in the future constructions from a more expressive formalism such as EFLUF (StrSmb~ck, 1997) could be needed.</Paragraph>
    <Paragraph position="1"> Flexibility in processing is achieved by one extension to ordinary PATR and some additions to a chart parser environment. Our version of PATR allows for a set of unknown words within phrases. This gives general grammar rules, and helps avoiding the analysis to be stuck in case of unknown words. In the chart parsing environment it is possible to define which of the inactive edges that constitute the result.</Paragraph>
    <Paragraph position="2"> The grammar is divided into five grammar modules where each module corresponds to some information requested by the dialogue manager. The modules can be used independently from each other.</Paragraph>
    <Paragraph position="3"> Domain concepts are captured using two grammar modules. The task of these grammars is to find keywords or sub-phrases in the expressions that correspond to the objects and properties in the database. The properties can be concept keywords or relational expressions containing concept keywords. Numbers are typed according to the property they describe, e.g.</Paragraph>
    <Paragraph position="4"> 40000 denote a price.</Paragraph>
    <Paragraph position="5"> To simplify the grammars we only require that the grammar recognizes all objects and properties mentioned. The results of the analyses are filtered through the heuristics that only the most specific objects are presented to the dialogue manager.</Paragraph>
    <Paragraph position="6"> Set operations. This grammar module  provides a marker to tell the dialogue manager what type of set operation the initiative requests, new, old or restrict. The user's utterance is searched for indicators of any of these three set operators. If no indicators are found we will assume that the operator is old.</Paragraph>
    <Paragraph position="7"> The chart is searched for the first and largest phrase that indicates a set operator.</Paragraph>
    <Paragraph position="8"> Recognizing interactional utterances.</Paragraph>
    <Paragraph position="9"> Many interactional utterances are not necessary to interpret for information retrieval systems, such as Thank you. However, Yes/Noexpressions are important. They can be recognized by looking for one of the keywords yes or no. One example of this is the utterance No, just the medium sized cars as an answer to if the user wants to see all cars in a large table. The Yes/No-grammar can conclude that it is a no answer and the property grammar will recognize the phrase medium sized cars.</Paragraph>
    <Paragraph position="10"> System/Task recognition. Utterances asking for information about a concept, e.g.</Paragraph>
    <Paragraph position="11"> Explain the numbers for rust, can be distinguished from utterances requesting information acquired from the background system How rust prone are these cars by defining keywords with a special meaning, such as explain. If any of these keywords are found in an utterance the dialogue manager will interpret the question as system-related. If not it will assume that the question is task-related.</Paragraph>
  </Section>
  <Section position="6" start_page="591" end_page="591" type="metho">
    <SectionTitle>
5 An example
</SectionTitle>
    <Paragraph position="0"> To illustrate the behaviour of the system consider an utterance such as show cars costing less than 100000 crowns. The word cars indicates that the set operator is new. The relational expression will be interpreted by the grammar  This results in two analyses \[Aspect: price\] and \[Aspect: price, Value: \[Relation: less, Arg: 100000\]\] which, when filtered by the heuristics, present the latter, the most specific analysis, to the dialogue manager. The dialogue manager inspects the result and as it is a valid database request forwards it to the background system.</Paragraph>
    <Paragraph position="1"> However, too many objects satisfy the request and the dialogue manager initiates a clarification request to the user to further specify the request. The user responds with remove audi 1985 and 1988. The keyword remove triggers the set operator restrict and the objects are interpreted by the rules:</Paragraph>
    <Paragraph position="3"> This results in three objects \[Manufacturer: audi\], \[Manufacturer: audi, Year: 1985\] and \[Manufacturer: audi, Year: 1988\]. When filtered the first interpretation is removed. This is integrated by the dialogue manager to provide a specification on both Objects and Properties which is passed to the background system and a correct response can be provided.</Paragraph>
  </Section>
  <Section position="7" start_page="591" end_page="593" type="metho">
    <SectionTitle>
6 Empirical evidence for the
</SectionTitle>
    <Paragraph position="0"> approach In this section we present results on partial interpretation i for a natural language interface for the CARS-application; a system for typed interaction to a relational database with information on second hand cars. The corpus contains 300 utterances from 10 dialogues. Five dialogues from the corpus were used when developing the interpretation methods, the Development set, and five dialogues were used for evaluation, the Test set.</Paragraph>
    <Section position="1" start_page="591" end_page="592" type="sub_section">
      <SectionTitle>
6.1 Results
</SectionTitle>
      <Paragraph position="0"> The lexicon includes information on what type of entity a keyword belongs to, i.e. Objects or Properties. This information is acquired automatically from the database with synonyms added manually from the background system description.</Paragraph>
      <Paragraph position="1"> The automatically generated lexicon of concepts consists of 102 entries describing Objects 1Results on dialogue management has been presented in JSnsson (1997).</Paragraph>
      <Paragraph position="2">  and Properties. From the system description information base 23 synonyms to concepts in the database were added to the lexicon. From the Development set another 7 synonyms to concepts in the database, 12 relational concepts and 7 markers were added.</Paragraph>
      <Paragraph position="3"> The five grammars were developed manually from the Development set. The object grammar consists of 5 rules and the property grammar consists of 21 rules. The grammar used for finding set indicators consists of 13 rules. The Yes/No grammar and System/Task grammar need no grammar rules. The time for developing these grammars is estimated to a couple of days.</Paragraph>
      <Paragraph position="4"> The obtained grammars and the lexicon of totally 151 entries were tested on both the Development set and on the five new dialogues in the Test set. The results are presented in table 1. In the first half of the table we present the number of utterances where the Yes/No, System/Task and Set parameters were correctly classified. In the second we present recall and precision for Objects and Properties.</Paragraph>
      <Paragraph position="5"> We have distinguished fully correct interpretations from partially correct. A partially correct interpretation provides information on the Aspect but might fail to consider Valuerestrictions, e.g. provide the Aspect value price but not the Value-restriction cheapest to an utterance such as what is the price of the cheapest volvo. This is because cheapest was not in the first five dialogues.</Paragraph>
      <Paragraph position="6"> The majority of the problems are due to such missing concepts. We therefore added information from the Test set. This set provided another 4 concepts, 2 relational concepts, and I mark-</Paragraph>
    </Section>
    <Section position="2" start_page="592" end_page="593" type="sub_section">
      <SectionTitle>
Fully Partial
Recall Precision Recall Precision
</SectionTitle>
      <Paragraph position="0"> Test set 92,3% 79,5% 93,8% 90,6% er and led us to believe that we have reached a fairly stable set of concepts. Adding these relational and domain concepts increased the correct recognition of set operations to 95,8%. It also increased the numbers for Properties recall and precision, as seen in table 2. The other results remained unchanged.</Paragraph>
      <Paragraph position="1"> To verify the hypothesis that the concepts are conveyed from the database and a small number of dialogues, we analyzed another 10 dialogues from the same setting but where the users know that a human interprets their utterance. From these ten dialogues only another 3 concepts and 1 relational concept were identified. Furthermore, the concepts are borderline cases, such as mapping the concept inside measurement onto the database property coupd, which could well result in a system-related answer if not added to the lexicon.</Paragraph>
      <Paragraph position="2"> As a comparison to this a traditional nonpartial PATR-grammar, developed for good coverage on only one of the dialogues consists of about 200 rules. The lexicon needed to cover all ten dialogues consists of around 470 words, to compare with the 158 of the lexicon used here.</Paragraph>
      <Paragraph position="3"> The principles have also been evaluated on a system with information on charter trips to the Greek archipelago, TRAVEL. This corpus contains 540 utterances from 10 dialogues. The information base for TRAVEL consists of texts from travel brochures which contains a lot of information. It includes a total of around 750 different concepts. Testing this lexicon on the corpus of ten dialogues 20 synonyms were found.</Paragraph>
      <Paragraph position="4"> When tested on a set of ten dialogues collected with users who knew it was a simulation (cf. the CARS corpus) another 10 synonyms were found.</Paragraph>
      <Paragraph position="5"> Thus 99% of the concepts utilized in this part of the corpus were captured from the information base and the first ten dialogues. This clearly supports the hypothesis that the relevant concepts can be captured from the background system and a fairly small number of dialogues.</Paragraph>
      <Paragraph position="6"> For the TRAVEL application we have also es- null timated how many of the utterances in the corpus that can be analyzed by this model. 90,4% of the utterances can easily be captured by the model. Of the remaining utterances 4,3% are partly outside the task of the system and a standard system message would be a sufficient response. This leaves only 4,8% of the utterances that can not be handled by the approach.</Paragraph>
    </Section>
    <Section position="3" start_page="593" end_page="593" type="sub_section">
      <SectionTitle>
6.2 Discussion
</SectionTitle>
      <Paragraph position="0"> When processing data from the dialogues we have used a system for lexical error recovery, which corrects user mistakes such as misspellings, and segmentation errors. This system utilizes a trained HMM and accounts for most errors (Ingels, 1996). In the results on lexical data presented above we have assumed a system for morphological analysis to handle inflections and compounds.</Paragraph>
      <Paragraph position="1"> The approach does not handle anaphora.</Paragraph>
      <Paragraph position="2"> This can result in erroneous responses, for instance, Show rust .for the mercedes will interpret the mercedes as a new set of cars and the answer will contain all mercedeses not only those in the previous discourse. In the applications studied here this is not a serious problem. However, for other applications it can be important to handle such expressions correctly. One possible solution is to interpret definite form of object descriptions as a marker for an old set.</Paragraph>
      <Paragraph position="3"> The application of the method have only utilized information acquired from the database, from information on the system's capabilities and from corpus information. The motivation for this was that we wanted to use unbiased information sources. In practice, however, one would like to augment this with common sense knowledge on human-computer interaction as discussed in JSnsson (1993).</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>