<?xml version="1.0" standalone="yes"?>
<Paper uid="W97-1406">
  <Title>I Active and Passive Gestures - Problems with the Resolution of Deictic and Elliptic Expressions in a Multimodal System</Title>
  <Section position="2" start_page="0" end_page="0" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> This paper deals with aspects of the resolution of deictic and elliptic expressions that are related to gestures. It discusses two approaches to distinguish between deictic pointing and manipulative gestures.</Paragraph>
    <Paragraph position="1"> The resolution methods for both approaches have been implemented in a project for the development of a multimodal interface for route-planning, traffic information and driver guidance MOFA (German acronym for &amp;quot;Multimodale Fahrerinformation&amp;quot;&amp;quot;Multimodal Driver Information&amp;quot;). 1 We proved the reusability of the methods and the architecture of MOFA in a completely different domain by implementing a prototype for multimodal calendar management (the system TALKY).</Paragraph>
    <Paragraph position="2"> The input modalities supported by our system are spoken and written natural language, deictic pointing gestures and the interaction methods known from direct manipulation systems. As input devices for gestures we use either a mouse or touch screen 1For a description of MOFA see \[Streit 96\].</Paragraph>
    <Paragraph position="3"> technology. The latter allows the user to perform deictical pointing in an (almost) natural way.</Paragraph>
    <Paragraph position="4"> Our approach to multimodal systems aims at a smooth integration of a conversational communication style with the features of direct manipulation. By conversational style of multimodal communication we understand the use of natural speech supported by deictical gestures, as in the combination of the utterance &amp;quot;an dem Tag gesch~iftlich in Kalserslautern&amp;quot;, (&amp;quot;at this day in Kalserslautern for business&amp;quot;) with a pointing act to a graphical presentation of the day under consideration. In this example the gesture acts only passively, supplying the spoken utterance with additional information. In contrast gestures used in direct manipulation are active, they are expected to trigger some action by the system, e.g. clicking at an icon which represents &amp;quot;1st of May&amp;quot; opens a representation of this day. Verbal utterances that accompany or follow such gestures may be related to this gesture: * If in this context the user utters &amp;quot;um 12 Uhr in Raum 17&amp;quot; (&amp;quot;at 12 o'clock in room 17&amp;quot;) it is most likely that he specifies an appointment at the 1st of May. In this case the elliptic verbal utterance can be resolved by the manipulative gesture.</Paragraph>
    <Paragraph position="5"> * But if the user utters &amp;quot;at 2nd of May meeting with Mr. X&amp;quot; he performs an unrelated tasks.</Paragraph>
  </Section>
class="xml-element"></Paper>