NATURAL AND SIMULATED POINTING 
Dagmar Schmauks 
Sonderforschungsbereich 314 
FB 10.2 - Informatik 
Universi6it des Saarlandes 
D-6600 Saarbrficken 11 
WEST GERMANY 
CSnet: schmauks % sbsvax.uucp@ germany.csnet 
Abstract 
Referent identification in human conversation is 
performed both by describing the objects in question 
and by pointing at them. Up till now, only the linguis- 
tic component could be simulated in dialog systems. But 
recently, technical innovations have made it possible to 
'point' at the objects on a display as well. 
The paper has two intentions. First, it investigates nat- 
ural pointing in more detail and offers some possibilities 
to classify the great variety of pointing actions. Then, 
it tries to clarify the extent to which pointing by techni- 
cal means (especially mouse-clicks) can be regarded as a 
simulation of natural pointing or as a functional equiv- 
alent. Furthermore, some steps towards even more ac- 
curate simulation are briefly mentioned. 
1. Introduction 
1.1 Terminological remarks 
The term 'deixis' denotes those referential devices 
whose interpretation requires a consideration of the sit- 
uation of utterance. "Local deixis' means the specifi- 
cation of directions, places and objects relative to the 
speaker's actual orientation. The closely related top- 
ics 'anaphora' and 'text-deixis' are not treated in this 
paper. One component of deictic actions are linguis- 
tic expressions, mainly demonstrative pronouns ('this', 
'that') and adverbs ('here', 'there'), the other being ex- 
tralinguistic means, especially pointing gestures. In this 
paper, the latter are represented by the sign '/" ' 
"Natural pointing' denotes pointing actions occur- 
ring during interhuman dialog. This includes the use of 
easily available aids like pencils or pointers. "Simulated 
pohuing' means the use of technical pointing devices 
during man-computer dialog. 
Following the terminology of Clark, Schreuder, and 
Buttrick (1983), the object pointed at is called the 
'demonstratum', and the descriptive part of the accom- 
panying noun phrase (if there is one) is called the 'de- 
scriptor'. The 'referent" is the object to which the whole 
pointing act is intended to refer. 
1.2 Motivation 
In face-to-face interaction, pointing gestures are 
used frequently and efficiently. Although their referen- 
tial power is beyond any doubt, they have, up till now, 
hardly ever been treated in more detail. The disciplines 
concerned with them are mainly semiotics, linguistics 
and psychology. 
Recently, the investigation of natural pointing has 
also become interesting for the area of Artificial Intel- 
ligence. In dialog systems developed to date, objects 
could be referred to by more or less complex verbal de- 
scriptions or unique artificial identifiers only. Techni- 
cal innovations (e.g., high-resolution graphic displays, 
touch-sensitive screens, pointing devices such as mice 
or joysticks) have made it possible to simulate pointing 
gestures to various degrees as well. Multimodal input 
is both more comfortable from the user's point of view, 
and a more natural simulation of interhuman commu- 
nication. 
Therefore, several systems have been developed re- 
cently which allow the combination of verbal descrip- 
tions and pointing gestures for referent identification 
(see section 5.2). One example is the dialog system 
XTRA (a natural-language access system for expert 
systems) which is currently under development at the 
University of Saarbriicken. Its current application do- 
main is to assist the user in filling out a tax form which 
is visible on the screen. In section 5.3, XTRA's deictic 
component TACTILUS is shortly presented. The term 
"form deixis" shall henceforth denote all those pointing 
actions which are performed in order to specify regions 
or entries of a form. 
An adequate simulation of pointing gestures pre- 
supposes a thorough investigation of the regularities 
which underlie natural pointing. Therefore, the next 
three sections investigate natural pointing in more de- 
tail. Section 2 shows that pointing actions (although 
functionally similar) are not a uniform phenomenon but 
differ with respect to various aspects. Semiotics, lin- 
guistics and psychology (study of nonverbal behavior) 
are concerned with these investigations. The interde- 
pendency of describing and pointing is the topic of sec- 
tion 3. This relationship is relevant with regard to tan- 
179 
guage processing, because natural and simulated dialog 
have a lot of problems in common. More details on 
the issues discussed in section 2 and 3 are to be found 
in Schmauks (1986b). Section 4 treats the peculiarities 
of form deixis, which is the special type of deixis oc- 
curing in the XTRA system. Section 5 tries to clarify 
the extent to which technical pointing devices already 
in existence can be regarded as a simulation of natural 
pointing, or as a functional equivalent. In section 6, 
some steps towards even more accurate simulation are 
briefly mentioned. (Thus, the last two sections are in- 
teresting from a cognitive science point of view as well.) 
2. Essential features of natural pointing 
All efforts to simulate natural pointing have to 
take into account that pointing is not a uniform phe- 
nomenon. This section shows that the goal 'pointing at 
something' is achieved by a great variety of body move- 
ments. Up till now, only a small part of these can be 
simulated (see sections 5 and 6). 
2.1 The variety of pointing actions 
Pointing actions are those body movements which 
are performed by a speaker to direct the hearer's at- 
tention to some part of the shared visual field. In the 
normal case, both for their encoding and their reception 
by the hearer no other means than the human body are 
involved. Successful reference by pointing requires that 
the addressee pays attention visually to the person who 
is pointing. One may suppose, therefore, that linguistic 
material such as demonstrative pronouns or deictic ad- 
verbs serve as a request to turn one's face to the speaker. 
Pointing can be performed by various body move- 
ments, mainly gestures. The most frequent one is the 
'finger point', by which the index finger is extended in 
the direction of the object or place indicated. A much 
more vague gesture is pointing with the thumb over 
one's shoulder. Other extralinguistic reference devices 
are head movements and line of sight. 
All these actions are only interpretable as 'Look 
there!' if the speaker uses a body movement which be- 
longs to the stock of signs s/he shares with the hearer. 
For example, the African 'mouth point' (Kirk, Burton 
1981) will not cause the intended reaction on the part 
of a European hearer. 
2.2 'Visual' and 'tactile' pointing 
Sometimes it is possible not only to point to an ob- 
ject, but also to touch an object within reach. In these 
cases, pointing becomes much more precise, because 
some of the ambiguities of natural pointing are dropped 
(see sections 3.3 and 4.1). If there is a physical con- 
tact between finger (or pencil etc.) and the indicated 
object, the action in question is called "tactile pointing' 
as opposed to 'visual pointing' where there is no such 
contact. 
So far. only a small subset of naturally occurring 
pointing gestures can be simulated on a terminal screen, 
namely certain kinds of tactile pointing gestures. The 
emphasis of the remainder of this paper will therefore 
rely upon this type of deictic gesture and its relation 
to verbal descriptions. However, many observations to 
follow will also hold for pointing gestures in general. 
Tactile pointing gestures can be classified according 
to various aspects of their observable appearance. Some 
distinguishing characteristics are: 
- body parts involved in execution of the ges- 
ture, i.e. number and position of fingers, 
- presence or absence of visual guidance, 
- use of aids (pencil, pointer .... ), 
- complexity of movement (singular, repeated, 
multiple pointing), and 
- duration and intensity of gesture. 
An adequate simulation of tactile pointing has to take 
into account at least some of these features. 
2.3 The relationship between pointing gesture 
and demonstratum 
One open problem is whether there are correlations 
between the physical features of pointing gestures and 
the objects thereby indicated. Up till now, it cannot be 
taken for granted that different persons point in an iden- 
tical manner at objects of a specific size, location, shape, 
depth of embedding etc. Empirical investigations are 
currently being carried out in the XTRA project to an- 
swer these questions. 
Pointing is called 'punctual', if the movement of the 
arm reaches only one apex and thus indicates one sin- 
gle point in space. This gesture is only adequate if the 
demonstratum is relatively small and motionless. Dur- 
ing non-punctual pointing actions, the apex itself per- 
forms a complex motion which corresponds in various 
ways to the object in question, e.g. follows its motion, 
gives its shape or indicates the part of space the object 
is supposed to be in. 
Furthermore, pointing gestures differ in accuracy. 
Pointing with a pencil, pointer etc. can be more precise 
than pointing with a finger or the whole hand. 
3. The interdependency of describing and point- 
ing 
In face-to-face interaction, objects are frequendy 
referred to by gestures and speech in parallel. Simu- 
lation of this multimodal process presupposes the inves- 
tigation of the specific limitations of each component 
and the advantages of their combination. This is done 
in the following section. 
180 
There exist both functional and temporal, relations 
between gestures and phrases. Gestures can substitute, 
repeat, contradict, modify or amplify the vocal output 
(Scherer 1979). Pointing gestures usually amplify deic- 
tic expressions and therefore belong to the kind of ges- 
tures called 'illustrators' (Ekman, Friesen 1969). Nor- 
really, pointing gestures and their correlated phrases are 
produced simultaneously (Levelt, Richardson, and La 
Heij 1985). 
3.1 Obligatory and optional pointing gestures 
Some deictic expressions must be accompanied by 
a pointing action (or a linguistic equivalent, Sennholz 
1985). These include: 
- demonstrative pronouns: 'this book', 
- heterodeictic lncal adverbs: 'the tree there', 
- personal pronouns with deictic function: 'he 
did it', and 
- 'such'-constructions: 'I hke such flowers'. 
Syntactically, obligatory pointing gestures are embed- 
ded in noun phrases or adverbial phrases. In the former 
case, they amplify a linguistic attribute. Within its cor- 
responding phrase, the location of the pointing gesture 
is arbitrary. Usually, it will accompany the most em- 
phasized expression. 
A lot of expressions can be accompanied by pointing 
gestures, in principle all those which refer to visible ob- 
jects, events etc. OptionM pointing gestures have var- 
ious functions, e.g. to mark whether the speaker uses 
adverbs deictically or relative to another orientation sys- 
tem. 
3.2 Pointing simplifies describing 
The use of purely verbal descriptions can fail for 
various~.reasons. For example, some descriptions may 
not completely specify their referents: They can be 
wrong, inconsistent or too subjective. But even ade- 
quate descriptions can cause misinterpretations. One 
extreme would include descriptions with little inten- 
sion and therefore too wide an extension, such 'whatsit' 
or 'thingamajig' (generally used if one doesn't know a 
more precise descriptor). The other extreme includes 
very detailed and complex descriptions which are dif- 
ficult to process (e.g., 'the small red book on the left 
side in the second shelf from the top'). A closely related 
problem is that of technical terms used in conversation 
with non-specialists: Although the description may be 
totally adequate, the hearer is not able to understand it. 
Therefore, verbal description alone may be too gen- 
eral or too specific. Within this range, the speaker has 
the task of specifying the referent in enough detail with- 
out constructing a verbal expression which is too com- 
plex. 
One frequent solution is the use of pointing gestures. 
They allow successful reference without the need of to- 
tally specified verbal descriptions (Pechmann, Deutsch 
1982). The use of pointing shortens the accompany- 
ing descriptor and the loss of intension is compensated 
by the gesture. General nouns amplified by pointing 
gestures can substitute for more specific nouns (e.g., 'I 
like cornflowers' is replacable by 'I like these / flow- 
ers'). Thus, additional pointing allows unambiguous 
(or at least relatively precise) referent specification even 
if one doesn't know an exact descriptor. The process of 
referent identification is speeded up, because the orien- 
tation to the object's direction and the processing of the 
verbal description are performed simultaneously. 
3.3 Describing disambiguates pointing 
One essential drawback of pointing gestures is their 
inevitable dependency on the here-and-now. Further- 
more, pointing without describing the referent is fun- 
damentaUy ambiguous (Wittgenstein 1958). Referent 
identification involves the following three steps: First, 
one has to recognize the direction indicated. This re- 
quires facing the speaker and following his/her gesture 
with gaze and eventually a body turn. Thus, the deictic 
spaces of both participants are co-oriented by physical 
means and not by mental acts (e.g., transformation of 
'left' into 'right' and vice versa, see Klein 1978). 
The second task is the identification of the object 
indicated. Usually, there is more than one object sit- 
uated in any one direction. Problems arise if possible 
demonstrata are: 
- next to each other, 
- behind each other, or 
- embedded in one another. 
In these cases, unambiguous reference requires the 
naming or describing of the demonstratum. 
Thirdly, one has to decide what aspect of the object is 
being referred to. Like the second step, this is usually 
done by consideration of the descriptor. For example, 
pointing at a moving car can refer to its colour ('Nice 
green 7 , isn't it?') or its kind of motion (' This speed 
.1" causes lots of accidents') etc. Pointing at sets of 
objects can even refer to aspects of higher degree such 
as number ('I'd like to have that man), ,I books'). 
4. Form deixis 
Pointing at two-dimensional objects (forms, dia- 
grams, maps, pictures etc.) differs in various aspects 
from pointing at objects within the entire visual field. 
This offers a definite advantage from a linguistic point 
of view: Some problems of local deixis are reduced in 
complexity without the communicative setting having 
to become unnatural (Schmauks 1986a). Furthermore, 
181 
this domain is interesting from an artificial intelligence 
point of view, since some of the pointing actions with 
regard to forms can now be simulated on a terminal 
screen. 
4.1 Reduction of problems 
Following Bfihler's terminology (1982), form deixis 
belongs to the kind of deixis called 'demonstratio ad 
oculos', because all objects pointed at are visible. Fur- 
thermore, it represents an example of the 'canonical sit- 
uation of utterance' (Lyons 1977): All the participants 
are co-present and can thus mutually perceive their 
(pointing) gestures etc. Form deixis is relatively pre- 
cise, because tactile pointing is always possible. Precise 
pointing at small objects (e.g. single words) is frequently 
performed by using a pencil etc., larger areas by encir- 
cling them. The ambiguity with regard to objects be- 
hind each other does not occur, because the deictic space 
is only two-dimensional. If speaker and hearer are sit- 
uated side by side, their deictic fields are co-oriented. 
Therefore, this position makes cooperation easier, and 
thus is the most advantageous one. 
4.2 Remaining problems 
Although form deixis implies a reduction of prob- 
lems, referent identification has not at all become a triv- 
ial task. It cannot be taken for granted that demonstra- 
tum and referent are identical. This might be due to the 
fact that the speaker has mistakenly pointed at a wrong 
place because s/he doesn't know the referent's actual 
location or misses the target by accident. Other diver- 
gencies emerge intentionally: The speaker doesn't want 
to cover the referent and therefore points a bit lower. 
Other essential problems arise because there exist 
subset relations among form regions. For example, the 
demonstratum can be a part of the referent - this is re- 
ferred to as 'pars-pro-toto deixis'. In those cases, one 
must take into account the verbal description to resolve 
the ambiguity. 
Furthermore, pointing at one form region can (de- 
pending on linguistic context) refer to three different 
entities: 
1. The form region itself: 'What is to be en- 
tered here? 
2. The actual entry: 'I want to increase this 
sun'/. ' 
3. Correlated concepts: 'Are these expenses to 
be verified?' 
5. Simulated pointing 
This section investigates the extent to which some 
features of natural pointing can already be simulated in 
dialog systems developed to date. In section 6, some 
steps towards more accurate simulation are briefly sug- 
gested. 
5.1 Different ways of simulating pointing ges- 
tures 
Face-to-face interaction is performed by gestures 
and speech in parallel. In many domains (e.g. form 
deixis), objects are often and efficiently referred to by 
pointing gestures. Thus, dialog systems will become 
more natural if the user has the possibility of 'pointing' 
at the objects which are visible on the screen. 
The goal 'reference by pointing' can be achieved 
by various strategies. One fundamental decision must 
be made first: whether one wants to simulate natural 
pointing (as is the aim of TACTILUS) or to offer func- 
tionM equivalents. In the former case, there is the pre- 
supposed but questionable demand that man-machine- 
communication should be performed by the same means 
as interhuman communication. 
If the main emphasis relies on simulation, then the 
pointing device and its use must correspond to natural 
pointing as accurately as possible. In this case, the most 
adequate simulation will be pointing at a touch-sensitive 
screen (see section 6). But other devices (e.g. input via 
mouse-clicks) can also partially simulate natural point- 
ing (see sections 5.3). 
Functional equivalents to natural pointing include 
the following devices: Framing the referent or zooming 
in on it, highlighting it in different colours etc. (see 
Ffthnrich et al. 1984). On the one hand, the system 
can 'point' by these means. On the other hand, the 
user gets immediate teedback as to whether the system 
has recognized the intended referent. This advantage is 
paid for by the loss of 'naturalness'. 
5.2 Historical remarks 
Multimodal input, especially the possibility of 
pointing at visible objects, offers certain crucial ad- 
vantages. For example, the use of simple pointing 
actions was already possible in the following systems: 
SCHOLAR (Carbonell 1970) allows pointing gestures 
in order to specify regions of geographic maps. Point- 
ing in Woods' (1979) system, combined with simple 
descriptions, refers to substructures of a parse tree dis- 
played on the screen. In NLG (Brown et al. 1979), the 
user can draw simple geometric objects through descrip- 
tive NL-commands and simultaneous tactile touches 
on the screen. SDMS (Bolt 1980)enables the user to 
create and manipulate geometric objects on a screen- 
arrangement called 'MEDIA ROOM'. In all those sys- 
tems, there exist predefined relations between the point- 
ing gesture and its demonstratum. Referent identifica- 
tion is not dependent on context etc. 
Currently, several projects are investigating prob- 
lems concerning the integration of pointing actions 
and NL input, e.g.: In NLMENU (Thompson 1986), 
the user can select parts of a street map by means 
182 
of a mouse-controlled rubber-band technique. Hayes 
(1986) oudines the integration of a deictic component 
into the Language Craft System, which should allow 
the user to click on items on the screen, e.g. the ma- 
chines on a blueprint of a factory floor. ACORD in- 
vestigates pointing actions with respect to various two- 
dimensional objects, e.g. a map of the planetary system 
(Hanne, Hoepelmann, and F~ihnrich 1986) and a form 
for university registration (Wetzel, Hanne, and Hoe- 
pelmann 1987). 
5.3 Pointing actions in TAGTILUS 
One aim of XTRA is the integration of (typed) ver- 
bal descriptions and pointing gestures (currently real- 
ized by mouse-clicks) for referent identification (Kobsa 
et al. 1986). The user should be able to efficiently 
refer to objects on the screen, even when s/he uses 
underspecified descriptions and/or imprecise pointing 
gestures (Allgayer, Reddig 1986). Hence the process 
of specifying referents is speeded up and requires less 
knowledge of specialist terms. 
The deictic component of XTRA (called TAC- 
TILUS) is completely implemented on a Symbolics Lisp 
Machine (Allgayer 1086). It offers four types of point- 
ing gestures which differ in accuracy. They correspond 
to three modes of punctual pointing (with pencil, in- 
dex finger, or hand) and to the possibility of encircling 
the demonstratum. Thus, pointing becomes a two-step 
process: First, one has to select the intended degree of 
preciseness and then to 'point'. 
These pointing actions are natural because of their 
ambiguity: There is no predefined relation between the 
spot where the mouse is activated and the object which 
is thereby referred to. Therefore, the system has to take 
into account additional knowledge sources for referent 
identification, e.g. verbal descriptions and dialog mem- 
ory. From the user's point of view, the essential indi- 
cation of this naturalness is the lack of visual feedback. 
In analogy to natural pointing, the identified referent is 
not highlighted. 
5.4 Problems in processing mixed input 
One essential problem is to assign a mouse-click to 
its corresponding verbal constituent. This task is not 
trivial since there is no guarantee that the user 'points' 
within the range of the deictic expression. Possibly, the 
click occurs too late because of the user being inatten- 
tive, not familiar with the system etc. One example is: 
What is this sum above the last entry/" ? 
Here, the pointing action occurs next to 'the last entry'. 
But this is an anaphor and doesn't need to be amplified. 
On the other hand, there is the deictic expression 'this 
sum' without its correlated obligatory pointing action. 
Therefore, the system has to recognize that '/ ' be- 
longs to 'this sum'. This problem is aggravated by the 
fact that the words 'here'/'there' and 'this'/'that' are 
not only the most frequent deictic expressions but have 
anaphoric and text- deictic readings as well. 
Matching mouse-clicks and phrases becomes even more 
difficult if a single utterance requires more than one 
pointing action. This case is called 'multiple pointing'. 
Examples include: 
This sum I would prefer to enter here. 
Hayes (1986) assumes that pointing actions are per- 
formed in the same order as their corresponding 
phrases. But until this hypothesis is confirmed empiri- 
cally, it can only serve as a heuristic rule. 
As soon as reference by pointing is possible, the use of 
incomplete expressions will increase. In these cases, ad- 
ditional knowledge sources are needed for referent iden- 
tification, like descriptor analysis and case frame analy- 
sis (Kobsa et al. 1986). For example, the expression 
'this' in the sentence 'I want to add Sis/" ' surely refers 
to a number in the present domain, because 'add' is cat- 
egorized as an action to be performed with numbers. 
5.5 Problems in generating mixed output 
If the pointing actions of the system are also con- 
ceived as a simulation of natural pointing, the user is 
confronted with the same problems that have already 
been identified in the last subsection (Reithinger 1987). 
But, whereas multiple pointing can be simulated during 
input, there seems to be no adequate mode for simulat- 
ing it during output as well: In normal communication, 
the hearer doesn't need to watch the speaker in order to 
understand him/her unless the occurence ofa deictic ex- 
pression (or the sound of touching during tactile point- 
ing) demands his/her visual attentiveness. Also, during 
typed dialog, there is no need to observe the output sen- 
tences permanently. In the case of multiple pointing, 
the possibility cannot be ruled out that the user might 
fail to notice one of the pointing actions. 
6. Prospects of more natural simulation 
Up till now, only certain kinds of tactile pointing 
gestures can be simulated on a screen. Negroponte 
(1981) oudines some future plans, e.g. the considera- 
tion of non-tactile actions such as eye tracking and body 
movements. 
Simulation of tactile pointing gestures by mouse- 
clicks has some serious limitations with regard to its 
'naturalness'. Empirical investigations are needed to 
determine the extent to which mouse-clicks can be re- 
garded as an equivalent of natural pointing. These 
investigations are currently carried out in the XTRA 
project. 
183 
In the case of natural pointing, the choice of a more 
or less precise pointing gesture is made automatically 
rather than consciously. But in TACTILUS, the user 
has tc, select explicitly the intended degree of accuracy. 
Empirical investigations must examine whether the user 
regards this as a disadvantage. 
Furthermore, pointing via mouse-clicks differs from 
natural tactile pointing, because there is no physical 
contact between finger and demonstratum. A better 
solution would be the use of a touch-sensitive screen 
on which 'real-world gestures' (see Minsky 1984) are 
possible. Touch-sensitive screens allow highly natural 
pointing gestures (see Picketing 1986), but have some 
shortcomings, e.g. a restricted degree of resolution. 
A problem just as serious as the aforementioned is 
the temporal dissociation of a pointing gesture and its 
corresponding phrase. This problem would be soluble if 
the system would accept input via voice. But this alone 
wouldn't be sufficient: There is no guarantee that spo- 
ken phrases and correlated mouse-clicks occur simul- 
taneously. Furthermore, current voice-input systems 
have too small a vocabulary and cannot process fluent 
speech. 
Therefore, the most adequate simulation would be 
the combination of voice input~output and gestures on 
a touch-sensitive screen. However, the state of the art 
with respect to the required devices is not yet sufficient. 
Acknowledgements 
The research described in this paper has been 
funded by the German Science Foundation (DFG) in its 
Special Collaborative Program on AI and Knowledge- 
Based Systems (SFB 314). I am indepted to my col- 
leagues of the XTRA project for their helpful comments 
on an earlier version of this paper. 
References 
Allgayer, J. (1986): Eine Graphikkomponente zur 
Integration yon Zeigehandlungen in natfirlich- 
sprachliche KI-Systeme. Proceedings der 16. GI- 
Jahrestagung. Berlin etc.: Springer. 
Allgayer, J. and C. Reddig (1986): Processing De- 
scriptions containing Words and Gestures. A Sys- 
tem Architecture. In: C.-R. RoUinger, Hrsg.: 
GWAI/OGAI 1986. Berlin etc.: Springer. 
Bolt, R. A. (1980): 'Put-That-There': Voice and 
Gesture at the Graphics Interface. Computer 
Graphics 14,262-270. 
Brown, D. C. et al. (1979): An Experimental 
Graphics System with Natural Language Input. 
Computer and Graphics 4, 13-22. 
Bfihler, K. (1982): The Deictic Field of Language 
and Deictic Words. Abridged translation of K. 
B/.ihler (1934): Sprachtheorie, part 2, chapters 7 
and 8. In: R. J. JarveUa and W. Klein, eds.: 
Speech, Place, and Action. Chichester etc.: Wi- 
ley. 
Carbonell, J. R. (1970): Mixed-Initiative Man- 
Computer Dialogues. Cambridge, MA: Bolt, Be- 
ranek and Newman. 
Clark, H. H., R. Schreuder and S. Buttrick 
(1983): Common Ground and the Understanding 
of Demonstrative Reference. Journal of Verbal 
Learning and Verbal Behavior 22, 245-258. 
Ekman, P. and W. V. Friesen (1969): The Reper- 
toire of Nonverbal Behavior: Categories, Origins, 
and Coding. Semiotica 1, 49-98. 
F~ihnrich, K. P. et al. (1984): The Role of 
Graphics. Technical Report E3/GR, FhG, IAO, 
Stuttgart. 
Hanne, K. H., J. P. Hoepelmann und K. P. 
Ffihnrirh (1986): Combined Graphics/Natural 
Language Interfaces to Knowledge Based Sys- 
tems. Proceedings of the Artificial Intelligence 
and Advanced Computer Technology Confer- 
ence, Wiesbaden, West Germany. 
Hayes, P. J. (1986): Steps towards Integrating 
Natural Language and Graphical Interaction for 
Knowledge-based Systems. Proceedings of the 
7th European Corlf,-rence on Artificial Intelli- 
gence, Brighton, England. 
Kirk, L. and M. Burton (1981): Physical ver- 
sus Semantic Classification of Nonverbal Forms: 
A Cross-Cultural Experiment. In: A. Kendon, 
ed.: Nonverbal Communication, Interaction, and 
Gesture. 
184 
Klein, W. (1978): Wo ist hier? Pr~iliminarien zu 
einer Untersuchung der lokalen Deixis. Lingui- 
stische Berichte 58, 18-40. 
Kobsa, A. et al. (1986): Combining Deictic Ges- 
tures and Natural Language for Referent Iden- 
tification. Proceedings of the llth International 
Conference on Computational Linguistics, Bonn, 
West Germany. 
Levelt, W. J. M., G. Richardson and W. La Heij 
(1985): Pointing and Voicing in Deietic Expres- 
sions. Journal of Memory and Language 24, 133- 
164. 
Lyons, J. (1977): Semantics, Vols 1 and 2. Cam- 
bridge: Cambridge University Press. 
Minsky, M. (1984): Manipulating Simulated Ob- 
jects with Real-world Gestures using a Force and 
Position Sensitive Screen. Computer Graphics 
18, 195-203. 
Negroponte, N. (1981): Media Room. Proceedings 
of the Society for Information Display 22, 109- 
113. 
Pechmann, T. and W. Deutsch (1982): The De- 
velopment of Verbal and Nonverbal Devices for 
Reference. Journal of Experimental Child Psy- 
chology 34, 330-341. 
Pickering, J. A. (1986): Touch-sensitive screens: 
the technologies and their application. Int. J. 
Man-Machine Studies 25,249-269. 
Reithinger, N. (1987): Generating Referring Ex- 
pressions and Pointing Gestures. To appear in: 
G. Kempen, ed.: Natural Language Generation. 
Dordrecht: Kluwer 
Scherer, K. R. (1979): Die Funktionen des Non- 
verbalen Verhahens im Gespr/ich. In: K. R. 
Scherer und H. G. Wallbott, Hrsg.: Nonverbale 
Kommunikation. Weinheim/Basel: Beltz. 
Schmauks, D. (1986a): Formulardeixis und ihre 
Simulation auf dem Bildschirm. Ein Oberblick 
aus linguistischer Sicht. Memo Nr. 4, Sonder- 
forschungsbereich 314, Dept. of Computer Sci- 
ence, University of Saarbralicken, FR Germany. 
Schmauks, D. (1986b): Form und Fun.ktion von 
Zeigegesten. Ein interdisziplin/irer Uberblick. 
Bericht Nr. 10, Sonderforschungsbereich 314, 
Dept. of Computer Science, University of 
Saarbriicken, FR Germany. 
Sennholz, K. (1985): Grundzfige der Deixis. 
Bochum: Brockmeyer. 
Thompson, C. (1986): Building Menu-Based Nat- 
ural Language Interfaces. Texas Engineering 
Journal 3, 140-150. 
Wetzel, R. P., K. H. Hanne and J. P. Hoe- 
pelmann (1987): DIS-QUE: Deictic Interaction 
System-Query Environment. LOKI Report KR- 
GR 5.3/KR-NL 5, FhG, IAO, Stuttgart. 
Wittgenstein, L. (1958): Philosophical investiga- 
tions. Oxford: Blackwell. 
Woods, W. A. et al. (1979): Research in Natural 
Language Understanding: Annual Report. TR 
4274, Bolt, Beranek and Newman, Cambridge, 
MA. 
185 
