Integrating Natural Language Components into Graphical Discourse 
Stephan Dilley, John Bateman, Ulrich Thiel, Anne Tissen 
GMD, Integrated Publication and Information Systems Institute (IPSI) 
DolivostralSe 15, D-6100 Darmstadt, FRG 
e-maih { bateman,dilley,thiel,tissen }@darmstadt.gmd.de 
Abstract 
In our current research into the design of cognitively well-mo- 
tivated interfaces relying primarily on the display of graphical 
information, we have observed that graphical information 
alone does not provide sufficient support to users particu- 
larly when situations arise that do not simply conform to the 
users' expectations. This can occur due to too much informa- 
tion being requested, too little, information of the wrong kind, 
etc. To solve this problem, we are working towards the integra- 
tion of natural language generation to augment the interaction 
functionalities of the interface. This is intended to support the 
generation of flexible natural language utterances which pin- 
point possible problems with a user's request and which fur- 
ther go on to outline the user's most sensible courses of action 
away from the problem. In this paper, we describe our first pro- 
totype, where we combine the graphical and interaction plan- 
ning capabilities of our graphical information system SIC! 
with the text generation capabilities of the Penman system. We 
illustrate the need for such a combined system, and also give 
examples of how a general natural language facility beneficial- 
ly augments the user's ability to navigate a knowledge base 
graphically. 
I Introduction 
Natural language interfaces to information systems allow the 
user to converse with the system in a natural way without being 
restricted by the syntax of a formal query language. However, 
users often have difficulties in stating their information need 
precisely (cf. Brooks et al., 1986) and so a user interface design 
emphasizing multimedia aspects of information systems may 
be more appropriate. An object oriented presentation meets 
this requirement because users are allowed to investigate in- 
formation items directly. Thus, the distance between the users' 
intentions and the objects they involve is reduced to a mini- 
mum (cf. Hutchins et al., 1986). However, visual presentations 
of abslract information items have to be designed carefully to 
support the user in perceiving them correctly. Often the in- 
herent limitations of available presentation forms, e.g. the 
number ofdisplayable items, are an impediment to an adequate 
visualization. These difficulties which are well known in visu- 
al languages can be attacked by combining the presentation of 
graphical objects to the user with cooperative system re- 
sponses in natural languages. 
Our approach to human-computer interaction provides 
the basis for an integration of different interaction styles, in our 
case natural language and graphics, in a multi-modal informa- 
tion system. The choice of an appropriate combination of 
modes is essential for a successful interface design. As there is 
not yet a complete theory of multi-modal interaction, this ques- 
tion has to be answered experimentally. Therefore, we propose 
a modular architecture for multi-modal interfaces composed 
of interaction tools. In this paper, we discuss the addition ot 
natural language generation to the interaction tools provided. 
This capability is embodied in the prototype IGiNG (_Integrat- 
ing Graphics and Natural Language Generation) which com- 
bines 
• the graphical information system SIC! (System for In- 
formation on Conferences) comprising a dialog man- 
ager which is able to track the user's goals and, conse- 
quently, to plan system reactions as cooperative 
responses (Tissen, 1991), a database access module 
providing relevant data (Kracker, 1991) and apresen- 
tation manager generating visualizations of the re- 
trieved information items (Kerner and Thiel, 1991). 
• the text generation component KOMET/Penman 
(Bateman et al., 1991; Penman, 1989), comprising ex- 
tensive grammatical, semantic, and text organization- 
al linguistic resources. 
The paper is organized as tbllows: first, we describe relate( 
work and how our approach goes beyond this, second, w~ 
introduce the basic components of our system and explain hov 
they can be appropriately interfaced. Then, we go on to discus: 
the benefits that including a natural language component caJ 
bring for augmenting the coherence and functionality of~ 
user's interaction with the system. We offer some examples o 
helpful responses that the system can make which would bq 
difficult, if not impossible, to represent graphically in a non a, 
hoc fashion. Finally, we describe some possible future direc 
tions. 
  
 72 
2 Related Work 
One body of related work has primarily intended to coordinate 
different modes of expression within a framework of natural 
communication (cf. Hayes, 1987; Neal and Shapiro, 1988; Co- 
hen et al., 1989, Feiner and McKeown, 1990; Bandyopadhyay, 
1990; S tock, 1991; Wahlster et al., 1991). The principle effort 
here is to ascertain factors that can motivate the distribution of 
information across different modes (e.g. Arens and Hovy, 
1990). A further body of related work moves towards prob- 
lems of interaction by exploring the potential of the combina- 
tion of natural language and deictic gestures (cf. Allgayer et 
al., 1989; Moore and Swartout, 1990). In a similar vein, ap- 
proaches to flexible graphical interaction based on the con- 
versationalmetaphor (cf. Reichman, 1986, 1989; Thiel, 1990) 
treat user inputs such as mouse clicks, menu selections, etc. not 
as invocations of methods that can be executed without regard- 
ing the dialog context, but instead as dialog acts expressing a 
discourse goal of the user. The direct manipulation of an object 
then becomes itself a part of the dialog of the user with the sys- 
tem, meaning that the system can respond in a more flexible 
way by taking into account the illocutionary and semantic as- 
pects of the user's input. Related work from generation in- 
cludes the correction of misconceptions in the work of McCoy 
(1986) and the explicit representation of information about a 
system's own knowledge and planning activities that is found 
in the Explainable Expert Systems system of Swartout and 
Smoliar (1987). None of this work, however, addresses the 
problems of meta-dialog concerning graphically supported in- 
teraction. 
In our approach we bring the kind of natural language ca- 
pabilities required by the first body of related work (i.e., graph- 
ical and natural language information functioning together) to 
bear on the kinds of problems that arise in the second body of 
related work when the direct manipulation of objects by a user 
creates goals that the system cannot fulfil. Here we must not 
only respond to the user's attempt to manipulate an object or 
the user's deictic gesture as a dialog act, but also be able to en- 
gage in a meta-interaction to debug that act if it creates a prob- 
lematic situation. We show that natural language possesses 
properties that make it preferable over the graphical mode of 
expression for such meta-interaction and hence natural lan- 
guage generation needs to be supported even in graphics-ori- 
ented interfaces. 
3 The SIC!-System 
We demonstrate the combination of graphics ,and natural lan- 
guage output in the context of the SIC! system, SIC\[ is imple- 
mented using HyperNews (HyperNews, 1989), a hypermedia- 
like user-interface management system which can be 
controlled from aLISP-client. SIC! offers information on con- 
ferences; its domain consists of abstract information on con- 
ferences including workshops, tutorials, persons, institutes, 
and conference topics. A user who wants to obtain information 
from SIC! poses a query whereafter he can inspect the data re- 
trieved. This is done by selecting a presentation form. In SIC! 
we use several cognitively motivated presentation forms 
(Kemer and Thiel; 1991). 
One of these forms is the ring presentation form (cf. Fig- 
ure 1 ) which surveys a structure of given data. All items are 
positioned on a virtual ring structure, the relations between the 
concepts being presented as single lines. In our example the 
ring presentation form contains the categories workshop and 
topic. The concepts of the workshops are on the left side of the 
figure, those of the topics on the right. However, this presenta- 
tion form is clearly limited with respect to the quantity of data 
that can be presented simultaneously. 
Consider, for example, the following situation: A user 
asks for a subset of the IJCAI89 workshops and their related 
topics. SIC! retrieves three workshops and several topics. If 
the user wants to get an overview of the data's structure, then 
for this goal the ring is the most adequate presentation form 
available in SIC! But this causes a problem when the ring form 
cannot display as many data as SIC! retrieved. There are sever- 
al ways of solving this problem, depending on the user's main 
Q Int Interfaces 
(0o Programming) 
( WS on graphs ( ) 
Figure I : 
Ring presentation form 
  
 73 
point of interest. If he wants to see all the data he might select a 
presentation form that can show unlimited amounts of in- 
formation, e.g. the table presentation form (cf. Figure 2 ). If he 
Workshops and their topics: \] 
Workshops 
O0 Programming 
Int. Interfaces 
WS on graphs 
Topics 
Kn-E ngineefinD, 10 
Prg-Langua ge*, I 
Learning, I Kn-Enginecring, i 
I 
Kn-Engineering~ I 
P-Classes, 
Figure 2 : Table presentation form 
is interested more in the structure of the data, he might change 
his initial query so that less data are retrieved. But, importantly, 
the user cannot be aware of the details of the current situation 
of retrieval and its implications simply from the graphical in- 
formation displayed. Hence the system has to inform him of 
the situation and offer possible alternatives. This needs to be 
done in a way that enables users to grasp the situation and to 
choose the appropriate alternatives for their purpose. To 
achieve this, the system has to create a coherent informative 
act that is concise and yet unambiguous (in context), giving all 
the information necessary for the user to determine his future 
actions. 
Based on work by Feiner and McKeown (1990) on the 
coordination of text and graphics in explanation generation 
and by Lombardi (1989), who examined the assignment of in- 
formation to media, we assume that text is the appropriate me- 
dium for informative acts in meta-dialogs, since a well- 
constructed text is not only concise and easy to understand, but 
also guarantees the necessary flexibility to meet any situation 
that ,arises. Graphical 'acts' cannot be constructed composi- 
tionally to express possibly unforeseen complex circum- 
stances: novel graphics must first be learnt by users--a situa- 
tion avoided by the generation of situationally appropriate 
natural language. Thus, generating natural language text, par- 
ticularly text involving controlled and appropriate deployment 
of text-forming resources such as rhetorical relations, en- 
hances the total coherence of the user's dialog with the system. 
Our hypothesis is that the user's understanding of the situation 
and its implications is increased by the natural language out- 
put, which becomes an intermediary between the various pos- 
sibilities for information presentation. 
4 KOMET/Penman Text Generation System 
We are using the KOMET/Penman I system (Mann and Mat- 
thiessen, 1983) for generating the natural language output our 
system requires. KOMET/Penman is a domain-independent 
text generation system based on systemic-functional grammar 
(Halliday, 1978). It consists of extensive grammars of English 
and German (Matthiessen, 1990; Teich, 1991), a linguistically 
motivated ontology, called the Upper Model (Bateman, Kasp- 
er, Moore and Whitney, 1989), a semantic interface that relates 
the categories of the conceptual ontology with their possible 
grammatical expressions in English and German (Matthies- 
sen, 1990), and a basic lexicon containing English and German 
closed-class items and default lexical realizations for the con- 
cepts in the Upper Model ontology. The definition of the lexi- 
cal items includes morphological information and sets of lexi- 
cal features that determine the grammatical contexts in which 
items are to be selected. 
The Upper Model is the component of the system that is 
primarily responsible for mediating between the knowledge 
specific to any given domain and the general lexical and gram- 
matical expressions that are provided by a language. Because it 
is possible to state how any particular Upper Model concept is 
to be realized, subordinating domain concepts to particular 
Upper Model concepts causes those domain concepts to inherit 
appropriate forms of expression. For example, concepts from 
the object-class are usually realized as nominal phrases, while 
concepts from theprocess-class (e.g., mental-process, verbal- 
process, action-process, relation-process) are often realized by 
clauses 2. The relationship between Upper Model and domain 
model is diagrammed in the context of its application for SIC! 
in Figure 3. 
Input to the KOMET/Penman text generation system is 
given in terms of the Sentence Plan Language (Kasper, 1989), 
of which we will see examples below. An SPL expression de- 
fines the semantic content of a sentence to be generated; it con- 
sists of a set of typed variables and relations defined between 
those variables. Both the types and the possible relations are 
defined either by the Upper Model directly or by concepts or 
relations in the domain model that have been subordinated to 
the Upper Model. In addition to this information, SPL expres- 
sions may also contain direct statements in terms of the gram- 
roar's semantic interface -- in practical applications these 
latter are often abbreviated by use of macros (e.g. :tense pres- 
ent) or are defaulted. 
1 The original Penman system was developed at the Information 
Science Institute of the University of Southern California; the 
KOMET system of GMD/IPSI builds on this, working towards 
multilinguality and enhanced text planning capabilities. 
z But not always: the existence of, for example, nominalizations 
motivates the maintenance of two distinct levels of representa- 
tion, 
  
 74 
~i~i;i::iiiii--i;i;:~::.::::::::iiii#ii~:~::;~iiiiii.:i.:i~.:.:~j::: I 
!ii 7 
. 
Figure 3 : Interfacing SIC! with KOMET/Penman on the Knowledge Level 
5 Interfacing SIC! with the Text Generation 
System 
To interface SIC! with KOMET/Penman we have to provide 
several types of knowledge (cf. Figure 3 ) 
• A domain model, which is a taxonomy of knowledge 
specific to our application-domain. We split the do- 
main into two parts: an Information-Domain (I-Do- 
main), which contains concepts related to the informa- 
tion that is shown by SIC!, e.g. workshops and topics 
(cf. Figure 1 ), and a Presentation-Domain (P-Do- 
main), which contains concepts related to the way this 
information is presented by SIC!, e.g. ring, table. By 
splitting the domain model we increase the adaptabil- 
ity in case of changes in the underlying application do- 
main, e.g. replacing the conference knowledge base 
with a knowledge base on research projects. Every 
concept in the domain model has to be linked to some 
Upper Model concept from which it inherits attributes 
which enable KOMET/Penman to express the concept 
in a way that is grammatically correct. The I-Domain 
concepts can be generated automatically flom the un- 
derlying SIC! knowledge bases (cf. Figure 3 ). Con- 
cepts can also be associated to lexical items. 
• A domain lexicon, containing the definitions of lexi- 
cat items of all the words that may appear in the ap- 
plication domain. 
6 Creating the Natural Language Output 
6.1 Planning Sentences 
As stated in our example above, we want to produce text that, 
in this case, informs the user that not all the information that 
was requested can be shown because the current presentation- 
form's capacity is limited. Furthermore, we need to offer pos- 
sible actions which solve this problem. In Figure 4, we show 
the semantic input to KOMET/Penman, expressed in SPL, that 
would cause KOMET/Penman to generate the first sentence 
I (a / ascription :domain (c / capacity :owned-by (p / presentation-form)) :range (e / exceeded)) 
Fig. 4 : SPL-Plan for 
"The presentation-form's capacity is exceeded." 
that we require: i.e., "The presentation-form's capacity is ex- 
ceeded." One type of abstract concept that the system requires 
is the status of a particular entity that may be displayed or used. 
Possible statuses are, for presentation-forms, exceeded, in- 
complete. These status concepts can then be attributed to ob- 
jects by means of the Upper Model relation ascription, which 
has roles ':domain' and ':range'. They represent the concepts 
which are related, in our example the presentation-form's ca- 
pacity and exceeded. In general, ':domain' contains the essen- 
tial concept of the relation while ':range' contains additional 
information. The P-Domain concept capacity has been mod- 
eled as an object. 
6.2 Using Rhetorical Relations 
Figure 5 shows a more complex SPL-plan which demon- 
strates some of the more advanced possibilities given by KO- 
MET/Penman. The most interesting aspect in this plan is the 
use of rhetorical relations based on Rhetorical-Structure- 
Theory (RST). 
RST is a theory of the organization of natural language 
texts (Mann and Thompson, 1987). Mann and Thompson stu- 
died a wide variety of English texts and observed that there are 
approximately 25 relations that usually occur between co- 
herent portions of English text. An RST relation consists of 
two parts, a nucleus and a satellite. The nucleus is that part that 
is most essential to the speaker's purpose, while the satellite 
contains additional information. The satellite is more easily re- 
placed than the nucleus because of the nucleus' central role in 
the thematical progression of the discourse. Even though there 
are some critics questioning the use of rhetorical relations in 
discourse structure theory (Grosz and Sidner, 1986) we use 
  
 75 
(rl /rst-nonvolitional-result 
:domain (el/ existence 
:domain (cl / concept 
:number mass 
:process (r2/show 
:saying cl 
:speechact denial 
:tense present))) 
:range (a/ascription 
:domain (c2 / capacity 
:owned-by (p/pres-form)) 
:range (ex / exceeded) 
:tense present))) 
Figure 5 : SPL-Plan for "There are concepts that are not 
shown, because the presentation-form's capacity is exceeded". 
RST relations because they proved to be quite useful when we 
link portions of information. In KOMET/Penman, RST-rela- 
tions are treated the same way as other relations, e.g. ascription 
which we used in the plan shown in Figure 4. 
The SPL-plan shown in Figure 5 combines two relations: 
the ascription-relation, which we used in the SPL-plan in Fig- 
ure 4, and the existence-relation. Existence is a so called one- 
place-relation, because it contains only a :domain-role but no 
:range. It is usually realized as "There is ...", where :domain 
defines what exists. We link these two relations via an RST- 
relation called rst-nonvolitional-result. This RST-relation im- 
plies that the nucleus, which is defined in our :domain-role is a 
result of the satellite, defined in :range. One possible output is 
<domain>, because <range>, in our case "<There are concepts 
• ..>, because <... capacity is exceeded>". Because what is de- 
fined in the :domain ("There are concepts that are not shown") 
is not volitional, we use rst-nonvolitional-result instead of rst- 
volitional-result. The fact that there is data that is not shown by 
the current presentation-form is essential to our informational 
purpose. Therefore this fact becomes the nucleus (represented 
by :domain) of our plan. 
RST-relations which ensure the connectivity between our text 
segments. Pragmatic coherence is supported by the mere fact 
that we are using text as a medium for meta-dialogs, as these 
are difficult to understand on a graphical level. 
7 Controlling Multimodal Discourse 
The dialog manager is one of the main components of our in- 
terface system (cf. Figure 6 ). It chooses interaction modes 
(graphic or text) and controls the navigation or exploration in 
the information space. 
In order to prevent the user from 'being lost in hyper- 
space', we guide the user by case-based dialog plans (Tissen, 
1991). In a case-based planning system a new plan will be gen- 
erated by retrieving the plan which is most appropriate to the 
user's goals and adapting it dynamically during the ongoing 
dialog. Two types of adaptations can be distinguished: In'st, 
system-driven modifications using domain dependent back- 
ground knowledge, and second, corrections of misconcep- 
tions, handled interactively in meta-dialogs with the user. 
The dialog manager detects misconceptions, i.e. situa- 
tions in which an intended goal cannot be realized, e.g. more 
items were retrieved than can be displayed in the current pre- 
sentationform. The corrector operates on knowledge bases of 
misconceptions and correction rules, e.g. "if there is a miscon- 
ception like 'ring presentation: not all requested data can be 
presented in the ring' and there is no automatic plan modifica- 
tion possible then start a meta-dialog, which informs the user 
about the situation and offers alternatives." Because meta- 
dialogs will be handled in text mode, the dialog manager re- 
quests the SPL creator to produce SPL plans. Therefore, the 
dialog manager informs the SPL creator on the current miscon- 
ception and possible alternatives the user has to choose from to 
resolve the situation. Then, the SPL creator produces the ap- 
propriate SPL plans by combining information on the miscon- 
ception and possible alternatives with elements from the SPL 
library. The SPL plans are transformed into natural language 
text by the KOMET/Penman system. The resulting text is re- 
turned to the dialog manager which presents it to the user. 
6.3 Supporting Coherence 
In his work on coherence in multi-modal discourse, Bandyo- 
padhyay (Bandyopadhyay, 1990) states that there are three 
levels of coherence: syntactic coherence, semantic coherence 
and pragmatic coherence. Syntactic coherence deals with the 
immediate connectivity among adjacent segments (in texts 
this is often called text cohesion). Semantic coherence ensures 
the wellformed thematic organization of a discourse. Dis- 
course segments are connected by semantic ties (Hobbs, 
1983). Bandyopadhyay defines a discourse to bepragmatical- 
ly coherent if it is compatible with the addressees' interpreta- 
tive ability. In our system syntactic coherence is enhanced by 
the way we present the natural language output in our graphical 
environment. Semantic coherence is supported by the use of 
8 Controlling utterance selection 
The IGiNG system intends to produce user adapted naturaJ 
language output. It is an object oriented system consisting ol 
several object classes (cf. Figure 7 ) 
When IGiNG is requested to produce an utterance it calls th~ 
utterance's express method, which In'st builds a list ofplan-ob. 
jects starting from the initial plan-object given by the utterance 
object. Then, it is determined whether complex or short state. 
ments are desired. This information is kept in a user-stereotype 
and determines in which direction the list of plan-objects is tc 
be traversed. Now IGiNG tests each plan-object's select condi 
tions. If all conditions are satisfied, the plan-object is selected 
otherwise IGiNG tries the succeeding plan-object. Finally th4 
plan def'med by the plan-object is passed to the KOMET\]Pen 
man system which generates the utterance. 
  
 76 
Figure 6 : Integrating SIC! and KOMET/Penman 
Example: Let us consider the IGiNG-objects given in Fig- 
ure 7. IGiNG is requested to express rood-1. It builds a list 
of possible plan-objects, which is (plan-1 plan-2). As con- 
cise statements are desired, plan-1 is tested first. Because 
con-1 is not satisfied (it demands that user-level is low while 
the current user level is advanced) plan-1 is rejected. Next, 
plan-2 is tested. As all conditions are satisfied, the plan given 
by plan-2 is generated. 
Adapting to new situations 
When a new situations is to be included, the following 
steps have to be performed: 
A new proposition-instance has to be defined as the suc- 
cessor of an existing proposition-class. If the new proposition 
is not a part of any of the existing proposition-classes, a new 
proposition-class should be defined first. 
For any of the possible utterances a parameterizable par- 
tial sentence plan has to be written, which is stored in a plan- 
object, together with a reference to the plan-object's select- 
condition-object. Of course, it is possible to use existing 
plan-objects, if they are suitable for the intended purpose. 
Finally the plan-objects have to be linked to select-condi- 
tions. As these are domain independent, preexisting select- 
conditions can be reused. 
These are all the steps necessary for defining new proposi- 
tions. The new objects inherit from their ancestors all the func- 
tionality which is necessary for selection and expression. 
9 Future Work 
We are going to extend the use of natural language output to 
other situations which seem to be suited for textual informa- 
tion rather than graphics. We are working on a closer examina- 
tion of what information has to be included in the user-stereo- 
type and how this knowledge can be obtained. We intend to 
integrate the conversational roles model approach by Sitter 
and Stein (1991), so that we are able to track meta-dialogs, and 
to incorporate the current work on text planning to further im- 
prove the dynamic generation of plan-objects. 
  
 77 
IGiNG-Root-Object 
proposition-object plan-object select-condition-object 
B ~ s 
I ! 
modification alternative 
user-stereotype-object 
m~difia~ t t ioo n-1 ! J alt.ernati.ve_l I I pnan-, J select-condition-I 
has_plan~ i ~ has-selcon C-,,,j next-plan ~ ~,~,-I user-levell°w 
l~ select-condition-2 
t , .,,,J~'l reasoning-desired 
plan-2 
has-selcon O" 
.,~ select-condition-3 
-l~ \[ give-alternatives 
has-selcon O" I 
m advanced-user 
user-level low reasoning-desired YES 
detailed NO give-alternatives YES 
Figure 7 : IGiNG object hierarchy and sample instances 

References 

Allgayer, J., Harbusch, K., Kobsa, A., Reddig, C., Reithin- 
ger, N., and Schmauks, D. 1989. XTRA: A Natural Lan- 
guage Access System to Expert Systems. In: Int. J. Man- 
Machine Studies, Vol. 31, No. 2, 1989, pp. 161-195 

Arens, Y. and Hovy, E.H. 1990. How to describe what ? 
Towards a theory of modality utilization. In: twelfth annual 
conference of the Cognitive Science Society, pp. 487-94, 
Lawrence Erlbaum Associates, Hillsdale, New Jersey, 
1990, July 25-28, 1990, Cambridge, USA 

Bandyopadhyay, S. 1990. Towards an Understanding of 
Coherence in Multi-Modal Discourse, Technical Memo 
TM-90-01, Deutsches Forschungsinstitut fuer Kuenst- 
liche InteUigenz GmbH, Saarbruecken, 1990 

Bateman, J., Kasper, B., Moore, J., and Whitney, R. 1989. 
A general organization of knowledge for natural language 
processing: the Penman Upper Model. USC/Information 
Sciences Institute, Technical Report, 1989 

Bateman, J. and Paris, C. 1989. Phrasing text in terms the 
user can understand. In: Proceedings of IJCAI-89 

Bateman, J., Maier, E., Teich, E., and Wanner,L. 1991
Towards an Architecture for Situated Text Generation,  
International Conference on Current Issues in Computational Linguistics, Penang, Malaysia, 1991 

Brooks, H.M., Daniels, P.J. and Belkin, N.J. 1986.
search on Information Interaction and Intelligent lnforrr 
tion Provision Mechanisms. In: J. of Information Scienl 
Vol. 12, 1986, pp. 37--44 

Cohen, P. R., Dalrymple, M., Moran, D., Pereira, F., Su\] 
van, J., Gargan Jr., R., Schlossberg, J., and Tyler, S. 19t 
Synergistic Use of Direct Manipulation and Natural Lc 
guage. In: Bice, K., and Lewis, C. (eds): Proceedings 
CHI '89, (Austin, Texas, April 30 - May 4, 1989), Nq 
York: ACM, 1989, pp. 227-233 

Feiner, S.K. and McKeown, K.R. 1990. Coordinating T, 
and Graphics in Explanation Generation. In: AAAI -c. 
Proc. 8th Nat. Conf. on Artificial Intelligence. July 
1990- Aug. 3, 1990. Vol. I. Menlo Park et al.: AAAI Pre: 
The MIT Press, 1990, pp. 442 ,149. 

Grosz, B.J. and Sidner, C.L. 1986.Attention, Intention, and 
the Structure of Discourse, Computational Linguistics, 
Vol. 12, No. 3, pp. 175-204, 1986 

Halliday, 1978. Language as Social Semiotic. London: Ed- 
ward Arnold. 

Hayes, Philip J. 1987. Steps towards Integrating Natural 
Language and Graphical Interaction for Knowledge- 
Based Systems. In: Boulay, B. du / Hogg, D. / Steels, L. 
(eds): Advances in Artificial Intelligence -- II. (Proc. 
ECAI-86), Amsterdam et al.: North-Holland, 1987, pp. 
543--552 

Hobbs, J. 1983. Why is Discourse Coherent ? In: Neubauer 
(Editor), Coherence in Natural Language Texts, Buske, 
1983 

Hutchins, E.L., Hollan, J.D. and Norman, D.A. 1986. Di- 
rect Manipulation Interfaces. In: Norman, D.A,, and Drap- 
er, S.W. (eds): User Centered System Design: New Per- 
spectives on Human-Computer Interaction. Hillsdale, NJ 
& London: Lawrence Erlbaum, 1986, pp. 87--124 

HyperNews 1989. HyperNeWS User's Guide, Hoff, Arthur 
van (Editor), The Turing Institute, Glasgow, UK, 1989 

Kasper, R. 1989. A flexible interface for linking applications to Penman's sentence generator. Proceedings of the 
DARPA Workshop on Speech and Natural Language. 

Kerner, A. and Thiel, U. 1991. Graphical Support for 
Users' Inferences within Retrieval Dialogues. In: Proceed- 
ings of the IEEE Workshop on Visual Languages 1991, 
pp.211-216, 

Kracker, M. 1991. Unscharfes assoziativesBegriffswissen 
zur Unterstiitzung der Formulierung von Datenbankabfra- 
gen, Dissertation, TU Wien, April 1991. Title in English: 
Fuzzy Associative Conceptual Knowledge for Supporting 
Query Formulation. 

Lombardi, C. 1989. Experiments for Determining the As- 
signment of Information to Media in COMET, Columbia 
University, New York, N.Y., USA, 1989 

Mann W.C. and Matthiessen C. 1983. Nigel: A Systemic 
Grammar for Text Generation, Technical Report 
RR-83-105, USC/Information Sciences Institute, 1983 

Mann, W.C. and Thompson, S.A. 1987. Rhetorical Structure Theory: A Theory of Text Organization. In: Polanyi, L. 
(Editor): The Structure of Discourse, Ablex Publishing 
Company, Norwood, N.J., 1987. Also available as US C/In- 
formation Sciences Institute Technical Report Number 
RS-87-190 

Matthiessen, C.M.I.M. 1990. Lexicogrammatical Cartog- 
raphy: the systems of English. 
Expanding ongoing draft. Department of Linguistics, Uni- 
versity of Sydney, Australia. 

McCoy, K.F. 1986. The ROMPER system: responding to 
object-related misconceptions using perspective. Pro- 
ceedings of the 24th. Annual Meeting of the Association 
for Computational Linguistics, New York. 

Moore, J.D. and Swartout, J.R. 1990. Pointing: A Way 
Toward Explanation Dialogue. In: AAAI-90: Proc. 8th 
Nat. Conf. on Artificial Intelligence. July 29, 1990-- Aug. 
3, 1990. Vol. I. Menlo Park et al.: AAAI Press / The MIT 
Press, 1990, pp. 457~464 

Neal, J.G. and Shapiro S.C. 1988. Intelligent Multi-Media 
Interface Technology. In: Sullivan, J.W./Tyler, S.W. (eds): 
Proceedings of the Workshop on Architectures for Intelli- 
gent Interfaces: Elements and Prototypes. ACM/Addison- 
Wesley, 1989, pp. 69--91 

Penman 1989. The Penman Project: The Penman Primer, 
User Guide and Nigel Manual. USC / Information 
Sciences Institute, Marina del Rey, CA, 1989 

Reichman, R. 1986. Communication Paradigms for a Win- 
dow System. In: Norman, D.A. / Draper, S.W. (eds): User 
Centered System Design: New Perspectives on Human- 
Computer Interaction. Hillsdale, NJ & London: Lawrence 
Erlbaum, 1986, pp. 285--313 

Reichman, R. 1989. Integrated Interfaces Based on a 
Theory of Context and Goal Tracking. In: Taylor, M.M. / 
Neel, F. / Bouwhuis, D.G. (eds): The Structure of Multimo- 
dal Dialogue. Amsterdam et al.: North-Holland, 1989, pp. 
209---228 

Sitter, S. and Stein, A. 1991. Modeling the lllocutionary 
Aspects of Information-Seeking Dialogues, Arbeitspa- 
piere der Gesellschaft fuer Mathematik und Datenverar- 
beitung 515, 1991 

Stock, O. 1991. Natural Language and the Exploration of 
an Information Space: the ALFresco Interactive System. 
IJCAI91, Sydney. pp972-978. 

Swartout, W.R. and Smoliar, S. 1987. Explaining the link 
between causal reasoning and expert behavior. In: Pro- 
ceedings of the Symposium on Computer Applications in 
Medical Care. Washington, D.C. 

Teich, E. 1991. A Systemic Grammar of German for Text 
Generation, to appear in: Davies, M., L. Ravelli, Advances 
in Systemic Linguistics: Recent Theory and Practice, in 
progress. 

Thiel, U. 1990. Konversationale graphische lnteraktion 
mit lnformationssystemen: Ein sprechakttheoretischer An- 
satz. Dissertation, Universitat Konstanz. Title in English: 
Conversational Graphical Interaction with Information- 
Systems. An approach based on Speechact Theory. 

Tissen, A. 1991. A Case-Based Architecture for a Dia- 
logue Manager for Information-Seeking Processes. In: 
Procceedings of SIGIR '91, October 13-16, 1991, Chica- 
go, USA 

Wahlster, W., Andrt, E., Graf, W., and Rist, T. 1991. Designing illustrated texts: how language production is influenced by the graphics generation. Proceedings of the 
5th Conference of the European Chapter of the Association 
for Computational Linguistics. 
