Authoring Multimedia Documents using WYSIWYM Editing 
Kees van Deemter and Richard Power 
Infbrmation Technology Research Institute 
UIfiversity of Brighton, Brighton, UK, 
{Kees. van. Deemter, Richard. Power}@itri. brighton, ac. uk 
Abstract 
(1) This paper outlines a future 'ideal' nmlti- 
media document authoring system that allows 
authors to speci\[y content and form of the docu- 
ment independently of each other and at a h igt~ 
level of abstraction; 
(2) It describes a working system that imple- 
ments a small but significant part of the flmc- 
tionality of such an ideal system, based on se- 
mantic modeling of tile pictures as well as the 
text of the docunmnt; and 
(3) It explains what needs to be done to bridge 
the gap between the implemented system and 
the ideal one. 
1 A Future ~Ideal' Multimedia 
Document Authoring System 
A Document Authoring System is a tool that 
helps an author to writ(; docmnents. If the sys- 
tem supports tile authoring of documents that 
combine 'presentations' in diflb.rent media (text 
and images, for example), we will speak of a 
multimedia document authoring system. Ide- 
ally, a multimedia document authoring system 
would allow authors to speci(y the content and 
fbrm of a high-quality document in ways that 
are both simple and etiicient. More specifically, 
an ideal system would aftbrd the tbllowing op- 
tions to the author: 
1. Easy determination of content. ~Content' 
is taken to mean the factual (i.e., proposi- 
tional) content of the docmnent - in other 
words, the content of the Knowledge Base 
(KB) that forms the input to tilt document 
authoring system. 
2. Easy determination of style and layout. In 
the absence of specific instructions from the 
author, style and layout should be deter- 
mined using intelligent defimlts. (For ex- 
ample, tile standard settings may require 
tilt document to be infi)rmal, with avoid- 
ancc of technical terms, lists and footnotes, 
without nlaximum paragraph length, and 
with numbered sections.) Defaults can be 
overridden by the author, whereupon other 
defaults mw become relevant. 
3. Easy allocation oJ' media. As in the case 
of style and layout, the system has to use 
judiciously chosen de.faults: perhaps using 
illustrative pictures wherever suitable pic- 
turks are available, and graphs or tables 
wherever large amomlts of homogeneously 
structured quantitative information are in- 
volved. As above, defaults may be over- 
ruled by specific requests from the author; 
if a request is impossible to fifllfil, an appro- 
priate error message should tm generatc(t. 
4. Easy annotation of non-generated presen- 
tations. In some cases, it will be possible 
tbr the system to generate presentations. In 
other cases, this mw be impossible: Liter- 
ally quoted texts, for example, or historic 
photogral)hs , m~y predate the use of the 
system, in which case it may be necessary 
to treat them as 'canned' and to annotate 
thenl to allow the system to make intelli- 
gent use of them. 
5. Easy post-editin.q. Once tile system has 
produced a document according to the 
specifications of the author, the ideal sys- 
tem would oiler tools to address remaining 
ilmdequacies using post-editing. 
'Easy' means efficient, protected against incon- 
sistencies, and not requiring specialist skills or 
knowledge. A domain specialist, - who may not 
know anything about knowledge representation, 
logic, or linguistics - could use such a system to 
222 
build KBS that l;he sysl;elll call turn into docu- 
menl;s in any desired bmguage using any desired 
(:olnbination of media. The 1)reduction and ut)- 
(b~ting of (;omplex documents would l)e greatly 
simplitied as a result. 
In present-day practice, these requirements tend 
to be far from realized: authoring docuxnents by 
means of such l;oots as MS WORD or POWER,- 
POINT requires much low-level int;eraction, such 
as the typing of (:haracters (m a keyl)o;trd anti 
the dragging of figures from one plwsic~d lo- 
cal;ion t;o m~other. In SOllle cases, all h~telli- 
gent Mult, inledia l)resentation Systc, ln (IMMPS 
e.g., Bordegoni et al. 1.997) can be used (see 
AIR 1995, Maybury and Wahlster 1998 for some 
surveys), which (nnploys techniques from Arti- 
ficial Intelligence to allow higher-lew~l intera('- 
l;ion. Present IMMI'S, howev(,r, meel; few of the 
\]'e(luirenmnl;s lnenl;ioned al)ove. Most ()f l;hem, 
fi)r exmnl)le ~ require intmt ()t' a highly Sl)e(:ial- 
ize.d 11al;llre (e.g., the (;omt)lex logical fornm- 
las ent;ere(t in the wIP sysl;em, An(trd and II.isI; 
1995) 1 and l;hey allow an author little (:ontrol 
over the tbnn (e.g., layout, textual style, me(lia 
allocation) of the (loellnlenl;. The issue of easy 
amlol.ation (d) is never even ad(lress(',(t, to tim 
be, st of our knowledge. 
The, next section descril)es an iml)hnnented sys- 
tem tbr l;he authoring of teztual (lo(:uinents l, hal. 
can |)e ~rgue(l to fltllill requirements (1) and 
(2) and which tbrms n suitnbh: st~rtin g point 
for working towards the 'ideal' multimedia sys- 
l;e111 outlined above,. Section 3 des(:ril)es ~tll ex- 
tension of this system in which signiticallt as- 
1)ect;s of re(luireln('nt;s 3-5 h~we also been ilnl)le- 
menl;ed. Key features of l;his new sysl;em nre its 
ability to use .semantic 'repre.s'entatio'n.s l;hat are. 
common to the different media, and the abil- 
ity to construct natural  feedback texts 
to help the author understand the contenl; and 
the form of |;lie document while it is still raider 
eonsl;ruction. The concluding section exl)lains 
what needs to be done to till the gap |)el, ween 
the iml)lemenl;ed sysl;eln and |,tie ideal one. 
1An exception is AIA,Tes(:o whit:h takes natural lan- 
guage input~ requiring the system to interpret uncon- 
strained natural  (Stock 1991). Avoiding the 
need for doing this is an important design motivation 
for WYSlWYM-based syst;enis. 
2 A WYSIWYM-based System tbr 
the Authoring of Textual 
Documents 
Elsewhere (Power mid Scott 1998, Scott el; al. 
1998, Scott 1999), a new knowledge-editing 
1net;hod called ~WYSIWYM editing' has been in- 
troduced and motivated. WYSIWYM editing 
allows a domain expert to edit a knowledge 
base (KB) by inl;er~wl;ing with a .\[~edbaek ic.:rt,, 
generated by the system, which pl'esents both 
the knowledge, already defined and the options 
for exl;ending and modit~ying it. Knowledge is 
added or modified by return-based choices which 
directly Mti;et the knowledge base; the result is 
displayed to the author by means of an auto- 
matic~lly generated feedback text: thus ~What 
You See ls What You Meant'. WYSIWYM in- 
stant|ares a general recent trend in dialogue sys- 
l;ems |;owards moving some of the initiative from 
the user to the sysl;em, ~dlowing such systenls to 
avoid the (titli(;ulties of t)ro(;essing %t)cn ~ (i.e., 
tureens|rained) input. 
Of parti(:ular importance, here, m:e at)plieations 
of WYSIWYM to the generation of documents 
(:ont~dning text mid 1)ietures; the t)resent sec- 
tion tbcuses on (multilingual) tezt generation: 
l;he KB (:re~Lted with the help of WYSIWYM is 
used as input to a natural  generation 
(NLG) l)rogrmn, pro(hu:ing as output a docu- 
ment of some sort, for I;he benelit of ~m end 
user. Present apt)lications of WYSIWYM \[;o i;exl; 
generation use a KL-ONE-I,yl)e knowledge rep- 
resentation  as input to two NLG sys- 
\[;elliS. ()lie NLG sys|;elll generates tb.edback texts 
(for l;he raft;her) ml(t I;h(' other gener~d;es on|trot 
l;exi;s (for all ell(t llser). Olle at)plication cur- 
rently under develotmmnl; has 1;11(; creation of 
Patient Informal;ion Leaflets (PILLS) aS its do- 
main. The present vt;rsion of this PILLs system 
allows authors to enter information about pos- 
sible side eft'cots ('if you are either pre.qnant o1" 
allergic to penicillin, then tell your doctor') and 
how to handle lnedical devices such as inhalers, 
inoculators, etc. By interacting with the feed- 
back texts generated by the system, the author 
can detine a procedure for perfornfing a task, 
e.g. preparing an inhaler for use. A llew KB 
leads to the creation of a procedure instance, 
e.g. p. The permanent part of the KB (i.e., 
the T-Box) spt, eifies l;haI; procedures ma, y be 
223 
complex or atomic, and lists a number of op- 
tions in both cases. In the atomic case, the op- 
tions include Clean, Store, Remove, etc., and 
these are made visible by means of a metal from 
which tile author can select, say, Remove. Tile 
program responds by adding a new instance, of 
type Remove, to the KB: 
Remove(p) 
('There is a procedure p whose type is Remove.') 
th'om the updated knowledge base, the genera- 
for produces a feedback text 
Remove this device or device-part 
from this device or device-part, 
making use of the infbrmation, in the T-Box of 
the system, that Remove procedures require an 
Actee and a Source. Such not yet defined at- 
tributes are shown through mouse-sensitive an- 
chors. By clicking on all anchor, the author 
obtains a pop-up metal listing the pernfissible 
values of the attrilmte; by selecting one of these 
options, the author updates the knowledge base. 
Clicking on this device or device part yields 
a pop-up menu that lists all the types of devices 
and their parts that the systenl knows about, in- 
eluding a Cover (which, according to the T-Box 
must have a Device as Owner). By continuing 
to make choices at anchors, the author might 
expand the knowledge base in the tbllowing se- 
quence: 
• Remove a device's cover from a device 
or device-part 
• Remove a device's cover from an inhaler 
of a person 
• Remove a device's cover from your inhaler 
• Remove your inhaler's cover from your in- 
haler 
At this point the knowledge base is potentially 
complete, so a (less stilted) outp'ut tczt can be 
generated and incorporated into the leaflet, e.g. 
Please remove the cover of your ill- 
haler. 
Longer output texts can be obtained by expand- 
ing the feedback text fitrther. A numl)er of 
proi)erties of the PILLS system are worth stress- 
ing. First, the system sut)ports a high-level di- 
alogue, allowing the author to disregard low- 
level details, such as the exact words used in the 
output text. This makes it possible to interact 
with the system using, say, French (provided a 
generator tbr French feedback texts is available), 
for the i)roduction of leatlets in Japanese (pro- 
vided a generator for Japanese output texts is 
available). The semantic model in the T-Box 
guarantees that many types of inconsistencies 
(e.g., a medicine that has to be taken both once 
and twice a day) are prevented. Second, a sire- 
ple version of WYSIWYM has also been applied 
to the tbrm of the text, allowing the author to 
specit) it separately from its content. This is 
done by allowing the author to use WYSIWYM 
tbr building a second, form-related KB which 
describes the st~.tlc and layo'ut of the docmnent. 
This KB, for example, may state that the maxi- 
mum paragraph length is 10 sentences and that 
there are no tbotnotes. (A second, form-related 
T-Box deternfines what the options determining 
layout are.) This form-related KB constrains the 
texts that are generated. By interacting with 
feedback texts describing the tbrm-related KB, 
the author changes the stylistic/layout proper- 
ties of the document. 
A WYSIWYM-based System for 
the Authoring of Multimedia 
Documents 
ILLUSTrl, ATE is ai1 extension of PILLS produc- 
ing documents that contain pictures as well as 
words. Consider a toy exalnl)le, adapted from 
ABPI (1997). Suppose the document says Re- 
move the cover of your inhaler. An instruction 
of this kind may be illustrated by the picture 
below. How can a document authoring system 
produce a document in which appropriate pic- 
tnres illustrate the text when this is desired? 
ILLUSTRATE does this by allowing an author to 
ask tbr pictorial illustration of the intbrmation 
in the document by interacting with the feed- 
back texts. The author can indicate, fbr a given 
mouse-sensitive stretch s of the feedback text, 
whether she would like to see the part of the 
document that corresponds to s illustrated. If 
so, the system searches its library to find a pic- 
ture that matches the meaning of s. In Fig.2, 
the author has requested illustration of the in- 
224 
i ....... 1_/ 
I 
Figure 1: One of the picture.s ill the lit)rary of 
the a, uthoring system 
struction (:orresl)onding with tim text 'Remove 
your inhaler's cov(,r fi'om your inhaler'. (The 
otlmr four options are irrelevant for t)resent pur- 
poses.) In domains where all the pictures are 
,i 
\[:Jle ;%.'.ture f,.',oda!~t¢ C,srltr0l 
\]{enlova your inhaler' s cover h', ...... ,.,,,. ;,.h .I..,. 
tiiili i:. '.t 2c..fvs,:l:t; 
C,:,I>'~ 
CLR 
(gel)ell : t~, a 
,%i.: !!,,,,d,, di, r ~ 
Figur(' 2: Screen(hunt): Author makes a re-. 
quests for illustration 
variations on a common theme, suitnble pic- 
tures can be generated. Ill the case. of l)atient; 
Information LeMtets, however, this was not a 
practical option because of the many ditDrent 
kinds of things depicted in the leaflets: medicine 
packages, body parts, medical at)l)licmmes, var-. 
ious t;yl)eS of actions, etc. Pictures, moreover, 
are he,wily reused in the diit'erent leatlets writ- 
tell \])y the S&lilO company. For these reasons, 
IIAAJSTRATt?, llses ~tii alternative al)proaeh, se- 
lecting pictures from a library, each of which is 
;tnnol;at,ed with ~t formal rel)r(;sentation of its 
meaning. We will explain the workings of IL- 
LUSTRATE by answering three questions: (\]) 
What kinds of rei)resent~tions are used ill the li- 
brary to annotate the pictures with relevant as- 
l)ects of their meaning'? (2) How is the .~;emanti- 
(:ally annotated library of t)i(:tures created'? and 
(3) What selection algorithm is employed to re- 
trieve all optimally approt)riate illustration for a 
given part of the, KB frolll the library? We shall 
assmne that the information whose illustration 
is requested con'esponds with the following for- 
mul~t in the KB, which tel)resents the metaling 
of the feedback text ill Fig. 2. 
R,'~,,,,o~,,'.(t,) ,V Acto,'(p) = .',: 
I~,.a~.,'(.~:) g So,,,,.,:c~(;,) = v a~ 
Inhalcr(y) ~ Actce(p) = z 
g = v. 
('There exists a 'Remove.' action whose Sere'co 
is an Inhah'.r and who.~e Ad, ee is a Cover of the 
same inhaler.') 
1. What kinds of representations are 
used? Representations say what information 
each pictm'e intends to convey. Irrelewmt de- 
tails shouht be omitted. It has been observed 
that photographic pictures express 'vivid' in- 
formation and that this intbrmation can be 
expressed by a conjmmtion of positive, literals 
(Levesque 1!)86}. In line with this obserwttion, 
ILLUSTI/ATE rei)resents the lllea, nillg of the pic- 
ture in Fig. 1, for example, as follows: 
;~+.',,,.o~,+.'(p) *+ So',,.r',:+.'(p) = :\] 
H-I*",'(V) ,V A,tc~;(~,) = 
a C,,',.',,'(~) a O',,,,,.','(~) = :,j. 
(The leattet.~; (le,~mril)e Inhalers, Autohale.rs, and 
Aerohalers.) If any of the wtriables c, :r, y, z has 
&Ii oc('urrelice ill the llle;tlling rel)resentation of 
mlother 1)i(:ture thei~ these occurrences coref'er. 
This ~:dlows the systmn to know wh(.'n two pi(:- 
tm'es depict the same. i)erson, for examt)le (Vail 
Demrd:er and Power 1999). 
2. How is the library created? This is a 
question of great imi)ortmlce because the library 
contains semantic representations that m'e lIlltCh 
more detailed than those in current picture re- 
trieval systems (e.g. Van de ~vVaal \]995) nml this 
couht potentially nmke the &llllOtat;iOll t;ask ex- 
tremely })ur(lensome (Enser 1995). The an,~wer 
to this t)rol)lem may be unext)e(:t;e(l: ILLUS- 
TI1ATE u.qi'.s WYSIWYM it;self to emtl)le authors 
t.t) associate ;t given t)icture with a novel rep- 
resenl;;tl;ion. The class of representations tlmt 
225 
are suitable for expressing the meaning of a pic- 
ture is, alter all, a ('vivid') subset of tile class 
of representations allowed by the T-Box tbr tim 
text of the document, and consequently, tim 
same WYSIWYM interlhce can be used to create 
snch representations. Fig. 3 contains a screen- 
dump of the annotation process, wtmre the cnr- 
rent annotation corresponds with the formula 
Cover(z) & Owner(z) = y. Note that this 
formula is still incomplete because the nature 
of the Source is undefined. (When it is finished, 
the feedback text will be eqniwtlent to that in 
Figure 2.) The top of the screendump shows the 
accompanying feedback text containing anchors 
tbr flsrther additions. 
Figure 3: Screendump: A stage during tile an- 
notation of a picture 
3. What is the selection algorithm? A pic- 
ture can illustrate ass item of information with- 
out expressing everything in it. For example, 
Fig. 1 does not show that the Actor is the 
Reader and it leaves the type of 'Haler' unspec- 
ified. (They all look alike.) So, a selection rule 
must allow pictures to omit intbrmation: 
Selection Rule: Use the logically 
strongest picture whose representa- 
tion is logically implied by the infor- 
mation to be illustrated. (Van Deemter 
1999) 
Logical strength is determined on the basis of 
the two semantic representations alone. Deter- 
mining whether one representation logically ira- 
lilies tim other, where one is an instance in the 
KB and tim other a representation of a picture, 
is easy, given that both are conjunctions of pos- 
itive literals. 
This brief description should suffice to highlight 
the following advantages of ILLUSTRATE: 
• One unifbrm interface is employed for all 
actions that involve the editing of semantic 
representations, regardless of the type of 
presentation involved (i.e., its media). 
• When used for the construction of anno- 
tations of pictures, the T-Box of the sys- 
tem snakes sure that only those properties 
can enter an annotation that are relevant in 
connection with it. In the present domain, 
for example, the height of the patient is ir- 
relevant, and consequently the T-Box does 
not make height all attribnte of a person. 
,, Pictures are retrieved by a reasoning pro- 
cess involving classical logic; since a match 
between a picture and a piece of the KB 
can never be inexact, there is no need tbr 
the retriewfl process to be probabilistic, as 
has to be done when the system has less 
control over the form of annotations (Van 
Rijsl)ergen 1985, Van Deesnter 1999). 
Specific aspects of ILLUSTRATE have been de- 
scribed elsewhere, but the assumptions belfind 
the system as a whole have not beess stated be- 
fore. (For tlm representation scheme and the 
selection scheme see Van Deemter 1999; tbr 
the treatment of sequences of pictures see Van 
Deemter and Power 1999.) We have so t'nr sin> 
plified by assuming there to be only one au- 
thor. In fact, however, an intelligent authoring 
system is most useflfl when there are several au- 
thors (each of which can be allowed to work in a 
different langnage). More specifically, it is plan- 
sible that the person anthoring the annotations 
in the library is not the santo as the person(s) 
who author(s) the document itself. 
4 Future Work Towards the Ideal 
The PILLS system (section 2) makes a first 
stab at fnlfilling text-related requirements 1 and 
2 nmntioned in section 1. The ILLUSTRATE 
demonstrator goes beyond this, fulfilling impor- 
tant aspects of requirements 3 and 4 as well. 
226 
Yet, there is a considerable ga t) l)etween the im- 
i)hmlented system and the ideal one of section 
1. Possit)le improvements do not only concern 
the (:overage of the sysl;em, but matters of sys- 
tem arehitectm:e as well. Three (titti;rent sets of 
improvements may l)e dis(:erned. Firstly, there 
is rcquirelnent 5 of section 1, whi(:h requires 
easy postediting. It is easy to allow at%hers to 
make h)w-levcl corrections in the document a.f- 
let the interaction with WYSIWYM, HIlL unless 
the system 'understands' the, meaning of the 
editing actions, i)ostediting destroys tlm con- 
ne(:tion t)etween the edited document an(l the 
(:ontent of l;he various knowledge l)nses. Conse- 
quenl;ly, l)OSt-editing is not a t)ractieal t)ossit)il- 
ity yet, giv(;n the state of the, m:t in text- and 
picture understanding. 
Other imt)rovements would 1)e less t)rol)lem- 
atic. On the one hand, there are issues that 
have been t~mkled by other research groups and 
whose sohltions we inten(t to (:arty over to a 
WYSlWYM-l)ased setl;ing. These (:on(:ern the 
generation ot' gral)hies Kern underlying rel)re- 
sentations (Wahlster et; al. 1993) and the 1)rot)- 
lem of ot)timizing tim layout of text & \])i(:ture 
do(:uments (e.g. Grat)h et al. 1996), l'or in- 
stall(;e. Three remaining imt)rovements , (m the 
Ot\]ler \]lall(t~ ~LFe lllal;t(~rs for fill;tire l;('.s(?~l;(;ll: 
• Media alloc,,tio'n,. \]LLUSTRATF eml)o(li(',s 
one way in whi(:h media may be allocated. 
Other mechanisms (:ould give the system 
more autonomy. For example, l;he system 
may use rules (e.g. I/.oth and Hettey 1.!)93) 
to decide alltOllOnlOllsly what illforlllation 
is in need of illustration. Simih~rly, authors 
may 1)e enabled to 1)oint at thmnlmail 1)ie- 
tures, whereul)On the system tries to fin(t a 
suitable place in the document to include 
them, based on the ret)resentation of their 
meaning and making use, of the Selection 
Rule of section 3. By thus allowing the au- 
thor and the system to coot)crate on media 
allocation, this ditiicult task will t)e lnade 
more tractable (see the recent discussions 
ill ETAI 1997-8). 
• Other media. Little in ILLUSTRATE hinges 
on the fact that the ol)jects in the lil)rary 
are t)ictures. The Salll(. ~ system, for exam- 
t)le, can be used tbr ammtating somul or 
canned tezt (for examl)le, a complex t)it of 
btw {:ode, which needs to be rendered liter- 
ally). Of great practical interest, finally, is 
the 1)ossil)ility of including docunlents au- 
thored previously (and possibly by a dif- 
ferent author), leading to iterative applica- 
tions of WYSIWYM. 
• bttcraction bctwc.cn media. Ide~flly, the 
words in a text should be Ml'ected by the 
inclusion of a picture: First, and most obvi- 
ously, texts im~y be cnlauicd by retbrences 
to 1)ictures (e.g., references like ~See Fig. 
3' may l}e.adde{t, {:f Paraboni an{t Van 
l)eemter 1999). Secondly, texts may 1)e 
red,uccd because information expressed in 
the 1)ictme can l)e shortened (or left out 
all;ogether). One type of situation where 
this h~q)l)ens is cxempliiied by t;he text '12,o- 
move the ('al)sule frolll the foil as shown 
in the \])i(:ture' (ABPI 1997), a(:(:omt)anie(t 
1)y a t)i(:ture showing how this may be 
done. Oth(;r tyt)es of situation in(:lude the 
case where quantil;ative inforln~ttion is ex- 
1)resse(t through a vague textual descril)l;ion 
('a blol) of (:ream', ~a tingertip of ointmead:') 
that is made more l)re(:ise by means of a 
picture showing t\]w, required amount. 
It should 1)e noted that each of these extensions 
del)ends ('ru('ially on ILLUSTRA'I'E:s at)ility to 
ma.nil)ulate the semanti(: rel)resentations ass()- 
('iated with multimedia objc(:ts, whoxc' one mid 
the same rel)resental;ion  is used fbr 
the difl'erent media: a lmfltimedia qnterlingua' 
(e.g. Barkcr-Plummer and Greeves 1995). 
In the ('ase of an author selecting a t)icturc 
using tlmmbnails, tbr exami)le , the semantic 
rel)resentation cnal)les the author to (a) tinda 
suitabh; local;ion for the t)ieture and (1)) adat)t 
the, (;ext l)y omitting fl'om i(; information that 
is now expressed by the picture. 
A final extension of the ideas outlined in this 
t)aper would involve completing the symmetry 
between feedl)ack and outl)ut: all t)resent 
WYSIWYM systems IlSe Imrely textual feedback. 
In prin(:it)le, however, feedt)ack can l)e as 
multimodal as the target document. We are 
currently exploring the 1)ossibility of allowing 
an author to express some of her choices by 
clicking on a mouse-sensitive part of a picture; 
the system could generate an ui)dated feedback 
text (possibly along with an updated t)icture) 
227 
as a result. Iu some technologically complex 
domains, for example, where a brief description 
of an object may be difficult to obtain, this 
might lead to a fllrther improvement of the 
WYSIWYM technique. 

References 

ABPI (1997). The Association of the British 
Pharmaceutical Industry, 1996-1997 ABPI 
Compendium of Patient Information Leaflets. 

Am (1995). Special Issue, edited by P. Mc 
Kevitt, on Integration of Natural Language 
and Vision Processing: Intelligent Multimedia. 
Artificial Intelligence Review 9, Nos.2-3. 

E. Andr6 and Th. Rist (1995). Generating 
Coherent Presentations Employing Textual and 
Visual Material. Art{ficial Intelligence Review 
9:147-165. 

D. Barker-Plummer and M. Greeves (1995). 
Architectures for Heterogeneous Reasoning. 
In J.Lee (Ed.) Proc. of First International 
Workshop on Intelligence and Multimodality in 
Multimedia Interfaces: Research and Applications (IMMI- 1), Edinburgh. 

M. Bordegoni, G. Faconti, S. Feiner, M.T. 
Maybury, T. I{ist, S. Ruggieri, P. Trahanias, 
and M. Wilson (1997). A Standard Reference 
Model for Intelligent Multimedia Presentation 
Systems. Computer Standards & Interfaces 18, 
pp. 477-496. 

P. Enser (1995). Progress in Docmnentation; 
Pictorial hffbrmation II,etrieval. Journal of 
Documentation, Vol.51, No.2, pp.126-170. 

ETM (1997, 1998). ETAI News Journal on 
Intelligent User Interfaces, Vol 1, No's 1 and 2. 

W.H. Graf, S. Neurohr, and R. Goebel (1996). 
A Constraint-Based Tool for the Pagination of 
Yellow-Page Directories. In U. Geske and H. 
Simonis (Eds.) Procs. of KI96 workshop on 
declarative constraint programnfing. GMD- 
Studien 297, St. Augustin. 

H.J. Levesque (1986). Ma~king Believers out of 
Computers. Artificial Intelligence 30, pp.81-108 

M. Maybury and W. Wahlster (1998). Read- 
ings in Intelligent User Interthces. Morgan 
Kaufmmm Publ., San Francisco. 

I. Paraboni and K. van Deemter (1999). Issues 
for Generation of Document Deixis. Ill E. 
Andrd ctal. (Eds) Procs. of workshop on 
Deixis, Demonstration and Deictic Belief in 
Multimedia Contexts, ill association with tile 
llth European Smnmers School in Logic, 
Language and Information (ESSLLI99). 

R. Power and D. Scott (1998). Multilingual 
Authoring using Feedback Texts. In Proc. of 
COLING/ACL conference, Montreal. 

S. Roth and W. Hefley (1993). Intelligent 
Multimedia Presentation Systems: Research 
and Principles. In M.Maybury (Ed.) Intelligent 
Multimedia Interlaces, AAAI Press, pp.13-58. 

D. Scott, R. Power, and R. Evans (1998). 
"Generation as a Solution to its own Problem", 
Accepted for Proc. of 9th International Workshop on Natural Language Generation. 

D. Scott (1999). The Multilingual Generation 
Game: authoring fluent texts in unfamiliar lan- 
guages. Proceedings of the 16th International 
Joint Conference on Artificial Intelligence 
(IJCAI'99). 

O. Stock (1991). Natural Language and Explo- 
ration of an Information Space: the aLFfl'esco 
Interactive Systein. In M. Maytmry and W. 
Wahlster (1998). 

K. van Deemter (1999). Docunlent Geueration 
and Picture Retrieval. In Prec. of Third 
Int. Conf. on Visual hfformation Systems, 
Amsterdam, Springer Lecture Notes. 

K. van Deemter and R. Power (1999). Inclusion 
of Picture Sequences in Generated Docmnents. 
In Prec. of iburth Portuguese Conf. on 
Artificial Intelligence, Evora, Springer Lecture 
Notes. 

H. van de Waal (1995). ICONGLASS; An leone- 
graphic classification system. Amsterdam 1973- 
1985 (17 vols). ISBN 0-7204-8264-X. See also 
<http ://iconelass. leg. ruu. nl/home, html>. 

C.J. van Rijsbergen (1989). Towards an inibr- 
mation logic. In: Prec. ACM SIGIR. 

W. Wahlster, E. Andrd, W. Finkler, H.-J. 
Profitlich, and Th.Rist (1993). Plan-based 
Integration of Natural Language and Graph- 
ics Generation. Artificial Intelligence 63, 
p.387-427. 
