INTERACTION BETWEEN LEXICON AND IMAGE: 
LINGUISTIC SPECIFICATIONS OF ANIMATION 
Maryvonne ABRAHAM *~, Jean-Pierre DESCL\]~S i 
atelier "Formalisation'" du GDR 957 (CNRS) Sciences Cognitives de Paris. France t 
ABSTRACT 
Certains textes d'une langue peuvent 6tre 
considdrds comma des spdcifications linguistiques 
capables d'engendrer des animations qui simu- 
lent la comprdhension des taxies d'entr6e. Pour 
rdafiser un tel programme, il nous sernble indis- 
pensable de construire des reprdsentations in- 
termddiaires compatibles, d'un c6t6 avec les 
descriptions sdmantiques d'unitds linguistiques, 
et d'un autre c6td avec des spdcifications d'ani- 
marion d'images. 
Comprendre un texte relatif h des mouvements 
spatiaux revient: i) it reprdsenter sdmantique- 
ment le texte; ii) it engendrer une animation 
mettant en jeu des images; caste animation 
vise it reproduire ce qui eat compris lots de 
la lecture du texte. Ce programme ambitieux 
suppose qu'au prdalable chaque verbs sit des 
reprdsentations qua nous appelons sch6mes sd- 
mantico-cognitifs (SSC), correspondant aux 
diffdrentes significations dn verbe. Tout schdma 
de mouvement associ~ it la signification d'un 
verbs repr6sente trois types de connaiesances: 
i) des ddclarations et des relations invariantes 
pendant le mouvement; ii) la description cind- 
antique du mouvement qui fait passer d'une 
phase initiale (Sih) it une phase finale (Sit2); 
iii) les conditions dynamiques dventuelles qui 
rendent possibles ou qui contrSlent le mouve- 
ment cindmatique. 
Nous reprenons l'approche de R.Schank ou 
de J.Sowa qui remet en cause une association 
directe mot-concept pour adopter une d6compo- 
sltion de la signification des unitds linguistiques 
en 61dments de Usens" plus petits. Notre forma- 
lisms a pour but d'une part, de mieux traiter 
les probl/~mes de temps, et d'autre part, de 
fonder notre choix de primitives, dana le cadre 
d'une th6~rie qui articule its niveaux de reprd- 
sentation linguistiques et cognitifs. 
Introduisons quelques distinctions: 
I- le verbs est une entitd lexicale qui peut ~tre 
polysdmique, ex: circular. 
2- le verbe syntaxique entre dans une construc- 
tion syntaxique, ex: NI circular prep Nloe. 
3- le prddicat logique n-airs est aesocid au verbe 
syntaxique, ex: CIROULER est l'expression du 
prddicat l-airs CIRCULER1. 
4- A un prddicat correspondent un ou plusieurs 
sens, d6signds par /verbei/, reprdsent~s cha- 
cun par un SSC. Un SSC est organisd g par- 
Sir de primitives cognitives; il conduit it une 
reprdsentation cognitive dAs qu'il est instanci6. 
5- l'archdtype cognitif notd/verbe/, s'il exists, 
sc prdsente alors comae "le sans abstrait" com- 
mun it tous Its SSC d'une m6me entrde lexicah 
polys~mique. La construction de l'arch6type 
repose sur une organisation des SSC en rdseau. 
Ella utilise une description analogue it cells de 
J.Picoche. 
A partir de l'exemple d'un texts fran~ais 
qui reprdsente des mouvements spatio-temporels, 
nous donnerons pour quelques verbe du texte 
lea SSC et les animations correspondantes. L'ani- 
mation complete ddclenchde par la comprehension 
de texts ndcessite cette premiere phase de recherche. 
Les sch6mes que nous propesons sont int6grables 
darts un dictionnalre s~mantique du fran~ais. 
Notre 6tude presenters /t partir de I'exemple 
choisi la mdthoclologie appuyde par une rdalisation. 
mots-cl~s: { comprehension de tezte, repr&entation 
des connaissances dynamiqttes, leziqa¢ verbs/, m~thodo- 
lofie, cholz dt primitives sdmantlco-toqnitiees, eerbes 
de mouvement, analyst d'un It=St, dictionaaire s~man- 
tiqtte, sp~cifitation linfeistiqae st animation } 
*D6partement Math~matique~ et Syet~m~ de Communication, LIASC, Telecom Bretagne, BP 832, 29285 Bre~t 
cedex, tel: 98 00 11 11 - FAX: 98 00 10 98 e-mail: ABRAHAMObrtst.enat.bretasne.yr 
tC.A.M.S., unitd mixte CNRS - EHF~S - Universit6 Paris-Sorbonne, 96 Boulevard Raspail, 750~ Paris - tel (16-1) 
45 48 08 04 - FAX : (16-1) 40 49 07 98 - e-mail : ishaO\[rci651.bitnel 
Acr~ DE COLING-92. NANTEs. 23-28 AOt~' 1992 1 0 4 3 PRoc. OF COLING-92. NAN'rES. AUG. 23-28. 1992 
key.words: understanding a text, dynamic 
know--l~ge representation, lexicon of verbs, method- 
ology, choice of semantico-cognitive primitives, 
verbs of motion, text analysis, semantic dictio- 
nary, linguistic specification and animation. 
1 understanding a text and a 
moving picture 
Certain texts of a given langage can be consid- 
ered as linguistic specifications able to produce 
motion pictures simulating the understanding 
of entry texts. To achieve such a program, we 
think that it is necessary to build intermedi- 
ate representations which are compatible, on 
the one hand with the semantic description of 
linguistic units, and on the other hand, with 
specifications for the moving of pictures. 
other approaches contest this direct associa- 
tion word- concept, and start from a break- 
down of the signification of the linguistic units l, 
following R.Schank in the 1975's. The meaning 
of the words is no longer considered as primi- 
tive but can be split up into smaller siguifica~ 
tion units. More recently, Sown has presented 
a formalism of representation based on graphs 
with two sorts of nodes representing concepts 
and relations. However, many researchers feel 
that the choice of the primitives looks arbitrary 
(\[SABS8\], pp.241-243). We propose a formal- 
ism similar to Sowa's graphs \[SOW84\], but we 
try to process time better and establish our 
choice of primitives z, using a theory which ar- 
ranges links between the levels of linguistic and 
cognitive representations. 
1.2 semantico-cognitive schemes of 
a verb 
System of semantico-cognitive Repr~sentations~ 
(~ animation specification) . J 
language .... .~. .......... (moving pict ure~ 
%'~talist ic sveciflcationJ 
Figure 1: interaction between lexicon and im- 
age 
Understanding a text relative to spatial mo- 
tion consists of: i) a semantic representation 
of this text; ii) movement with pictures; this 
animation aims to reproduce what is under- 
stood during the reading of the text. As a first 
step, this ambitions program requires that each 
verb should have representations which we call 
semantico-cognitive schemas (SCS). These rep- 
resentations correspond to the different mean- 
ings of the verbs. Each scheme of motion as- 
sociated with the meaning of a verb represents 
three kinds of knowledge: i) declarations and 
relations which are invariant during the mo- 
tion; ii) the cinematic description of the mo- 
tion from an initial phase (Sitl) to a final phase 
(Sit2); iii) the dynamic conditions which make 
or control kinetic motion. 
1.1 semantico-cognitive representa- 
tions 
Although several formalisms of knowledge rep- 
resentations (e.g. semantic networks) suppose 
that the word is the elementary semantic unit, 
static 
declarations 
static relations 
~t 
FAIRE and CONTR~LE 
dynamic 
Figure 2: formal diagram representing seman- 
rico-cognitive schemes (SCS) 
The cognitive archetypes (or semantico-cognitive 
schemes, written SCS, see figure 2), were intro- 
duced in 1985 \[DES85\] in order to represent the 
verbal knowledge underlying the language. Our 
method to discover them is to collect linguistic 
data which allows us to characterize prototypic 
schemes of meanings from which an abstract 
invariant may appear, in some cases. 
The following levels can be taken into consid- 
eration: 
1- The verb, written verb, is a lexical entity; it 
can be polysemic (e.g., sortir). 
2- The syntactic verb written verb/ is a part 
1See \[KAY89\] about the notion of variable depth. 
2The Applicative and Cognitive Grammar model in- 
creases S.Schaumyan's initial model with more cog- 
nitive levels where we aim to discern operations 
about cognitive categories; We have described the 
maln primitives (MOUVT, FAIllE, CONTROL/g, 
...) which are used in several publications DES90 ; 
see \[JPD911 for time processing, and \[DES91 for the 
theory underlying the choice of the primitives. 
ACl~ DE COLING-92. NANTES, 23-28 AOUT 1992 1 0 4 4 PROC. OF COLING-92. NANTES. AUG. 23-28, 1992 
of a syntactic construction (e.g., N1 sortir prep I%,). 
3- The n-ary predicate is associated with the 
syntactic verb (e.g., SORTIR1) a. 
4- With a predicate, one or several meanings 
are associated, written /verb//4. A SCS is or- 
ganised from cognitive primitives. It produces 
a cognitive representation the moment it is in- 
stanciated. 
5- Then, if it exists, the cognitive archetype 
(written/verb/) appears as the "abstract mean- 
ing" common to each of the SCS of one lexical 
polysemic entry. Its construction is organised 
from SCSs. It uses a description similar to the 
one in J.Picoche's method \[PIC86\] 5. 
From a sample text representing spatio-temporal 
motion, we give the semantics-cognitive schemes 
and the corresponding motions for each verb of 
the text. This first step is required in order for 
an understanding of the text to trigger com- 
plete animation. 
2 A few verbs to represent a 
text 
The schemes that we propose are integrated in 
a semantic dictionary of French. From a cho- 
sen example, our study presents the methodol- 
ogy based on demonstration. To illustrate our 
method, we analyse the verbs in the following 
French text and we give cognitive representa- 
tions of them: 
Madame Dupont circulalt dana use 
petite rue ~ sees unique od stationnaient 
des voitures de ehaque c6t& Un vdhicule 
ap_r3dt_rd d'un parking surgit brusque. 
meet sur sa droite. 
which means: 
Mrs Dupont__was driving along a small 
one-way street where cars were parked 
on each side. A vehicle leaving a car 
park shot out on her leff . 
sin agreement with the conceptual language of 
R.Jackendoff \[JAC83\] 
4i indexes one of the prototypical meanings of the 
polyeemes. 
SSee two examples a~eociated with the different mean- 
ingB of the French verbs fouler \[ABR92\] and raon- 
ter (J.P.Descl~!s, article to be published). We need 
syntactic descriptions such as those of M.Grosa 
(\[GRO81\]). For identifying the SCSs, we use work8 
on semantics such a~ 1COL83\] and \[MEL84\]. 
6Notice that we make more than a lexical translation 
from French into English: cognitive laws of the high- 
way code are ~translated ~ too. 
As we have not enough room in this paper to 
describe each verb of this text, we have chosen 
to describe the French verbs stationner, and 
surer. 
2.1 SCS of the French verb 
stationne r 
ddclarations: 
relations: 
3 y : 
/eollectlf / pluriel / indlvidu/: x; 
lieu: loci, 1oc2; 
1ocl :loca ; 
FAIRE IL 
y 
Figure 3: SCS of the French verb "stationner" 
In x stationne, we have a static positional re- 
lation. This situation is implicitly bound by a 
constraint (which can be more or less explicit). 
The signification of/stationner/in this propo- 
sition is given by the dynamic scheme in figure 
3. The quantification which binds the variable 
y blocks every instanciation of y. In our exam- 
ple, tile SCS is instanciated with: 
x :-: des voitures 
y := de chaque c6tg d'une petite rue ~ sees 
unique 
A constraint is needed to keep a car parked. It 
is expressed here using the primitive "FAIRE" 
which keeps the movement from Sit1 to Sit2 in 
the same place. We say that /x stationne/ if 
the behaviour of x implies a constraint on its 
own movement which keep it in the same place. 
This constraint is expressed by the equality be- 
tween the localisations loci of Sit1 and loc2 of 
Sitz. 
2.2 SCS of the French verb surgir 
x surgit means a sudden transition, a move- 
went entering the perception zone of an entity 
able to perceive. The perception zone is gener- 
ally taken from the speaker's point of view. We 
have a rule (IDES901 pp.285-287, \[ABR91\]§2.1) 
concerning localisation: 
AcYgs DE COLING-92, NANTES, 23-28 AOfrr 1992 I 0 4 5 PROC. OF COLING-92, NANrES, AUG. 23-28, 1992 
d~clarations: indivldu: x, y; lieu: 
PERCEP{y); 
relations: 
y a~o.. PERCEP(y}; 
LOC# PERCEP(y); 
sitl \[ \[ Sit2 
x ~o (in (LOC)) \] ~o~uvr \[ x eo ( in PERCEP (y)) 
Figure 4: SCS of the French verb "surgir" 
rule: if z is an entity typed lndlvldu z deter- 
mines a localisation loc(z} including the entity 
Z. 
In fact, to each x is associated a family of lo- 
calisations, among which a proper loealisation, 
a peripheral localisation depending on cultural 
habits, and a neighbouring localisation depen- 
dent on the context. The localisation loe (x) 
can be determined by taking its internal area, 
its external area, or its frontier. The family of 
localisations can be extended to the localisa- 
tions perceived by the individual entity. 
definition of perception: The localisation PER- 
CEP(x) is defined as a result of the intersection 7 
of an external area to be perceived and of the 
properties of perception of the person perceiv- 
ing (its senses). 
x surgit if x moves from Sitx to Sit2. In Sitl, x 
is situated in any localisation; in Sit2, x is sit- 
uated in the place of perception of the witness, 
or, by default, in the area of perception of the 
speaker. 
/surgir/can be associated with a preposition, 
for example, sur (on), which partially speci- 
fies the place of arrival. The French relation 
ETRE-SUR can be represented formally by the 
following set of relations s: 
definition of the relation ETRE-SUR: 
i) The one referred to is outside a domain D 
(closed localisation) 
ii) The one referred is in contact with the ex- 
ternal frontier of the domain D. 
iii) An czternal action (by default, gravitation) 
keeps the referred one in contact with this ex- 
ternal frontier. 
VPERCEP(x) can be considered ae the result of the 
interjection of the area SENSE(x) (of an entity that 
x can perceive) and of the external domain which is 
examined. VIS(A) is the vhdbility area of A. 
SFor any explanation, see \[ABR.91\], Iii.2.3 
FRONT 
HORIZON (A) 
field of invisibility 
Figure 5: on Mrs Dupont's left (left in GBI) 
Concerning visual space perception (figure 5), 
the notions of right and left are relative: they 
can be organised from the point of view ei- 
ther of the subject observed, or of the speaker. 
Let us take an orientation from the subject 
observed A, here, Madame Dupont, which has 
an intrinsic orientation. A defines FRONT(A) 
REAR(A) LEFT(A), RIGHT(A), VIS(A). The 
intersection of these areas defines frontiers. In 
our example, we can consider that /to be on 
the left of Mrs Dupont/means /to be on the 
external frontier between the areas VIS(Mrs 
Dupont) and LEFT(Mrs Dupont)/. 
3 Towards a semantlco-cognitive 
lexicon of verbs 
We implemented a semantics-cognitive lexicon 
on a work-station. The lexicon appears as a set 
of several windows showing: 
- a list of entry points: the lexical verbs; by 
clicking on a verb, a new window is opened, 
showing: 
- a list of examples of the entry verb. 
- for each of the examples, a button helps ob- 
tain the SCS associated with a given verb (and 
the corresponding archetype, if it exists). 
- Visual animation is associated with the SCS, 
and it is obtained from the archetype. This 
animation is the cognitive representation of a 
skeleton of motion; it may then be "clothed" 
with the textures specific to the moving object. 
Our work is limited to the cognitive specifica- 
tion of verbs of motion; it does not take into 
account the distortion specific to the objects. 
- the archetype of the verb, if it exists. 
Our demonstration shows the movement cor- 
responding to the verbs in the text. The SCSs 
(and the corresponding archetype, if it exists) 
establish a specification from which: i) infor- 
ACT, S DE COLING-92. NANTES, 23-28 AO~r 1992 1 0 4 6 PROC. OF COLING-92, NANTES, AUG. 23-28, 1992 
marion for inferences is extracted; ii) virtual \[KAY89\] 
animation can he generated on a screen. In par- 
allel, we have defined the general language for 
computer specifications which use the semantico- 
cognitive primitives that we propose in build- 
ing the SCSs and the cognitive archetype \[ChJ91\]. \[MEL&I\] 
We thank Janet Ormrod, Telerom Bretagne, 
/or her help in producing the English version of 
this text. \[PiCa6\] 
R6f6rences {SABaa\] 
\[ABR91\] Maryvonne ABRAHAM J.P.DESCLI~,S 
R.AFZALI J.BERRI CH.JOUIS H.G.Ott. \[SOW&t\] 
R~prgsentation du mouvement ver6alisg. 
AFCET, Actes du 8~me congr~ RFIA, 
vol.1, pp.ll-21, novembre 1991. 
Maryvonne ABRAHAM Ralek AFZALI 
Christephe JOUIS. Lea archFtypes cogni- 
tifs 8ont des 9gn~rateurs de representation 
des connalssancee. ~ para~tre in T.A.Information, 
1902. 
B.GAUDINAT J-M.ROUSSEAU Ch.TORA 
Ch.JOUIS, F.COMPAGNON. METODAC: 
uric mdthodolotTie pour l'acqulsition cg la 
mod~lieation de~ connaissances. AFCET, 
Actes du 8~me congr~ RFIA, novembre 
1991. 
COLLECTIF cons la direction de H.Lewicka 
et K.Bogacki. Dictionnairc Sdmantique 
et Syntaxlquc des Verbes Francois. Pafistwowe 
Wydawnictwo Naukowe, Warsovie, 1983. 
Jean-Pierre DESCL~S. Reprdsentation des 
connaiseanees: archetypes cognitifs, schemes 
conceptuels, schemas qrammaticauz. EHESS- 
CNRS, Actee S~miotiques, VII, no 69-70, 
pp. 5-51, 1985. 
Jean-Pierre DESCL\]~S. Langages appli- 
catlfs, langues naturclle~ et cognition. Hermes, 
Paris, 1990. 
Jean-Pierre DESCLI~,S. La prddication opdrge 
par lea lanques (on h propos de l'interaction 
entre lanfa9e et perception), in LANGAGE, 
NO 103, Larou~ee, Paris, sept 1991. 
Maurlce GROSS. Formes syntaziqucs et 
prgdlcate s~mantiqucs. Larouaee, Langages, 
nO 63, 1981. 
Ray JACKENDOFF. Semantics and Cog. 
nition. MIT Prees, Cambridge, 1983. 
J.P.DESCL\]~S CH.JOUIS H.G.OH D.REPPERT 
F.SEGOND. Reprgecntations et recherches 
des valeurs sgmantlques des tempe de Uindi~atif 
du francois pour unc raise en oeuvre in- 
formatique. CNRS GDR 957, Centre d'Analyee 
et de Math~matiques Sociahs - ISHA - 
Paris, Mai 1991. 
\[ABR921 
\[ChJ911 
\[COLa31 
\[DESa51 
\[DESQ01 
\[DES91\] 
\[GROg1\] 
\[JAC83\] 
\[JPD91\] 
Daniel KAYSER. Ad~quation inad~quation 
de la logique au tradement s~mantique 
de8 languee. Actee du colloqae s6maatique 
et cognition, ~ paraitre &ux biitions du 
CNRS, Universit~ Paris-Sorbonne, 1989. 
Igor MELCHOCK & alii. dictionnaire cz- 
plicatif et combinatoire du fran~aie con. 
temporain - recherche lc::ico.s~mantique. 
Presses de l'Univereit~, Montreal, Canada, 
1984. 
Jacqueline PICOCHE. Structures s~mantique~ 
du lczique franfaie. Nathan, France, 1986. 
G~rard SABAH. L 'intelligence artifi.ciellc 
et le langage, tl; reprgeentaZion de8 con- 
naissancce. Hermes, Paris, 1988. 
J.F. SOWA. Conceptual structures: In- 
formation Processin9 in Mind and Ma- 
chine. Addison-Wesley, London~ 1984. 
AcrEs DE COLING-92, NANTES, 23-28 AO~V 1992 1 0 4 7 PRec. OF COLING-92, NANTES, AUG. 23-28, 1992 
