Interactive Multimedia Navigation 
Prof. Dr. Dr. Mihai NADIN 
Computational Design, University of Wuppertal 
Hofaue 35-39 
D-42103 Wuppertal, Germany 
nadin @ code.uni-wuppertal.de 
Dipl. Des. Clemens LANGO 
Computational Design, University of Wuppertal 
Hofaue 35-39 
D-42103 Wuppertal, Germany 
lango@code.uni-wuppertal.de 
Abstract 
We understand knowledge as an infinite sequence 
of associations between what we know and what 
we are in the process of acquiring as knowledge. 
The most important cognitive processes that 
support acquisition of knowledge are associations. 
How associations come into being is difficult to 
describe because association processes occur in 
open systems. Associations that have taken place 
can be documented, refined, or, if some turn out 
to be irrelevant, discarded. Associations are almost 
always multimedial, i.e., we associate texts, 
sounds, pictures, movement etc. 
The structure of an interactive, multimedia 
encyclopedia that is based on associations includes 
1) a knowledge space/domain, 
2) an associative search procedure, 
3) a function for storing associative traces. 
Interactive Multimedia Navigation 
Until now, problems of navigation have in the main 
been reduced to the research of interface. Aspects 
of metaphor constitution, of the role of visual 
language, as well as of the understanding of 
feedback mechanisms, and the role of back 
channels were pushed into the background. 
Moreover, questions pertinent to knowledge 
acquisition, expression, understanding, and 
application are still only marginally approached. 
Consequently, navigation is defined for 
applications pertaining to homogenous sets of data, 
such as numerical representation, images, and 
texts. Indeed, relatively stable navigation methods 
and tools support text-based dissemination of 
knowledge, applications of mathematical intent, 
and simple ways to handle visual information by 
means of zooming or by animation procedures. 
Although the community of scientists dedicated 
to the design and implementation of distributed 
collaborative leaming systems is aware that the 
real potential of the digital dissemination of 
knowledge lies in media integration, very little has 
been done to specifically deal with the integration 
of image (produced as computer graphics, photo, 
film, animation, video), text, audio, and other 
media. Even less has been done in developing 
navigation concepts, methods, and instruments 
appropriate to this complex task. Until now, 
navigation has been predominantly considered 
independent of forms and means of representation. 
Representation, however, is a constitutive 
dimension, of gnoseological activity, not just a 
theme of formal description and focus. 
Accordingly, those involved in the cognitive 
aspects of learning realized that it is essential to 
conceive, design, and optimize navigation tools in 
respect to the representation modi, and not only to 
what is still naively defined as the content. Various 
possibilities and restrictions are intrinsic to a 
chosen representational mode-assuming that the 
requirement of adequacy is effectively met. It is 
not the same thing when navigation takes place in 
a 3-dimensional space as when it takes place in 2- 
dimensional text display. We can read of molecular 
docking or we can experience it. The representation 
chosen (text description, graphical illustration, or 
virtual reality context) affects to a high degree the 
nature and the outcome of the learning experience. 
Texts can be disseminated in various forms (from 
e-mail to scholarly publications). Graphical 
representations can range from simple diagrams 
47 
to animated sequences. However, these forms are 
interactive in a limited sense only. They can be 
integrated in Web interactions, but not in 
cooperative forms of learning. A virtual reality 
environment can be distributed and accessed from 
many parts of the world, and experienced in a 
cooperative/collaborative effort. In recent time, 
with the emergence of channels and active 
desktops, the dynamic characteristics of navigation 
tools have again changed. We know "pull" methods 
(ieamer finds a site and pulls a subject in some 
format varying between a PS download of text to 
a chat room), as well as "push" methods. These 
apply to "subscriptions" made in the network by 
those pursuing a specific course of learning. In 
both cases-push and pull-the subject of navigation 
must reflect the need for customization (learning 
is an individual experience regardless of where 
and when it takes place). Our research at the 
Computational Design Program is focused on the 
concept of mind ergonomics. This term addresses 
exactly the expectation of customization, i.e., the 
individual nature of learning based on how minds, 
as opposed to systems, work. The three aspects 
that make up the object of our research are 1) the 
expression of knowledge through interactive 
multimedia; 2) navigation tools corresponding to 
the expression of knowledge through interactive 
multimedia; and 3) mechanisms for updating both 
the body of knowledge represented through 
interactive multimedia methods and the 
appropriate navigation tools. A multimedia system 
for digital dissemination of knowledge usually 
consists of editors, search tools, and browser. 
Within this model, those using it still face 
shortcomings such 1) "getting lost in hyperspace," 
2) "spaghetti appearance" (interference of data 
streams), and 3) desynchronization (between 
navigation methods and new knowledge, between 
the representation and interior connection of this 
new knowledge). Our alternative mind ergonomic 
approach addresses the following aspects: the 
constitution of vast knowledge spaces, the best 
example being the Web; the evolution of 
communication models (one-to-one, one-to-many, 
many-to-many); the variety of forms of access, 
from the traditional question-answer model to 
hyperlinks and timescaling. This approach 
suggests the following da ta-mining procedure: the 
constitution of associative links and the automatic 
generation of associative maps corresponding to 
different types of learning and investigation. In 
order to accomplish this goal, we suggest a 
multimedia database with powerful indexing and 
classification functions. (eventually with resident 
intelligent procedures). This database is 
complemented by an intelligent search procedure 
able to handle multimedia representations of 
knowledge. The underlying information is provided 
with a gnoseological map of known interpretations, 
as well as with an indexing utility in which the 
learner or researcher advances new hypotheses or 
new interpretations as these arise through the 
continuous progress of knowledge acquisition and 
discovery. In developing the navigational tools, we 
address the constitutive elements of the mind- 
ergonomic structure. Specifically, this refers to the 
variety of cognitive types of learners, for instance, 
exemplified in question formulation through text 
queries or through visual or aural interrogation. 
Answers, although independent of these cognitive 
types, are made available in a variety of formats 
corresponding to those types. They are collected 
and submitted to further evaluation through the 
program and through the users themselves. The 
major cognitive process on which the project 
focuses at this time are associations. The difficulty 
in dealing with these processes is that they 
constitute an open system. Once constituted, 
associations can be documented, refined, discarded 
if they are irrelevant, or submitted for future 
evaluation by the leamer and by researchers. In 
terms of programming, associations require the 
involvement of fuzzy models and a fully supported 
interactive multimedia environment. (A text can 
be associated to sound, image, motion, etc. and 
combinations thereof.) In the knowledge space that 
we define, learners can pursue a one-dimensional 
inquiry, the classic example being historical 
analysis, i.e., the gnoseological sequence. Or they 
can pursue a multi-dimensional inquiry, for 
instance, the relations between a theory in physics 
and the broader context of knowledge and action, 
including identifiers such as aesthetic value, 
dominant cognitive characteristics, the state of 
technology, etc. We are motivated in this approach 
by the notion that knowledge dissemination cannot 
and should not be reduced to retrieval mechanisms. 
48 
Learning minds, i.e., people interacting in the 
process of learning in a digitally supported 
environment, are actually defined in a process of 
discovery. The active component of the act of 
learning is what we are in fact pursuing. 
Accordingly, the navigation tools developed are 
tools for making such interaction possible. 
Especially in a context of information overload, 
learners have to be able to prune what appears to 
them as the seemingly endless knowledge trees 
provided by digital systems. The result of learning 
is expressed through the associative map generated 
by the system as the learner gets closer to 
understanding the subject researched. After several 
inquiries, the learner will "own" several such 
associative maps and will be able to continue the 
learning process at the level of generality or 
abstraction at which new knowledge can be 
generated. Obviously, we no longer support 
learning as a form of information consumption, 
but rather as a form of knowledge generation. 
This research project is funded by the German 
Research Center (DFG). 
49 
50 
",~., 
I 
I 
O 0m 
C~ 
51 
= 
A 
v 
i 
A 
i 
m 
A 
A 
i 
A 
i 
A 
i 
A 
i 
