Interactive Multimedia Explanation for 
Equipment Maintenance and Repair 
Kathleen McKeown and Steven Feiner 
Department of Computer Science 
450 Computer Science Building 
Columbia University 
New York, N.Y. 10027 
Objectives 
We are developing COMET, an interactive system that 
generates multimedia explanations of how to operate, 
maintain, and repair equipment. Our research stresses the 
dynamic generation of the content and form of all material 
presented, addressing issues in the generation of text and 
graphics, and in coordinating text and graphics in an in- 
tegrated presentation. 
COMET contains a static knowledge base describing 
objects and plans for maintenance and repair, and a 
dynamic knowledge source for diagnosing failures. Ex- 
planations are produced using a content planner that deter- 
mines what information should be communicated, a media 
coordinator that determines which information should be 
re.allzcd in graphics and which in text, and separate text and 
graphics generators. The graphics and text for a single 
explanation are laid out on the screen by a media layout 
component. A menu interface allows users to request ex- 
planations of specific procedures or to specify failure 
symptoms that will invoke a diagnostic component. The 
diagnostic component can ask the user to carry out 
procedures that COMET will explain if requested. In con- 
wast to hypermedia systems that present previously au- 
thored material, COMET has underlying models of the user 
and context that allow each aspect of the explanation 
generated to be based on the current situation. 
Our focus in the text generation component has been on 
the development of the Functional Unification Formalism 
(FUF) for non-syntactic tasks, of a large syntactic grammar 
in FUF, of lexical choice in FUF using constraints from 
underlying knowledge sources and from past discourse, and 
of models of conslraints on several classes of word choice. 
Important results in the graphics component include the 
knowledge-hased design of 3D illustrations, a 
design-grid-hased methodology for display layout, and the 
development of a novel graphical history interface that al- 
lows users to review previous sessions. 
Finally, we have had significant results in the develop- 
ment of our media coordinator which, unlike other systems, 
features a common description language that allows a fine- 
grained division of information between text and graphics. 
The media coordinator maps information to media specific 
resources, and allows information expressed in one media 
to influence realization in the other. This allows for tight 
integration and coordination between different media. 
Recent Accomplishments 
• Integrated the different systems of which COMET is 
comprised to produce a running testbed that creates 
multimedia explanations on-the-fly, in response to 
user requests. Our testbed system generates and lays 
out a full display of text and graphics in 15--25 
seconds. 
• Incorporated user model constraints on content 
generation, to avoid presenting information that has 
been presented previously. 
• Extended the support for using FUF in vocabulary 
choice, adding the ability to access the knowledge 
base from the lexicon to influence the choice of 
vocabulary. 
• Added the ability to use previous discourse to 
influence the choice of vocabulary. 
• Began implementing support for coordinating picture 
and text breaks. 
• Began implementing support for cross-references 
between text and graphics. 
• Extended the ability of the graphics generator to 
represent actions by means of metaobject arrows. 
• Began the preliminary design of facilities for user 
exploration of the generated explanation, including 
the ability to examine depicted objects from different 
viewpoints than those chosen by the system. 
413 
