Proceedings of the Analyzing Conversations in Text and Speech (ACTS) Workshop at HLT-NAACL 2006, pages 62–70,
New York City, New York, June 2006. c©2006 Association for Computational Linguistics
RETROSPECTIVE ANALYSIS OF COMUNICATION EVENTS - 
Understanding the Dynamics of Collaborative Multi-Party Discourse. 
 
Andrew J. Cowel Jereme Haack Adrienne Andrew 
 
Rich Interaction Environments 
Pacific Northwest National Laboratory 
Richland, WA 936 
{andrew.cowell | jereme.haack | adrienne.andrew}@pnl.gov 
 
 
Abstract 
This research is aimed at understanding the 
dynamics of colaborative multi-party dis-
course acros multiple communication mo-
dalities. Before we can truly make significant 
strides in devising colaborative communica-
tion systems, there is a need to understand 
how typical users utilize computationally sup-
ported communications mechanisms such as 
email, instant messaging, video conferencing, 
chat rooms, etc., both singularly and in con-
junction with traditional means of communi-
cation such as face-to-face meetings, 
telephone calls and postal mail. Atempting to 
understand an individual’s communications 
profile with access to only a single modality is 
challenging at best and often futile. Here, we 
discus the development of RACE – Retro-
spective Analysis of Communications Events 
– a test-bed prototype to investigate isues re-
lating to multi-modal multi-party discourse. 
We also examine future avenues of research 
for further enhancing our prototype and inves-
tigating this area. 
1 Introduction 
Communication is the heart of what makes us so-
cial creatures. Today, we have a myriad of tech-
nologies that allow us to communicate in ways our 
forefathers could never have imagined. Computa-
tionally suported modalities such as email and 
instant messaging have had immeasurable effect on 
the way we work, play and generally interact with 
those in our lives. Being able to understand how 
individuals communicate, the methods they use, 
their personal preferences etc., and are all part of a 
field called anthroposemiotics
1
. This field loks to 
uncover the mystery behind how we communicate 
with ourselves (intrapersonal communication), 
with others (interpersonal communication), within 
groups (group dynamics) and acros cultures 
(cros-cultural communication). While there is a 
great deal of literature in these fields, there are few 
operational applications that allow for true hands-
on investigation. 
Perhaps nowhere is the application of this 
field more important than in the field of intelli-
gence analysis. Intelligence analysts must make 
sound judgments, coherently constructed from 
scattered heterogeneous fragments of information 
while being faced with significant time constraints. 
The information they use is rarely complete, often 
unreliable and usually temporally and spatially 
diverse. These dimensions need to be aligned and 
the information understod to enable the analyst to 
recognize sequences of inter-related events and 
hypothesize about future actions. 
Our aim has been to aid the analyst by 
researching, designing and implementing a test-bed 
for the investigation of colaborative, multi-party 
discourse. The focus is on reducing the complexity 
of analyzing communications data through a triage 
process; from a large corpus to a small handful of 
relevant conversations to finally a highly detailed 
view of one or two conversations, with incorpo-
rated socio-behavioral dimensions. Below we pre-
sent our design methodology and discus the latest 
version of the prototype. 
                                                             
1
 htp:/en.wikipedia.org/wiki/Human_comunication 
62
2 Method 
The design methodology we used included a 
review of the literature, folowed by in-depth focus 
group discusions with working analysts to deter-
mine requirements. Folowing the group sesion, a 
participatory design process was used to gather 
more information from our user group, leading to a 
set of sketches that were used in the initial proto-
type implementation. From these an initial proto-
type was created. The test-bed is currently in its 
second phase of implementation that includes the 
integration of components developed under other 
auspices into the RACE environment. These in-
clude indicators of affect and social roles. Finally, 
we have built and colected a number of data 
sources that we intend to use to evaluate the sys-
tem. We describe these stages in the folowing sec-
tions. 
2.1 Prior Art 
This effort began with a thorough literature 
review acros the fields of ubiquitous computing, 
visualization, multi-party discourse and communi-
cations theory. A number of research systems with 
similarities to our goal were reviewed in order to 
be able to understand the landscape and determine 
where specific oportunities may lie. Here we dis-
cus some of the systems (mainly research proto-
types) that are available for reviewing 
communications data. 
As both Internet communications and com-
plex graphics capabilities have become more per-
vasive in modern computing, there has been much 
interest in visualizing conversations. Due to the 
ease of data capture with computationally sup-
ported communications, such communication mo-
dalities as email, chat, and forum/newsgroup 
threads appear to be the most researched. Several 
systems have represented vast, multi-threaded 
newsgroup or forum posts such as USENET. 
‘Lom’ can represent the activity patterns of indi-
viduals relative to one another, helping to charac-
terize individuals’ participation and roles (Donath 
et al, 199). In another view, linked posts are 
graphed to represent threads, characterizing the 
newsgroup as a whole. ‘Discourse Diagrams’ de-
scribes newsgroups with semantic graphs of related 
concepts, and also graphs people’s conectedness 
to one another in social networks (Sack, 200). 
‘Conversation Thumbnails’ uses an over-
view/detail display to contextualize a user’s post in 
the group as a whole while it is being composed 
(Wattenberg and Milen, 203). ‘PeopleGarden’ 
represents each individual participant as a compos-
ite of their history of posting. Having all partici-
pants represented in the same screen provides 
insight into the dynamics of the group as a whole 
acros its recorded history, although there is no 
way to track conections between individuals or 
threading (Xiong and Donath, 199). 
In RACE, the topics of a multitude of con-
versations are explored by an analyst loking for 
both episodic and social information. Through an 
iterative filtering process, the analyst examines 
individual conversations. Like the newsgroup visu-
alizations above, the goals are (in addition to a 
general desire to understand what is going on) to 
determine an individual’s social role and dynamic 
of the group, but the concept of “conversations” is 
more granular. Whereas the newsgroup visualiza-
tions may represent hundreds or even thousands of 
users and conversation threads, the detailed visu-
alization in RACE’s final screen represents a sin-
gle discourse with as few as two people. Thus, the 
systems above deal with a higher level of abstrac-
tion and do not convey information on “lurkers” 
who may read but not post, emotional qualities of 
contributions, or the temporal information present 
in synchronous communication. RACE has the 
additional goals of denoting presence, affect, and 
what Viegas and Donath call “negotiation of con-
versational synchrony” (199). 
Research on chat room conversation has 
produced some interesting visualizations that start 
to deal with these concepts. The ‘Babble’ system 
both facilitates and visualizes synchronous and 
asynchronous chat (Erickson and Laff, 201). Us-
ers are represented as colored dots on a social 
proxy called a ‘cookie’. The more interactions they 
have with the system, whether posting or only 
reading, the more central they become in the visu-
alization. With inactivity, the dots move slowly 
back out to the periphery of the cookie, conveying 
information about presence and activity level. 
‘Chat Circles’ is designed for synchronous chat 
and creates a strong sense of location by situating 
participants (represented as colored circles) in a 
large 2D space and only allowing them to see the 
text posted by others positioned nearby (Viegas 
and Donath, 199). The circles expand to encom-
pass posted text and shrink when ample time to 
63
read the uterance has passed. Even people who are 
idling or only listening are represented spatially so 
others can see them. People can position their cir-
cles to avoid the ‘noise’ of unrelated conversations 
(as one could do at a cocktail party) or signify 
whom they are addressing. Each post leaves a cu-
mulative translucent trace, indicating how long the 
poster has been there and how active they have 
been. Thus, group dynamics such as a group con-
versation fragmenting into smaller ones, relative 
verbosity, and relative position are available for 
interpretation. 
While each of the systems above is designed 
for a particular modality, RACE integrates email, 
instant messaging, text messaging, phone conver-
sations and teleconferences, in-person meetings in 
addition to chat or newsgroup participation. The 
goal is to get a more holistic sense of an individual 
throughout their discrete conversations and com-
munication methods. As a post-hoc analysis tol, 
RACE aids the analyst by adding system interpre-
tations of affect and social dynamics to the infor-
mation represented in the prior art. It should be 
noted that this effort violates one of Erickson’s six 
claims about social visualization: “Portray actions, 
not interpretation… users understand the context 
better than the system ever wil” (203). We agree 
in theory, but the needs of our analysts differ from 
those of a contributor to the conversation. Content-
driven interpretations of group dynamics, affect, 
and social role complement ful-text transcripts of 
the conversations, providing shortcuts to insight. 
Below we discus further the requirements of our 
user group. 
2.2 Requirements Elicitation 
To ensure our research was applicable to our or-
ganization’s misions and fulfiled the require-
ments and expectations of our user group, we 
enlisted the help of four analysts to determine spe-
cific requirements. These were to be our subject 
matter experts (SME’s). Through interactions with 
our SME’s we determined that while it is important 
to being able to understand a single conversation in 
time, it is just as, if not more, important to be able 
to comprehend the stream of conversations that 
occurs over longer periods, related to the same 
topic. For example, it is important to be able to 
intercept, process, and analyze a discusion be-
tween two individuals talking about making a 
homemade bomb, but it is even more important to 
place such a discusion within the context of the 
set of comunications leading to an understanding 
of the overarching plot. Such review can provide 
additional information that could be invaluable to 
the analyst. Other requirements identified as part of 
these sesions included: 
• The system should allow the analyst to get 
back to original source documents and be able 
to review the provenance. 
• The system should allow the analyst to anno-
tate the communication events. 
• Consider the use of color for note taking and 
marking modalities. 
• The system should allow the analyst to high-
light conversation fragments (i.e., small parts 
of a larger conversation that are considered 
important). 
• The system should provide basic translation 
mechanisms for foreign language suport as 
well as provide some form of lexicon for terms 
that fall outside an analyst’s field of expertise. 
• The system should be able to import and ex-
port conversation fragments using common 
formats. The system should allow multiple 
analysts to work colaboratively within the 
same workspace. 
• The system should allow the analyst to cus-
tomize the environment to their preferences. 
In addition to an informal list of requirements, a 
great deal of brainstorming was performed during 
this session. Folowing a participatory design proc-
ess, system designers worked with SME’s to put 
together a work process and some initial sketches 
of the overall system that could be fed into the im-
plementation stage. 
The process was designed so that the analyst 
could (Figure 1) interact with the conversation 
corpus available to them (potentially produced as a 
result of a search), viewing the conversations as 
dots, clustered around major topics. This view 
could be filtered based on time period, participants 
involved and communications modality used. 
 
64
 
Figure 1: Sketch of the Corpus View. 
 
On selecting a subset of conversations to 
review further (Figure 2) the analyst moves 
through to a second screen (the sequence view) 
where they can analyze the conversations in rela-
tion to when they occurred (the view is reminiscent 
of Microsoft Project’s Gant view). 
 
Figure 2: Sketch of the Sequence View. 
 
While icons and text wil continue to de-
pict the modality the conversation utilized, the fo-
cus at this level is of fusing the conversations to 
build a sequenced stream of communications traf-
fic so the underlying thread or purpose can be un-
derstod. Finally (Figure 3), conversations of 
specific interest to the analyst can be pursued in 
further detail in a third screen, called the ‘detail 
view’. Here, the ful transcript is displayed and can 
be ‘played’ uterance by uterance in real time. As 
each uterance is reached, a text-to-speech engine 
speaks the words, while a number of visual repre-
sentations indicate social constructs such as social 
roles and the dynamics between the individuals. 
 
 
Figure 3: Sketch of the Detail View. 
2.3 Implemented Prototype 
Using a participatory design process, in-
formed by the sketches and requirements of our 
analysts and the limitations of current research sys-
tems, we implemented a three-screen prototype 
analytical environment that allows a user to visual-
ize a large corpus of communications events 
(Figure 4). 
 
 
 
Figure 4: Analyst using the RACE Environment. 
 
The environment can run on three screens simulta-
neously, be split acros three panes (useful for per-
forming analysis on large displays like wall-
mounted plasma displays) or on a single screen 
with the use of a window manager seen in the top 
right of each view. 
For the ‘corpus view’ (left hand screen, 
Figure 5) we customized some commercially 
available visualization software to present the con-
versation corpus, clustered by topic. Zoming in to 
individual items brings up metadata about that spe-
cific conversation. The different modalities may 
also be represented by different icons or colors, 
65
depending on the type of style sheet loaded. Filters 
currently available include the modality used, the 
participants involved and the time/date the conver-
sation occurred (and shortcuts to selecting all or 
none, or the current inverse are also available). 
Finally, a navigation window ensures the user does 
not get lost when interacting with a massive data 
that is topically diverse. 
 
 
Figure 5: The Corpus View. 
 
The ‘sequence view’ (Figure 6) is where 
we envision the majority of an analyst’s time wil 
be spent. It is here that they wil review, in detail, a 
small subset of conversations that they found of 
interest in the corpus space. For example, in their 
exploration of the visualization, the analyst may 
find a group of discusions about a particular 
chemical substance. Knowing that this is relevant 
to a study they are performing, they simply drag a 
box around that subset and immediately those con-
versations are shown in the sequence view. Each 
conversation has an independent time line and can 
be zoomed out to show the entire conversation or 
zoomed in to see the individual uterances (these 
may also be accessed using tol-tips). The conver-
sation titles on the left hand side of the screen can 
be unexpanded to show all the participants in-
volved. Clicking on the participant opens a dialog 
box containing known information about that indi-
vidual (including any known aliases and other 
names they may use online). A global timeline at 
the botom of the screen shows where each conver-
sation falls in sequence. 
 
 
Figure 6: The Sequence View. 
 
Once an important conversation is uncov-
ered through the triage process, it can be selected 
for deeper investigation in the details view (Figure 
7). This view can enable the analyst to see beyond 
the individual uterances. Utilizing other research 
performed at the Pacific Northwest National Labo-
ratory, the details view enables the analyst to gain 
insight into an individual’s opinion on the topics 
discused. The transcript is color-coded to show 
the seven dimensions of affect (expression, power, 
ethics, attainment, skil, accomplishment and 
transactions), while a graph representation alows 
the analyst to compare individuals’ affect against 
each other. In order to ingest the text in different 
ways, a ‘text-to-speech’ engine can be used to have 
the computer ‘speak’ the transcript. As it steps 
through the uterances, a group dynamics graphic 
(based on Erickson’s Social Proxy) shows how the 
individuals relate to each other, highlighting those 
involved in the conversation and those that are 
idle. This view also provides a hierarchical view of 
the topics discused with the ability to triger a 
multi-dimensional visualization that maps partici-
pants to topics. 
66
 
Figure 7: The Detail View. 
3 Evaluation and Data Sets 
In addition to the prototype system, an evalua-
tion plan was developed. The current dataset being 
used to demonstrate the system was synthesized 
from news reports about the London bombings of 
7th July 205. The evaluation wil use a new 
dataset build up from telephone transcripts from 
the regional August 14, 203 blackout
2
 to ensure 
any analysts used that were involved in the devel-
opment of the prototype wil not benefit from any 
potential learning effects. This data is made up of 
several participants involved in many different 
conversations. These characteristics are exactly 
what RACE was designed for. Another dataset is a 
transcript of a murder mystery held on a chat room. 
While there was only space for characters to inter-
act, there are many different threads of conversa-
tion going on at once. This data set wil be useful 
for exploring the social dynamic part of RACE. 
We hope to show ho the conversational “drivers” 
were and explore what characteristics give some-
one away as hiding details they do not want other 
characters to discover. 
4 Sumary & Further Work 
The ultimate goal of the RACE project is to 
assist analysts as they try to extract meaning from 
a myriad of sources.  To this end, we started by 
talking with analysts themselves. This is in recog-
nition of the fact that no matter how powerful a 
tol might seem to its developers, it is useless un-
                                                             
2
 htp:/ww.nerc.com/~filez/blackout.html 
less the end users actually adopt it.  By working 
with analysts every step of the way, we are keeping 
that goal in sight. 
RACE’s design as a test bed enables other re-
search to get in front of the analyst soner.  The 
quick insertion of the text affect work ilustrates 
the capability to make functionality available to the 
user for evaluation. Showing an analyst a concrete 
example of an idea allows them to get a better un-
derstanding of it and an easier way to elicit feed-
back for future work. 
While this is an exciting first step, there are 
many avenues of crucial research stil to be per-
formed. In many fields, having access to all the 
communications events that occurred is rare. Re-
search needs to be performed to determine how 
best to enable the analyst to fil in these blanks. 
Potential approaches include hypothesized infer-
ence or the use placeholders. 
 Currently, the prototype analytical envi-
ronment only processes and displays textual tran-
scripts of communication events. This decision 
was made to handle textual content first so to en-
sure proof of principle prior to expending effort on 
the more challenging aspect of fusing video, audio, 
stil images and text (VAST). Some effort has been 
expended on loking for suitable design metaphors 
that could aid an analyst in making sense of such 
diverse media (e.g., video production user inter-
faces such as Final Cut Pro) but more research, 
design and evaluation is required. 
 More effort needs to be expended on un-
derstanding how best to fuse different modalities 
of communication. Currently, a time-shifting ap-
proach is used to normalize an asynchronous email 
thread with similar-topic synchronous communica-
tions (e.g., telephone call, instant messaging ses-
sion). This aproach works but needs to be refined 
in order to be successful. At one level, the modal-
ity used is irrelevant – it is the esence of the event 
that is of primary concern. Being able to boil down 
the associated threads into one specific stream 
(e.g., multiple conversations acros a number of 
modalities, all around the topic of ploting to ex-
plode a device at a particular location) is crucial in 
being able to suport the analytical tradecraft and 
allow analysts to provide actionable intelligence to 
their superiors. 
 Conversations rarely keep to one single 
focused topic, and this can cause problems in the 
cluster visualization type approach used so far. 
67
Topic segmentation is a difficult research area and 
not one that we intend to pursue. There are at least 
three projects currently on the way at our institu-
tion that deal with this area and this work intends 
to utilize the fruits of those labors. 
 Finally, there are many elements of multi-
party discourse that exist outside linguistic bounda-
ries. The words we use, how often we make an 
uterance, etc., all speak to who we are as individu-
als. While some of this is obvious and can be ob-
served with just a cursory review of a transcript of 
the source material, other elements are discrete and 
hiden. For example, conversational statistics can 
be recorded and used to determine an individual’s 
level of engagement in a topic. Detection of fa-
miliarity (e.g., either by specific words not cur-
rently found in the present conversation or through 
the use of casual rather than formal speech) can 
indicate personal relationships between individuals 
in a dyad. Personality types can be inferred by 
markers indicative of leadership (e.g., number of 
interruptions performed/received, ability to change 
topic, use of power terms) or weaker, subversive 
roles (e.g., use of weak terms, submision of flor, 
deference to others). Analysts are rarely able to 
access such rich personality profiles of their sub-
jects without performing an exhaustive analysis or 
calling in specialized help. While we are just be-
gining to integrate certain elements of social dis-
course, there are many other dimensions to be 
considered.
 
References 
Donath, J., Karahalios, K., and Viegas, F. 199. 
"Visualizing conversations," Proceedings of 
HICSS-32, Persistent Conversations Track. Maui, 
Hawaii, January 5-8. 
Sack, W. 200. "Discourse Diagrams: Interface 
Design for Very Large Scale Conversations," Pro-
ceedings of the HICSS-3, Persistent Conversa-
tions Track . Maui, Hawaii, January. 
Wattenberg, M. M & Milen, D. R., (203) Con-
versation Thumbnails for Large-Scale Discusions. 
. In Extended Abstracts – CHI 203. Ft. Lauder-
dale, April 8-10. 
Xiong, R. and Donath, J. PeopleGarden: Creating 
Data Portraits for Users, in Proceedings of the 12 
th Anual ACM Symposium on User Interface 
Software and Technology, New York: ACM, p 
37-4. 
Viegas , F. B. & Donath, J. S. (199). Chat Circles. 
CHI 199 Conference Proceedings: Conference on 
Human Factors in Computing Systems. New York: 
ACM Press, 9-16. 
Erickson T and Laff MR (201) The design of the 
‘Babble’ timeline: A social proxy for visualizing 
group activity over time. Extended Abstracts: Hu-
man Factors in Computing System (CHI 201), 
329-20. 
Erickson, T. Designing Visualizations of Social 
Activity: Six Claims, The Proceedings of CHI 
203: Extended Abstracts. New York: ACM Press, 
203.
