Multi-Speaker Language Modeling
Gang Ji and Jeff Bilmes  
SSLI Lab, Department of Electrical Engineering
University of Washington
Seattle, WA 98195-2500
fgang,bilmesg@ee.washington.edu
Abstract
In conventional language modeling, the words
from only one speaker at a time are repre-
sented, even for conversational tasks such as
meetings and telephone calls. In a conversa-
tional or meeting setting, however, speakers
can have signi cant in uence on each other.
To recover such un-modeled inter-speaker in-
formation, we introduce an approach for con-
versational language modeling that considers
words from other speakers when predicting
words from the current one. By augmenting a
normal trigram context, our new multi-speaker
language model (MSLM) improves on both
Switchboard and ICSI Meeting Recorder cor-
pora. Using an MSLM and a conditional mu-
tual information based word clustering algo-
rithm, we achieve a 8.9% perplexity reduction
on Switchboard and a 12.2% reduction on the
ICSI Meeting Recorder data.
1 Introduction
Statistical language models (LMs) are used in many ap-
plications such as speech recognition, handwriting recog-
nition, spelling correction, machine translation, and in-
formation retrieval. The goal is to produce a probability
model over a word sequence P(W) = P(w1;   ;wT ).
Conventional language models are often based on a fac-
torized form P(W)  Qt P(wtj (ht)), where ht is the
history for wt and  is a history mapping.
The case of n-gram language modeling, where
 (ht) = wt n+1    wt 1, is widely used. Typically,
n = 3, which yields a trigram model. A re nement of
this model is the class-based n-gram where the words are
partitioned into equivalence classes (Brown et al., 1992).
 This work was funded by NSF under grant IIS-0121396.
In general, smoothing techniques are applied to lessen
the curse of dimensionality. Among all methods, mod-
i ed Kneser-Ney smoothing (Chen and Goodman, 1998)
is widely used because of its good performance.
Modeling conversational language is a particularly dif-
 cult task. Even though conventional techniques work
well on read or prepared speech, situations such as tele-
phone conversations or multi-person meetings pose great
research challenges due to dis uencies, and odd syn-
tactic/discourse patterns. Other dif culties include false
starts, interruptions, and poor or unrepresented grammar.
Most state-of-the-art language models consider word
streams individually and treat different phrases indepen-
dently. In this work, we introduce multi-speaker lan-
guage modeling (MSLM), which models the effects on
a speaker of words spoken by other speakers partici-
pating in the same conversation or meeting. Our new
model achieves initial perplexity reductions of 6.2% on
Switchboard-I, 5.8% on Switchboard Eval-2003, and
10.3% on ICSI Meeting data. In addition, we devel-
oped a word clustering procedure (based on a standard
approach (Brown et al., 1992)) that optimizes conditional
word clusters. Our class-based MSLMs using our new
algorithm yield improvements of 7.1% on Switchboard-I,
8.9% on Switchboard Eval-2003, and 12.2% on meetings.
A brief outline follows: Section 2 introduces multi-
speaker language modeling. Section 3 provides ini-
tial evaluations on Switchboard and the ICSI Meeting
data. Section 4 presents evaluations using our class-based
multi-speaker language models, and Section 5 concludes.
2 Multi-speaker Language Modeling
In a conversational setting, such as during a meeting or
telephone call, the words spoken by one speaker are af-
fected not only by his or her own previous words but
also by other speakers. Such inter-speaker dependency,
however, is typically ignored in standard n-gram lan-
guage models. In this work, information (i.e., word to-
kens) from other speakers (A) is used to better predict
word tokens of the current speaker (W). When predict-
ing wt, instead of using P(wtjw0;   ;wt 1), the form
P(wtjw0;   ;wt 1;a0;   ;at) is used. Here at repre-
sents a word spoken by some other speaker with appro-
priate starting time (Section 3). A straight-forward im-
plementation is to extend the normal trigram model as:
P(wtj (ht)) = P(wtjwt 1;wt 2;at): (1)
Hi
Hi
Diana  Take care
 Bye
A:
W:
(a)
Saturday
A:
W: Saturday
on Sundayon 0.2s or Saturday0.4s
C5:
C4:
C3:
C1:
C0:
Saturday
Saturday
Saturday 1.1s SaturdayC2:
on Sundayon or Saturday
(b)
Figure 1: Examples of phone conversation (a) and meet-
ing (b). (Frame sizes are not proportional to time scale.)
Figure 1 shows an example from Switchboard (a) and
one from a meeting (b). In (a), only two speakers are in-
volved and the words from the current speaker, W, are
affected by the other speaker, A. At the beginning of a
conversation, the response to  Hi is likely to be  Hi or
 Hello. At the end of the phone call, the response to
 Take care might be  Bye , or  You too , etc. In (b),
we show a typical meeting conversation. Speaker C2 is
interrupting C3 when C3 says  Sunday . Because  Sun-
day is a day of the week, there is a high probability that
C2’s response is also a day of the week. In our model, we
only consider two streams at a time, W and A. There-
fore, when considering the probability of C2’s words, it
is reasonable to collapse words from all other speakers
(C0,C1,C3,C4, and C5) into one stream A as shown in
the  gure. This makes available to C2 the rest of the
meeting to potentially condition on, although it does not
distinguish between different speakers.
Our model, Equation 1, is different from most lan-
guage modeling systems since our models condition on
both previous words and another potential factor A. Such
a model is easily represented using a factored language
model (FLM), an idea introduced in (Bilmes and Kirch-
hoff, 2003; Kirchhoff et al., 2003), and incorporated into
the SRILM toolkit (Stolcke, 2002). Note that a form of
cross-side modeling was used by BBN (Schwartz, 2004),
where in a multi-pass speech recognition system the out-
put of a  rst-pass from one speaker is used to prime words
in the language model for the other speaker.
3 Initial Evaluation
We evaluate MSLMs on three corpora: Switchboard-
I, Switchboard Eval-2003, and ICSI Meeting data. In
Switchboard-I, 6.83% of the words are overlapped in
time, where we de ne w1 and w2 as being overlapped
if s(w1)  s(w2) < e(w1) or s(w2)  s(w1) < e(w2),
where s( ) and e( ) are the starting and ending time of a
word.
The ICSI Meeting Recorder corpus (Janin et al., 2003)
consists of a number of meeting conversations with three
or more participants. The data we employed has 32 con-
versations, 35,000 sentences and 307,000 total words,
where 8.5% of the words were overlapped. As mentioned
previously, we collapse the words from all other speakers
into one stream A as a conditioning set for W. The data
consists of all speakers taking their turn being W.
To be used in an FLM, the words in each stream need to
be aligned at discrete time points. Clearly, at should not
come from wt’s future. Therefore, for each wt, we use
the closest previous A word in the past for at such that
s(wt 1)  s(at) < s(wt). Therefore, each at is used
only once and no constraints are placed on at’s end time.
This is reasonable since one can often predict a speaker’s
word after it starts but before it completes.
We score using the model P(wtjwt 1;wt 2;at).1 Dif-
ferent back-off strategies, including different back-off
paths as well as combination methods (Bilmes and Kirch-
hoff, 2003), were tried and here we present the best re-
sults. The backoff order (for Switchboard-I and Meeting)
 rst dropped at, then wt 2, wt 1, ending with the uni-
form distribution. For Switchboard eval-2003, we used
a generalized parallel backoff mechanism. In all cases,
modi ed Kneser-Ney smoothing (Chen and Goodman,
1998) was used at all back-off points.
Results on Switchboard-I and the meeting data em-
ployed 5-fold cross-validation. Training data for Switch-
board eval-2003 consisted of all of Switchboard-I. In
Switchboard eval-2003, hand-transcribed time marks are
unavailable, so A was available only at the beginning of
utterances of W.2 Results (mean perplexities and stan-
dard deviations) are listed in Table 1 (Switchboard-I and
meeting) and the jV j column in Table 3.
Table 1: Perplexities from MSLM on Switchboard-I
(swbd-I) and ICSI Meeting data (mr)
data trigram four-gram mslm reduction
swbd-I 73.2 0.4 73.7 0.4 68.5 0.3 6.2%
mr 87.4 4.6 89.5 4.9 78.4 2.7 10.3%
1In all cases, end of sentence tokens, </s>, were not scored
to avoid arti cially small perplexities arising when wt = at =
</s>, since P(</s>j</s>) yields a high probability value.
2Having time-marks, say, via a forced alignment would
likely improve our results.
In Table 1, the  rst column shows data set names. The
second and third columns show our best baseline trigram
and four-gram perplexities, both of which used interpo-
lation and modi ed Kneser-Ney at every back-off point.
The trigram outperforms the four-gram. The fourth col-
umn shows the perplexity results with MSLMs and the
last column shows the MSLM’s relative perplexity reduc-
tion over the (better) trigram baseline. This positive re-
duction indicates that for both data sets, the utilization of
additional information from other speakers can better pre-
dict the words of the current speaker. The improvement is
larger in the highly conversational meeting setting since
additional speakers, and thus more interruptions, occur.
3.1 Analysis
It is elucidating at this point to identify when and how
A-words can help predict W-words. We thus computed
the log-probability ratio of P(wtjwt 1;wt 2;at) and the
trigram P(wtjwt 1;wt 2) evaluated on all test set tuples
of form (wt 2;wt 1wt;at). When this ratio is large and
positive, conditioning on at signi cantly increases the
probability of wt in the context of wt 1 and wt 2. The
opposite is true when the ratio is large and negative. To
ensure the signi cance of our results, we de ne  large 
to mean at least 101:5  32, so that using at makes wt at
least 32 times more (or less) probable. We chose 32 in a
data-driven fashion, to be well above any spurious prob-
ability differences due to smoothing of different models.
At the  rst word of a phrase spoken by W, there are
a number of cases of A words that signi cantly increase
the probability of a W word relative to the trigram alone.
This includes (in roughly decreasing order of probabil-
ity) echos (e.g., when A says  Friday , W repeats it),
greetings/partings (e.g., a W greeting is likely to follow
an A greeting), paraphrases (e.g.,  crappy followed by
 ugly , or  Indiana followed by  Purdue ), is-a relation-
ships (e.g., A saying  corporation followed by W saying
 dell , A- actor followed by W- Swayze , A- name 
followed by W- Patricia , etc.), and word completions.
On the other hand, some A contexts (e.g., laughter) sig-
ni cantly decrease the probability of many W words.
Within a W phrase, other patterns emerge. In
particular, some A words signi cantly decrease the
probability that W will  nish a commonly-used phrase.
For example, in a trigram alone, p(biggerjand; bigger),
p(forthjand; back), and p(easyjand; quick), all have
high probability. When also conditioning on A, some
A words signi cantly decrease the probability of
 nishing such phrases. For example, we  nd that
p(easyjand; quick;  uh-hmm )  p(easyjand; quick).
A similar phenomena occurs for other com-
monly used phrases, but only when A has uttered
words such as  yeah ,  good ,  ok ,  [laugh-
ter] ,  huh , etc. While one possible explanation
of this is just due to decreased counts, we found
that for such phrases p(wtjwt 1;wt 2;at)  
minwt 32S p4(wtjwt 1;wt 2;wt 3) where p4 is a
four-gram, S = fw : C(wt;wt 1;wt 2;w) > 0g, and
C is the 4-gram word count function for the switchboard
training and test sets. Therefore, our hypothesis is
that when W is in the process of uttering a predictable
phrase and A indicates she knows what W will say, it is
improbable that W will complete that phrase.
The examples above came from Switchboard-I, but we
found similar phenomena in the other corpora.
4 Conditional Probability Clustering
100 500 1000 1500 2000 |V|
68
70
72
74
76
78
80
82
84
86
88
number of classes
p
e
r
p
l
e
x
i
t
y
mr baselinemr
swbd baselineswbd
Figure 2: Class-based MSLM from MCMI clustering on
Switchboard-I (swbd) and ICSI Meeting (mr) data.
Table 2: Three types of class-based MSLMs on
Switchboard-I (swbd) and ICSI Meeting (mr) corpora
# of swbd mr
classes BROWN MMI MCMI BROWN MMI MCMI
100 68.9 0.3 68.4 0.3 68.2 0.3 78.9 3.0 77.3 2.8 76.8 2.8
500 68.9 0.3 68.3 0.3 67.9 0.3 78.7 3.1 77.1 2.8 76.7 2.8
1000 68.9 0.3 68.2 0.3 67.9 0.3 79.0 3.1 77.2 2.7 76.9 2.8
1500 69.0 0.3 68.2 0.3 68.0 0.3 79.6 3.1 77.4 2.7 77.4 2.7
2000 69.0 0.3 68.3 0.3 68.0 0.3 80.1 3.1 77.6 2.7 77.9 2.7
jV j 68.5 0.3 78.3 2.7
Table 3: Class-based MSLM on Switchboard Eval-2003
size 100 500 1000 1500 2000 jV j 3-gram 4-gram
ppl 65.8 65.5 65.6 65.7 66.1 67.9 72.1 76.3
% reduction 8.6 8.9 8.8 8.7 8.3 5.8 0 -5.8
Class-based language models (Brown et al., 1992;
Whittaker and Woodland, 2003) yield great bene ts when
data sparseness abounds. SRILM (Stolcke, 2002) can
produce classes to maximize the mutual information
between the classes I(C(wt);C(wt 1)), as described
in (Brown et al., 1992). More recently, a method for clus-
tering words at different positions was developed (Ya-
mamoto et al., 2001; Gao et al., 2002). Our goal is
to produce classes that improve the scores P(wtjht) =
P(wtjwt 1;wt 2;C1(at)), what we call class-based
MSLMs. In our case, the vocabulary for A is partitioned
into classes by either maximizing conditional mutual
information (MCMI) I(wt;C(at)jwt 1;wt 2) or just
maximizing mutual information (MMI) I(wt;C(at)).
While such clusterings can perform poorly under low
counts, our results show further consistent improvements.
Our new clustering procedures were implemented into
the SRILM toolkit. When partitioned into smaller
classes, the A-tokens are replaced by their correspond-
ing class IDs. The result is then trained using the same
factored language model as before. The resulting per-
plexities for the MCMI case are presented in Figure 2,
where the horizontal axis shows the number of A-stream
classes (the right-most shows the case before clustering),
and the vertical axis shows average perplexity. In both
data corpora, the average perplexities decrease after ap-
plying class-based MSLMs. For both Switchboard-I and
the meeting data, the best result is achieved using 500
classes (7.1% and 12.2% improvements respectively).
To compare different clustering algorithms, results
with the standard method of (Brown et al., 1992)
(SRILM’s ngram-class) are also reported. All the per-
plexities for these three types of class-based MSLMs are
given in Table 2. For Switchboard-I, ngram-class does
slightly better than without clustering. On the meeting
data, it even does slightly worse than no clustering. Our
MMI method does show a small improvement, and the
perplexities are further (but not signi cantly) reduced us-
ing our MCMI method (but at the cost of much more
computation during development).
We also show results on Switchboard eval-2003 in Ta-
ble 3. We compare an optimized four-gram, a three-
gram baseline, and various numbers of cluster sizes us-
ing our MCMI method and generalized backoff (Bilmes
and Kirchhoff, 2003), which, (again) with 500 clusters,
achieves an 8.9% relative improvement over the trigram.
5 Discussions and Conclusion
In this paper, novel multi-speaker language modeling
(MSLM) is introduced and evaluated. After simply
adding words from other speakers into a normal trigram
context, the new model shows a reasonable improvement
in perplexity. This model can be further improved when
class-based cross-speaker information is employed. We
also presented two different criteria for this clustering.
The more complex criteria gives similar results to the
simple one, presumably due to data sparseness. Even
though Switchboard and meeting data are different in
terms of topic, speaking style, and speaker number, one
might more robustly learn cross-speaker information by
training on the union of these two data sets.
There are a number of ways to extend this work. First,
our current approach is purely data driven. One can imag-
ine that higher level information (e.g., a dialog or other
speech act) about the other speakers might be particularly
important. Latent semantic analysis of stream A might
also be usefully employed here. Furthermore, more than
one word from stream A can be included in the context
to provide additional predictive ability. With the meet-
ing data, there may be a bene t to controlling for spe-
ci c speakers based on their degree of in uence. Alterna-
tively, an MSLM might help identify the most in uential
speaker in a meeting by determining who most changes
the probability of other speakers’ words.
Moreover, the approach clearly suggests that a multi-
speaker decoder in an automatic speech recognition
(ASR) system might be bene cial. Once time marks for
each word are provided in an N-best list, our MSLM
technique can be used for rescoring. Additionally, such
a decoder can easily be speci ed using graphical mod-
els (Bilmes and Zweig, 2002) in  rst-pass decodings.
We wish to thank Katrin Kirchhoff and the anonymous
reviewers for useful comments on this work.
References
J. Bilmes and K. Kirchhoff. 2003. Factored language
models and generalized parallel backoff. In Human
Language Technology Conference.
J. Bilmes and G. Zweig. 2002. The graphical models
toolkit: An open source software system for speech
and time-series processing. In Proc. ICASSP, June.
P. Brown, V. Della Pietra, P. deSouza, J. Lai, and R. Mer-
cer. 1992. Class-based n-gram models of natural lan-
guage. Computational Linguistics, 18(4):467 479.
S. Chen and J. Goodman. 1998. An empirical study of
smoothing techniques for language modeling. Techni-
cal Report TR-10-98, Computer Science Group, Har-
vard University, August.
J. Gao, J. Goodman, G. Cao, and H. Li. 2002. Exploring
asymmetric clustering for statistical langauge model-
ing. In Proc. of ACL, pages 183 190, July.
A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart,
N. Morgan, B. Peskin, T. Pfau, E. Shriberg, A. Stolcke,
and C. Wooters. 2003. The ICSI Meeting Corpus. In
Proc. ICASSP, April.
K. Kirchhoff, J. Bilmes, S. Das, N. Duta, M. Egan, G. Ji,
F. He, J. Henderson, D. Liu, M. Noamany, P. Schone,
R. Schwartz, and D. Vergyri. 2003. Novel approaches
to arabic speech recognition: Report from the 2002
Johns-Hopkins workshop. In Proc. ICASSP, April.
R. Schwartz. 2004. Personal communication.
A. Stolcke. 2002. SRILM  an extensible language mod-
eling toolkit. In Proc. Int. Conf. on Spoken Language
Processing, September.
E. Whittaker and P. Woodland. 2003. Language mod-
elling for Russian and English using words and classes.
Computer Speech and Language, pages 87 104.
H. Yamamoto, S. Isogai, and Y. Sagisaka. 2001. Multi-
class composite n-gram language model for spoken
language processing using multiple word clusters. In
Proc. of ACL, pages 531 538.
