AN INTEGRATED MODEL FOR ANAPHORA RESOLUTION 
Ruslan Mitkov 
Institute of Mathematics 
Acad. G. Bonchev str. bl.8, 1113 Sofia, Bulgaria 
ABSTRACT 
The paper discusses a new knowledge- 
based and sublanguage-oriented model 
for anaphora resolution, which integrates 
syntactic, semantic, discourse, domain 
and heuristical knowledge for the 
sublanguage of computer science. Special 
attention is paid to a new approach for 
tracking the center throughout a discourse 
segment, which plays an imtx~rtant role in 
proposing the most likely antecedent to 
the anaphor in case of ambiguity. 
INTRODUCTION 
Anaphora resolution is a complicated 
problem in computational linguistics. 
Considerable research has been done by 
computational linguists (\[Carbonell & 
Brown 88\], IDahl & Ball 90\], 
\[Frederking & Gchrke 87\], \[Hayes 81\], 
\[Hobbs 78\], \[lngria & Stallard 89\], 
\[Preug et al. 9411, \[Rich & LuperFoy 88\[, 
\[Robert 89\]), but no complete theory has 
emerged which offers a resolution 
procedure with success guaranteed. All 
approaches developed - even if we restrict 
our attention to pronominal anaphora, 
which we will do throughout this paper - 
from purely syntactic ones to highly 
semantic and pragmatic ones, only 
provide a partial treatment of the problem. 
Given the complexity of the problem, we 
think that to secure a comparatively 
successful handling of anaphora 
resolution one should adhere to the 
following principles: l) restriction to a 
domain (sublanguage) rather than focus 
on a particular natural language as a 
whole; 2) maximal use of linguistic 
information integrating it into a uniform 
architecture by means of partial theories. 
Some more recent treatments of anaphora 
(\[Carbonell & Brown 88\], \[Preug et al. 
941, \[Rich & LuperFoy 8811) do express 
the idea of "multi-level approach", or 
"distributed architecture", but their ideas 
a) do not seem to capture enough 
discourse and heuristical knowledge and 
b) do not concentrate on and investigate a 
concrete domain, and thus risk being too 
general. We have tried nevertheless k) 
incorporate some of their ideas into our 
proposal. 
THE ANAPttORA RESOLUTION 
MODEL 
Our anaphora resolution model integrates 
modules containing different types of 
knowledge - syntactic, semantic, domain, 
discourse and heuristical knowledge. All 
the modules share a common 
representation of the cunent discourse. 
The syntactic module, for example, 
knows that the anaphor and antecedent 
must agree in number, gender and 
person. It checks if the c-command 
constraints hold and establishes disjoint 
reference. In cases of syntactic 
parallelism, it prefers the noun phrase 
with the same syntactic role as the 
anaphor, as the most probable antecedent. 
It knows when cataphora is possible and 
can indicate syntactically topicalized noun 
phrases, which are more likely to be 
antecedents than non-topicalized ones. 
The semantic module checks for semantic 
consistency between the anaphor and the 
possible antecedent. It filters out 
semantically incompatible candidates 
following the cun-ent verb semantics or 
the animacy of the candidate. In cases of 
semantic parallelism, it prefers the noun 
phrase, having the same semantic role as 
the anaphor, as a most likely antecedent. 
Finally, it generates a set of possible 
antecedents whenever necessary. 
The domain knowlcdge module is 
practically a knowlcdge basc of the 
concepts of the domain considered and 
1170 
thc discourse knowledge module knows 
how to track the center throughout the 
current discourse segment. 
The heuristical knowledge module can 
solnetimes bc helpful in assigning the 
antecedent. It has a set of useful rules 
(e.g. the antecedent is to be located 
preferably in thc current sentence or in the 
previous one) and can forestall certain 
impractical search procedures. 
The use of colnmon sense and world 
knowledge is in general commendable, 
but it requires a huge knowledge base and 
set of inference rules. The present version 
of the model does not have this mcxtule 
implementcd; its development, however, 
is envisaged for later stages of the project. 
The syntactic and semantic modules 
usually filter the possible candidates and 
do not propose an antecedent (with the 
exception of syntactic and semantic 
parallelism). Usually the proposal for an 
antecedent comes from the domain, 
heuristical, and discourse modules. The 
latter plays an important role in tracking 
the center and proposes it in many cases 
as the most probable candidate for an 
antecedent. 
Figurc 1 illustrates the general structure of 
our anaphom resolution model. 
IIIiURISTICAI, 
KNOWI ,I ';l X;t i 
l)omain lleuristics 
Rating Rules 
Recency 
\[ Rl:ilqlilil iNTIA\], l ihJ 
ANAPIIOR-~--~.I  ixptaisstoN 
SYNTACTIC KNOW\] ,t il )(}l i 
Number Agrccmenl 
Gender Agfeelncnl 
PCI'SOll A~l'Celllellt 
l)isjoim Reference 
(~-(~ommaud Constraints 
Cataphora 
Syntactic Paralldislll 
Syntactic Topicalization 
1)OMAIN I)|SCOURSI ~, 
KNOWI ,El X\]! i KNOW I ,l ilX\]l'~ 
l)omain Concept '1 'racking Center Ktlowledgc 1 ~ase 
ANAPIIORA 
P,I {SOl ,VI {R 
ANTI iCI il)ENT 
St,',MANTI( 
KNOW1,1 '~I)GE 
Semm~tic Consistency 
Case Roles 
Semantic Parallelism 
Animacy 
Set Generalion 
Figure 1: Anaphora resolution model 
THE NEED FOR DISCOURSE 
CRITERIA 
Although the syntactic and semantic 
criteria for the selection of an antecedent 
are already very strong, they are not 
always sufficient to discriminate alnong a 
set of possible candidates. Moreover, 
they serve more as filters to eliminate 
unsuitable candidates than as protx)sers of 
the most likely candidate. Additional 
criteria are therefore needed. 
As an illustration, considerthe following 
text. 
Chapter 3 discusses these additional or 
auxiliary storage devices, wlfieh mc 
similar to our own domestic tape 
cassettes and record discs. Figure 2 
illustrates lheir connection to the main 
cenlral memory. 
1171 
In this discourse segment neither the 
syntactic, nor the semantic constraints can 
eliminate the ambiguity between "storage 
devices", "tape cassettes" or "record 
discs" as antecedents for "their", and thus 
cannot turn up a plausible antecedent from 
among these candidates. A human reader 
would be in a better position since he 
would be able to identify the central 
concept, which is a primary candidate for 
pronominalization. Correct identification 
of the antecedent is possible on the basis 
of the pronominal reference hypothesis: in 
every sentence which contains one or 
more pronouns must have one of its 
pronouns refer to the center 1 of the 
previous sentence. Therefore, whenever 
we have to find a referent of a pronoun 
which is alone in the sentence, we have to 
look for the centered clement in the 
previous sentence. 
Following this hypothesis, and 
recognizing "storage devices" as the 
center, an anaphora resolution model 
would not have problems in picking up 
the center of the previous sentence 
("storage devices") as antecedent for 
"their". 
We see now that the main problem which 
arises is the tracking of the center 
throughout the discourse segment. 
Certain ideas and algorithms for tracking 
focus or center (e.g. \[Brennan et al.87\]) 
have been proposed, provided that one 
knows the focus or center of the first 
sentence in the segment. However, they 
do not try to identify this center. Our 
approach determines the most probable 
center of the first sentence, and then 
tracks it all the way through the segment, 
correcting the proposed algorithm at each 
step. 
TRACKING THE CENTER IN "\['HE 
SUBLANGUAGE OF COMPUTER 
SCIENCE 
Identifying center can be very helpful in 
1 Though "center" is ml uncrancc specific notion, 
we refer to "sentence center", because in many 
cases the centers of the uttermmes a senlence may 
consist of, coincide. In a complex sentence, 
however, we distinguish also "clause centers" 
anaphora resolution. Usually a center is 
the most likely candidate for 
pronominalization. 
There are different views in literature 
regarding the preferred candidate for a 
center (focus). Sidner's algorithm 
(\[Sidner 811), which is based on thematic 
roles, prefers the theme o\[ the previous 
sentence as the focus of the current 
sentence. This view, in general, is 
advocated also in (\[Allen87\]). PUNDIT, 
in its current implementation, considers 
the entire previous utterance to be the 
potential focus (\[Dahl&Ball 901). Finally, 
in the centering literature (\[Brennan et al. 
87\]), the subject is generally considered 
to be preferred. We have found, 
however, that there are many additional 
interrelated factors which influence upon 
the location of the center. 
Wc studied the "behaviour" of center in 
various computer science texts (30 
different sources totally exceeding 1000 
pages) and the empirical observations 
enabled us to develop efficient 
sublanguage-dependent heuristics for 
tracking the center in the sublanguage of 
computer science. We summarize the 
most important conclusions as follows: 
1) Consider the primary candidates 
for center from the priority list: 
subject, object, verb phrase. 
2) Prefer the NP, representing a 
domain concept to the NPs, which 
are not domain concepts. 
3) If the verb is a member of the 
Verb set = {discuss, present, 
illustrate, summarize, examine, 
describe, define, show, check, 
develop, review, report, outline, 
consider, investigate, explore, 
assess, analyze, synthesize, study, 
survey, deal, cover}, then consider 
the object as a most probable 
center. 
4) If a verbal adjective is a member 
of" the Adj set = {defined, called, 
so-called}, then consider the NP 
they refer to as the probable center 
of the subsequent clause/current 
sentence. 
1172 
5) If the subject is "chapter", 
"section", "table", or a personal 
pronoun - 'T', "we", "you", then 
consider the object as most likely 
center. 
6) If a NP is repeated throughout 
the discourse section, then consider 
it as the most probable center. 
7) Kf an NP occurs in the head of 
the section, part of which is the 
current discourse scgment, then 
consider it as the probable center. 
8) If a NP is topicaKized, then 
consider it as a probable center. 
9) Prefer definite NPs to indefinite 
ones. 
K0) Prefer the NPs in the main 
cKausc to NPs in the subordinate 
clauses. 
K 1) If the sentence is complex, then 
prefer for an antecedent a noun 
phrase from the previous chmse 
within the same sentence. 
As far as rule I is concerned, we found 
that the subject is a primary candidate for 
center in about 73% of the cases. The 
second most likely center would be the 
object (25%) and the third most likely 
one the verb phrase as a whole (2%). 
Therefore, the priority list \[subject, 
object, verb phrase\] is considered in 
terms of the apriori estimated probability. 
There are certain 'symptoms' which 
determine the subject or the object as a 
center with very high probability. Cases 
in point are 3) and 5). Other cases are not 
st) certain, but to some extent quite likely. 
For example, iK a non-concept NP is in 
subject position and if a repeated concept 
NP, which is also in a head, is in object 
position, it is ahnost certain that the latter 
is the unambiguous center. Moreover, 
certain preferences are stronger than 
others. For example an NP in subject 
position is preferred over an NP in a 
section head, but not in subject position. 
We have made use of our empirical 
results (with approximating probability 
lneasures) and AK techniques to develop a 
proposer module which identifies the 
most likely center. We must point out that 
even iK we do not need one for immediate 
antecedent disambiguation, a center must 
still be proposed for each sentence. O" 
else we will have to go all the way back to 
track it from the beginning of the segment 
when one is needed later on. 
The rules 1)- 11) should be ordered 
according to their priority - a problem, 
which is being currently investigated. 
Tracking the center in a discourse 
segment is very important since knowing 
the center of each current sentence helps 
in many cases to make correct decisions 
about an antecedent in the event that 
syntactic and semantic constraints cannot 
discriminate among the available 
candidates. 
AN ARTIFICIAL INTELLIGENCE 
APPROACH FOR CALCULATING 
THE PROBABILITY OF A NOUN 
(VERB) PHRASE.TO BE IN THE 
CENTER 
On the basis of the results described in the 
previous section, we use an artificial 
intelligence approach to determine the 
probability of a noun (verb) phrase to be 
the center of a sentence. Note that this 
approach allows us to calculate this 
probability in every discourse sentence, 
including the first one and to propose the 
most probable center. This approach, 
combined with the algorithm tbr tracking 
the center (\[Brcnnan et al. 87\]), is 
expected to yield improvcxl results. 
Our model incorporatcs an AI algorithm 
for calculating the probability of a noun 
(verb) phrase to be in the center of a 
discourse segment. The algorithm uses an 
inference engine based on Bayes' 
theorem: 
P(HK) P(AII-IK) 
P(I t\[dA) ................... 
P(I KOP(AIH~) 
forK = 1,2,... 
Under the conditions of our model Bayes' 
theorem allows the following 
1173 
interpretation: there are only two possible 
hypotheses for a certain noun (verb) 
phrase - that it is the center of the current 
sentence (clause) or that it is not. Let Hy 
be the positive, while HN - the negative 
hypothesis. If we call the presence of 
some of the pieces of evidence, described 
in the previous section, a "symptom", 
then let A denote the occurrence of that 
symptom with the examined phrase. 
P(AIHy) would be the apriori probability 
of the symptom A being observed with a 
noun (verb) phrase which is the center 
(we will henceforth refer to this factor as 
Py). By analogy P(AIHN) is the 
probability of the symptom being 
observed with a phrase which is not the 
center (henceforth referred to as PN). The 
aposteriori probability P(HKtA) is defined 
in the light of the new piece of evidence - 
the presence of an empirically obtained 
symptom, indicating the higher 
probability the examined phrase to be in 
the center of the discourse segment. 
In other words, inference engine based on 
Bayes' theorem draws an inference in the 
light of some new piece of evidence. This 
formula calculates the new probability, 
given the old probability plus some new 
piece of evidence. 
Consider the following situation. 
According to our investigation so far, the 
probability of the subject being a center is 
73%. Additional evidence (symptom), 
e.g. if the subject represents a domain 
concept, will increase the initial 
probability. If this NP is also the head of 
the section, the probability is increased 
further. If the NP occurs more than once 
in the discourse segment, the probability 
gets even higher. 
An estimation of the probability of a 
subject, (direct or indirect) object or verb 
phrase (the only possible centers in our 
texts) to be centers, can be represented as 
a predicate with arguments: 
center (X, PI, \[symptoml (weight 
factorl 1' weight factorl2 ) .... symptomN 
(weight factorN~, weight factolNz)\] ) 
where center (X, I, list) represents the 
estimated probability of X to be the center 
of a sentence (clause), X E {subjec t, 
objectl, object2 .... verb phrase} and Pl 
is the initial probability of X to be the 
center of the sentence (clause). 
Weight factorl is the probability of the 
symptom being observed with a noun 
(verb) phrase which is the center (Py). 
Weight factor2 is the probability of the 
symptom being observed with a noun 
(verb) phrase wiaich is not the center 
(PN). 
Following our preliminary results, wc can 
write in Prolog notation: 
center (object, 25, \[sylnptoln (verb_set, 
40, 3), symptom (subject set, 40,2), 
symptom (domain concept (95, 80), 
symptom (repeated, 10, 5), symptom 
(headline, 10, 9)11, symptoln (topicalizexl, 
6, 2), symptom (main chmsc (85, 30), 
symptom (definite.form (90, 7t))\]). 
center (subject, 73, Isymptonl 
(domain concept (95, 70), symptom 
(repeated, 10, 4), symptom (headline, 
10, 8), symptom (topicalized, 10, 3), 
symptom (main_clause (85, 30), 
symptom (definite_form (85, 20)11). 
The first fact means that the object is the 
center in approximately 25% of the cases. 
Moreover, it suggests that in 40% of the 
cases where the center is the object, the 
verb belongs to the set of verbs {discuss, 
illustrate, summarize, examine, describe, 
define...} and it is possible with 3% 
probability for the verb to be a member of 
this set while the center of the sentence is 
not the object. 
The above Prolog facts are part of a 
sublanguage knowledge base. 
The process of estimating the probability 
of a given phrase being the center of a 
sentence (clause), is repetitive, beginning 
with an initial estimate and gradually 
working towards a more accurate answer. 
More systematically, the "diagnostic" 
process is as follows: 
- start with the initial probability 
- consider the symptoms one at a time 
- for each symptom, update the current 
probability, taking into account: a) 
whether the sentence has the symptom 
and b) the weight factors Py and PN. 
1174 
The probability for an NP to be the center 
is calculated by the inference engine 
represented as a Prolog program (left out 
here for reasons of space), which 
operates on the basis of the sublanguage 
knowledge base and the "local" 
knowledge base. The latter gives 
information on the current discourse 
segment. Initially, our program works 
with manual inputs. The local knowledge 
base can be represented as Prolog facts in 
the following way: 
observcd (lmadlinc). 
observed (domailL conccl)O. 
obsmvcd (repeated). 
The inference engine's task is to match 
the expected symptoms of the possible 
syntactic function as center in the 
knowledge base of the sentence's actual 
symptoms, and produce a list of 
(reasonably) tx)ssible candidates. 
THE PROCEDURE: 
AN INTEGRATED KNOWLEDGE 
APPROA Ctt 
Our algorithm for assigning (proposing) 
an antecedent to an anaphor is 
sublanguage-oriented because it is based 
on rules resulting from studies of 
colnputer science texts. It is also 
knowledge-based because it uses at least 
syntactic, semantic and discourse 
knowledge. Discourse knowledge and 
especially knowing how to track the 
center play a decisive role in proposing 
the most likely antecedent. 
The initial version of our project handles 
only pronominal anaphors. However, not 
all pronomls may have specific reference 
(as in constructions like "it is necessary", 
"it should be pointed out", "it is clear", 
.... ). So be\[ore the input is given to the 
anaphor resolver, the pronoun is checked 
to ensure that it is not a part of such 
grammatical construction. This function is 
carried out by the "referential expression 
filter". 
The proccdurc for proposing an 
antecedent to an anaphor operates on 
discourse segments and can be described 
itffonnally in the lollowing way: 
l) Propose the center of the first 
sentence of the discourse segment 
using the method described. 
2) Use the algorithm proposed in 
IBrennan et al. 871, ilnproved by an 
additional estimation of the cotTcct 
probability supplied by our method, 
in order to track the center throughout 
the discourse segment (in case the 
anaphor is in a complex sentence, 
identify clause centers too). 
3) Use syntactic and semantic 
constraints to eliminate antecedent 
candidates. 
4) Propose the noun phrase that has 
been filtered out as the antecedent in 
case no other candidates have come 
up; otherwise propose the center of 
the preceding sentence (clause) as the 
antecedent. 
The information obtained in 1) and 2) 
may not be used; however, it may be vital 
for proposing an antecedent in case of 
ambiguity. 
To illustrate how the algorithm works, 
consider the tollowing sample text: 
SYSTI '~M PROGRAMS 
We should note that, unlike user 
progrants, system pmgran~s such as the 
supervisor and the language translator 
should not have to bc translated every 
lime they are used, olherwisc lhis would 
result ill a serious increase ill the time 
spent in processing a user's program. 
System t)rogr~|lns are usually written ill 
the assembly version of the machine 
langtmgc and are tnmslated once into Ihe 
nladlillC code itself, l;rom thCll oi1 they 
can be loaded into memory in machine 
code without the need for any immediate 
transhuion phases. They are written by 
specialist programmers, who arc "called 
system programmers and who know a 
great deal about the computer and the 
comlmtcr system 12)1" which their progrmns 
arc wrincn. They know Ihc exact ntmlber 
of location which each program will 
occupy mid in consequence can make use 
of these mmlbcrs in the supervisor and 
t l',qllslalor t)rograms. 
117,5 
The proposed center of the first sentence 
is "system programs". The center remains 
the same in the second, third and forth 
sentences. Syntactic constraints are 
sufficient to establish the antecedent of 
"they" in the third sentence as "system 
programs". In the forth sentence, 
syntactic constraints only, however, are 
insufficient. Semantic constraints help 
here in assigning "system programs" as 
antecedent to "they". In the fifth sentence 
neither syntactic nor semantic constraints 
can resolve the ambiguity. The correct 
decision comes from proposing the center 
of the previous sentence, in this case 
"system programmers" (and not 
"programs"!), as the most likely 
antecedent. 
CONCLUSION 
The model proposed has two main 
advantages. First, it is an integrated 
model of different types of knowledge 
and uses existing techniques for anaphora 
resolution. Second, it incorporates a new 
approach for tracking the center, which 
proposes centers and subsequently 
antecedents with maximal likelihood. 
Since we regard our results still as 
preliminary, further research is necessary 
to confirm/improve the approach/model 
presented. 
ACKNOWLEDGEMENT 
I would like to express nay gratitude to 
Prof. Pieter Seuren for his useful 
comments and to the Machine Translation 
Unit, Universiti Sains Malaysia, Penang, 
where a considerable part of the described 
research has been carried out. 
REFERENCES 
\[Aonc & MeKee 93\] Ch. Aonc, D. McKee - 
Language-independent anaphora resolution system 
for understanding multilingual texts. ProceeAings 
of the 31st Annual Meeting of the ACL, The 
Olfio State University, Colmnbus, Ohio, 1993 
\[Allen87\] J. Allen Natural language 
understanding. The Benjamin/Cummings 
Publishing Company Inc., 1987 
\[Brennan ct al. 87\] S. Brennml, M. Fridman, C. 
Pollard - A centering approach to pronouns. 
Proceedings of the 25th Annual Meeting of file 
ACL, Statfford, CA, 1987 
\[Cmbonell & Brown 88\] J. Carbonell, R. 13town 
- Anaphora resolution: a multi-strategy approach. 
Proceedings of the 12. International Colfferencc 
on Computational Linguistics COLING'88, 
Budapest, 1988 
\[Dahl & Ball 90\] 1). Dahl, C. Ball - Reference 
resolution in PUNDIT. Research Report CAIT- 
SLS-9004, March 1990. Center for Advanced 
Information Teclmology, Paoli, PA 9301 
\[Frederking & Gehrkc 87\] R. Frederking, M. 
Gchrke - Resolving anaphoric references in a 
DRT-based dialogue system: Part 2: Focus at~l 
Taxonomic inference. Siemens AG, WlSBER, 
Bericht Nr. 17, 1987 
\[Grosz & Sidncr 86\] B. Grosz, C. Sidncr - 
Attention, Intention and the Structure of 
Discourse. Computalional Linguistics, Vol. 12, 
1986 
\[Hayes 8111 P.J. Hayes - Anaphorafor limited 
domain systems. Proceedings of the 7th IJCAI, 
V~mcouver, Canada, 1981 
\[Hirst 81\] G. Hirst - Anaphora in natural 
language understanding. Berlin Springer Verlag, 
1981 
\[Hobbs 78\] J. IIobbs - Resolving pronoun 
refere~es. Lingua, Vol. 44, 1978 
\[lngria & StaUm'd 8911 R. lngria, D. Stallard - A 
computational mechanism for pronominal 
reference. Proceedings of the 27th Annual 
Meeting of the ACL, Vancouver, British 
Columbia, 1989 
\[Mitkov 93\] R. Mitkov - A knowledge-basedand 
sublanguage-oriented approach for anaphora 
resolution. Proceedings of the Pacific Asia 
Conference on Formal and Computational 
Linguistics, Taipei, 1993 
\[Preug et al. 94l PrcuB S., Schmitz B., 
Hauenschild C., Umbach C. Anaphora 
Resolution in Machine Translation. In W. 
Ramm, P. Schmidt, J. Schiitz (eds.) Studies in 
Machine Translation and Natural I.~mguagc 
Processing, Volume on "l)iscoursc in Machine 
Trmlslation" 
\[Rich & LupcrFoy 88\] 1~. Rich, S. lalpcrFoy - 
An architecture for anaphora resolution. 
Proceedings of the Second Conference on Applitxt 
Natnral l~anguagc Processing, Austin, Texas, 
1988 
\[Robert 8911 M. Robert - RSsolution de formes 
pronominales dans l'interface d'interogation d'une 
base de donndes. Th6sc de doctorat. Facult6 des 
sciences de Luminy, 1989 
\[Sidner 8111 C.L. Sidncr - Focusing for 
Interpretation of Pronouns. Americau Journal of 
Computational Linguistics, 7, 1981 
\[Walker 89\] M. Walker -- Evaluating discourse 
processing algorithms. Proceedings of the 27th 
Annual Meeting of the ACL, Vancouver, 
Colmnbia, 1989 
7776 
