GE NLTOOLSET :DESCRIPTION OF THE SYSTEM AS USED FOR MUC- 4
George Krupka, Paul Jacobs and Lisa Ra u
Artificial Intelligence Laboratory
GE Research and Developmen t
Schenectady, NY 12301 USA
E-mail: rau@crd.ge.com
Phone : (518) 387 - 505 9
and
Lois Childs and Ira Sider
Management and Data Systems Operatio n
GE Aerospace
Abstract
The GE NLTooLsET is a set of text interpretation tools designed to be easily adapted to ne w
domains. This report summarizes the system and its performance on the MUG-4 task .
INTRODUCTIO N
The GE NLTooLsET aims at extracting and deriving useful information from text using a knowledge-based ,
domain-independent core of text processing tools, and customizing the existing programs to each new task .
The program achieves this transportability by using a core knowledge base and lexicon that adapts easil y
to new applications, along with a flexible text processing strategy that is tolerant of gaps in the program 's
knowledge base .
The language analysis strategy in the NLTooLsET uses fairly detailed, chart-style syntactic parsing
guided by conceptual expectations . Domain-driven conceptual structures provide feedback in parsing, con -
tribute to scoring alternative interpretations, help recovery from failed parses, and tie together information
across sentence boundaries. The interaction between linguistic and conceptual knowledge sources at the leve l
of linguistic relations, called "relation-driven control" was added to the system in a first implementation be -
fore MUC-4 .
In addition to flexible control, the design of the NLTooLsET allows each knowledge source to influenc e
different stages of processing . For example, discourse processing starts before parsing, although many deci-
sions about template merging and splitting are made after parsing . This allows context to guide language
analysis, while language analysis still determines context .
The NLTooLsET, now in Version 3 .0, has been developed and extended during the three years since th e
MUCK-II evaluation . During this time, several person-years of development have gone into the system . The
fundamental knowledge-based strategy has remained basically unchanged, but various modules have been
extended and replaced, and new components have been added while the system has served as a testbed for
a variety of experiments . The only new module added for MUC-4 was a mechanism for dealing with spatia l
and temporal information ; most of the other improvements to the system were knowledge base extensions ,
enhancements to existing components, and bug fixes .
The next section briefly describes the major portions of the NLTooLsET and its control flow ; the re-
mainder of the paper will discuss the application of the Toolset to the MUC-4 task .
SYSTEM OVERVIEW
Processing in the NLTooLsET divides roughly into three stages : (1) pre-processing, consisting mainly of
a pattern matcher and discourse processing module, (2) linguistic analysis, including parsing and semanti c
177
interpretation, and (3) post-processing, or template filling . Each stage of analysis applies a combination o f
linguistic, conceptual, and domain knowledge, as shown in Figure 1 .
Pre-processing Analysis Post-processing
Syntax Tagging, bracketing Parsing ,attachment Scoring
Semantics Collocations ,
cluster analysis
Sense disambiguation ,
role mapping
Role
extension
Domain
Templates Template activation Ambiguity pruning,recovery Default filling
Figure 1 : Stages of data extraction
The pre-processor uses lexico-semantic patterns to perform some initial segmentation of the text, iden-
tifying phrases that are template activators, filtering out irrelevant text, combining and collapsing som e
linguistic constructs, and marking portions of text that could describe discrete events . This component i s
described in [1] . Linguistic analysis combines parsing and word sense-based semantic interpretation wit h
domain-driven conceptual processing . The programs for linguistic analysis are largely those explained i n
[2, 3]—the changes made for MUC-4 involved mainly some additional mechanisms for recovering from faile d
processing and heavy pruning of spurious parses . Post-processing includes the final selection of template s
and mapping semantic categories and roles onto those templates . This component used the basic elements
from MUCK-II, adding a number of specialized rules for handling guerrilla warfare, types, and refines th e
discourse structures to perform the template splitting and merging required for MUC-3 and MUC-4.
The control flow of the system is primarily from linguistic analysis to conceptual interpretation to domai n
interpretation, but there is substantial feedback from conceptual and domain interpretation to linguisti c
analysis. The MUC-4 version of the Toolset includes a version of a strategy called relation-driven control,
which helps to mediate between the various knowledge sources involved in interpretation . Basically, relation-
driven control gives each linguistic relation in the text (such as subject-verb, verb-complement, or verb -
adjunct) a preference score based on its interpretation in context . Because these relations can apply to a
great many different surface structures, relation-driven control provides a means of combining preference s
without the tremendous combinatorics of scoring many complete parses . Effectively, relation-driven contro l
permits a "beam" strategy for considering multiple interpretations without producing hundreds or thousand s
of new paths through the linguistic chart .
The knowledge base of the system, consisting of a feature and function (unification-style) grammar wit h
associated linguistic relations, and a core sense-based lexicon, still proves transportable and largely generic .
The core lexicon contains over 10,000 entries, of which 37 are restricted because of specialized usage in th e
MUC-4 domain (such as device, which always means a bomb, and plant, which as a verb usually means to
place a bomb and as a noun usually means the target of an attack) . The core grammar contains abou t
170 rules, with 50 relations and 80 additional subcategories . There were 23 MUC-specific additions to this
grammatical knowledge base, including 8 grammar rules, most of them dealing with unusual noun phrase s
that describe organizations in the corpus .
The control, pre-processing, and transportable knowledge base were all extremely successful for MUC-4 ;
remarkably, lexical and grammatical coverage, along with the associated problems in controlling search an d
selecting among interpretations, proved not to be the major stumbling blocks for our system . While the
program rarely produce an incorrect answer as a result of a sentence interpretation error, it frequently fail s
17 8
to distinguish multiple events, resolve vague or subtle references, and pick up subtle clues from non-ke y
sentences. These are the major areas for future improvements in MUC-like tasks .
ANALYSIS OF TST2-0048
Overview of Example
TST2-0048 is faily representative of how the NLTooLSET performed on MUC-4 . The program successfully
interpreted most of the key sentences but missed some references and failed to tie some additional informatio n
in to the main event . As a result, it filled two templates for what should have been one event and misse d
some additional fills . The program thus derived 53 slots out of a possible 52, with 34 correct, 19 missing ,
and 19 spurious for .65 recall, .64 precision, and .35 overgeneration . We made no special effort to adapt th e
system or fix problems for this particular example ; in fact, we used TST2 as a "blind" test and did not d o
any development on that set at all .
Detail of Message Ru n
This example is actually quite simple at the sentence level 1 : The sentences are fairly short and grammatical ,
especially when compared to some of the convoluted propaganda stories, and TRUMP had no real problems
with them . The story is difficult from a discourse perspective, because it returns to the main event (th e
attack on Alvarado) essentially without any cue after describing a background event (the attack on Merino' s
home). In addition, the story is difficult and a bit unusual in the implicit information that is captured in
the answer key—that the seven children, because they were home when Merino's house was attacked, ar e
targets . Most of the difference between our system's response and the correct templates was due to thes e
two story-level problems .
The program made one or two other minor mistakes ; for example, it was penalized for filling in "INDI-
VIDUAL" as a perpetrator (from the phrase AN INDIVIDUAL PLACED A BOMB ON THE ROOF OF
THE ARMORED VEHICLE), an apparently correct fill that could have been resolved to "URBAN GUER-
RILLAS". It missed the SOME DAMAGE effect for the vehicle, which should have been inferred from th e
fact that the story later says the roof of the vehicle collapsed .
The system correctly parsed most of the main sentences, correctly linked the accusation in the firs t
sentence to the murder of the Attorney General in the same sentence, and correctly separated the secon d
event, which was distinguished by the temporal expression 5 days ago .
As explained earlier, the Toolset uses pattern matching for pre-processing, followed by discourse pro-
cessing, parsing and semantic interpretation, and finally template-filling . The pre-processor in this example
filters out most of the irrelevant sentences (and, in this case, two of the relevant ones), recognizes mos t
of the compound names (e .g. SALVADORAN PRESIDENT-ELECT ALFREDO CRISTIANI and AT-
TORNEY GENERAL ROBERTO GARCIA ALVARADO) . The pre-processor marks phrases that activate
templates (such as A BOMB PLACED and CLAIMED CREDIT), brackets out phrases like source an d
location (ACCORDING TO CRISTIANI and IN DOWNTOWN SAN SALVADOR), and tags a few word s
with part-of-speech to help the parser (e .g. auxiliaries (HAS), complementizers (THAT), and certain verb s
following "to" (COLLAPSE)) .
The last stage of pre-processing is a discourse processing module, which attempts a preliminary segmen-
tation of the input story using temporal, spatial, and other cues, event types, and looking for certain definit e
and indefinite descriptions of events . In this case, the module identifies five potential segments. The first
three turn out to be different descriptions of the same event (the killing of Alvarado), but they are late r
correctly merged into one template . The fourth segment is correctly identified as a new event (the attack o n
Merino's home). The fifth segment (describing the injury to Alvarado's bodyguards) is correctly treated a s
a new description, but is never identified as being part of the same event as the attack on Alvarado .
Linguistic analysis parses each sentence and produces (possibly alternative) semantic interpretations a t
the sentence level . These interpretations select word senses and roles, heavily favoring domain-specific senses .
The parser did fail in one important sentence in TST2-0048 : In the sentence "A 15-YEAR-OLD NIECE O F
I See Appendix F for the text and answer templates for the example .
179
MERINO 'S WAS INJURED " , it could not parse the apostrophe-s construct . This was a harmless failur e
because it occurs between a noun phrase and a verb phrase, and one of the parser's recovery strategie s
attaches any remaining compatible fragments that will contribute to a template fill.
The interpretation of each sentence is interleaved with domain-driven analysis . The conceptual ana-
lyzer, TRUMPET, takes the results of interpreting each phrase and tries to map them onto domain-base d
expectations, determining, for example, the appropriate role for the FMLN in "ACCUSED THE FMLN" a s
well as associating "support" events (such as accusations and effects) with main events (such as attacks o r
bombings). Because the discourse pre-processing module is prone to error, TRUMPET has begun to play a
major role in resolving references as well as in guiding semantic interpretation .
Post-processing maps the semantic interpretations onto templates, eliminating invalid fills (in this cas e
none), combining certain multiple references (in the attack on Alvarado), and "cleaning up" the final output .
Interpretation of Key Sentence s
The TRUMP parser of the NLTooLSET successfully parsed and interpreted the first sentence (Si) and cor-
rectly applied conjunction reduction to get Cristiani as the accuser and get the `"SUSPECTED OR AC-
CUSED BY AUTHORITIES" fill. Embedded clauses are typically handled in much the same way as main
clauses, except that the main clauses often add information about the CONFIDENCE slot . The syste m
correctly treats the main event and the accusing as a single event, in spite of ignoring the definite referenc e
"THE CRIME" . In our system, linking an accusation (C-BLAME-TEMPLATE in the output below) to an
event is the default .
The following is the pre-processed input and final sentence-level interpretation of Si :
Pre-processed input:
[byline : SAN SALVADOR, 19 APR 89 (ACAN-EFE) --] [bracket : [TEXT]] [fullname : SALVADORAN PRESIDENT-ELECT
ALFREDO CRISTIIII] CONDEMNED THE TERRORIST KILLING OF [fullname : ATTORIEY GENERAL ROBERTO GARCIA
ALVARADO] AND (comp : ACCUSED THE FARABUIDO MARTI NATIONAL LIBERATION FROIT [bracket : (FMLN)) OF) THE CRIME .
Interpretation :
Calling Trumpet with FINAL Interpretation :
(COORDCOIJ_AND1
(R-PART
(VERB ACCUSE1 (R-REL-TIME +PAST+ )
(R-PATIENT
(TERRORIST-NAME_FML11 (R-NAME FMLN )
(R-PART (C-ENTITY))))
(R-(UMBER +SIIGULARs)
(R-NAME ALFREDO-CRISTIAIIa)
(R-NATIONALITY
(C-NATI01-NAME_EL-SALVADOR(-QUAL))) )
(R-ACCUSATIOI
(IOUN_CRIME1 (R-NUMBER +SINGULAR+ )
(R-DEFINITE (DET_THE1))))))
(R-COMMUNICATOR
(FULLIAME_ALFREDO-CRISTIANI+1
(R-PART
(VERB_COIDEMI1 (R-REL-TIME +PAST+ )
(R-PATIENT
(C-ACT-OF-VERB_KILL1 (R-NUMBER *SINGULAR+)
(R-EFFECT (C-DEAD-QUAL))
(R-DEFINITE (DET_THE1) )
(R-CAUSE
(C-VERB_TERRORIZE1-ER
(R-NUMBER +SINGULAR+)
(R-IIHEREIT-ACTIVITY
(VERB_TERRORIZE1))) )
(R-EFFECTED
(FULLIAME_ROBERTO-GARCIA-ALVARAD01
(R-NUMBER +SINGULAR+)
(R-NAME ROBERTO-GARCIA-ALVARADO)))))
(R-COMMUNICATOR
(FULLBAME_ALFREDO-CRISTIANI+1 (R-NUMBER +SINGULAR*)
180
(R-LAME ALFREDO-CRISTIAII* )
(R-IATIOIALITY
(C-NATION-BANE EL-SALVADORI-QUAL)))))) )
TRUMPET WARS: Splitting connective COORDCONJ AND1 into part s
Activating new sense
(C-BLAME-TEMPLATE (R-REL-TIME *PAST* )
(R-MODALITY (C-QUALIFIER) )
(R-POLARITY (C-QUALIFIER) )
(R-PERPETRATOR
(TERRORIST-IAME_FMLN1 (R-NAME FMLN)
(R-PART (C-ENTITY))))
(R-ACCUSATION
(NOUN CRIMES (R-NUMBER *SINGULAR*)
(R-DEFINITE (DET_THE1))))
(R-ACCUSER
(FULLNAME_ALFREDO-CRISTIANI*1 (A-NUMBER *SINGULAR+)
(R-NAME ALFREDO-CRISTIAII*)
(A-NATIONALITY
(C-NATION-BAME_EL-SALVADOR1-QUAL)))) )
(R-NUMBER *SINGULAR*)
(R-MODALITY (C-QUALIFIER) )
(R-POLARITY (C-QUALIFIER) )
(R-DEFINITE (DET_THE1) )
(R-TARGET
CFULLIAME_ROBERTO-GARCIA-ALVARADO1
(A-NUMBER +SINGULAR*)
(R-NAME ROBERTO-GARCIA-ALVARADO)))
(A-PERPETRATOR
(C-VERB_TERRORIZEI-ER
(R-NUMBER *SINGULAR+)
(R-INHERENT-ACTIVITY
(VERB TERRORIZE1))))) )
(R-REPORTER
(FULLIAME_ALFREDO-CRISTIANI*1 (R-NUMBER +SINGULAR~ )
(R-NAME ALFREDO-CRISTIAII* )
(R-NATIONALITY
(C-NATION-SAME EL-SILVADOR1-QUAL)))))
TRUMPET YARN: Breaking out core templates (C-DEATH-TEMPLATE )
TRUMPET WAR': Linking (special) C-REPORT-TEMPLATE as filler for R-SUPPORT of C-DEATH-TEMPLATE
TRUMPET WARN : Linking (special) C-BLAME-TEMPLATE as filler for R-SUPPORT of C-DEATH-TEMPLAT E
Adding TERRORIST-1AME_FMLN1 from C-BLAME-TEMPLATE to R-PERPETRATOR of C-DEATH-TEMPLATE
The next set of examples sentences (S11-13) are more difficult . There was one parser failure, with a
successful recovery . As we have mentioned, we correctly identify this as a new event based on tempora l
information, but filter out S12 because it has no explicit event reference . This is not a bug—this sort of
implicit target description is fairly infrequent, so we chose not to address it at this stage .
Pre-processed input:
GUERRILLAS ATTACKED MERINO'S HOME (location: In SAN SALVADOR) [ago : 5 DAYS AGO] WITH EXPLOSIVES .
[filtered: THERE WERE SEVEN CHILDREN, INCLUDING FOUR OF THE VICE PRESIDENT'S CHILDREN, IN THE HOME AT TH E
TIME.] A (age: 15-YEAR-OLD) NIECE OF MERINO'S WAS INJURED .
Interpretation:
Calling Trumpet with FINAL Interpretation :
(VERB_ATTACKI (R-REL-TIME *PAST*)
(A-INSTRUMENT (IOUN_EXPLODE-IVE-X (R-NUMBER *PLURAL*)))
(R-PATIENT
(NOUI_HOME1 (R-NUMBER *SINGULAR*)
(R-OBJECTHOLDER
Activating new sense
(C-REPORT-TEMPLATE (R-REL-TIME *PAST* )
(R-MODALITY (C-QUALIFIER) )
(R-POLARITY (C-QUALIFIER) )
(R-OBJECT
(C-DEATH-TEMPLATE
181
(FULLNAME_FRANCISCO-MERINO*1
(R-NUMBER *SINGULAR*)
(R-NAME FRANCISCO-MERI10*)))))
CR-DATE
(C-DATE-OF-OCCURRENCE (R-RELATIVE NO )
(R-YEAR 1891 )
(R-DAY 1141 )
(R-MONTH 141)) )
(R-AGENT (NOUB_GUERRILLA1 (R-NUMBER *PLURAL*)))
(R-LOCATION (CITY-IAME_SAN-SALVADOR1 (R-NAME SAN-SALVADOR))) )
Activating new sense
(C-ATTACK-TEMPLATE (R-REL-TIME *PAST* )
(R-MODALITY (C-QUALIFIER))
(R-POLARITY (C-QUALIFIER))
(R-LOCATION (CITY-IAME_SAN-SALVADOR1 (R-NAME SAD-SALVADOR)) )
CR-DATE
(C-DATE-OF-OCCURRENCE (R-RELATIVE 10 )
(R-YEAR 1891 )
(R-DAY 1141 )
(R-MONTH 141)) )
(R-INSTRUMENT (NOUI_EXPLODE-IVE-I (R-(UMBER *PLURAL*)))
(A-TARGET
(NOUN_HOME1 (R-LUMBER *SINGULAR*)
CR-LOCATION
(CITY-IAME_SAI-SALVADOR1
(R-LAME SAN-SALVADOR)) )
(R-OBJECTHOLDEA
CFULLNAME_FRANCISCO-MERINO*1
(R-NUMBER *SINGULAR*)
(R-NAME FRANCISCO-MERINO*)))))
(A-PERPETRATOR (NOUN GUERAILLA1 (R-NUMBER *PLURAL*))) )
Calling Trumpet with FRAGMENT Interpretation :
(VERB_INJURE1 (R-REL-TIME *PAST+)
(R-EFFECT (C-INJURY) )
(R-EFFECTED
(NOUI_IIECE1 (R-/UMBER +SINGULAR*)
(R-DEFINITE (DET_A1) )
(R-POSSESSES
(FULLIAME_FRAICISCO-MERI10+1
(R-LUMBER +SINGULAR*)
(A-LAME FRANCISCO-MERINO~))))))
Activating new sense
(C-INJURY-TEMPLATE (R-REL-TIME *PAST*)
(R-MODALITY (C-QUALIFIER) )
(R-POLARITY (C-QUALIFIER) )
(A-TARGET
(NOUN_IIECE1 (R-NUMBER *SINGULAR*)
(R-DEFINITE (DET_A1))
(R-POSSESSES
(FULLIAME_FRANCISCO-MERINO+1
(R-NUMBER *SINGULAR*)
(R-NAME FRANCISCO-MERI10*))))) )
TRUMPET YARN : Linking (special) C-INJURY-TEMPLATE as filler for R-TARGET-EFFECT of C-BOMBING-TEMPLAT E
The system filters S21 (this is an omission, because "ESCAPED UNSCATHED" should be recognize d
as an effect), but successfully interprets S22 and resolves "ONE OF THEM" to "BODYGUARDS" . Note
that it is the pronoun "THEM", not "ONE", that gets resolved, using a simple reference resolution heuristi c
that looks for the most recent syntactically and semantically compatible noun phrase . However, this action
results in a penalty rather than a reward because the system does not tie the injury to the attack on Alvarado
at the beginning of the story.
Pre-processed input :
182
[filtered: ACCORDING TO THE POLICE AND GARCIA ALVARADO'S DRIVER, WH O
ESCAPED UNSCATHED, THE ATTORNEY GENERAL WAS TRAVELING WITH TWO
BODYGUARDS .] [numvord : ONE] OF THEM WAS INJURED.
Interpretation:
Calling Trumpet with FINAL Interpretation :
(VERB_INJURE1 (R-REL-TIME *PASTS)
(R-EFFECT (C-INJURY) )
(R-EFFECTED
(C-NUMBER (R-VALUE Iii )
(R-WHOLE (PNOUN_THEM1 (R-NUMBER *PLURAL*))))) )
Activating new sense
(C-INJURY-TEMPLATE (R-REL-TIME *PASTS)
(R-MODALITY CC-QUALIFIER) )
(R-POLARITY (C-QUALIFIER) )
(R-TARGET
(C-NUMBER (R-VALUE Ill )
(R-WHOLE (PIOUN_THEM1 (R-NUMBER *PLURAL*))))) )
Applying TR_WHOLE-OF-PARTS transform on NUMWORD_ONE1 for (AID (OR C-HUMAN C-HUMAN-GROUP NOUN_BODY3) EXP-NEG-TARGET )
Adding PNOUN_THEM1 from C-INJURY-TEMPLATE to R-TARGET of C-BOMBING-TEMPLAT E
TRUMPET DANGER : Resolving exp PNOUI_THEM1 to :EIISTIIG TON BODYGUARDS
TRUMPET DANGER : Resolving exp PIOUN_THEM1 to :EXISTING TOR BODYGUARDS
Comparison of Program Answers with Answer Ke y
The NLTooLSET results for TST2-0048 were the following templates (Annotations have been added in lowe r
case preceded by %, and blank slot (-) fills have been deleted to save space) .
0. MESSAGE: ID TST2-MUC4-0048
1. MESSAGE: TEMPLATE 1
2. INCIDENT: DATE - 19 APR 89
3. INCIDENT: LOCATION EL SALVADOR : SAI SALVADOR (CITY )
4. INCIDENT: TYPE BOMBING
5. INCIDENT: STAGE OF EXECUTION ACCOMPLISHED
6. INCIDENT: INSTRUMENT ID "BOMB"
7. INCIDENT: INSTRUMENT TYPE BOMB: "BOMB"
8. PERP: INCIDENT CATEGORY TERRORIST ACT
9. PERP: INDIVIDUAL ID "URBAN GUERRILLAS"
"INDIVIDUAL" % spurious fill
10. PERP: ORGANIZATION ID "FARABUNDO MARTI NATIONAL LIBERATION FRONT"
11. PERP: ORGANIZATION.CONFIDENCE SUSPECTED OR ACCUSED BY AUTHORITIES : "FARABUNDO MARTI NATIONAL LIBERATION FRONT"
12. PRYS TGT: ID "HIS VEHICLE"
13. PRYS TGT: TYPE TRANSPORT VEHICLE: "HIS VEHICLE"
14. PRYS TGT: NUMBER 1: "HIS VEHICLE"
16. PHYS TGT: EFFECT OF INCIDENT - I missed SOME DAMAGE : "HIS VEHICLE"
18. HUM TGT: LAME "ROBERTO GARCIA ALVARADO"
19. HUM TGT: DESCRIPTION "ATTORNEY GENERAL" : "ROBERTO GARCIA ALVARADO"
% missed fills DRIVER
% missed fills BODYGUARDS
"ATTORNEY GENERAL" % spurious fill
20. HUM TGT: TYPE GOVERNMENT OFFICIAL : "ROBERTO GARCIA ALVARADO"
LEGAL OR JUDICIAL: "ATTORNEY GENERAL" % spurious
% missed fills CIVILIAN: "DRIVER"
I missed fills SECURITY GUARD : "BODYGUARDS"
21. HUM TGT: NUMBER 1: "ROBERTO GARCIA ALVARADO"
1 : "ATTORNEY GENERAL" I spurious
% missed fills : 1 : "DRIVER"
% missed fills : 2: "BODYGUARDS"
I missed fills : 1 : "BODYGUARDS"
23. HUM TGT: EFFECT OF INCIDENT
	
DEATH: "ATTORNEY GENERAL" I spurious
DEATH: "ROBERTO GARCIA ALVARADO"
% missed fills : NO INJURY : "DRIVER"
% missed fills : INJURY: "BODYGUARDS"
24. HUN TGT : TOTAL NUMBER
	
-
183
O. MESSAGE: ID TST2-MUC4-0048
1. MESSAGE: TEMPLATE 2
2. INCIDENT: DATE 14 APR 89
3. INCIDENT: LOCATION EL SALVADOR: SAI SALVADOR (CITY)
4. INCIDENT: TYPE BOMBING
5. INCIDENT: STAGE OF EXECUTION ACCOMPLISHED
6. INCIDENT: INSTRUMENT ID "EXPLOSIVES"
7. INCIDENT: INSTRUMENT TYPE EXPLOSIVE : "EXPLOSIVES"
8. PERP: INCIDENT CATEGORY TERRORIST ACT
9. PERP: INDIVIDUAL ID "GUERRILLAS"
10. PERP: ORGANIZATION ID "FARABUNDO MARTI NATIONAL LIBERATION FRONT"
11. PERP: ORGANIZATION CONFIDENCE SUSPECTED OR ACCUSED BY AUTHORITIES: "FARABUNDO MARTI NATIONAL LIBERATION FRONT"
12. PHYS TGT: ID "MERINO'S HOME"
13. PHYS TGT: TYPE CIVILIAN RESIDENCE: "MERINO'S HOME" % missed type should be GOVERNMENT OFFICE OR RESIDENn
14. PHYS TGT: NUMBER 1: "MERINO'S HOME"
19. HUM TGT : DESCRIPTION "NIECE OF MERINO"
% missed fill - "CHILDREII"
% missed fill - "VICE PRESIDENT'S CHILDREN"
CIVILIAN: "NIECE OF MERIIIO"
% missed fill CIVILIAN: "CHILDREN"
% missed fill CIVILIAN: "VICE PRESIDENT'S CHILDREN"
1: "NIECE OF MERINO"
% missed fill 7: "CHILDREN"
% missed fill 4: "VICE PRESIDENT'S CHILDREN"
INJURY: "NIECE OF MERINO"
DEATH: "NIECE OF MERINO"
- 1 missed fill 7
1 completely spurious template; should have been merged with template 1
20. HUM TGT: TYPE
21. HUM TCT: LUMBER
23. HUM TGT : EFFECT OF INCIDENT
24. HUM TGT : TOTAL NUMBER
0. MESSAGE : ID
1. MESSAGE : TEMPLATE
2. INCIDENT: DATE
3. INCIDENT: LOCATION
4. INCIDENT: TYPE
6. INCIDENT : STAGE OF EXECUTION
6. INCIDENT: INSTRUMENT ID
7. INCIDENT: INSTRUMENT TYPE
8. PERP: INCIDENT CATEGORY
10. PEEP: ORGANIZATION ID
11. PERP: ORGANIZATION CONFIDENCE
16. PHYS TOT : EFFECT OF INCIDENT
19. HUM TGT: DESCRIPTION
20. HUM TGT: TYPE
21. HUM TGT: NUMBER
23. HUM TGT: EFFECT OF INCIDENT
TST2-MUC4-0048
3
- 19 APR 89
EL SALVADOR
BOMBING
ACCOMPLISHED
"BOMB"
BOMB: "BOMB"
TERRORIST ACT
"FARABUNDO MARTI NATIONAL LIBERATION FRONT"
POSSIBLE: "FARABUNDO MARTI NATIONAL LIBERATION FRONT"
DESTROYED: "-"
"BODYGUARDS"
SECURITY GUARD : "BODYGUARDS"
PLURAL: "BODYGUARDS"
INJURY: "BODYGUARDS"
Some of the missing information in the response template comes from failing to tie information in t o
the main event or failing to recover implicit information . This is the case with the damage to the vehicle ,
which is described in passing, the children who were in Merino's home, and the driver who escaped unscathed .
Almost all the rest of the departures owe to some aspect of reference resolution—from failing to recognize th e
injury to the bodyguards as part of Alvarado 's murder, to the extra fills "INDIVIDUAL" and "ATTORNE Y
GENERAL" that were co-referential with others. One of these turned out to be a simple bug, in that the title
"ATTORNEY GENERAL" in our system was interpreted as a different type (GOVERNMENT OFFICIAL )
from the noun phrase "ATTORNEY GENERAL" (LEGAL OR JUDICIAL) ; thus the system failed to unify
the references. However, the general problem of reference resolution is certainly one of the main areas wher e
future progress can come .
The other illustrative problem with this example is the degree to which relatively inconsequential fact s
can be pieced together into an interpretation . There is no theoretical reason why our system didn't know
about different forms of damage to vehicles, but we certainly wouldn 't want to spend a lot of time encoding
this sort of knowledge . This turned out to be a rather tedious part of the MUC task. We did go so far as to
have template filling heuristics, for example, that tell the system: (1) When vehicles explode near buildings ,
it is the buildings and not the vehicles that are the targets, (2) When parts of buildings are destroyed o r
damaged (e.g. "the bomb shattered windows") this means that the buildings sustained some damage, an d
18 4
(3) When body parts are damaged (e .g. "the bomb destroyed his head"), it is the owner of the body part s
that is affected . However, such rules only scratch the surface of the reasoning that contributes to templat e
filling.
While the reference resolution problem is quite general and very interesting from a research perspective ,
the reasoning problem seems more MUC-specific, and it's hard to separate general reasoning issues from th e
peculiar details of the fill rules .
Aside from these problems, our system performed pretty well on this example, as for MUC on the whole .
The recall and precision for this message were both over .60, with the program recovering most of the
information from the text . As is typical from our MUC experience, the local processing of sentences was
very accurate and complete, while the general handling of story level details and template filling had som e
loose ends .
SUMMARY AND CONCLUSION
MUC-4 is a very difficult task, combining language interpretation at many levels with a variety of rules an d
strategies for template filling. The examples here illustrate some of the important characteristics of ou r
system as well as where future progress can be made . Not surprisingly, the major problems that remai n
after MUC-4 are very similar to the ones that we identified at the end of MUC-3 . This by itself might see m
discouraging, but the fact that the system did much better on MUC-4 suggests that we can expect mor e
improvements in the future. While there is a class of phenomena that we haven't really begun to address
(the body of world knowledge that contributes to interpreting events), there is also the ripe problem o f
interpreting text in context, in which MUC has given the field a leg up .
References
[1] Paul S . Jacobs, George R . Krupka, and Lisa F . Rau. Lexico-semantic pattern matching as a companio n
to parsing in text understanding . In Fourth DARPA Speech and Natural Language Workshop, pages
337-342, San Mateo, CA, February 1991 . Morgan-Kaufmann .
[2] Paul Jacobs and Lisa Rau. SCISOR: Extracting information from on-line news. Communications of th e
Association for Computing Machinery, 33(11) :88-97, November 1990 .
[3] Paul S. Jacobs . TRUMP : A transportable language understanding program . International Journal of
Intelligent Systems, 7(3) :245-276, 1992 .
185
