File Information
File: 05-lr/acl_arc_1_sum/cleansed_text/xml_by_section/metho/95/m95-1009_metho.xml
Size: 39,336 bytes
Last Modified: 2025-10-06 14:13:59
<?xml version="1.0" standalone="yes"?> <Paper uid="M95-1009"> <Title>SLOT POS ACTT COR PAR INC' SPU MIS NONIREC PRE UND OVG ERR SUB ----------------------------------------------------------------------------- - ALL OBJECTS 2276 22961 2128 0 741 94 74 01 93 93 3 4 10 3 ----------------------------------------------------------------------------- - F-MEASURES P&R</Title> <Section position="1" start_page="0" end_page="0" type="metho"> <SectionTitle> LOCKHEED MARTIN: LOUELLA PARSING , AN NLTOOLSET SYSTEM FOR MUC- 6 by </SectionTitle> <Paragraph position="0"/> </Section> <Section position="2" start_page="0" end_page="0" type="metho"> <SectionTitle> BACKGROUN D </SectionTitle> <Paragraph position="0"> During the 1980s, General Electric Corporate Research and Development began the design and implementation of a set of text-processing tools known as the NLToolset . This suite of tool s developed, over time, in cooperation with a subgroup of the Management and Data Systems Operations component of General Electric Aerospace. Through corporate mergers, this subgrou p has become the Language Exploitation Technologies group of the Lockheed Martin Management and Data Systems division . Over the years, the toolset has evolved into a robust set of aids for tex t analysis . It has been used to build a variety of applications, and the knowledge gained from each application has been utilized to improve the toolset . The LOUELLA PARSING system was de signed with the latest version of the NLToolset .</Paragraph> <Paragraph position="1"> The Lockheed Martin group's LOUELLA PARSING system participated in three of the fou r from a text . This task is called the &quot;scenario template&quot; (ST), and requires a deeper anlysis of th e text than the other tasks .</Paragraph> <Paragraph position="2"> This paper will give an overview of our systems, describe our performance on each of the task s and the walkthrough article, and discuss areas where our systems need to be improved .</Paragraph> <Paragraph position="3"> LOUELLA's NE system was developed using the &quot;spotters&quot; from the NLToolset . The TE and ST systems were developed using a range of tools for reference resolution, for information extraction , for simple discourse processing, and for template generation, as well as a variety of spotters for spotting entities . Below we describe the processing stages that are used in LOUELLA's NE, TE an d ST systems . The first three stages are used in the NE task .</Paragraph> </Section> <Section position="3" start_page="0" end_page="98" type="metho"> <SectionTitle> PROCESSING STAGE S </SectionTitle> <Paragraph position="0"> The NLToolset contains a core knowledge base, a large sense-disambiguated lexicon, and a variety of text-processing tools for extracting information, organizing information, and generating output. Typically, these tools are sequentially applied to text.</Paragraph> <Paragraph position="1"> Text Tokenization and Segmentatio n LOUELLA uses sequential processing which simplifies the text with each phase . The first text-processing module used is NLlex, a lexical-analyzer-development package which handles the character string to word translation and tokenizes the text. Next, the text segmenter interprets the SGML markers and common punctuation, and stores the text in a structure that holds the origina l version of the text as a whole ; it also stores each section of the text, such as paragraph, sentence , headline, dateline, etc. Throughout processing, the structure holds an original and a &quot;latest &quot; version of each sentence ; the &quot;latest&quot; version is updated with each processing phase . Lexical Look-up At this point, the text is stored in tokenized form, and any unambiguous names and phrase s that are stored in the lexicon are identified . Named entities, such as organizations and people, ar e stored on an active token list to allow the system to link occurrences of the same entity, based o n name variations. Next, each word is analyzed morphologically and tagged with its possible parts o f speech, which are found in the lexicon. The following is an example of the first check against th e lexicon. Here the phrasal &quot;less than&quot; and the company &quot;Coke&quot; are identified and marked as multitokens (MT): &quot;I would be [MT{LESS-THAN} : less than ] honest to say I'm not disappointed not to be able to clai m creative leadership for [MT{COCA-COLA}: Coke],&quot; Mr. Dooner says .</Paragraph> <Section position="1" start_page="97" end_page="98" type="sub_section"> <SectionTitle> Text Reduction </SectionTitle> <Paragraph position="0"> LOUELLA uses a non-deterministic, lexico-semantic, finite-state pattern matcher . By this definition, we refer to the pattern matcher as a finite-state machine which matches against bot h syntactic and semantic features of text . The pattern matcher uses a knowledge base of pattern action rules, grouped in rule packages . These rule packages are applied to the text in successiv e passes to mark primitive text elements such as time, money, locations, person, and compan y names. These marked phrases are then reduced into single tokens .</Paragraph> <Paragraph position="1"> LOUELLA uses eight rule-packages to reduce elements ranging from the most primitive tim e expressions and organization noun phrases to the more complex IN_AND_OUT objects . This phase of text processing extends from the Named Entity system through to the Template Element , and includes the Scenario Template phase . Here are several examples : Matches: (TIME-RULE-0 TIME-RULE-36 TIME-RULE-38 ) [TIME-ABSOLUTE{36} : Yesterday ], McCann made official what had been widely anticipated: Mr.</Paragraph> <Paragraph position="2"> James, 57 years old, is stepping down as chief executive officer on [DATE{4} : July 1 ] and will retire as chairman [TIME-RELATIVE{38} : at the end of the year ] .</Paragraph> <Paragraph position="3"> Matches: (PNAME-RULE-2) Yesterday, McCann made official what had been widely anticipated : [PNAME{2} : Mr. James ], 57 years old, is stepping down as chief executive officer on July 1 and will retire as chairman at the end o f the year .</Paragraph> <Paragraph position="4"> Matches: (OTHERNP-RULE-3) [PPNOUN{3} : He ] will be succeeded by Dooner, 45 .</Paragraph> <Paragraph position="5"> Matches: (ORGNP-RULE-3) But the bragging rights to Coke's ubiquitous advertising belongs to Creative Artists Agency , [ORGNP{3} : the big Hollywood talent agency ] .</Paragraph> <Paragraph position="6"> Matches: (ORGANIZATION-RULE-7 PERSON-RULE) Now, [PERSON{0} : James] is preparing to sail into the sunset, and [PERSON{0} : Dooner] is poised to rev up the engines to guide [ORGANIZATION{7}: Interpublic Group ] 's [ORGANIZATION{7} : McCann-Erickson ] into the 21st century.</Paragraph> <Paragraph position="7"> Matches: (IN_AND_OUT-RULE-O IN_AND_OUT-RULE-1 ) Yesterday, McCann made official what had been widely anticipated : [IN_AND_OUT{O} : James] , 57 years old, is stepping down as [OFFICERTOK{1}: chief executive officer] on July 1 and will retire as [OFFICERTOK{1} : chairman ] at the end of the year.</Paragraph> </Section> <Section position="2" start_page="98" end_page="98" type="sub_section"> <SectionTitle> Reference Resolution </SectionTitle> <Paragraph position="0"> Reference resolution is ongoing throughout processing . As soon as a named entity is recognized, it is stored--along with its variations--on an active token list so that variations of the nam e can be recognized and linked to the original occurrence . When organization names are recognized, they can often be directly linked to their appositives or prenominal phrases . In addition, noun-phrase recognition prompts a backward search through a stack of named entities in orde r to identify its referent. This search uses several tactics to find the correct referent . If the noun-phrase is semantically rich, a content filter is constructed and compared against content filters for known, named entities . If this is not successful, various heuristics are used based on entity typ e and position in text .</Paragraph> </Section> <Section position="3" start_page="98" end_page="98" type="sub_section"> <SectionTitle> Information Extraction </SectionTitle> <Paragraph position="0"> LOUELLA uses the same pattern matcher for information extraction that it uses for text reduc tion; however, there is a difference in the way the pattern matcher is used . While extracting, the pattern matcher is allowed to overlap patterns because it is not changing the text found ; it is merely extracting information of interest and sending it to the text organizer. The text organize r tries to assemble the extracted information into a lucid account of events . It performs this assembly by using a model of the domain as delineated in the task specification . For example, it is permissible to have more than one IN_AND_OUT object participating in a SUCCESSION_EVENT, bu t there must be one SUCCESSION_ORG involved .</Paragraph> <Paragraph position="1"> Postprocessing Postprocessing is the final review of the extracted information before the templates are gener ated. It is LOUELLA's chance to apply any heuristics which may seem helpful to an accurate re porting of information. This part of the system is entirely dependent on the domain and can b e customized at will by the developer.</Paragraph> <Paragraph position="2"> Template Generating LOUELLA has a template generator which uses an object-oriented mapping script for generat ing the final template. The script is based on the task specification and contains the path whic h the template generator should follow through the objects. The script also contains pointers to the functions which print each slot fill.</Paragraph> </Section> </Section> <Section position="4" start_page="98" end_page="100" type="metho"> <SectionTitle> SYSTEM MODULE S </SectionTitle> <Paragraph position="0"> For MUC-6, LOUELLA is comprised of three system modules, one for each of the MUC-6 tasks addressed: Named Entity (NE), Template Element (TE), and Scenario Template (ST) . Below, we give a brief description of each of these systems .</Paragraph> <Paragraph position="1"> The NE System LOUELLA's Named Entity system is a multi-pass process which builds upon entities whic h are found in previous passes. In addition to the segmentation and lexical look-up stages of ou r system, early passes of the reduction phase identify time, date, money, and percent components . The system then searches for locations, knowing that the entities found previously will not be par t of the location phrase . The person and company-name passes also use the previous informatio n to identify contexts which indicate the presence of a company name, such as : &quot;ABC stock ros e *percentage*&quot; . The NE system generates all possible variations for each person and compan y name it finds; another pass tries to find these variations.</Paragraph> <Paragraph position="2"> LOUELLA's NE system uses a variety of matching methods. Entities such as dates are found by combining structure format with a list of valid items, i .e. a valid month followed by a number . Mr. <ENAMEX TYPE=&quot;PERSON&quot;>James</ENAMEX>, 57 years old, is stepping down as chief executiv e officer on <TIMEX TYPE=&quot;DATE&quot;>July 1</TIMEX> and will retire as chairman at the end of the year. In cases where none of the entity parts are in a list of known things, we use surrounding con text to identify the name .</Paragraph> <Paragraph position="3"> One of the many differences between <ENAMEX TYPE=&quot;PERSON&quot;>Robert L. James</ENAMEX>, chairman and chief executive officer of <ENAMEX TYPE=&quot;ORGANIZATION&quot;>McCann-Erickson</ENAMEX> null In the above example, LOUELLA does not know what McCann-Erickson is, however, she does know that people are &quot;chairman and chief executive officer of' an organization . Other widely known companies such as &quot;Coca-Cola&quot; are identified through a list of known organizations . This list also helps identify &quot;Coke&quot; as referring to the &quot;Coca-Cola&quot; company . LOUELLA's NE system makes a basic assumption that any organizations that appear in th e headline are the same as, or variations of, the organizations found in the text . Therefore, the system does not examine the headline for organizations until it processes the body of the text . In the previous example, the reference to &quot;McCann-Erickson&quot; in the headline of the walkthrough text i s found only after the body of the text is processed during the variation matching phase .</Paragraph> <Paragraph position="4"> Each variation of a person or organization found is linked to the original name . This link is used in the TE system to identify aliases found in the text for that entity . Only people and named companies found by the NE system will be processed by the TE system .</Paragraph> <Paragraph position="5"> One additional NE system feature used by the TE system is a &quot;company rename function .&quot; If an organization changed--or plans to change--its name, the old or future name is linked to the current name, and the system symbol for the current name is used by all references to eithe r name. The TE system then uses the current name to find all references to the current and old o r future names.</Paragraph> <Paragraph position="6"> The TE Syste m The TE system builds an object for each organization and person name that contains all of th e related information it can find in the article .</Paragraph> <Paragraph position="7"> An organization object consists of: 1)the organization's name 2) all aliases for that name found in the text 3) one descriptor phrase, 4) the organization type , 5) the organization's locale an d 6) country.</Paragraph> <Paragraph position="8"> A person object consists of : 1) the person's nam e 2) any aliases for that name in the article, and 3) any titles for that individual which appear in the text .</Paragraph> <Paragraph position="9"> Much of the information related to the entity name is found during the initial phases of the NE module in the context surrounding the entity name . Appositives, for example, are often good descriptor phrases . The system also links other descriptive phrases and pronouns to the named entity, and these additional descriptions are used to assist the ST system in its information-extraction task . Later, these links will allow for the replacing of noun phrases with, for example , normalized organization template elements .</Paragraph> <Paragraph position="10"> Once an organization noun-phrase or personal pronoun is identified, the reference resolution module seeks to find its referent . For persons, LOUELLA uses the simple heuristic of assigning the last person mentioned as the referent, keeping in mind gender constraints . For organizations, th e process involves several steps. First, the phrase is checked to make sure it hasn't already been recognized and linked by the NE system . If no match is recognized, a content filter for the phrase i s run against a content filtered version of each known organization name ; if there is a match, the link is made .</Paragraph> <Paragraph position="11"> Content Filters: &quot;the jewelry chain&quot; => ( jewelry jewel chain ) &quot;Smith Jewelers&quot; => ( smith jewelers jeweler jewel ) For example, if the organization noun phrase &quot;the jewelry chain&quot; is identified, its content filte r would be applied to the list of known company names . When it reaches &quot;Smith Jewelers,&quot; it will compare the filter against a filtered version of the name . The best match is considered the referent. If there is a tie, file position is considered as a factor, and the closest name is the most likely refer ent. For generic phrases like &quot;the company,&quot; reference is currently determined solely by file position and type .</Paragraph> <Paragraph position="12"> When a descriptor is linked to an organization name, the syntactic relationship of the descrip tor to the organization name is also stored with the phrase . For example, appositives and prenom inal phrases recognized by the NE system are tagged with &quot; :APP&quot; and &quot; :PRENOM&quot;, respectively . Likewise, references resolved by the reference resolution module are appropriately tagged . The template generator uses a heuristic to choose the descriptor which is most likely correct . The choice is based on a hierarchy which begins with appositives, prenominals, and predicate nominatives, and ends with references resolved by the reference resolution module . Once an organization or person has been linked to all its variations in the article, the TE sys tem chooses the best name for the element and relegates the rest of the names to the alias catego ry. Assigning the same symbol name to each instance of a template element greatly simplifies th e work of the subsequent ST system.</Paragraph> <Section position="1" start_page="100" end_page="100" type="sub_section"> <SectionTitle> The ST System </SectionTitle> <Paragraph position="0"> The ST system extracts information about complex events that involve template elements lik e organizations and people . LOUELLA's scenario is about changes in corporate management .</Paragraph> <Paragraph position="1"> The top-level template of interest is the SUCCESSION_EVENT, which is comprised of: SUCCESSION_ORG : an organization template element , POST: a string fill, IN_AND_OUT: a relational object about each person involve d (may be more than one),</Paragraph> </Section> </Section> <Section position="5" start_page="100" end_page="100" type="metho"> <SectionTitle> VACANCY REASON : </SectionTitle> <Paragraph position="0"> a set fill .</Paragraph> <Paragraph position="1"> LOUELLA's strategy is to repeatedly simplify the text before information extraction take s place. This approach allows the most basic elements of the scenario to be identified first. The TE system identifies the primitive template elements (person and organization) involved in a particu lar scenario. In addition, NE-style methods are applied, at this point, to recognize and tag management-position titles within the text.</Paragraph> <Paragraph position="2"> The next level of complexity is to find the relational object, IN_AND_OUT. This object is filled by: a pointer to a person template element, the IO_PERSON ; a set-fill indicating whether the per son is in or out, the NEW_STATUS ; a set fill indicating whether the person is currently on the job , the ON THE JOB ; a pointer to an organization template element representing another corporat e entity involved in the change, the OTHER_ORG, (if known) ; and a set-fill indicating the relation ship of the other organization, the REL_OTHER_ORG .</Paragraph> <Paragraph position="3"> It makes sense to first convert all person template elements into potential IN_AND_OUT objects. In most cases, the sentence clues which will tell the system whether a person is in or out of a position and whether the person is still on the job are also the clues for the succession event itself. 10 1 It is preferable, then, to instantiate an empty IN_AND_OUT object around each person element , and then to fill in the rest of the information if an event is extracted .</Paragraph> <Paragraph position="4"> LOUELLA's ST application consists of three rule packages : ingress.k, which holds all rules for entering corporate posts ; egress.k, which holds all rules for leaving corporate posts ; and activations.k, which holds all macros for the ingress and egress rule packages .</Paragraph> <Paragraph position="5"> The Lockheed Martin approach to information extraction is to build sets of floating phrases , i.e. rules, which can glide over each sentence, binding to the right configuration of information . This information is then extracted and reorganized into a lucid account of the events . This approach is similar to the model-based segmentation method used in image-understanding systems. Portions of images are recognized easily, and their configuration is ultimately used to iden tify the complete image .</Paragraph> <Paragraph position="6"> By examining a training set of articles for the sentences which report the events of interest , rules are developed . As training progresses, the rules are generalized to cover more and more pos sible constructs . A typical ingress rule, made up of macros, might look something like this : $subjphr $conjphr ?IN=$appointvb { $postorg } => c-reassigning-template The binding macros are $subjphr, $appointvb, and an optional $postorg. $conjphr is a buffer macro allowing the pattern matcher to skip over irrelevant material . This rule contains a variable ?I N which is bound when the rule is matched . This binding is then conveyed to the IN_AND_OUT object as its NEW_STATUS . Other variables are also present within the macro definitions. These variables, when bound, will convey information about VACANCY_REASON, ON THE JOB, an d the OTHER_ORG to the objects involved in the event. For example, if the ?ACTING, ?IN, and ?FU-TURE variables are all bound in a match, then the IN_AND_OUT's NEW_STATUS is IN an d ON THE JOB is NO, because the text is reporting that the person will be acting in a position .</Paragraph> <Paragraph position="7"> A difficulty occurs with this method when a sentence identifies a person as leaving one position and entering another. For example, &quot;Judy Jones, president of Exxon, has been hired as CE O of GE.&quot; In this case, the person element will have two different NEW_STATUS values, dependin g on the position being discussed . When this happens, the person element must be re-instantiate d as an additional IN_AND_OUT object in order to collect the correct value .</Paragraph> <Paragraph position="8"> Once an article's information has been extracted, it is then organized into a sensible account based on a model of the domain . This model, along with the final template model (which guides th e system's template generator), is constructed at the beginning of training . Both models are base d on the scenario specifications .</Paragraph> <Paragraph position="9"> In the postprocessing stage, we apply any heuristics learned during the course of system development. For this application, the OTHER_ORG portion of the IN_AND_OUT object was filled-in here, based on the information gathered about that person . For example, if at this point LOUELLA knows that a person is leaving one organization and joining another, she can conclud e that each organization can be the OTHER_ORG in the IN_AND_OUT object for the other organization's SUCCESSION_EVENT; in effect, the system swaps SUCCESSION_ORGs between succession events to supply their respective IN_AND_OUT objects with OTHER_ORG fillers .</Paragraph> </Section> <Section position="6" start_page="100" end_page="102" type="metho"> <SectionTitle> WALKTHROUGH PERFORMANC E </SectionTitle> <Paragraph position="0"/> <Section position="1" start_page="100" end_page="102" type="sub_section"> <SectionTitle> Scenario Template </SectionTitle> <Paragraph position="0"> WARNING : this message does not represent LOUELLA's typical performance. Its F-measure is less than half of LOUELLA's average performance . In fact, this was her next-to-worst score of al l the messages . Nonetheless, the system produced the following template for the walkthrough mes sage. The template represents recall of only 15, with precision of 80.</Paragraph> <Paragraph position="1"> Performance on this message reveals two areas in which our system can be improved . First, our method of generalizing had not reached fruition by the time of the evaluation . This message was improved by expanding the definition for one of the floating phrases, i .e. macros, which make up all ingress and egress patterns, and by inserting a buffer phrase into one of the egress patterns . Adding a buffer phrase allows the pattern matcher to jump over part of the conjunctive phrase in the following sentence : James, 57 years old, is stepping down as chief executive officer on July 1 and will retire as chairman a t the end of the year.</Paragraph> <Paragraph position="2"> The following sections show the extraction process taking place . The italicized words represen t the rule which is being matched and the variables which are being bound. The buffer addition allows two rules to overlap the sentence, extracting both succession events .</Paragraph> </Section> </Section> <Section position="7" start_page="102" end_page="103" type="metho"> <SectionTitle> EVALUATION EXTRACTION: </SectionTitle> <Paragraph position="0"> cer {0}] on July 1 and ?FUTURE=will ?VACANCY REASON=?OUT=?HEAD=refire as ?POST-chairman {0}] at the end of the year.</Paragraph> <Paragraph position="1"> The secondary factors that hurt LOUELLA's performance were unsatisfactory post-processin g decisions . During development, generic patterns were instantiated to extract organization s which would likely be involved in a succession event . These companies are usually in the act of announcing some event. Then, if succession events are extracted without an organization bein g directly involved in the event statement, the announcing organization can be tied to the organiza tion-less events. This heuristic worked well ; however, no allowance had been made for the case in which a SUCCESSION_ EVENT was extracted in the absence of any SUCCESSION_ORG . This was remedied, post-evaluation, by allowing the system to collect all organizations, and to choose a n organization during postprocessing to act as a default SUCCESSION_ORG for all organization less events. Even though post-processing chose the wrong organization for the walk-throug h message, it still got two extra points for having an organization .</Paragraph> <Paragraph position="2"> The most striking effect of a deficient post-processing heuristic was the decision to eliminat e any succession events which contained only an IN_AND_OUT object, with no other information .</Paragraph> <Paragraph position="3"> Removal of this heuristic alone, with no other pattern modifications, increased the recall on the walk-through message from 15R/80P to 44R/61P! This is due to the fact that LOUELLA was no w producing four succession events, instead of one, each with its own IN_AND_OUT object . This increased the number of correct slots from 8 to 23, even though the additional succession event s had no post and no succession organization.</Paragraph> <Paragraph position="4"> Changes in post-processing, while prompted by performance on the walk-through message , affect system performance as a whole . Consequently, the entire evaluation set was rerun with the changes made to improve the walk-through message . Performance increased from 43R/64P wit h a 51 .63 F-measure to 49R/60P with a 54 .04 F-measure .</Paragraph> <Paragraph position="5"> Of course, the best performance occurs when LOUELLA recognizes all of the organization s present. When improvements were made to the Named Entity task for the walk-through message ,</Paragraph> <Paragraph position="7"> Notice that all of the person objects have actually been extracted . The discrepancy in the score for the person object is due to the incorrect string-fill for the name of Alan Gottesman . LOUELLA incorrectly added the word &quot;Even. &quot; The main improvement to LOUELLA for this walk-through message was the recognition o f &quot;McCann&quot; as an alias for &quot;McCann-Erickson,&quot; instead of as a location . This allowed the mapping '10 5 of the two McCann-Erickson organization objects, which improved our score to 76R/79P fro m 71R/67P.</Paragraph> <Paragraph position="8"> Named Entity Our official NE scores for the walk-through document were 91R/88P. We found two system problems that drastically reduced this score. One problem was the variation for McCann-Erickson, &quot;McCann.&quot; LOUELLA threw out the variation because it was known in the gazetteer as a city name. By testing that the variation is part of a hyphenated name, we could then allow the variation to be valid. This one change raised this particular document's score to 96R/93P. Additionally, LOUELLA found &quot;Even Alan Gottesman&quot; as a person, as well as the variatio n &quot;Even&quot; later in the document . By forcing LOUELLA to accept the match that starts with a known first name, instead of another part of speech, we threw out this match and raised the documen t total score to 97R/94P.</Paragraph> <Paragraph position="9"> With the addition of these two modifications, our total NE F-measure rose to : 94.08. This document also contains an example of the difficulty in recognizing when a company nam e is being used as a modifier to a product .</Paragraph> <Paragraph position="10"> ... the agency still is dogged by the loss of the key creative assignment for the prestigious <ENAME X TYPE=&quot;ORGANIZATION&quot;>Coca-Cola</ENAMEX> Classic account .</Paragraph> <Paragraph position="11"> We are currently looking into expanding the NE module to include a products package. This package will use knowledge about the use of products in text, i .e., how they are referred to and when they include the company name as a premodifier. This type of information may be useful t o the analyst who notices a particular person frequently associated with the purchase of certai n products, such as Winchester Rifles .</Paragraph> <Paragraph position="12"> Another interesting ambiguous phrase, which our system did not handle correctly, is : Mr. <ENAMEX TYPE=&quot;PERSON&quot;>Dooner</ENAMEX>, who recently lost <NUMEX TYPE=&quot;MONEY&quot;>60 pounds</NUMEX> over three--and--a--half months, says now that he has &quot;rein vented&quot; himself, he wants to do the same for the agency .</Paragraph> <Paragraph position="13"> Since it is conceivable that Mr . Dooner could have lost 60 pounds of currency, this makes fo r an interesting discussion of how smart our systems should be at the named entity level . By possibly making the reference between &quot;reinventing himself' and &quot;lost 60 pounds,&quot; the system coul d throw out the money tag. Another argument could be made that since McCann-Erickson is referred to as &quot;world-wide&quot; in many places, it is even more possible that Mr . Dooner could lose 6 0 pounds of money. Another possibility is to give our systems the notion of money value vs . weight value; that is, is 60 pounds of currency significant enough to outweigh 60 pounds of weight loss ?</Paragraph> </Section> <Section position="8" start_page="103" end_page="108" type="metho"> <SectionTitle> AN ANALYSIS OF SYSTEM PERFORMANC E </SectionTitle> <Paragraph position="0"> LOUELLA experienced two bugs during the evaluation which caused at least one documen t not to be scored in each task . Therefore, we will report two sets of results : the official scores for the incomplete responses, and the unofficial scores for the complete responses which were generated after the bugs were fixed . We consider our true performance to be the complete responses .</Paragraph> <Paragraph position="1"> Overall, LOUELLA's performance was near the top in all tasks, with F-measures within si x percentage points of the top F-measures in Named Entity, within four in Template Element, an d within five in Scenario Template.</Paragraph> <Paragraph position="2"> Named Entity The Named Entity performance was severely effected by a bug which virtually eliminated one entire response out of the set of thirty ; accordingly, the difference in scores between official an d unofficial is most dramatic here. The bottom line scot es for Named Entity performance follow , along with the Task Subcategorization Scores for the complete response .</Paragraph> <Paragraph position="3"> Note that LOUELLA has achieved near-perfection in four of the six subcategories . It is expected that performance would be even greater over a larger corpus . Since the NE component is a reusable module, it is expected to increase--over time--in recall and precision as it is exercised over a larger corpus.</Paragraph> <Section position="1" start_page="106" end_page="108" type="sub_section"> <SectionTitle> TemplateElement </SectionTitle> <Paragraph position="0"> Official performance on the Template Element task was degraded by two bugs which cause d LOUELLA to lose two articles from the set of 100 . Official bottom-line and unofficial total slo t scores are: LOUELLA had very high recall in the Template Element task . She also had very high F-measure for the locale and country slots, and for the descriptor slot. Figures 1 and 2 illustrate F-mea- null sure rankings in the descriptor and locale/country slots, respectively. Since location information is often found in the descriptor phrase, these three slots are somewhat related . High performanc e on these slots may be due to the attention given to reference resolution during the development of LOUELLA for MUC-6 .</Paragraph> <Paragraph position="1"> liszt brahms verdi dvorak bizet borodin puccinichopin .nwagnerchopin .bmahler grieg grieg* A difficulty with the descriptor slot is its mixed role . One function of the slot is to contain any descriptor phrase which is related to an organization's name. This is a true reference resolution task. In addition, however, the slot may also contain a phrase describing an un-named organization. This then requires LOUELLA to differentiate between the two types of phrases and may lea d her to overgenerate un-named organization objects, thereby suppressing precision .</Paragraph> </Section> <Section position="2" start_page="108" end_page="108" type="sub_section"> <SectionTitle> Scenario Template </SectionTitle> <Paragraph position="0"> Official performance on the Scenario Template task was degraded by two bugs which cause d us to lose two articles from the set of 100 . Fortunately, only one of these articles was relevant t o the task. Official bottom-line and unofficial total slot scores are : LOUELLA recognized 60% of the succession events after one person-month of development . In fact, she had an F-measure of 69 .65 for that slot . This performance shows the system's adapt ability. This fact is even more remarkable because of the necessity to write specialized code to handle the peculiarities of this task. Unlike previous extraction tasks in which the event template is built from lower-level relational and primitive elements, this specific task requires that informa tion, such as IN or OUT status, be recognized at the event level but instantiated in the lower leve l relational element, the IN_AND_OUT object .</Paragraph> </Section> </Section> <Section position="9" start_page="108" end_page="108" type="metho"> <SectionTitle> TRAINING LOUELLA </SectionTitle> <Paragraph position="0"/> <Section position="1" start_page="108" end_page="108" type="sub_section"> <SectionTitle> Methodology </SectionTitle> <Paragraph position="0"> Ten percent of the 100 development messages were set aside as a blind set for the developmen t phase. This ten percent was chosen based on the size of their keys, so as to accurately represen t the complexity of the development set . Over four weeks, the Scenario Template task was able to achieve F-measure of 70.16 on the development set and 52 .05 on the blind set. This measure i s quite close to our evaluation F-measure of 51 .63.</Paragraph> <Paragraph position="1"> During training, the system is run over both the blind set and the development set of message s overnight, several times a week. Developers can then check the scores at the start of the day and determine which area of the system is most in need of improvement at that time . This method allows us to check our progress frequently, and to backtrack quickly if a regression is noticed . Effort The NE and TE modules of LOUELLA were developed over the Spring and Summer of 1995 b y two experienced system developers, one focusing on the NE task and the other on the TE task, wit h an emphasis on reference resolution . When the evaluation period started, the NE person shifte d attention to the TE task, while the TE person shifted to the ST task . Two inexperienced developers were then assigned to the NE task for the evaluation period .</Paragraph> <Paragraph position="2"> The bulk of the NE effort was directed toward perfecting the rules for recognition . The TE task was more code-intensive because of its reference resolution component, i .e. that task requires an assembling of information gathered up from throughout the article for each organization and per son object. The ST effort runs the gamut from domain-specific application design through rul e construction and specialized coding ; however, the Lockheed Martin NLToolset system provides a basic framework for building an information-extraction application which greatly reduces th e amount of effort required . The NE and TE modules themselves are now available for any information-extraction task, and the object-oriented template generator allows the system to easily pro duce any new template based on the task specifications .</Paragraph> </Section> </Section> <Section position="10" start_page="108" end_page="111" type="metho"> <SectionTitle> DIRECTION </SectionTitle> <Paragraph position="0"> The reference resolution strategies used for MUC-6 will be expanded to provide more accuracy in identifying related and unrelated organization descriptors, as well as pronoun references . Inclusion of linguistic theory, in addition to other techniques that have been successful for the coref erence participants, is a possibility . Research into this area is currently underway .</Paragraph> <Paragraph position="1"> The procedure for building an extraction system is currently too labor-intensive and haphaz ard a process, dependent to a great extent on the abilities of the developer. The first step toward remedying this procedure is to build a rigorous syntactic framework which can be used as a tem plate for rule variations. A further step is to investigate the possibility of building a self-trainin g system. Since, at the point of extraction, the system knows a great deal about the components o f each sentence, it may be possible to have the system itself generate a set of interesting patterns for a particular domain.</Paragraph> <Paragraph position="2"> A preliminary effort at linking sub-parts of succession events was attempted for MUC-6 . This entailed extracting generic events which were disposable if not linked to task-relevant events .</Paragraph> <Paragraph position="3"> Expansion in this area will include layering of events, as well as an incorporation of time elements , and will ultimately improve the system understanding of the texts being processed .</Paragraph> </Section> class="xml-element"></Paper>