<?xml version="1.0" standalone="yes"?>
<Paper uid="M91-1024">
  <Title>Training Stories Recall Precision</Title>
  <Section position="1" start_page="0" end_page="0" type="metho">
    <SectionTitle>
HUGHES TRAINABLE TEXT SKIMMER :
DESCRIPTION OF THE TTS SYSTEM AS USED FOR MUC-3
</SectionTitle>
    <Paragraph position="0"/>
  </Section>
  <Section position="2" start_page="0" end_page="0" type="metho">
    <SectionTitle>
OVERVIEW OF CAPABILITIE S
</SectionTitle>
    <Paragraph position="0"> The objective of the Hughes Trainable Text Skimmer (TTS) Project is to create text skimming software that: (1) can be easily re-configured for new applications, (2) improves its performanc e with use, and (3) is fast enough to process megabytes of text per day . The TTS-MUC3 system i s our first full scale prototype .</Paragraph>
    <Paragraph position="1"> TTS-MUC3 incorporates semi-automated lexicon generation and almost fully automated phras e pattern generation. Associative retrieval from a case memory provides raw data for computing se t fills and string fills . TTS-MUC3's modular process model integrates the results of case memor y retrieval over sentences from multiple stories, extracts the date and location of incidents, an d computes cross-reference information for various slots.</Paragraph>
  </Section>
  <Section position="3" start_page="0" end_page="0" type="metho">
    <SectionTitle>
SYSTEM COMPONENT S
</SectionTitle>
    <Paragraph position="0"> The TTS-MUC3 system incorporates a number of different modules shown below :</Paragraph>
  </Section>
  <Section position="4" start_page="0" end_page="0" type="metho">
    <SectionTitle>
TEXT
TEXT DATABASE
PROCESS +
SENTENCE CLASSIFICATIO N
TOPIC GROUPING
DATE PROCESSING
LOCATION PROCESSING
TEMPLATE GENERATIO N
</SectionTitle>
    <Paragraph position="0"/>
  </Section>
  <Section position="5" start_page="0" end_page="159" type="metho">
    <SectionTitle>
SENTENCE
CASES
MEMORY
ASSOCIATIVE LOOKU P
FEATURE MATCHIN G
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="155" end_page="155" type="sub_section">
      <SectionTitle>
Text Database
</SectionTitle>
      <Paragraph position="0"> The text database provides the capability to retrieve a fragment of text from a large collection tha t may be spread over multiple disk files . In 'ITS-MUC3, the text database was used to store : (1) the database of training stories, (2) the database of testing stories, (3) the database of training template s for user browsing, and (4) the database of parsed templates for use by the associative cas e memory. Retrievals from the text database may return : (1) a raw text string (used for templates), (2) a recursive token structure with individual words at the leaf nodes (used for stories), or (3) a n s-expression (used for parsed templates) .</Paragraph>
    </Section>
    <Section position="2" start_page="155" end_page="156" type="sub_section">
      <SectionTitle>
Phrasal Parse r
</SectionTitle>
      <Paragraph position="0"> The phrasal parser is a fast, shallow, conceptual parser . The parser accepts a token structure, a lexical hierarchy, and a phrase pattern set. The parser returns an ordered list of text features . A text feature includes: (1) a member of the concept hierarchy, (2) the string covered by the phrase, and (3) a recursive token structure spanning the tokens covered by the phrase.</Paragraph>
      <Paragraph position="1"> Lexicon entries are created by adding word stems to a concept hierarchy as follows, (ks :isa h-lex &amp;quot;PRIEST&amp;quot; :religious-individual-w) (ks :isa h-lex &amp;quot;MISSIONARY&amp;quot; :religious-individual-w) (ks :isa h-lex &amp;quot;CONFERENCE&amp;quot; :conference-w) (ks :isa h-lex &amp;quot;SUMMIT&amp;quot; :conference-w) (ks :isa h-lex &amp;quot;RECEPTION&amp;quot; :conference-w) Phrasal patterns may reference either elements of the concept hierarchy, or specific words :</Paragraph>
      <Paragraph position="3"> The features are extracted using a depth first search of the patterns, with a preference for pattern s that have specific words over those the have only concept names and a preference for longe r patterns.</Paragraph>
      <Paragraph position="4"> Case Memory The case memory takes an ordered list of text features and returns the K-nearest neighbors . For TTS-MUC3, K was 12 and the metric was the Euclidean distance in a binary vector space. The case memory also accepts a set of slots to fill (set fill and string fill) . For each sentence, the case memory returns weighted suggestions for filling each of the requested slots . Case indices are kept in main memory. Parsed templates, used for computing slot fillers, are loaded as needed.  The process model has four phases: (1) memory access, (2) topic grouping, (3) slot filling, and (4 ) template generation. There is also an initial training phase which initializes the case memory .</Paragraph>
    </Section>
    <Section position="3" start_page="156" end_page="156" type="sub_section">
      <SectionTitle>
Training Phase
</SectionTitle>
      <Paragraph position="0"> The training phase uses the provided templates to build up the phrase lexicon and the case memory .</Paragraph>
      <Paragraph position="1"> Phrases are generated from the fillers for the template slots . Cases are generated from th e sentences that provided the fillers . The word lexicon is generated by performing a word frequency analysis on the raw text. For TTS-MUC3, all words that occurred between 10 and 105 times wer e included in the lexicon.</Paragraph>
    </Section>
    <Section position="4" start_page="156" end_page="159" type="sub_section">
      <SectionTitle>
Memory Access
</SectionTitle>
      <Paragraph position="0"> For each sentence of a story, the memory access phase queries the case memory to obtai n suggestions for all slots. The resulting structure contains all the weighted suggestions and th e source cases.</Paragraph>
      <Paragraph position="1"> Topic grouping and relevance assessmen t Topic grouping (analogous to discourse processing) is based on the TYPE OF INCIDENT slot. The weight for each type of incident is computed for every sentence . The weights are then passed through a competitive filter, resulting in binary signals . The competitive filter first normalizes th e topic weights using a Gaussian mask on a sentence by sentence basis, then computes the bes t topic. A topic is a set of contiguous sentences with the same computed value for TYPE OF INCIDENT .</Paragraph>
      <Paragraph position="2"> Figure 2 shows the inputs and outputs to the topic grouping process . Note that moderately high evidence of kidnapping throughout the story is suppressed in favor of the bombing interpretation , which turns out to be correct. This filter used is topic grouping is designed to pick out signals tha t are high but that &amp;quot;drop out&amp;quot; from time to time, as one can see in the smoothing over the arson  Slot filling Slot filling consists of five parts : (1) pure set fills, (2) string fills, (3) cross-referenced slots, (4) date extraction, and (5) location extraction. The first three parts consider only relevant sentences . A relevant sentence shares the same topic with the previous sentence or contains no competin g topic. There are two distinct types of processing for slot filling. Most slots are filled using hypotheses returned by the associative memory, two, date and location, are filled by domai n specific procedures. Three types of slots are filled from the associative memory : 1.</Paragraph>
      <Paragraph position="3"> Set fills--Pure set fills are computed by averaging the weights over all sentence s for a given topic and picking the highest score .</Paragraph>
      <Paragraph position="4">  2. String fills--String fills are computed in a similar manner to set fills . The difference is that the suggestions returned by the case memory are subject to a threshold on the weights . For the official run of TTS-MUC3, the string fil l threshold was set at 0.1 .</Paragraph>
      <Paragraph position="5"> 3. Cross reference generation--Cross reference generation is performed by choosing the most likely tag (as suggested by the case-memory) for the sentence that contain s the string fill.</Paragraph>
      <Paragraph position="6">  Based on semantic features such as : POLICE-W and : PLACE-NAME, the following template (along with approximately 11 others) is retrieved from memory .</Paragraph>
      <Paragraph position="7">  (feature :place-name &amp;quot;UNITED STATES&amp;quot;) ) Comparing the strings in the retrieved template with the strings for the indexing features, TTS-MUC3 looks for a feature in the new sentence that matches the features (FEATURE : EMBASSY-W &amp;quot;EMBASSY&amp;quot;) . Using the semantic feature, :EMBASSY-W, TTS-MUC3 proposes (FEATURE : EMBASSY-W &amp;quot;EMBASSIES&amp;quot; #&lt;TOKEN: EMBASSIES&gt;) , as a hypothesis for the physical target in the new story.</Paragraph>
      <Paragraph position="8">  Processing proceeds in a like manner for the rest of the story to produce the following template , 0 . MESSAGE ID 1. TEMPLATE ID 2. DATE OF INCIDENT 3. TYPE OF INCIDENT 4. CATEGORY OF INCIDENT 5. PERPETRATOR: ID OF INDIV(S) 6. PERPETRATOR: ID OF ORG(S) 7. PERPETRATOR: CONFIDENCE 8. PHYSICAL TARGET : ID(S) 9. PHYSICAL TARGET: TOTAL NUM 10. PHYSICAL TARGET: TYPE(S) 11. HUMAN TARGET : ID(S) 12. HUMAN TARGET : TOTAL NUM 13. HUMAN TARGET : TYPE(S) 14. TARGET: FOREIGN NATION(S) 15. INSTRUMENT : TYPE(S) 16. LOCATION OF INCIDENT 17. EFFECT ON PHYSICAL TARGET(S)</Paragraph>
    </Section>
  </Section>
  <Section position="6" start_page="159" end_page="159" type="metho">
    <SectionTitle>
SOME DAMAGE: &amp;quot;VEHICLES&amp;quot;
SOME DAMAGE: &amp;quot;CAR&amp;quot;
SOME DAMAGE: &amp;quot;PRC EMBASSY&amp;quot;*
</SectionTitle>
    <Paragraph position="0"> TTS-MUC3 produces a reasonably good fill for this template . Three features are worth noting.</Paragraph>
    <Paragraph position="1"> First, the string fills &amp;quot;PRC EMBASSY&amp;quot; and &amp;quot;USSR EMBASSY&amp;quot; are extracted from sentences after the introductory sentence,</Paragraph>
  </Section>
  <Section position="7" start_page="159" end_page="159" type="metho">
    <SectionTitle>
&amp;quot;A CAR-BOMB EXPLODED IN FRONT OF THE PRC EMBASSY, WHICH IS IN
THE LIMA RESIDENTIAL DISTRICT OF SAN ISIDRO. MEANWHILE, TWO
BOMBS WERE THROWN AT A USSR EMBASSY VEHICLE THAT WAS PARKED
IN FRONT OF THE EMBASSY LOCATED IN ORRANTIA DISTRICT, NEAR
SAN ISIDRO. &amp;quot;
</SectionTitle>
    <Paragraph position="0"> The second feature worth noting is that &amp;quot;CAR&amp;quot; is picked up as a target, even though it is actually a part of the instrument &amp;quot;CAR-BOMB&amp;quot;. The reason for this mistake is a deficiency in the phrases that pick out semantic features.</Paragraph>
    <Paragraph position="1"> The third feature is that TTS-MUC3 produced only one template where there should have been tw o bombings . This merging of templates with the same incident type is an inevitable result of th e topic grouping used in TTS-MUC3.</Paragraph>
  </Section>
  <Section position="8" start_page="159" end_page="161" type="metho">
    <SectionTitle>
SENSITIVITY TO TRAINING SET
</SectionTitle>
    <Paragraph position="0"> To test the sensitivity to different training sets, we loaded the associative memory with differen t templates from the development corpus . To show the difference in performance, Table 1 show s the overall recall and performance for the MATCH/MISSING row of the scoring, with variou s  portions of the training data loaded. Whenever a training set is loaded, the number of case with a given incident type is limited to prevent sampling bias. For Table 1 the maximum cases per topic is 10. Note that these training sets are much smaller than the full 1200 stories in the DEV corpus , and therefore the recall performance is substantially lower than the 31% achieved with the ful l training set on TST2.</Paragraph>
    <Paragraph position="1">  Intuitively, one would imagine that recall at least would fall drastically . Table 2 confirms that intuition as, for all but one training set, the recall drops when fewer cases per incident type ar e loaded. Both Tables 1 and 2 are the result of running the first 10 stories in the TST1 corpus through TTS-MUC3. The first ten stories contain two ARSON templates, and even after limiting the number of cases per topic to 10, ARSON still has fewer than half as many cases as the more common types: ATTACK, MUDER, BOMBING, and KIDNAPPING. However, when the numbe r of cases per topic is limited to four (4), ARSON is perfectly balanced with the others . This under representation of ARSON in the training data may account for the anomaly between Tables 1 and 2 for stories 626-700.</Paragraph>
  </Section>
  <Section position="9" start_page="161" end_page="162" type="metho">
    <SectionTitle>
SUMMAR Y
</SectionTitle>
    <Paragraph position="0"> To understand the performance of TTS-MlUC3, one should look at the the inter-dependenc e between the various processing modules. ]Figure 3 shows these dependencies . Each module points to the modules it depends on. Our contention is that improving a module will enable improvement of the behavior of its dependents.</Paragraph>
    <Paragraph position="1"> For example, the case memory alone has recall and precision rates above 50% . Subsequent processing results in information loss that accounts for our final rates of 31% and 36% , respectively.</Paragraph>
    <Paragraph position="2"> Figure 3 : Module Dependency Graph We believe that this ability to analyze, from a system wide perspective, where the errors occur i s unique to TTS . From Figure 3, we can see that even a perfect case memory would not completely solve all performance problems, as every other component depends on topic grouping. Therefore we conclude that topic grouping is the system component where the most work is needed . We might also deduce that in topic group, we will find the largest leverage for adding knowledge to th e processing. This conclusion concurs with conventional wisdom in natural language, that understanding text across sentence boundaries requires more knowledge that understanding within a sentence.</Paragraph>
  </Section>
class="xml-element"></Paper>