<?xml version="1.0" standalone="yes"?>
<Paper uid="P98-2135">
  <Title>Discourse Cues for Broadcast News Segmentation</Title>
  <Section position="6" start_page="820" end_page="821" type="evalu">
    <SectionTitle>
4. Evaluation
</SectionTitle>
    <Paragraph position="0"> We evaluated segmentor performance by measuring both the precision and recall of segment boundaries compared to manual annotation of story boundaries where:  1. Precision - # of correct segment tags # of total segment tags 2. Recall = # of correct segment tags # of hand tags  Table 1 presents average precision and recall results for multiple programs after applying generalized cue patterns developed first for ABC as described in Section 2.2. Recall degrades when porting these same algorithms to different news programs (e.g., CNN, Jim Lehrer) given the genre differences as described in Section 2.1.</Paragraph>
    <Paragraph position="1"> Errors in story boundary detection include erroneously splitting a single story segment into two story segments, and merging two contiguous story segments into a single story segment. Furthermore, given our error-driven transformation based proper name taggers operate at approximately 80% precision and recall, this can adversely impact discourse cue detections. Also, our preliminary evaluation of speech transcription results in word error rates of approximately 50%, which suggest non captioned text is not yet feasible for this class of segmentation.</Paragraph>
    <Paragraph position="2"> We have just completed an empirical study (Merlino and Maybury, forthcoming) with BNN users that explores the optimal mixture of media elements show in Figure 3 (e.g., keyframes, named entities, topics) in terms of speed and accuracy of story identification and comprehension tasks. Key findings include that users perform better and prefer mixed media presentations over just one media (e.g., named entities or topic lists), and they are quicker and more accurate working from extracts and summaries than from the source transcript or video.</Paragraph>
  </Section>
class="xml-element"></Paper>