<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-0710">
  <Title>Reference Resolution over a Restricted Domain: References to Documents</Title>
  <Section position="7" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
6 Results and Observations
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
6.1 Baseline and Best Scores
</SectionTitle>
      <Paragraph position="0"> We provide first some baseline scores on the set of 15 meetings and 322 REs, that is, scores of very simple methods against which our algorithms must be compared (rather than against a 0% score). For RE $ document association, always choosing the most frequent newspaper leads to 82% accuracy (265 REs out of 322). But some meetings deal only with one document; if we look only at meetings that involve more than one newspaper, then the score of this baseline procedure is 50% (46/92), a much lower value. Regarding RE $ document element association, if the referent is always the front page as a whole (/Newspaper), then accuracy is 16%. If the referent is always the main article (/MasterArticle[ID='1']), then accuracy is 18%--in both cases quite a low value.</Paragraph>
      <Paragraph position="1"> The word co-occurrence algorithm (described in Section 5.1) correctly solves more than 50% of the selected REs, in a preliminary evaluation performed on six meetings. This simple algorithm gives interesting results especially when REs belong to an utterance that is thematically close to the content of a document's logical block. However, the method uses only thematic linking and, furthermore, does not take advantage of all the various document structures.4 The 50% score should thus be considered more as another baseline.</Paragraph>
      <Paragraph position="2"> The second algorithm (described in Section 5.2) reaches 98% accuracy for the identification of documents referred to by REs, or 93% if we take into account only the meetings with several documents; remember that baseline was 82%, respectively 50%.</Paragraph>
      <Paragraph position="3"> The accuracy for document element identification is 73% (237 REs out of 322). If we score only REs for which the document was correctly identified, the accuracy is 74% (236 REs out of 316), a little higher.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
6.2 Score-based Analysis of the Algorithm
</SectionTitle>
      <Paragraph position="0"> The best scores quoted above are obtained when only the right context of the RE is considered for matching (i.e. the words after the RE), not the left one. Also, the optimal number of words to look for in the right context is about ten. If the right context is not considered either, the score drops at 40%.</Paragraph>
      <Paragraph position="1"> Regarding the weights, a match between the RE and the title of an article appears to be more important than one between the right context and the title, and much more important than matches with the content of the article: weights are about 15 vs.</Paragraph>
      <Paragraph position="2"> 10 vs. 1. All these values have been determined empirically, by optimizing the score on the available data. It is possible that they change slightly when more data is available.</Paragraph>
      <Paragraph position="3"> If anaphor tracking is disabled, the accuracy of document element identification drops at 65%, i.e.</Paragraph>
      <Paragraph position="4"> 35% of the REs are linked to the wrong document element. Anaphor tracking is thus useful, though apparently not essential: dropping it leads to an algorithm close to our first attempt (Section 5.1).</Paragraph>
      <Paragraph position="5"> Since the automatic scorer provides a detailed evaluation report for each meeting, we are in the  ument topological information (e.g. 'the figure at the bottom'), or related to the document logical structure (e.g. 'the author of the first article'), which need a semantic analysis of the REs. process of analyzing the errors to find systematic patterns, which could help us improve the algorithm. Rules depending on the lexical items in the RE seem to be required.</Paragraph>
    </Section>
  </Section>
  <Section position="8" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
7 Applications
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
7.1 Speech to Document Alignment
</SectionTitle>
      <Paragraph position="0"> The resolution of references to documents is part of a cross-channel process aimed at detecting links between what was said during a meeting and the documents related to the meeting. The process enhances dialog and document processing, as well as the multi-media rendering of the results. Transcriptto-document alignment allows the generation of an enhanced transcript which is aligned also with the relevant documents, thanks to hyperlinks from transcript to document zones. Such a mechanism is integrated in the query and browsing interfaces that we are building.</Paragraph>
      <Paragraph position="1"> Reference-based alignment is not the only way to align documents with the speech transcript. We have proposed two other techniques (Mekhaldi et al., 2003; Lalanne et al., 2004). Citation-based alignment is a pure lexicographic match between terms in documents and terms in the speech transcription. Thematic alignment is derived from semantic similarity between sections of documents (sentences, paragraphs, logical blocks, etc.) and units of the dialog structure (utterances, turns, and thematic episodes). We have implemented an algorithm that uses various state-of-the-art similarity metrics (cosine, Jaccard, Dice) between bags of weighted words.</Paragraph>
      <Paragraph position="2"> For matching spoken utterances with document logical blocks, using cosine metric, recall is 0.84, and precision is 0.77, which are encouraging results. And when matching speech turns with logical blocks, recall stays at 0.84 and precision rises to 0.85. On the other hand, alignment of spoken utterances to document sentences is less precise but is more promising since it relies on less processing.</Paragraph>
      <Paragraph position="3"> Using Jaccard metric, recall is 0.83, and precision is 0.76 (Lalanne et al., 2004). Thematic units have not been considered yet, for want of reliable automatic segmentation.</Paragraph>
      <Paragraph position="4"> Reference-based alignment is complementary to other methods; these could be integrated in a common framework, so that they can be consolidated and compared. Their fusion should allow for more robust document-to-speech alignment.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
7.2 Overall Application: Meeting Processing
and Retrieval
</SectionTitle>
      <Paragraph position="0"> A promising use of human dialog understanding is for the processing and retrieval of staff or business meetings (Armstrong et al., 2003). When meetings deal with one or several documents, it is important to link in a precise manner each episode or even utterance of the meeting to the sections of the documents that they refer to. Considering users who have missed a meeting or want to review a meeting that they attended, this alignment is required for two types of queries that appear in recent studies of user requirements (Lisowska et al., 2004). First, the users could look for episodes of a meeting in which a particular section of a given document was discussed, so that they can learn what was said about that section. Second, the relevant documents could automatically be displayed when the users browse a given episode of a meeting--so that a rich, multi-modal context of the meeting episode is presented.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>