<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-3405">
  <Title>Shallow Discourse Structure for Action Item Detection</Title>
  <Section position="4" start_page="31" end_page="31" type="metho">
    <SectionTitle>
3 Baseline Experiments
</SectionTitle>
    <Paragraph position="0"> We applied Gruenstein et al. (2005)'s flat annotation schema to transcripts from a sequence of 5 short related meetings with 3 participants recorded as part of the CALO project. Each meeting was simulated in that its participants were given a scenario, but was not scripted. In order to avoid entirely dataor scenario-specific results (and also to provide an acceptable amount of training data), we then added a random selection of 6 ICSI and 1 ISL meetings from Gruenstein et al. (2005)'s annotations. Like (Corston-Oliver et al., 2004) we used support vector machines (Vapnik, 1995) via the classifier SVM-light (Joachims, 1999). Their full set of features are not available to us, but we experimented with combinations of words and n-grams and assessed classification performance via a 5-fold validation on each of the CALO meetings. In each case, we trained classifiers on the other 4 meetings in the CALO sequence, plus the fixed ICSI/ISL training selection.</Paragraph>
    <Paragraph position="1"> Performance (per utterance, on the binary classification problem) is shown in Table 1; overall f-score figures are poor even on these short meetings. These figures were obtained using words (unigrams, after text normalization and stemming) as features - we investigated other discriminative classifier methods, and the use of 2- and 3-grams as features, but no improvements were gained.</Paragraph>
    <Paragraph position="2"> Mtg. Utts AI Utts. Precision Recall F-Score</Paragraph>
  </Section>
  <Section position="5" start_page="31" end_page="32" type="metho">
    <SectionTitle>
4 Hierarchical Annotations
</SectionTitle>
    <Paragraph position="0"> Two problems are apparent: firstly, accuracy is lower than desired; secondly, identifying utterances related to action items does not allow us to actually identify those action items and extract their properties (deadline, owner etc.). But if the utterances related to these properties form distinct sub-classes which have their own distinct features, treating them separately and combining the results (along the lines of (Klein et al., 2002)) might allow better performance, while also identifying the utterances where each property's value is extracted.</Paragraph>
    <Paragraph position="1"> Thus, we produced an annotation schema which distinguishes among these four classes. The first three correspond to the discussion and assignment of the individual properties of the action item (task description, timeframe and owner); the final agreement class covers utterances which explicitly show that the action item is agreed upon.</Paragraph>
    <Paragraph position="2"> Since the task description subclass extracts a description of the task, it must include any utterances that specify the action to be performed, includingthosethatproviderequiredantecedentsfor anaphoric references. The owner subclass includes any utterances that explicitly specify the responsible party (e.g. &amp;quot;I'll take care of that&amp;quot;, or &amp;quot;John, we'll leave that to you&amp;quot;), but not those whose function might be taken to do so implicitly (such as agreements by the responsible party). The timeframe subclass includes any utterances that explicitly refer to when a task may start or when it is expected to be finished; note that this is often not specified with  a date or temporal expression, but rather e.g. &amp;quot;by the end of next week,&amp;quot; or &amp;quot;before the trip to Aruba&amp;quot;. Finally, the agreement subclass includes any utterancesinwhichpeopleagreethattheactionshould null and will be done; not only acknowledgements by the owner themselves, but also when other people express their agreement.</Paragraph>
    <Paragraph position="3"> A single utterance may be assigned to more than one class: &amp;quot;John, you need to do that by next Monday&amp;quot; might count asownerandtimeframe.</Paragraph>
    <Paragraph position="4"> Likewise, there may be more than one utterance of each class for a single action item: John's response &amp;quot;OK, I'll do that&amp;quot; would also be classed as owner (as well as agreement). While we do not require all of these subclasses to be present for a set of utterances to qualify as denoting an action item, we expect any action item to include most of them.</Paragraph>
    <Paragraph position="5"> We applied this annotation schema to the same 12 meetings. Initial reliability between two annotators on the single ISL meeting (chosen as it presented a significantly more complex set of action items than others in this set) was encouraging. The best agreement was achieved on timeframe utterances (k = .86), with owner utterances slightly less good (between k = .77), and agreement and description utterances worse but still acceptable (k = .73). Further annotation is in progress.</Paragraph>
  </Section>
class="xml-element"></Paper>