<?xml version="1.0" standalone="yes"?>
<Paper uid="W05-1524">
  <Title>Vancouver, October 2005. c(c)2005 Association for Computational Linguistics TFLEX: Speeding up Deep Parsing with Strategic Pruning</Title>
  <Section position="4" start_page="194" end_page="194" type="intro">
    <SectionTitle>
3 Evaluation
</SectionTitle>
    <Paragraph position="0"> Our evaluation data is an excerpt from the Monroe corpus that has been used in previous TRIPS research on parsing speed and accuracy (Swift et al., 2004). The test contained 1042 utterances, from 1 to 45 words in length (mean 5.38 words/utt, st. dev.</Paragraph>
    <Paragraph position="1"> 5.7 words/utt). Using a hold-out set, we determined that a beam width of 3 was an optimal setting for TFLEX. We then compared TFLEX at beam width 3 to the TRIPS parser with chart size limits of 1500, 5000, and 10000. As our evaluation metrics we report are average parse time per sentence and probability of nding at least one parse, the latter being a measure approximating parsing accuracy.</Paragraph>
    <Paragraph position="2"> The results are presented in Figure 1. We grouped sentences into equivalence classes based on length with a 5-word increment. On sentences greater than 10 words long, TFLEX is signi cantly more likely to produce a parse than any of the TRIPS parsers (evaluated using a binary logistic regression, p &lt; .001). Moreover, for sentences greater than 20 words long, no form of TRIPS parser returned a complete parse. TFLEX is signi cantly faster than TRIPS-10000, statistically indistinguishable in terms of parse time from TRIPS-5000, and signi cantly slower than TRIPS-1500 (p &lt; .001).</Paragraph>
    <Paragraph position="3"> Thus, TFLEX presents a superior balance of coverage and ef ciency especially for long sentences (10 words or more) since for these sentences it is signi cantly more likely to nd a parse than any version of TRIPS, even a version where the chart size is expanded to an extent that it becomes signi cantly slower (i.e., TRIPS-10000).</Paragraph>
  </Section>
class="xml-element"></Paper>