<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-0308">
  <Title>Incrementality in Deterministic Dependency Parsing</Title>
  <Section position="6" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
5 Experimental Evaluation
</SectionTitle>
    <Paragraph position="0"> In order to measure the degree of incrementality achieved in practical parsing, we have evaluated a parser that uses the arc-eager parsing algorithm in combination with a memory-based classifier for predicting the next transition. In experiments reported in Nivre et al. (2004), a parsing accuracy of 85.7% (unlabeled attachment score) was achieved, using data from a small treebank of Swedish (Einarsson, 1976), divided into a training set of 5054 sentences and a test set of 631 sentences. However, in the present context, we are primarily interested in the incrementality of the parser, which we measure by considering the number of connected components in (S,AS) at different stages during the parsing of the test data.</Paragraph>
    <Paragraph position="1"> The results can be found in Table 1, where we see that out of 16545 configurations used in parsing 613 sentences (with a mean length of 14.0 words), 68.9% have zero or one connected component on the stack, which is what we require of a strictly incremental parser. We also see that most violations of incrementality are fairly mild, since more than 90% of all configurations have no more than three connected components on the stack.</Paragraph>
    <Paragraph position="2"> Many violations of incrementality are caused by sentences that cannot be parsed into a well-formed dependency graph, i.e. a single projective dependency tree, but where the output of the parser is a set of internally connected components. In order to test the influence of incomplete parses on the statistics of incrementality, we have performed a second experiment, where we restrict the test data to those 444 sentences (out of 613), for which the parser produces a well-formed dependency graph. The results can be seen in Table 2. In this case, 87.1% of all configurations in fact satisfy the constraints of incrementality, and the proportion of configurations that have no more than three connected components on the stack is as high as 99.5%.</Paragraph>
    <Paragraph position="3"> It seems fair to conclude that, although strict word-by-word incrementality is not possible in deterministic dependency parsing, the arc-eager algorithm can in practice be seen as a close approximation of incremental parsing.</Paragraph>
  </Section>
class="xml-element"></Paper>