<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-2407">
  <Title>Memory-Based Dependency Parsing</Title>
  <Section position="5" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
4 Results
</SectionTitle>
    <Paragraph position="0"> Table 3 shows the prediction accuracy achieved with memory-based learning for the lexical and non-lexical model, with two different parameter settings for the learner. The results in the first column were obtained with the default settings of the TiMBL package, in particular:  tances rather than k nearest neighbors, which means that, even with k = 1, the nearest neighbor set can contain several in- null The second column shows the accuracy for the best parameter settings found in the experiments (averaged over both models), which differ from the default in the following respects:  For more information about the different parameters and settings, the reader is referred to Daelemans et al. (2003). The results show that the lexical model performs consistently better than the non-lexical model, and that the difference increases with the optimization of the learning algorithm (all differences being significant at the .0001 level according to McNemar's test). This confirms previous results from statistical parsing indicating that lexical information is crucial for disambiguation (Collins, stances that are equally distant to the test instance. This is different from the original IB1 algorithm, as described in Aha et al. (1991).</Paragraph>
    <Paragraph position="1"> 1999; Charniak, 2000). As regards optimization, we may note that although there is a significant improvement for both models, the magnitude of the difference is relatively small.</Paragraph>
    <Paragraph position="2"> Table 4 shows the parsing accuracy obtained with the optimized versions of the MBL models (lexical and nonlexical), compared to the MCLE model described in section 3. We see that MBL outperforms the MCLE model even when limited to the same features (all differences again being significant at the .0001 level according to a paired t-test). This can probably be explained by the fact that the similarity-based smoothing built into the memory-based approach gives a better extrapolation than the fixed back-off sequence in the MCLE model. We also see that the lexical MBL model outperforms both the other models. If we compare the labeled attachment score to the prediction accuracy (which also takes dependency types into account), we observe a substantial drop (from 89.7 to 81.7 for the lexical model, from 87.4 to 76.5 for the non-lexical model), which is of course only to be expected. The unlabeled attachment score is naturally higher, and it is worth noting that the relative difference between the MBL lexical model and the other two models is much smaller. This indicates that the advantage of the lexical model mainly concerns the accuracy in predicting dependency type in addition to transition.</Paragraph>
    <Paragraph position="3">  If we compare the results concerning parsing accuracy to those obtained for other languages (given that there are no comparable results available for Swedish), we note that the best unlabeled attachment score is lower than for English, where the best results are above 90% (attachment score per word) (Collins et al., 1999; Yamada and Matsumoto, 2003), but higher than for Czech (Collins et al., 1999). This is encouraging, given that the size of the training set in our experiments is fairly small, only about 10% of the standard training set for the Penn Treebank. One reason why our results nevertheless compare reasonably well with those obtained with the much larger training set is probably that the conversion to dependency trees is more accurate for the Swedish treebank, given the explicit annotatation of grammatical functions. Moreover, the fact that our parser uses labeled dependencies is probably also significant, since the possibility of using information from previously assigned (labeled) dependencies during parsing seems to have a positive effect on accuracy (Nivre, 2004).</Paragraph>
    <Paragraph position="4"> Finally, it may be interesting to consider the accuracy for individual dependency types. Table 5 gives labeled precision, labeled recall and unlabeled attachment score for four of the most important types with the MBL lexical model. The results indicate that subjects have the highest accuracy, especially when labels are taken into account. Objects and predicative complements have comparable attachment accuracy, but are more often misclassified with respect to dependency type. For adverbial modifiers, finally, attachment accuracy is lower than for the other dependency types, which is largely due to the notorious PP-attachment problem.</Paragraph>
  </Section>
class="xml-element"></Paper>