<?xml version="1.0" standalone="yes"?>
<Paper uid="W05-1513">
  <Title>Vancouver, October 2005. c(c)2005 Association for Computational Linguistics A Classifier-Based Parser with Linear Run-Time Complexity</Title>
  <Section position="7" start_page="129" end_page="130" type="evalu">
    <SectionTitle>
2 http://chasen.org/~taku/software/TinySVM
</SectionTitle>
    <Paragraph position="0"> trees were lexicalized using similar head-table rules as those mentioned in (Collins, 1996). The trees were then converted into trees containing only unary and binary branching, using the binarization transform described in section 2. Classifier training instances of features paired with classes (parser actions) were extracted from the trees in the training set, as described in section 2.3. The total number of training instances was about 1.5 million.</Paragraph>
    <Paragraph position="1"> The classifier in the SVM-based parser (denoted by SVMpar) uses the polynomial kernel with degree 2, following the work of Yamada and Matsumoto (2003) on SVM-based deterministic dependency parsing, and a one-against-all scheme for multi-class classification. Because of the large number of training instances, we used Yamada and Matsumoto's idea of splitting the training instances into several parts according to POS tags, and training classifiers on each part. This greatly reduced the time required to train the SVMs, but even with the splitting of the training set, total training time was about 62 hours. Training set splitting comes with the cost of reduction in accuracy of the parser, but training a single SVM would likely take more than one week. Yamada and Matsumoto experienced a reduction of slightly more than 1% in de- null racy, and time required to parse the test set. The parsers of Yamada and Matsumoto (Y&amp;M) and Nivre and Scholz (N&amp;S) do not produce constituent structures, only dependencies. &amp;quot;unk&amp;quot; indicates unknown values. Results for MBLpar and SVMpar using correct POS tags (if automatically produced POS tags are used, accuracy figures drop about 1.5% over all metrics).  pendency accuracy due to training set splitting, and we expect that a similar loss is incurred here.</Paragraph>
    <Paragraph position="2"> When given perfectly tagged text (gold tags extracted from the Penn Treebank), SVMpar has labeled constituent precision and recall of 87.54% and 87.61%, respectively, and dependency accuracy of 90.3% over all sentences in the test set.</Paragraph>
    <Paragraph position="3"> The total time required to parse the entire test set was 11 minutes. Out of more than 2,400 sentences, only 26 were rejected by the parser (about 1.1%). For these sentences, partial analyses were created by combining the items in the stack in flat structures, and these were included in the evaluation. Predictably, the labeled constituent precision and recall obtained with automatically POS-tagged sentences were lower, at 86.01% and 86.15%. The part-of-speech tagger used in our experiments was SVMTool (Gimenez and Marquez, 2004), and its accuracy on the test set is 97%.</Paragraph>
    <Paragraph position="4"> The MBL-based parser (denoted by MBLpar) uses the IB1 algorithm, with five nearest neighbors, and the modified value difference metric (MVDM), following the work of Nivre and Scholz (2004) on MBL-based deterministic dependency parsing. MBLpar was trained with all training instances in under 15 minutes, but its accuracy on the test set was much lower than that of SVMpar, with constituent precision and recall of 80.0% and 80.2%, and dependency accuracy of 86.3% (24 sentences were rejected). It was also much slower than SVMpar in parsing the test set, taking 127 minutes. In addition, the total memory required for running MBLpar (including the classifier) was close to 1 gigabyte (including the trained classifier), while SVMpar required only about 200 megabytes (including all the classifiers).</Paragraph>
    <Paragraph position="5"> Table 1 shows a summary of the results of our experiments with SVMpar and MBLpar, and also results obtained with the Charniak (2000) parser, the Bikel (2003) implementation of the Collins (1997) parser, and the Ratnaparkhi (1997) parser.</Paragraph>
    <Paragraph position="6"> We also include the dependency accuracy from Yamada and Matsumoto's (2003) SVM-based dependency parser, and Nivre and Scholz's (2004) MBL-based dependency parser. These results show that the choice of classifier is extremely important in this task. SVMpar and MBLpar use the same algorithm and features, and differ only on the classifiers used to make parsing decisions. While in many natural language processing tasks different classifiers perform at similar levels of accuracy, we have observed a dramatic difference between using support vector machines and a memory-based learner. Although the reasons for such a large disparity in results is currently the subject of further investigation, we speculate that a relatively small difference in initial classifier accuracy results in larger differences in parser performance, due to the deterministic nature of the parser (certain errors may lead to further errors). We also believe classifier choice to be one major source of the difference in accuracy between Nivre and Scholz's parser and Yamada and Matsumoto's parser.</Paragraph>
    <Paragraph position="7"> While the accuracy of SVMpar is below that of lexicalized PCFG-based statistical parsers, it is surprisingly good for a greedy parser that runs in linear time. Additionally, it is considerably faster than lexicalized PCFG-based parsers, and offers a good alternative for when fast parsing is needed.</Paragraph>
    <Paragraph position="8"> MBLpar, on the other hand, performed poorly in terms of accuracy and speed.</Paragraph>
  </Section>
class="xml-element"></Paper>