<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-3211">
  <Title>Mixing Weak Learners in Semantic Parsing</Title>
  <Section position="6" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
5 Discussion and Future Research
</SectionTitle>
    <Paragraph position="0"> The version of Random Forests described here out-performs the Bayesian algorithm (Gildea and Jurafsky, 2002; Gildea and Palmer, 2002) by 1.8% on the same feature set and outperforms the boosted decision tree classifier (Surdeanu et al., 2003) by 3.5% on the extended feature set with 5 fewer features.</Paragraph>
    <Paragraph position="1"> The SVM classifier (Pradhan et al., 2003) was 2.3% better training on the same data, but only 0.6% better than our best results.</Paragraph>
    <Paragraph position="2"> The Random Forest (RF) approach has advantages that might make it a better choice than an SVM in certain circumstances. Conceptually, it is simpler to understand and can be implemented more easily. This also makes it easier to modify the algorithm to evaluate new techniques. RFs allow one to more easily implement multi-class classifiers. The RFs here were implemented as a single classifier, rather than as the 22 one-against-all classifiers required by the SVM approach. Since RFs are not overly sensitive to noise in the training data (Breiman, 2001), it might be the case that they will narrow the performance gap when training is based on automatically parsed sentences. Further research is required in this area. Additionally, RFs have an advantage in training time. It takes about 40% of the SVM time (8 versus 20 hours) to train on the extended feature set for the classification task and we expect this time to be cut by up to a factor of 10 in porting from MatLab to C. Classification time is generally faster for RFs as well, which is important for real-time tasks.</Paragraph>
    <Paragraph position="3"> In a class-by-class comparison, using the same features, the RF performed significantly better than the SVM on Arg0 roles, the same or slightly better on 12 of the other 21 arguments, and slightly better overall on the 14 adjunctive arguments (77.8% versus 77.3% accuracy on 1882 observations). Reviewing performance on data not seen during training, both algorithms degraded to about 94% of their accuracy on seen data.</Paragraph>
    <Paragraph position="4"> The RF algorithm should be evaluated on the identification task and on the combined identification and classification task. This will provide additional comparative evidence to contrast it with the SVM approach. Further research is also required to determine how RFs generalize to new genres.</Paragraph>
    <Paragraph position="5"> Another area for future research involves the estimation of class probabilities. MOB-ESP, a variant of Random Forests which outputs class probability estimates, has been shown to produce very good results (Nielsen, 2004). Preliminary experiments suggest that using these probability estimates in conjunction with an SVM classifier might be more effective than estimating probabilities based on the example's distance from the decision surface as in (Platt, 2000). Class probabilities are useful for several semantic parsing and more general NLP tasks, such as selective use of labeled examples during training (c.f., Pradhan et al., 2003) and N-best list processing.</Paragraph>
  </Section>
class="xml-element"></Paper>