<?xml version="1.0" standalone="yes"?>
<Paper uid="W00-0720">
  <Title>Genetic Algorithms for Feature Relevance Assignment in Memory-Based Language Processing</Title>
  <Section position="4" start_page="103" end_page="104" type="evalu">
    <SectionTitle>
2,3 Results
</SectionTitle>
    <Paragraph position="0"> In Table 1 we show the results of our experiments (average accuracy and standard deviation over ten folds). We can see that applying any feature selection scheme when no weights are used (IB1) significantly improves classification performance (p&lt;0.01) 5. Selection also improves accuracy when using the IBI-IG or IGTREE algorithm. These differences are significant on the morphology dataset (p&lt;0.05), but for the unknown words dataset only the difference between (IB1) and (IBi-~-GASEL) is significant (p&lt;0.01). In both cases, however, the results in Table 1 do not reveal significant differences between evolutionary, backward or forward selection.</Paragraph>
    <Paragraph position="1"> With respect to feature weighting by means of a GA the results are much less clear: for the morphology data, the GA-weights significantly improve upon IB1, refered to as IB1-GA in the table, (p&lt;0.01) but not IGTREE (GATREE in the table). For the other dataset OA-weights do not even improve upon IB1. But in general, those weights found by the genetic algorithm lead to comparable classification accuracy as with gain ratio based weighting. The same applies to the combination of aA-weights with further selection of irrelevant features (GATREE-bGASEL).</Paragraph>
    <Section position="1" start_page="103" end_page="104" type="sub_section">
      <SectionTitle>
2.4 The Effect of GA Parameters
</SectionTitle>
      <Paragraph position="0"> We also wanted to test whether the GA would benefit from optimisation in the crossover and mutation probabilities. To this end, we used the morphology dataset, which was split into an 80% trainfile, a 10% validationfile and a held-out 10% testfile. The mutation rate was var- null the experiments. Boldface marks the best results for each basic algorithm per data set.</Paragraph>
      <Paragraph position="1"> ied stepwise adding a value of 0.001 at each experiment, starting at a 0.004 value up to 0.01. The different values for crossover ranged from 0.65 to 0.95, in steps of 0.05. The effect of changing crossover and mutation probabilities was tested for IBl-IG+GA-selection, for IB1 with CA weighting, for IGTREE+GA-selection, and for IGTREE with GA-weight settings.</Paragraph>
      <Paragraph position="2"> These experiments show considerable fluctuation in accuracy within the tested range, but different parameter settings could also yield same results although they were far apart in value.</Paragraph>
      <Paragraph position="3"> Some settings achieved a particularly high accuracy in this training regime (e.g. crossover: 0.75, mutation: 0.009). However, when we used these in the ten-fold cv setup of our main experiments, this gave a mean score of 97.4 (+- 0.9) for IBi-IG with CA-selection and a mean score of 97.1 (+- 1.1) for IGTREE with GA-selection.</Paragraph>
      <Paragraph position="4"> These accuracies are similar to those achieved with our default parameter settings.</Paragraph>
    </Section>
    <Section position="2" start_page="104" end_page="104" type="sub_section">
      <SectionTitle>
2.5 Discussion
</SectionTitle>
      <Paragraph position="0"> Feature selection on the morphology task shows a significant increase in performance accuracy, whereas on the unknown words task the differences are less outspoken. To get some insight into this phenomenon, we looked at the average probabilities of the features that were left out by the evolutionary algorithm and their average weights.</Paragraph>
      <Paragraph position="1"> On the morphology task this reveals that nucleus and coda of the last syllable are highly relevant, they are always included. The onset of all three syllables is always left out. Further, in all partitions the nucleus and coda of the second syllable are left out. 6 For part-of-speech tagging of unknown words all features appear to be more or less equally relevant. Over the ten partitions, either no omission is suggested at all, or the features that carry the pos-tag of n-2 word before and the n+2 word after the focus word are deleted. This is comparable to reducing the context window of this classification task to one word before and one after the focus.</Paragraph>
      <Paragraph position="2"> The fact that all features seem to contribute to the classification when doing POS-tagging (making selection irrelevant) could also explain why the IGTREE algorithm seems to benefit less from the feature orders suggested and why the non-weighted approach IB1 already has a high score on the tagging task. The IGTREE algorithm is more suited for problems where the features can be ordered in a straightforward way because they have significantly different relevance. null</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>