<?xml version="1.0" standalone="yes"?>
<Paper uid="P02-1030">
  <Title>Scaling Context Space</Title>
  <Section position="8" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
6 Results
</SectionTitle>
    <Paragraph position="0"> Since MINIPAR performs morphological analysis on the context relations we have added an existing morphological analyser (Minnen et al., 2000) to the other extractors. Table 4 shows the improvement gained by morphological analysis of the attributes and relations for the SEXTANT 150M corpus.</Paragraph>
    <Paragraph position="1"> The improvement in results is quite signi cant, as is the reduction in the representation space and number of unique context relations. The reduction in the number of terms is a result of coalescing the plural nouns with their corresponding singular nouns, which also reduces data sparseness problems. The remainder of the results use morphological analysis of both the words and attributes.</Paragraph>
    <Paragraph position="2"> Table 5 summarises the average results of applying all of the extraction systems to the two 150M word corpora. The rst thing to note is the time spent extracting contextual information: MINIPAR takes signi cantly longer to run than the other extractors. Secondly, SEXTANT and MINIPAR have quite similar results overall, but MINIPAR is slightly better across most measures. However, SEXTANT runs about 28 times faster than MINI-PAR. Also, MINIPAR extracts many more terms and relations with a much larger representation than SEXTANT. This is partly because MINIPAR extracts more types of relations from the parse tree  ods, W(L1R1) and W(L1;2) give reasonable results.</Paragraph>
    <Paragraph position="3"> The larger windows with low correlation between the thesaurus term and context, extract a massive context representation but the results are about 10% worse than the syntactic extractors.</Paragraph>
    <Paragraph position="4"> Overall the precision and recall are relatively poor. Poor recall is partly due to the gold standard containing some plurals and multi-word terms which account for about 25% of the synonyms.</Paragraph>
    <Paragraph position="5"> These have been retained because the MINIPAR and CASS systems are capable of identifying (at least some) multi-word terms.</Paragraph>
    <Paragraph position="6"> Given a xed time period (of more than the four days MINIPAR takes) and a xed 150M corpus we would probably still choose to use MINIPAR unless the representation was too big for our learning algorithm, since the thesaurus quality is slightly better. Table 6 shows what happens to thesaurus quality as we decrease the size of the corpus to 164 th of its original size (2.3M words) for SEXTANT. Halving the corpus results in a signi cant reduction for most of the measures. All ve evaluation measures show the same log-linear dependence on the size of the corpus. Figure 1 shows the same trend for Inverse Rank evaluation of the MINIPAR thesaurus with a log-linear tting the data points.</Paragraph>
    <Paragraph position="7"> We can use the same curve tting to estimate the- null saurus quality on larger corpora for three of the best extractors: SEXTANT, MINIPAR and W(L1R1). Figure 2 does this with the direct match evaluation. The estimate indicates that MINIPAR will continue to be the best performer on direct matching. We then plot the direct match scores for the 300M word corpus to see how accurate our predictions are. The SEXTANT system performs almost exactly as predicted and the other two slightly under-perform their predicted scores, thus the tting is accurate enough to make reasonable predictions.</Paragraph>
    <Paragraph position="8"> Figure 2 is a graph for making engineering decisions in conjunction with the data in Table 5. For instance, if we x the total time and computational</Paragraph>
    <Paragraph position="10"> MINIPAR can process 75M words, we get a best direct match score of 23.5. However, we can get the same resultant accuracy by using SEXTANT on a corpus of 116M words or W(L1R1) on a corpus of 240M words. From Figure 5, extracting contexts from corpora of these sizes would take MINIPAR 37 hours, SEXTANT 2 hours and W(L1R1) 12 minutes.</Paragraph>
    <Paragraph position="11"> Interpolation on Figure 3 predicts that the extraction would result in 10M unique relations from MINIPAR and SEXTANT and 19M from W(L1R1). Figure 4 indicates that extraction would result in 550k  MINIPAR terms, 200k SEXTANT terms and 600k W(L1R1) terms.</Paragraph>
    <Paragraph position="12"> Given these values and the fact that the time complexity of most thesaurus extraction algorithms is at least linear in the number of unique relations and squared in the number of thesaurus terms, it seems SEXTANT may represent the best solution.</Paragraph>
    <Paragraph position="13"> With these size issues in mind, we nally consider some methods to limit the size of the context representation. Table 7 shows the results of performing various kinds of ltering on the representation size. The FIXED and LEXICON lters run over the  full 300M word corpus, but have size limits based on the 150M word corpus. The FIXED lter does not allow any object/attribute pairs to be added that were not extracted from the 150M word corpus. The LEXICON lter does not allow any objects to be added that were not extracted from the 150M word corpus. The &gt; 1 and &gt; 2 lters prune relations with a frequency of less than or equal to one or two. The FIXED and LEXICON lters show that counting over larger corpora does produce marginally better results. The &gt; 1 and &gt; 2 lters show that the many relations that occur infrequently do not contribute signi cantly to the vector comparisons and hence don't impact on the nal results, even though they dramatically increase the representation size.</Paragraph>
  </Section>
class="xml-element"></Paper>