<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-0701">
  <Title>Sydney, July 2006. c(c)2006 Association for Computational Linguistics Dimensionality Reduction Aids Term Co-Occurrence Based Multi-Document Summarization</Title>
  <Section position="7" start_page="3" end_page="3" type="evalu">
    <SectionTitle>
5 Discussion and Future Work
</SectionTitle>
    <Paragraph position="0"> The positive message from the experimental results is that SVD dimensionality reduction improves performance over a term co-occurrence model for computing relevance and redundancy in a MMR framework. We note that we cannot conclude that the DS or DS+SVD systems outperform a conventional tf.idf -weighted term x sentence representation on this task. However, results from Jagarlamudi et al. (2005) suggest that the DS and term x sentence representations may be complementary in which case we would expect a further improvement through an ensemble technique.</Paragraph>
    <Paragraph position="1"> Previous results comparing SVD with unreduced representations show mixed results. For example, Pedersen et al. (2005) experiment with term co-occurrence representations with and without SVD on a name discrimination task and find 8Pairwise effect size estimates over datasets aren't sensible. Averaging of differences between pairs was affected by outliers, presumably caused by Rouge's error distribution. that the unreduced representation tends to perform better. Rohde et al. (In prep), on the other hand, find that a reduced matrix does perform better on word pair similarity and multiple-choice vocabulary tests. One crucial factor here may be the size of the corpus. SVD may not offer any reliable 'latent semantic' advantage when the corpus is small, in which case the efficiency gain from dimensionality reduction is less of a motivation anyway. We plan to address the question of corpus size in future work by comparing DS and DS+SVD derived from corpora of varying size. We hypothesise that the larger the corpus used to compile the term co-occurrence information, the larger the potential contribution from dimensionality reduction. This will be explored by running the experiment described in this paper a number of times using corpora of different sizes (e.g. 0.5m, 1m, 10m and 100m words).</Paragraph>
    <Paragraph position="2"> Unlike official DUC evaluations, which rely on human judgements of readability and informativeness, our experiments rely solely on Rouge n-gram evaluation metrics. It has been shown in DUC 2005 and in work by Murray et al. (2005b; 2006) that Rouge does not always correlate well with human evaluations, though there is more stability when examining the correlations of macro-averaged scores. Rouge suffers from a lack of power to discriminate between systems whose performance is judged to differ by human annotators.</Paragraph>
    <Paragraph position="3"> Thus, it is likely that future human evaluations would be more informative. Another way that the evaluation issue might be addressed is by using an annotated sentence extraction corpus. This could proceed by comparing gold standard alignments between abstract and full document sentences with predicted alignments using correlation analysis.</Paragraph>
  </Section>
class="xml-element"></Paper>