<?xml version="1.0" standalone="yes"?>
<Paper uid="N06-1051">
  <Title>A Machine Learning based Approach to Evaluating Retrieval Systems</Title>
  <Section position="7" start_page="404" end_page="405" type="concl">
    <SectionTitle>
6 Conclusions and Discussion
</SectionTitle>
    <Paragraph position="0"> This study has well illustrated that two algorithms of RBoost and rSVM are quite suitable for qrels construction task. The final qrels are not only small enough to ask for human judgment but also result in reliable conclusion about system effectiveness in  comparison with the counterpart of TREC methodology and that of MTF.</Paragraph>
    <Paragraph position="1"> It is necessary to include other metasearch methods for further study. This will allow us to validate not only the impact of the metasearch training principle based on pairwise ranking error RLoss but also the capacity of automatic feature selection of the two ranking algorithms used in this paper.</Paragraph>
    <Paragraph position="2"> This method needs to be further verified on challenging ad-hoc retrieval scenarios such as Terabyte, Web Topic Distillation or Robust Tracks in TREC context. The hardness of these scenarios involves two main issues. First, the number of document judged relevant varies largely across the whole topic set. Second, some topics might even have no relevant document in shallow pools. These matter any statistical inference on shallow pools.</Paragraph>
    <Paragraph position="3"> Acknowledgement The authors thank M.-R. Amini, B. Piwowarski, J. Zobel and the anonymous reviewers for their thorough comments. We acknowledge NIST to make accessible the TREC submissions. This work was supported in part by the IST Programme of the European Community, under the PASCAL Network of Excellence, IST-2002-506778. The publication only reflects the authors' views.</Paragraph>
  </Section>
class="xml-element"></Paper>