<?xml version="1.0" standalone="yes"?>
<Paper uid="N06-1050">
  <Title>Creating a Test Collection for Citation-based IR Experiments</Title>
  <Section position="7" start_page="396" end_page="397" type="concl">
    <SectionTitle>
6 Conclusions and Future Work
</SectionTitle>
    <Paragraph position="0"> We have presented an approach to building a test collection from an existing collection of research papers and described the application of our method to the ACL Anthology. We have collected 170 queries with relevance data, centered around the ACL-2005 and HLT-EMNLP-2005 conferences. We  have sanity-checked the usability of our data by running the queries through a retrieval system and evaluating the results using standard software. The collection currently has a low number of judged relevant documents and further experimentation is needed to determine if this poses a real problem. We plan a second stage of collecting relevance judgements, in line with the original Cran eld design, whereby authors who have contributed queries will be asked to judge the relevance of documents in retrieval rankings from standard IR models and, ideally, from our eventual citation-based experiments. Nevertheless, our test collection is likely to suffer from incomplete relevance information. The bpref measure (Buckley and Voorhees, 2004) gauges retrieval effectiveness solely on the basis of judged documents and is more stable to differing levels of completeness than measures such as MAP, R-precision or precision at xed document cutoffs.</Paragraph>
    <Paragraph position="1"> Thus, bpref may offer a solution to the incompleteness problem and we intend to investigate its potential use in our future evaluations.</Paragraph>
    <Paragraph position="2"> When nished, we hope our test collection will be a generally useful IR resource. In particular, we expect the collection to be useful for experimentation with citation information, for which there is currently no existing test collection with the properties that ours offers.</Paragraph>
    <Paragraph position="3"> Acknowledgements Thanks to the reviewers for their useful comments and to Karen Spcurrency1arck Jones for many instructive discussions.</Paragraph>
  </Section>
class="xml-element"></Paper>