<?xml version="1.0" standalone="yes"?>
<Paper uid="N04-4024">
  <Title>Direct Maximization of Average Precision by Hill-Climbing, with a Comparison to a Maximum Entropy Approach</Title>
  <Section position="3" start_page="0" end_page="0" type="intro">
    <SectionTitle>
2 Motivation
</SectionTitle>
    <Paragraph position="0"> Our initial approach to the study of optimal queries employed a conditional maximum entropy model. This model exhibited some problematic behavior, which motivated the development of the weight search algorithm described here.</Paragraph>
    <Paragraph position="1"> The maximum entropy model is used as follows. It is given a set of relevant and non-relevant documents and a vector of terms (the query). For any document, the model predicts the probability of relevance for that document based on the Okapi term frequency (tf ) scores (Robertson and Walker, 1994) for the query terms within it. Queries are developed by starting with the best possible one-term query and adding individual terms from a candidate set chosen according to a mutual information criterion. As each term is added, the model coef cients are set to maximize the probability of the empirical data (the document set plus relevance judgments), as described in Section 4.</Paragraph>
    <Paragraph position="2"> Treating the model coef cients as term weights yields a weighted query. This query produces a retrieval status value (RSV) for each document that is a monotonically increasing function of the probability of relevance, in accord with the probability ranking principle (Robertson, 1977). We can then calculate the average precision of the document set as ordered by these RSVs.</Paragraph>
    <Paragraph position="3"> As each additional query term represents another degree of freedom, one would expect model performance to improve at each step. However, we noted that the addition of a new term would occasionally result in a decrease in average precision despite the fact that the model could have chosen a zero weight for the newly added term.</Paragraph>
    <Paragraph position="4"> Figure 1 shows an example of this phenomenon for one TREC topic.</Paragraph>
    <Paragraph position="5"> This is the result of what might be called metric divergence . While we use average precision to evaluate the queries, the maximum entropy model maximizes the likelihood of the training data. These two metrics occasionally disagree in their evaluation of particular weight vectors. In particular, maximum entropy modeling may favor increasing the estimation of documents lower in the ranking at the expense of accuracy in the prediction of highly ranked documents. This can increase training data likelihood yet have a detrimental effect on average precision. null The metric divergence problem led us to consider an alternative approach for setting term weights which would hill-climb on average precision directly. In particular, we were interested in evaluating the results produced by the maximum entropy approach how much was the maximization of likelihood affecting the ultimate performance as measured by average precision? The algorithm described in the following section was developed to this end.</Paragraph>
  </Section>
class="xml-element"></Paper>