<?xml version="1.0" standalone="yes"?>
<Paper uid="C04-1083">
  <Title>Browsing Help for Faster Document Retrieval</Title>
  <Section position="8" start_page="1" end_page="1" type="evalu">
    <SectionTitle>
6 Results
</SectionTitle>
    <Paragraph position="0"> During the evaluation, participants could take a break between each research because of the 3 hours required for the full experiment. Several criteria have been used for performance  judgement: * Time to find the first relevant document, * Number of relevant documents retrieved, * Average recall.</Paragraph>
    <Paragraph position="1">  They are described in the following sections.</Paragraph>
    <Section position="1" start_page="1" end_page="1" type="sub_section">
      <SectionTitle>
6.1 Relevance judgment
</SectionTitle>
      <Paragraph position="0"> For each visited article, the subjects were asked to click on one of the two following buttons: * VALIDATION: document is judged relevant, * ANNULATION: document is judged non-relevant.</Paragraph>
      <Paragraph position="1"> An average of 4.9 documents was assessed relevant per query and user. Table 3 shows the average of relevant and non-relevant documents found by every user:</Paragraph>
    </Section>
    <Section position="2" start_page="1" end_page="1" type="sub_section">
      <SectionTitle>
User
</SectionTitle>
      <Paragraph position="0"/>
    </Section>
    <Section position="3" start_page="1" end_page="1" type="sub_section">
      <SectionTitle>
6.2 Time to first relevant document
</SectionTitle>
      <Paragraph position="0"> Time is a good criterion for navigation effectiveness judgment. How long does it take for users to find the first relevant document? This question is probably one of the most important in order to judge navigability gain over the six interfaces. When no-relevant documents were found for a query, the time was set to the maximum research time: 600s.</Paragraph>
      <Paragraph position="1"> The results, presented in Table 4, show the mean time over users/queries to the first relevant document. Responding to our expectations, Interface6 obtains the best result (smallest mean time).</Paragraph>
      <Paragraph position="2"> Interface Mean time to first rel. doc. (in s)</Paragraph>
    </Section>
    <Section position="4" start_page="1" end_page="1" type="sub_section">
      <SectionTitle>
document
</SectionTitle>
      <Paragraph position="0"> It shows that an interface with all features is better than having only one or none of them According to the different results, it also appears that a search interface featuring the named entities as navigation alternative the search time toward the first relevant docum The other interfaces seem to be of little help. In some way, that was predictable since and Interface5 do not present alternative at the summary page level.</Paragraph>
      <Paragraph position="1"> In this table, no standard deviation is given because the considered data are not hom (different users with different interfaces for different queries). For instance, spent by User 1 (naive user) on while the expert user 6 spent an average tim 31 s in order to find the first relevant docum</Paragraph>
    </Section>
    <Section position="5" start_page="1" end_page="1" type="sub_section">
      <SectionTitle>
6.3 Number of relevant documents retrieved
</SectionTitle>
      <Paragraph position="0"> The time to first relevant docum be the only criterion in order to judge the navigation effectiveness. Therefore, to the first  As expected, Interface6 (all features available to users) gives maximum relevant documents in average. It scores almost twice as Interface1. Concerning the non-relevant documents, we see that interfaces 2,3,5 and 6 allow the filtering of non-relevant documents or the navigation from a relevant document to another one. The consistency between Interface1 and Interface4 is logical because the user has to look in both cases at the document to know it is not relevant.</Paragraph>
    </Section>
    <Section position="6" start_page="1" end_page="1" type="sub_section">
      <SectionTitle>
6.4 Average recall
</SectionTitle>
      <Paragraph position="0"> In order to combine the two previous criteria, we computed the average recall over all users and all queries, for a given interface. In order to compute the recall for a query q, the total number of relevant documents was approximated to the total  number of documents marked as relevant over subjects for q. The recall at time t for a query q, a user u is then computed with the following formula:</Paragraph>
      <Paragraph position="2"> where N(q,u,t) is the number of relevant documents assessed by user u at time t for query q and N(q) is the total number of unique relevant documents found by all the users for query q.</Paragraph>
      <Paragraph position="3"> The average recall at time t is computed by averaging the recall over the users and the queries. Figure 4 presents the curves of average recall according to time at a sampling rate of 10 seconds.</Paragraph>
      <Paragraph position="4"> First of all, this figure shows that using any of the browsing features improves the document retrieval performances. The two better curves are obtained with entity filtering or using all the features. It is however a little bit strange that Interface3 rises over Interface6 on the first 120 seconds. Extensive tests should be carried on to corroborate these results.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>