<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-1209">
  <Title>Support Vector Machine Approach to Extracting Gene References into Function from Biological Documents</Title>
  <Section position="5" start_page="54" end_page="56" type="evalu">
    <SectionTitle>
4 Results and Discussions
</SectionTitle>
    <Paragraph position="0"> The performance measures are based on Dice coefficient, which calculates the overlap between the candidate GeneRIF and actual GeneRIF.</Paragraph>
    <Paragraph position="1"> Classic Dice (CD) is the classic Dice formula using a common stop word list and the Porter stemming algorithm. Due to lack of space, we referred you to the Genomics track overview for the other three modifications of CD (Hersh and Bhupatiraju, 2003).</Paragraph>
    <Paragraph position="2"> The evaluation results are shown in Table 2. The 1st row shows the official run of Jelier's team, the first place in the official runs. The 2nd row shows the performance when the Naive Bayes classifier adopted by Jelier is replaced with SVMs. The 3rd row is the performance of our weighting scheme without a classifier. The 4th row then lists the performance when our weighting scheme is combined with SVMs. The 5th row is the result when our weighting scheme and the sentence-wise bag of words model are combined together. The 6th row is the result when two gene/protein name detectors are incorporated into the combined model.</Paragraph>
    <Paragraph position="3"> The next two rows were obtained after two feature selection methods were applied. The 9th row shows the performance when the classifier always proposes a sentence most similar to the actual GeneRIF. The last row lists the baseline, i.e., title is always picked.</Paragraph>
    <Paragraph position="4"> A comparative study on text categorization (Joachims, 1998) showed that SVMs outperform other classification methods, such as Naive Bayes, C4.5, and k-NN. The reasons would be that SVMs are capable of handling large feature space, text categorization has few irrelevant features, and document vectors are sparse. The comparison  between SVMs and the Naive Bayes classifier again demonstrated the superiority of SVMs in text categorization (rows 1, 2).</Paragraph>
    <Paragraph position="5"> The performance greatly improved after introducing position information (rows 3, 4), showing the sentence position plays an important role in locating the GeneRIF sentence. The 2% difference between rows 2 and 4 indicates that the features under sentence-wise bag of words model are more informative than those under our weighting scheme. However, with only 9 features, our weighting scheme with SVMs performed fairly well. Comparing the performance before and after combining our weighting scheme and the sentence-wise bag of words model (rows 2, 5 and rows 4, 5), we can infer from the performance differences that both models provide mutually exclusive information in the combined model. The result shown in row 6 indicates that the information of gene/protein name occurrences did not help identify the GeneRIF sentences in these 139 test abstracts. We performed feature selection on the combined model to reduce the dimension of feature space.</Paragraph>
    <Paragraph position="6"> There were two methods applied: a supervised heuristic method (denoted as BWRatio feature selection in Table 2) (S. Dutoit et al., 2002) and another unsupervised method (denoted as Graphical feature selection in Table 2) (Chang et al., 2002). The number of features was then reduced to about 4,000 for both methods. Unfortunately, the performance did not improve after either method was applied. This may be attributed to over-fitting training data, because the cross-validation accuracies are indeed higher than those without feature selection. The result may also imply there are little irrelevant features in this case.</Paragraph>
  </Section>
class="xml-element"></Paper>