File Information
File: 05-lr/acl_arc_1_sum/cleansed_text/xml_by_section/evalu/06/w06-2915_evalu.xml
Size: 9,033 bytes
Last Modified: 2025-10-06 13:59:54
<?xml version="1.0" standalone="yes"?> <Paper uid="W06-2915"> <Title>Which Side are You on? Identifying Perspectives at the Document and Sentence Levels</Title> <Section position="7" start_page="112" end_page="114" type="evalu"> <SectionTitle> 5 Experiments </SectionTitle> <Paragraph position="0"/> <Section position="1" start_page="112" end_page="112" type="sub_section"> <SectionTitle> 5.1 Identifying Perspective at the Document Level </SectionTitle> <Paragraph position="0"> We evaluate three different models for the task of identifying perspective at the document level: two na&quot;ive Bayes models (NB) with different inference methods and Support Vector Machines (SVM) (Cristianini and Shawe-Taylor, 2000). NB-B uses full Bayesian inference and NB-M uses Maximum a posteriori (MAP). We compare NB with SVM not only because SVM has been very effective for classifying topical documents (Joachims, 1998), but also to contrast generative models like NB with discriminative models like SVM. For training SVM, we represent each document as a V -dimensional feature vector, where V is the vocabulary size and each co-ordinate is the normalized term frequency within the document. We use a linear kernel for SVM and search for the best parameters using grid methods.</Paragraph> <Paragraph position="1"> To evaluate the statistical models, we train them on the documents in the bitterlemons corpus and calculate how accurately each model predicts document perspective in ten-fold cross-validation experiments. Table 2 reports the average classification accuracy across the the 10 folds for each model. The accuracy of a baseline classifier, which randomly assigns the perspective of a document as Palestinian or Israeli, is 0.5, because there are equivalent numbers of documents from the two perspectives. null</Paragraph> </Section> <Section position="2" start_page="112" end_page="113" type="sub_section"> <SectionTitle> Document Level </SectionTitle> <Paragraph position="0"> The last column of Table 2 is error reduction relative to SVM. The results show that the na&quot;ive Bayes models and SVM perform surprisingly well on both the Editors and Guests subsets of the bitterlemons corpus. The na&quot;ive Bayes models perform slightly better than SVM, possibly because generative models (i.e., na&quot;ive Bayes models) achieve optimal performance with a smaller number of training examples than discriminative models (i.e., SVM) (Ng and Jordan, 2002), and the size of the bitterlemonscorpus is indeed small. NB-B, which performs full Bayesian inference, improves on NB-M, which only performs point estimation.</Paragraph> <Paragraph position="1"> The results suggest that the choice of words made by the authors, either consciously or subconsciously, reflects much of their political perspectives. Statistical models can capture word usage well and can identify the perspective of documents with high accuracy. null Given the performance gap between Editors and Guests, one may argue that there exist distinct editing artifacts or writing styles of the editors and guests, and that the statistical models are capturing these things rather than &quot;perspectives.&quot; To test if the statistical models truly are learning perspectives, we conduct experiments in which the training and testing data are mismatched, i.e., from different subsets of the corpus. If what the SVM and na&quot;ive Bayes models learn are writing styles or editing artifacts, the classification performance under the mismatched conditions will be considerably degraded.</Paragraph> <Paragraph position="2"> The results on the mismatched training and testing experiments are shown in Table 3. Both SVM and the two variants of na&quot;ive Bayes perform well on the different combinations of training and testing data. As in Table 2, the na&quot;ive Bayes models perform better than SVM with larger error reductions, and NB-B slightly outperforms NB-M. The high accuracy on the mismatched experiments suggests that statistical models are not learning writing styles or editing artifacts. This reaffirms that document perspective is reflected in the words that are chosen by the writers.</Paragraph> <Paragraph position="3"> We list the most frequent words (excluding stopwords) learned by the the NB-M model in Table 4. The frequent words overlap greatly between the Palestinian and Israeli perspectives, including &quot;state,&quot; &quot;peace,&quot; &quot;process,&quot; &quot;secure&quot; (&quot;security&quot;), and &quot;govern&quot; (&quot;government&quot;). This is in contrast to what we expect from topical text classification (e.g., &quot;Sports&quot; vs. &quot;Politics&quot;), in which frequent words seldom overlap. Authors from different perspectives often choose words from a similar vocabulary but emphasize them differently. For example, in documents that are written from the Palestinian perspective, the word &quot;palestinian&quot; is mentioned more frequently than the word &quot;israel.&quot; It is, however, the reverse for documents that are written from the Israeli perspective. Perspectives are also expressed in how frequently certain people (&quot;sharon&quot; v.s. &quot;arafat&quot;), countries (&quot;international&quot; v.s. &quot;america&quot;), and actions (&quot;occupation&quot; v.s. &quot;settle&quot;) are mentioned. While one might solicit these contrasting word pairs from domain experts, our results show that statistical models such as SVM and na&quot;ive Bayes can automatically acquire them.</Paragraph> </Section> <Section position="3" start_page="113" end_page="114" type="sub_section"> <SectionTitle> 5.2 Identifying Perspectives at the Sentence Level </SectionTitle> <Paragraph position="0"> In addition to identifying the perspective of a document, we are interested in knowing which sentences of the document strongly conveys perspective information. Sentence-level perspective annotations do not exist in the bitterlemons corpus, which makes estimating parameters for the proposed Latent Sentence Perspective Model (LSPM) difficult.</Paragraph> <Paragraph position="1"> The posterior probability that a sentence strongly covey a perspective (Example (6)) is of the most interest, but we can not directly evaluate this model without gold standard annotations. As an alternative, we evaluate how accurately LSPM predicts the perspective of a document, again using 10-fold cross validation. Although LSPM predicts the perspective of both documents and sentences, we will doubt the quality of the sentence-level predictions if the document-level predictions are incorrect.</Paragraph> <Paragraph position="2"> The experimental results are shown in Table 5.</Paragraph> <Paragraph position="3"> We include the results for the na&quot;ive Bayes models from Table 3 for easy comparison. The accuracy of LSPM is comparable or even slightly better than that of the na&quot;ive Bayes models. This is very encouraging and suggests that the proposed LSPM closely captures how perspectives are reflected at both the document and sentence levels. Examples 1 and 2 from the introduction were predicted by LSPM as likely to Palestinian palestinian, israel, state, politics, peace, international, people, settle, occupation, sharon, right, govern, two, secure, end, conflict, process, side, negotiate Israeli israel, palestinian, state, settle, sharon, peace, arafat, arab, politics, two, process, secure, conflict, lead, america, agree, right, gaza, govern</Paragraph> </Section> <Section position="4" start_page="114" end_page="114" type="sub_section"> <SectionTitle> Document and Sentence Levels </SectionTitle> <Paragraph position="0"> contain strong perspectives, i.e., large Pr(~S = s1).</Paragraph> <Paragraph position="1"> Examples 3 and 4 from the introduction were predicted by LSPM as likely to contain little or no perspective information, i.e., high Pr(~S = s0).</Paragraph> <Paragraph position="2"> The comparable performance between the na&quot;ive Bayes models and LSPM is in fact surprising. We can train a na&quot;ive Bayes model directly on the sentences and attempt to classify a sentence as reflecting either a Palestinian or Israeli perspective. A sentence is correctly classified if the predicted perspective for the sentence is the same as the perspective of the document from which it was extracted. Using this model, we obtain a classification accuracy of only 0.7529, which is much lower than the accuracy previously achieved at the document level. Identifying perspectives at the sentence level is thus more difficult than identifying perspectives at the document level. The high accuracy at the document level shows that LSPM is very effective in pooling evidence from sentences that individually contain little perspective information.</Paragraph> </Section> </Section> class="xml-element"></Paper>