<?xml version="1.0" standalone="yes"?>
<Paper uid="P99-1032">
  <Title>Development and Use of a Gold-Standard Data Set for Subjectivity Classifications</Title>
  <Section position="9" start_page="251" end_page="251" type="concl">
    <SectionTitle>
6 Conclusion
</SectionTitle>
    <Paragraph position="0"> This paper demonstrates a procedure for automatically formulating a single best tag when there are multiple judges who disagree. The procedure is applicable to any tagging task in which the judges exhibit symmetric disagreement resulting from bias. We successfully use bias-corrected tags for two purposes: to guide a revision of the coding manual, and to develop an automatic classifier. The revision of the coding manual results in as much as a 16 point improvement in pairwise Kappa values, and raises the average agreement among the judges to a Kappa value of over 0.87 for the sentences that can be tagged with certainty.</Paragraph>
    <Paragraph position="1"> Using only simple features, the classifier achieves an average accuracy 21 percentage points higher than the baseline, in 10-fold cross validation experiments. In addition, the average accuracy of the classifier is 81.5% on the sentences the judges tagged with certainty. The strong performance of the classifier and its consistency with the judges demonstrate the value of this approach to developing gold-standard tags.</Paragraph>
  </Section>
class="xml-element"></Paper>