<?xml version="1.0" standalone="yes"?>
<Paper uid="W99-0502">
  <Title>A Case Study on Inter-Annotator Agreement for Word Sense Disambiguation Hwee Tou Ng</Title>
  <Section position="7" start_page="10" end_page="10" type="evalu">
    <SectionTitle>
6 Results
</SectionTitle>
    <Paragraph position="0"> For each word w from the list of 121 nouns and 70 verbs, ~e applied the greedy search algorithm to each set of sentences in the intersected corpus contaming w For a subset of 95 words (53 nouns and 42 verbs), the algorithm was able to derive a coarser set of 2 or more senses for each of these 95 words such that the resulting Kappa ~alue reaches 0 8 or higher For the other 96 words, m order for the Kappa value to reach 0 8 or higher, the algorithm collapses all senses of the ~ord to a single (trivial) class Table 2 and 3 summarizes the results for the set of 53 nouns and 42 ~erbs, respectively Table 2 md~cates that before the collapse of sense classes, these 53 nouns have an average of 7 6 senses per noun There is a total of 5,339 sentences in the intersected corpus containing these nouns, of which 3,387 sentences were assigned the same sense by the two groups of human annotators The average Kappa statistic (computed as a simple average of the Kappa statistic of ~he mdlwdual nouns) is 0 463 After the collapse of sense classes by the greedy search algorithm, the average number of senses per noun for these 53 nouns drops to 40 Howe~er, the number of sentences which have been asmgned the same coarse sense by the annotators increases to 5,033 That is, about 94 3% of the sentences have been assigned the same coarse sense, and that the average Kappa statistic has improved to 0 862, mgmfymg high rater-annotator agreement on the derived coarse senses Table3 gl~es the analogous figures for the 42 verbs, agmn mdmatmg that high agreement is achieved on the coarse sense classes den~ed for verbs</Paragraph>
  </Section>
class="xml-element"></Paper>