<?xml version="1.0" standalone="yes"?>
<Paper uid="J02-3001">
  <Title>Automatic Labeling of Semantic Roles</Title>
  <Section position="4" start_page="248" end_page="251" type="intro">
    <SectionTitle>
2 The FrameNet annotation includes both the relative pronoun and its antecedent in the target word's
</SectionTitle>
    <Paragraph position="0"> clause.</Paragraph>
    <Paragraph position="1">  Computational Linguistics Volume 28, Number 3 the final database. Thus, the focus of the project was on completeness of examples for lexicographic needs, rather than on statistically representative data. Sentences from each subcorpus were then annotated by hand, marking boundaries of each frame element expressed in the sentence and assigning tags for the annotated constituent's frame semantic role, syntactic category (e.g., noun phrase or prepositional phrase), and grammatical function in relation to the target word (e.g., object or complement of a verb). In the final phase of the process, the annotated sentences for each target word were checked for consistency. In addition to the tags just mentioned, the annotations include certain other information, which we do not make use of in this work, such as word sense tags for some target words and tags indicating metaphoric usages.</Paragraph>
    <Paragraph position="2"> Tests of interannotator agreement were performed for data from a small number of predicates before the final consistency check. Interannotator agreement at the sentence level, including all frame element judgments and boundaries for one predicate, varied from .66 to .82 depending on the predicate. The kappa statistic (Siegel and Castellan 1988) varied from .67 to .82. Because of the large number of possible categories when boundary judgments are considered, kappa is nearly identical to the interannotator agreement. The system described in this article (which gets .65/.61 precision/recall on individual frame elements; see Table 15) correctly identifies all frame elements in 38% of test sentences. Although this .38 is not directly comparable to the .66-.82 interannotator agreements, it's clear that the performance of our system still falls significantly short of human performance on the task.</Paragraph>
    <Paragraph position="3"> The British National Corpus was chosen as the basis of the FrameNet project despite differences between British and American usage because, at 100 million words, it provides the largest corpus of English with a balanced mixture of text genres. The British National Corpus includes automatically assigned syntactic part-of-speech tags for each word but does not include full syntactic parses. The FrameNet annotators did not make use of, or produce, a complete syntactic parse of the annotated sentences, although some syntactic information is provided by the grammatical function and phrase type tags of the annotated frame elements.</Paragraph>
    <Paragraph position="4"> The preliminary version of the FrameNet corpus used for our experiments contained 67 frame types from 12 general semantic domains chosen for annotation. A complete list of the semantic domains represented in our data is shown in Table 1, along with representative frames and predicates. Within these frames, examples of a total of 1,462 distinct lexical predicates, or target words, were annotated: 927 verbs, 339 nouns, and 175 adjectives. There are a total of 49,013 annotated sentences and 99,232 annotated frame elements (which do not include the target words themselves). null How important is the particular set of semantic roles that underlies our system? For example, could the optimal choice of semantic roles be very dependent on the application that needs to exploit their information? Although there may well be application-specific constraints on semantic roles, our semantic role classifiers seem in practice to be relatively independent of the exact set of semantic roles under consideration. Section 9.1 describes an experiment in which we collapsed the FrameNet roles into a set of 18 abstract thematic roles. We then retrained our classifier and achieved roughly comparable results; overall performance was 82.1% for abstract thematic roles, compared to 80.4% for frame-specific roles. Although this doesn't show that the detailed set of semantic roles is irrelevant, it does suggest that our statistical classification algorithm, at least, is relatively robust to even quite large changes in role identities.</Paragraph>
  </Section>
class="xml-element"></Paper>