<?xml version="1.0" standalone="yes"?>
<Paper uid="W03-0411">
  <Title>Preposition Semantic Classification via PENN TREEBANK and FRAMENET</Title>
  <Section position="4" start_page="1" end_page="1" type="metho">
    <SectionTitle>
2 Background
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="1" end_page="1" type="sub_section">
      <SectionTitle>
2.1 Semantic roles in the PENN TREEBANK
</SectionTitle>
      <Paragraph position="0"> The second version of the Penn Treebank (Marcus et al., 1994) added additional clause usage information to the parse tree annotations that are popular for natural language learning. This includes a few case-style relation annotations, which prove useful for disambiguating prepositions. For example, here is a simple parse tree with the new annotation format: null</Paragraph>
      <Paragraph position="2"> This shows that the prepositional phrase (PP) is providing the location for the state described by the verb phrase. Treating this as the preposition sense would yield the following annotation: This every man contains within</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="1" end_page="1" type="metho">
    <SectionTitle>
LOC
</SectionTitle>
    <Paragraph position="0"> him The main semantic relations in TREEBANK are beneficiary, direction, spatial extent, manner, location, purpose/reason, and temporal. These tags can be applied to any verb complement but normally occur with clauses, adverbs, and prepositions. Frequency counts for the prepositional phrase (PP) case role annotations are shown in Table 1.</Paragraph>
    <Paragraph position="1"> The frequencies for the most frequent prepositions that have occurred in the prepositional phrase annotations are shown later in Table 7. The table is ordered by entropy, which measures the inherent ambiguity in the classes as given by the annotations. Note that the Baseline column is the probability of the most frequent sense, which is a common estimate of the lower bound for classification experiments. null</Paragraph>
    <Section position="1" start_page="1" end_page="1" type="sub_section">
      <SectionTitle>
2.2 Semantic roles in FRAMENET
</SectionTitle>
      <Paragraph position="0"> Berkeley's FRAMENET (Fillmore et al., 2001) project provides the most recent large-scale annotation of semantic roles. These are at a much finer granularity than those in TREEBANK, so they should prove quite useful for applications that learn detailed semantics from corpora. Table 2 shows the top semantic roles by frequency of annotation. This illustrates that the semantic roles in Framenet can be quite specific, as in the roles cognizer, judge, and addressee. In all, there are over 140 roles annotated with over 117,000 tagged instances.</Paragraph>
      <Paragraph position="1"> FRAMENET annotations occur at the phrase level instead of the grammatical constituent level as in TREEBANK. The cases that involve prepositional phrases can be determined by the phrase-type attribute of the annotation. For example, consider the following annotation.</Paragraph>
      <Paragraph position="2">  The constituent (C) tags identify the phrases that have been annotated. The target attribute indicates the predicating word for the overall frame. The frame element (FE) attribute indicates one of the semantic roles for the frame, and the phrase type (PT) attribute indicates the grammatical function of the phrase. We isolate the prepositional phrase annotation and treat it as the sense of the preposition. This yields the following annotation: It had a sharp, pointed face and a feathery tail that arched over Path its back.</Paragraph>
      <Paragraph position="3"> The annotation frequencies for the most frequent prepositions are shown later in Table 8, again ordered by entropy. This illustrates that the role distributions are more complicated, yielding higher entropy values on average. In all, there are over 100 prepositions with annotations, 65 with ten or more instances each.</Paragraph>
    </Section>
    <Section position="2" start_page="1" end_page="1" type="sub_section">
      <SectionTitle>
Tag Freq Description
</SectionTitle>
      <Paragraph position="0"> pp-loc 17220 locative pp-tmp 10572 temporal pp-dir 5453 direction pp-mnr 1811 manner pp-prp 1096 purpose/reason pp-ext 280 spatial extent pp-bnf 44 beneficiary  is the label for the role in the annotations. Freq is frequency of the role occurrences.</Paragraph>
    </Section>
  </Section>
  <Section position="6" start_page="1" end_page="2" type="metho">
    <SectionTitle>
3 Classification experiments
</SectionTitle>
    <Paragraph position="0"> The task of selecting the semantic roles for the prepositions can be framed as an instance of word-sense disambiguation (WSD), where the semantic roles serve as the senses for the prepositions.</Paragraph>
    <Paragraph position="1"> A straightforward approach for preposition disambiguation would be to use standard WSD features, such as the parts-of-speech of surrounding words and, more importantly, collocations (e.g., lexical associations). Although this can be highly accurate, it will likely overfit the data and generalize poorly. To overcome these problems, a class-based approach is used for the collocations, with WordNet high-level synsets as the source of the word classes.</Paragraph>
    <Paragraph position="2"> Therefore, in addition to using collocations in the form of other words, this uses collocations in the form of semantic categories.</Paragraph>
    <Paragraph position="3"> A supervised approach for word-sense disambiguation is used following Bruce and Wiebe (1999). The results described here were obtained using the settings in Figure 1. These are similar to the settings used by O'Hara et al. (2000) in the first SENSEVAL competition, with the exception of the hypernym collocations. This shows that for the hypernym associations, only those words that occur within 5 words of the target prepositions are considered. null  The main difference from that of a standard WSD approach is that, during the determination of the class-based collocations, each word token is replaced by synset tokens for its hypernyms in Word-Net, several of which might occur more than once. This introduces noise due to ambiguity, but given the conditional-independence selection scheme, the preference for hypernym synsets that occur for different words will compensate somewhat. O'Hara and Wiebe (2003) provide more details on the extraction of these hypernym collocations. The feature settings in Figure 1 are used in two different configurations: word-based collocations alone, and a combination of word-based and hypernym-based collocations. The combination generally produces  This window size was chosen after estimating that on average the prepositional objects occur within 2.35+/[?] 1.26 words of the preposition and that the average attachment site is within 3.0 +/[?] 2.98 words. These figures were produced by analyzing the parse trees for the semantic role annotations in the  tion classification experiments. CI refers to conditional independence; the per-class-binary organization uses a separate binary feature per role (Wiebe et al., 1998).</Paragraph>
    <Paragraph position="4"> the best results. This exploits the specific clues provided by the word collocations while generalizing to unseen cases via the hypernym collocations.</Paragraph>
    <Section position="1" start_page="2" end_page="2" type="sub_section">
      <SectionTitle>
3.1 PENN TREEBANK
</SectionTitle>
      <Paragraph position="0"> To see how these conceptual associations are derived, consider the differences in the prior versus class-based conditional probabilities for the semantic roles of the preposition 'at' in TREEBANK.Table 3 shows the global probabilities for the roles assigned to 'at'. Table 4 shows the conditional prob-</Paragraph>
    </Section>
    <Section position="2" start_page="2" end_page="2" type="sub_section">
      <SectionTitle>
Relation P(R) Example
</SectionTitle>
      <Paragraph position="0"> locative .732 workers at a factory temporal .239 expired at midnight Tuesday manner .020 has grown at a sluggish pace direction .006 CDs aimed at individual investors  tic relations for 'at' in TREEBANK. Category is WordNet synset defining the category. P(R|C) is probability of the relation given that the synset category occurs in the context.</Paragraph>
    </Section>
    <Section position="3" start_page="2" end_page="2" type="sub_section">
      <SectionTitle>
Relation P(R) Example
</SectionTitle>
      <Paragraph position="0"> addressee .315 growled at the attendant other .092 chuckled heartily at this admission phenomenon .086 gazed at him with disgust goal .079 stationed a policeman at the gate content .051 angry at her stubbornness  tic relations for 'at' in FRAMENET abilities for these roles given that certain high-level WordNet categories occur in the context. These category probability estimates were derived by tabulating the occurrences of the hypernym synsets for the words occurring within a 5-word window of the target preposition. In a context with a concrete concept (ENTITY#1), the difference in the probability distributions shows that the locative interpretation becomes even more likely. In contrast, in a context with an abstract concept (ABSTRACTION#6), the difference in the probability distributions shows that the temporal interpretation becomes more likely.</Paragraph>
      <Paragraph position="1"> Therefore, these class-based lexical associations reflect the intuitive use of the prepositions.</Paragraph>
      <Paragraph position="2"> The classification results for these prepositions in the PENN TREEBANK show that this approach is very effective. Table 9 shows the results when all of the prepositions are classified together. Unlike the general case for WSD, the sense inventory is the same for all the words here; therefore, a single classifier can be produced rather than individual classifiers. This has the advantage of allowing more training data to be used in the derivation of the clues indicative of each semantic role. Good accuracy is achieved when just using standard word collocations. Table 9 also shows that significant improvements are achieved using a combination of both types of collocations. For the combined case, the accuracy is 86.1%, using Weka's J48 classifier (Witten and Frank, 1999), which is an implementation of Quinlan's (1993) C4.5 decision tree learner. For comparison, Table 7 shows the results for individual classifiers created for each preposition (using Naive Bayes). In this case, the word-only collocations perform slightly better: 78.5% versus 77.8% accuracy.</Paragraph>
    </Section>
    <Section position="4" start_page="2" end_page="2" type="sub_section">
      <SectionTitle>
3.2 FRAMENET
</SectionTitle>
      <Paragraph position="0"> It is illustrative to compare the prior probabilities (i.e., P(R)) for FRAMENET to those seen earlier for 'at' in TREEBANK. See Table 5 for the most frequent roles out of the 40 cases that were assigned to it. This highlights a difference between the two sets of annotations. The common temporal role from TREEBANK is not directly represented in FRAMENET, and it is not subsumed by another specific role. Similarly, there is no direct role corresponding to locative, but it is partly subsumed by  tion with TREEBANK semantic roles. Instances is the number of role annotations. Classes is the number of distinct roles. Entropy measures non-uniformity of the role distributions. Baseline selects the most-frequent role. The Word Only experiment just uses word collocations, whereas Combined uses both word and hypernym collocations. Accuracy is average for percent correct over ten trials in cross validation. STDEV is the standard deviation over the trails. The difference in the two experiments is statistically significant at p&lt;0.01.</Paragraph>
      <Paragraph position="1">  biguation with FRAMENET semantic roles. See Table 9 for the legend.</Paragraph>
      <Paragraph position="2">  frequency for the prepositions. Entropy measures non-uniformity of the role distributions. The Baseline experiment selects the most-frequent role. The Word Only experiment just uses word collocations, whereas Combined uses both word and hypernym collocations. Both columns show averages for percent correct over ten trials. Total averages the values of the individual experiments (except for Freq).  the legend.</Paragraph>
      <Paragraph position="3"> goal. This reflects the bias of FRAMENET towards roles that are an integral part of the frame under consideration: location and time apply to all frames, so these cases are not generally annotated.</Paragraph>
      <Paragraph position="4"> Table 9 shows the results of classification when all of the prepositions are classified together. The overall results are not that high due to the very large number of roles. However, the combined collocation approach still shows slight improvement (49.4% versus 49.0%). Table 8 shows the results when using individual classifiers. This shows that the combined collocations produce better results: 70.3% versus 68.5%. Unlike the case with Treebank, the performance is below that of the individual classifiers. This is due to the fine-grained nature of the role inventory. When all the roles are considered together, prepositions are prone to being misclassified with roles that they might not have occurred with in the training data, such as whenever other contextual clues are strong for that role. This is not a problem with Treebank given its small role inventory.</Paragraph>
    </Section>
  </Section>
  <Section position="7" start_page="2" end_page="3" type="metho">
    <SectionTitle>
4 Related work
</SectionTitle>
    <Paragraph position="0"> Until recently, there has not been much work specifically on preposition classification, especially with respect to general applicability in contrast to special purpose usages. Halliday (1956) did some early work on this in the context of machine translation.</Paragraph>
    <Paragraph position="1"> Later work in that area addressed the classification indirectly during translation. In some cases, the issue is avoided by translating the preposition into a corresponding foreign function word without regard to the preposition's underlying meaning (i.e., direct transfer). Other times an internal representation is helpful (Trujillo, 1992). Taylor (1993) discusses general strategies for preposition disambiguation using a cognitive linguistics framework and illustrates them for 'over'. There has been quite a bit of work in this area but mainly for spatial prepositions (Japkowicz and Wiebe, 1991; Zelinsky-Wibbelt, 1993).</Paragraph>
    <Paragraph position="2"> There is currently more interest in this type of classification. Litkowski (2002) presents manually-derived rules for disambiguating prepositions, in particular for 'of'. Srihari et al. (2001) present manually-derived rules for disambiguating prepositions used in named entities.</Paragraph>
    <Paragraph position="3"> Gildea and Jurafsky (2002) classify semantic role assignments using all the annotations in FRAMENET, for example, covering all types of verbal arguments. They use several features derived from the output of a parser, such as the constituent type of the phrase (e.g., NP) and the grammatical function (e.g., subject). They include lexical features for the headword of the phrase and the predicating word for the entire annotated frame. They report an accuracy of 76.9% with a baseline of 40.6% over the FRAMENET semantic roles. However, due to the conditioning of the classification on the predicating word for the frame, the range of roles for a particular classification is more limited than in our case.</Paragraph>
    <Paragraph position="4"> Blaheta and Charniak (2000) classify semantic role assignments using all the annotations in TREE-BANK. They use a few parser-derived features, such as the constituent labels for nearby nodes and part-of-speech for parent and grandparent nodes. They also include lexical features for the head and alternative head (since prepositions are considered as the head by their parser). They report an accuracy of 77.6% over the form/function tags from the PENN TREEBANK with a baseline of 37.8%,  Their task is somewhat different, since they address all adjuncts, not just prepositions, hence their lower baseline. In addition, they include the nominal and adverbial roles, which are syntactic and presumably more predictable than the others in this group. Van den Bosch and Bucholz (2002) also use the Tree-bank data to address the more general task of assigning function tags to arbitrary phrases. For features, they use parts of speech, words, and morphological clues. Chunking is done along with the tagging, but they only present results for the evaluation of both tasks taken together; their best approach achieves 78.9% accuracy.</Paragraph>
  </Section>
class="xml-element"></Paper>