<?xml version="1.0" standalone="yes"?>
<Paper uid="P05-1006">
  <Title>The Role of Semantic Roles in Disambiguating Verb Senses</Title>
  <Section position="4" start_page="42" end_page="44" type="metho">
    <SectionTitle>
2 Basic automatic system
</SectionTitle>
    <Paragraph position="0"> Our WSD system was built to combine information from many different sources, using as much linguistic knowledge as could be gathered automatically by NLP tools. In particular, our goal was to see the extent to which sense-tagging of verbs could be improved by adding features that capture information about predicate-arguments and selectional restrictions. null We used the Mallet toolkit (McCallum, 2002) for learning maximum entropy models with Gaussian priors for all our experiments. In order to extract the linguistic features necessary for the models, all sentences containing the target word were automatically part-of-speech-tagged using a maximum entropy tagger (Ratnaparkhi, 1998) and parsed using the Collins parser (Collins, 1997). In addition, an automatic named entity tagger (Bikel et al., 1997) was run on the sentences to map proper nouns to a small set of semantic classes.1</Paragraph>
    <Section position="1" start_page="42" end_page="42" type="sub_section">
      <SectionTitle>
2.1 Topical features
</SectionTitle>
      <Paragraph position="0"> We categorized the possible model features into topical features and several types of local contextual features. Topical features for a verb in a sentence look for the presence of keywords occurring anywhere in the sentence and any surrounding sentences provided as context (usually one or two sentences).</Paragraph>
      <Paragraph position="1"> These features are supposed to show the domain in which the verb is being used, since some verb senses are used in only certain domains. The set of key-words is specific to each verb lemma to be disambiguated and is determined automatically from training data so as to minimize the entropy of the probability of the senses conditioned on the keyword. All alphabetic characters are converted to lower case.</Paragraph>
      <Paragraph position="2"> Words occuring less than twice in the training data or that are in a stoplist2 of pronouns, prepositions, and conjunctions are ignored.</Paragraph>
    </Section>
    <Section position="2" start_page="42" end_page="43" type="sub_section">
      <SectionTitle>
WordNet/words.txt
2.2 Local features
</SectionTitle>
      <Paragraph position="0"> The local features for a verb a0 in a particular sentence tend to look only within the smallest clause containing a0 . They include collocational features requiring no linguistic preprocessing beyond part-of-speech tagging, syntactic features that capture relations between the verb and its complements, and semantic features that incorporate information about noun classes for subjects and objects: Collocational features: Collocational features refer to ordered sequences of part-of-speech tags or word tokens immediately surrounding a0 . They include: null a1 unigrams: words a0a3a2a5a4 , a0a3a2a7a6 , a0a9a8 , a0a3a10a7a6 , a0a3a10a5a4 and parts of speech a11</Paragraph>
      <Paragraph position="2"> Syntactic features: The system uses heuristics to extract syntactic elements from the parse for the sentence containing a0 . Let commander VP be the lowest VP that dominates a0 and that is not immediately dominated by another VP, and let head VP be the lowest VP dominating a0 (See Figure 1). Then we define the subject of a0 to be the leftmost NP sibling of commander VP, and a complement of a0 to be a node that is a child of the head VP, excluding NPs whose head is a number or a noun from a list of common temporal nouns (&amp;quot;week&amp;quot;, &amp;quot;tomorrow&amp;quot;,  &amp;quot;Monday&amp;quot;, etc.). The system extracts the following binary syntactic features: a1 Is the sentence passive? a1 Is there a subject, direct object (leftmost NP complement of a0 ), indirect object (second left-most NP complement ofa0 ), or clausal complement (S complement of a0 )? a1 What is the word (if any) that is the particle or head of the subject, direct object, or indirect object?  a1 If there is a PP complement, what is the preposition, and what is the object of the preposition? Semantic features: a1 What is the Named Entity tag (PERSON, OR-GANIZATION, LOCATION, UNKNOWN) for each proper noun in the syntactic positions above? a1 What are the possible WordNet synsets and hypernyms for each noun in the syntactic positions above? (Nouns are not explicitly disambiguated; all possible synsets and hypernyms for the noun are included.) This set of local features relies on access to syntactic structure as well as semantic class information, and attempts to model richer linguistic information about predicate arguments. However, the heuristics for extracting the syntactic features are able to identify subjects and objects of only simple clauses. The heuristics also do not differentiate between arguments and adjuncts; for example, the feature sent-comp is intended to identify clausal complements such as in (S (NP Mary) (VP (VB called) (S him a bastard))), but Figure 1 shows how a purpose clause can be mistakenly labeled as a clausal complement.</Paragraph>
    </Section>
    <Section position="3" start_page="43" end_page="44" type="sub_section">
      <SectionTitle>
2.3 Evaluation
</SectionTitle>
      <Paragraph position="0"> We tested the system on the 1806 test instances of the 29 verbs from the English lexical sample task for Senseval-2 (Palmer et al., 2001). Accuracy was defined to be the fraction of the instances for which the system got the correct sense. All significance testing between different accuracies was done using a one-tailed z-test, assuming a binomial distribution of the successes; differences in accuracy were considered to be significant if a11a21a20a23a22a25a24a27a26a28a24a27a29a27a24 . In Senseval-2, senses involving multi-word constructions could be identified directly from the sense tags themselves, and the head word and satellites of multi-word constructions were explicitly marked in the training and test data. We trained one model for each of the verbs and used a filter to consider only phrasal senses whenever there were satellites of multi-word constructions marked in the test data.</Paragraph>
      <Paragraph position="1">  using topical features and different subsets of local features.</Paragraph>
      <Paragraph position="2"> Table 1 shows the accuracy of the system using topical features and different subsets of local fea- null tures. Adding features from richer linguistic sources always improves accuracy. Adding lexical syntactic (&amp;quot;syn&amp;quot;) features improves accuracy significantly over using just collocational (&amp;quot;co&amp;quot;) features (a11a30a22 a24a27a26a28a24a27a29a27a24 ). When semantic class (&amp;quot;sem&amp;quot;) features are added, the improvement is also significant.</Paragraph>
      <Paragraph position="3"> Adding topical information to all the local features improves accuracy, but not significantly; when the topical features are removed the accuracy of our system falls only slightly, to 62.0%. Senses based on domain or topic occur rarely in the Senseval-2 corpus. Most of the information provided by topical features already seem to be captured by the local features for the frequent senses.</Paragraph>
      <Paragraph position="4">  Semantic class information plays a significant role in sense distinctions. Table 2 shows the relative contribution of adding only named entity tags to the collocational and syntactic features (&amp;quot;co+syn+ne&amp;quot;), versus adding only the WordNet classes (&amp;quot;co+syn+wn&amp;quot;), versus adding both named entity and WordNet classes (&amp;quot;co+syn+ne+wn&amp;quot;). Adding all possible WordNet noun class features for arguments contributes a large number of parameters to the model, but this use of WordNet with no separate disambiguation of noun arguments proves to be very useful. In fact, the use of WordNet for common nouns proves to be even more beneficial than the use of a named entity tagger for proper nouns.</Paragraph>
      <Paragraph position="5"> Given enough data, the maximum entropy model is able to assign high weights to the correct hypernyms of the correct noun sense if they represent defining selectional restrictions.</Paragraph>
      <Paragraph position="6"> Incorporating topical keywords as well as collocational, syntactic, and semantic local features, our system achieves 62.5% accuracy. This is in comparison to the 61.1% accuracy achieved by (Lee and Ng, 2002), which has been the best published result on this corpus.</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="44" end_page="47" type="metho">
    <SectionTitle>
3 PropBank semantic annotations
</SectionTitle>
    <Paragraph position="0"> Our WSD system uses heuristics to attempt to detect predicate arguments from parsed sentences. However, recognition of predicate argument structures is not straightforward, because a natural language will have several different syntactic realizations of the same predicate argument relations.</Paragraph>
    <Paragraph position="1"> PropBank is a corpus in which verbs are annotated with semantic tags, including coarse-grained sense distinctions and predicate-argument structures. PropBank adds a layer of semantic annotation to the Penn Wall Street Journal Treebank II. An important goal is to provide consistent predicate-argument structures across different syntactic realizations of the same verb. Polysemous verbs are also annotated with different framesets. Frameset tags are based on differences in subcategorization frames and correspond to a coarse notion of word senses.</Paragraph>
    <Paragraph position="2"> A verb's semantic arguments in PropBank are numbered beginning with 0. Arg0 is roughly equivalent to the thematic role of Agent, and Arg1 usually corresponds to Theme or Patient; however, argument labels are not necessarily consistent across different senses of the same verb, or across different verbs, as thematic roles are usually taken to be. In addition to the core, numbered arguments, verbs can take any of a set of general, adjunct-like arguments (ARGM), whose labels are derived from the Treebank functional tags (DIRection, LOCation, etc.).</Paragraph>
    <Paragraph position="3"> PropBank provides manual annotation of predicate-argument information for a large number of verb instances in the Senseval-2 data set. The intersection of PropBank and Senseval-2 forms a corpus containing gold-standard annotations of fine-grained WordNet senses, coarse-grained PropBank framesets, and PropBank role labels.</Paragraph>
    <Paragraph position="4"> The combination of such gold-standard semantic annotations provides a unique opportunity to investigate the role of predicate-argument features in word sense disambiguation, for both coarse-grained framesets and fine-grained WordNet senses.</Paragraph>
    <Section position="1" start_page="44" end_page="45" type="sub_section">
      <SectionTitle>
3.1 PropBank features
</SectionTitle>
      <Paragraph position="0"> We conducted experiments on the effect of using features from PropBank for sense-tagging verbs.</Paragraph>
      <Paragraph position="1"> Both PropBank role labels and PropBank framesets were used. In the case of role labels, only the  gold-standard labels found in PropBank were used, because the best automatic semantic role labelers only perform at about 84% precision and 75% recall (Pradhan et al., 2004).</Paragraph>
      <Paragraph position="2"> From the PropBank annotation for each sentence, we extracted the following features:  1. Labels of the semantic roles: rel, ARG0, ARG1, ARG2-WITH, ARG2, ..., ARGM-LOC, ARGM-TMP, ARGM-NEG, ...</Paragraph>
      <Paragraph position="3"> 2. Syntactic labels of the constituent instantiating each semantic role: ARG0=NP, ARGM-TMP=PP, ARG2-WITH=PP, ...</Paragraph>
      <Paragraph position="4"> 3. Head word of each constituent in (2): rel=called, sats=up, ARG0=company, ARGM-TMP=day, ...</Paragraph>
      <Paragraph position="5"> 4. Semantic classes (named entity tag, WordNet hypernyms) of the nouns in (3): ARGOsyn=ORGANIZATION, AR null GOsyn=16185, ARGM-TMPsyn=13018, ...</Paragraph>
      <Paragraph position="6"> When a numbered role appears in a prepositional phrase (e.g., ARG2-WITH), we take the &amp;quot;head word&amp;quot; to be the object of the preposition. If a constituent instantiating some semantic role is a trace, we take the head of its referent instead.</Paragraph>
      <Paragraph position="8"> an agreement by next September at the latest] .</Paragraph>
      <Paragraph position="9"> For example, the PropBank features that we extract for the sentence above are:</Paragraph>
      <Paragraph position="11"/>
    </Section>
    <Section position="2" start_page="45" end_page="46" type="sub_section">
      <SectionTitle>
3.2 Role labels for frameset tagging
</SectionTitle>
      <Paragraph position="0"> We collected all instances of the Senseval-2 verbs from the PropBank corpus. Only 20 of these verbs had more than one frameset in the PropBank corpus, resulting in 4887 instances of polysemous verbs.</Paragraph>
      <Paragraph position="1"> The instances for each word were partitioned randomly into 10 equal parts, and the system was tested on each part after being trained on the remaining nine. For these 20 verbs with more than one PropBank frameset tag, choosing the most frequent frameset gives a baseline accuracy of 76.0%.</Paragraph>
      <Paragraph position="2"> The sentences were automatically pos-tagged with the Ratnaparki tagger and parsed with the Collins parser. We extracted local contextual features as for WordNet sense-tagging and used the local features to train our WSD system on the coarse-grained sense-tagging task of automatically assigning PropBank frameset tags. We tested the effect of using only collocational features (&amp;quot;co&amp;quot;) for frameset tagging, as well as using only PropBank role features (&amp;quot;pb&amp;quot;) or only our original syntactic/semantic features (&amp;quot;synsem&amp;quot;) for this task, and found that the combination of collocational features with Prop-Bank features worked best. The system has the worst performance on the word strike, which has a high number of framesets and a low number of training instances. Table 3 shows the performance of the system on different subsets of local features.</Paragraph>
      <Paragraph position="3">  task for verbs with more than one frameset, using different types of local features (no topical features); all features except pb were extracted from automatically pos-tagged and parsed sentences.</Paragraph>
      <Paragraph position="4"> We obtained an overall accuracy of 88.3% using our original local contextual features. However, the system's performance improved significantly when we used only PropBank role features, achieving an accuracy of 90.1%. Furthermore, adding collocational features and heuristically extracted syntactic/semantic features to the PropBank features do not provide additional information and affects the accuracy of frameset-tagging only negligibly. It is not surprising that for the coarse-grained sense-tagging task of assigning the correct PropBank frameset tag to a verb, using the PropBank role labels is better than syntactic/semantic features heuristically extracted from parses because these heuristics are meant to capture the predicate-argument informa- null tion that is encoded more directly in the PropBank role labels.</Paragraph>
      <Paragraph position="5"> Even when the original local features were extracted from the gold-standard pos-tagged and parsed sentences of the Penn Treebank, the system performed significantly worse than when PropBank role features were used. This suggests that more effort should be applied to improving the heuristics for extracting syntactic features.</Paragraph>
      <Paragraph position="6"> We also experimented with adding topical features and ARGM features from PropBank. In all cases, these additional features reduced overall accuracy, but the difference was never significant (a11a46a45a23a22a47a24a27a26a28a48a27a24a27a24 ). Topical features do not help because frameset tags are based on differences in subcategorization frames and not on the domain or topic. ARGM features do not help because they are supposedly used uniformly across verbs and framesets.</Paragraph>
    </Section>
    <Section position="3" start_page="46" end_page="47" type="sub_section">
      <SectionTitle>
3.3 Role labels for WordNet sense-tagging
</SectionTitle>
      <Paragraph position="0"> We experimented with using PropBank role labels for fine-grained WordNet sense-tagging. While ARGM features are not useful for coarse-grained frameset-tagging, some sense distinctions in Word-Net are based on adverbial modifiers, such as &amp;quot;live well&amp;quot; or &amp;quot;serves someone well.&amp;quot; Therefore, we included PropBank ARGM features in our models for WordNet sense-tagging to capture a wider range of linguistic behavior. We looked at the 2571 instances  tagging for instances in both Senseval-2 and Prop-Bank, using different types of local features (no topical features).</Paragraph>
      <Paragraph position="1"> Table 4 shows the accuracy of the system on WordNet sense-tagging using different subsets of features; all features except pb were extracted from automatically pos-tagged and parsed sentences. By adding PropBank role features to our original local feature set, accuracy rose from 0.666 to to 0.694 on this subset of the Senseval-2 verbs (a11a49a22a50a24a27a26a28a24a27a51a27a24 ); the extraction of syntactic features from the parsed sentences is again not successfully capturing all the predicate-argument information that is explicit in PropBank.</Paragraph>
      <Paragraph position="2"> The verb &amp;quot;match&amp;quot; illustrates why accuracy improves using additional PropBank features. As shown in Figure 2, the matched objects may occur in different grammatical relations with respect to the verb (subject, direct object, object of a preposition), but they each have an ARG1 semantic role label in PropBank.3 Furthermore, only one of the matched objects needs to be specified, as in Example 3 where the second matched object (presumably the company's prices) is unstated. Our heuristics do not handle these alternations, and cannot detect that the syntactic subject in Example 1 has a different semantic role than the subject of Example 3.</Paragraph>
      <Paragraph position="3">  Our basic WSD system (using local features extracted from automatic parses) confused WordNet Sense 1 with Sense 4: 1. match, fit, correspond, check, jibe, gibe, tally, agree - (be compatible, similar or consistent; coincide in their characteristics; &amp;quot;The two stories don't agree in many details&amp;quot;; &amp;quot;The handwriting checks with the signature on the check&amp;quot;; &amp;quot;The suspect's fingerprints don't match those on the gun&amp;quot;)  labels, one for each of the matching objects. Other verbs that have more than a single ARG1 in PropBank include: &amp;quot;attach, bolt, coincide, connect, differ, fit, link, lock, pin, tack, tie.&amp;quot;  quality or ability; &amp;quot;Nothing can rival cotton for durability&amp;quot;; &amp;quot;Your performance doesn't even touch that of your colleagues&amp;quot;; &amp;quot;Her persistence and ambition only matches that of her parents&amp;quot;) The senses are differentiated in that the matching objects (ARG1) in Sense 4 have some quantifiable characteristic that can be measured on some scale, whereas those in Sense 1 are more general. Gold-standard PropBank annotation of ARG1 allows the system to generalize over the semantic classes of the arguments and distinguish these two senses more accurately. null</Paragraph>
    </Section>
    <Section position="4" start_page="47" end_page="47" type="sub_section">
      <SectionTitle>
3.4 Frameset tags for WordNet sense-tagging
</SectionTitle>
      <Paragraph position="0"> PropBank frameset tags (either gold-standard or automatically tagged) were incorporated as features in our WSD system to see if knowing the coarse-grained sense tags would be useful in assigning fine-grained WordNet sense tags. A frameset tag for the instance was appended to each feature; this effectively partitions the feature set according to the coarse-grained sense provided by the frameset. To automatically tag an instance of a verb with its frameset, the set of all instances of the verb in Prop-Bank was partitioned into 10 subsets, and an instance in one subset was tagged by training a maximum entropy model on the instances in the other nine subsets. Various local features were considered, and the same feature types were used to train the frameset tagger and the WordNet sense tagger that used the automatically-assigned frameset.</Paragraph>
      <Paragraph position="1"> For the 20 Senseval-2 verbs that had more than one frameset in PropBank, we extracted all instances that were in both Senseval-2 and PropBank, yielding 1468 instances. We examined the effect of incorporating the gold-standard PropBank frameset tags into our maximum entropy models for these 20 verbs by partitioning the instances according to their frameset tag. Table 5 shows a breakdown of the accuracy by feature type. Adding the gold-standard frameset tag (&amp;quot;*fset&amp;quot;) to our original local features (&amp;quot;orig&amp;quot;) did not increase the accuracy significantly. However, the increase in accuracy (from 59.7% to 62.8%) was significant when these frameset tags were incorporated into the model that used both our original features and all the PropBank features.</Paragraph>
      <Paragraph position="2">  tagging of 20 Senseval-2 verbs with more than one frameset, with and without gold-standard frameset tag.</Paragraph>
      <Paragraph position="3"> However, partitioning the instances using the automatically generated frameset tags has no significant effect on the system's performance; the information provided by the automatically assigned coarse-grained sense tag is already encoded in the features used for fine-grained sense-tagging.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>