<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-2705">
  <Title>The NomBank Project: An Interim Report</Title>
  <Section position="4" start_page="0" end_page="0" type="metho">
    <SectionTitle>
3 Lexical Entries and Noun Classes
</SectionTitle>
    <Paragraph position="0"> Before we could begin annotation, we needed to classify all the common nouns in the corpus. We needed to know which nouns were markable and make initial approximations of the inventories of senses and arguments for each noun. Toward this end, we pooled a number of resources: COMLEX Syntax (Macleod et al., 1998a), NOMLEX (Macleod et al., 1998b) and the verb classes from (Levin, 1993). We also used string matching techniques and hand classi cation in combination with programs that automatically merge crucial features of these resources. The result was NOMLEX-PLUS, a NOMLEX-style dictionary, which includes the original 1000 entries in NOMLEX plus 6000 additional entries (Meyers et al., 2004). The resulting noun classes include verbal nominalizations (e.g., destruction, knowledge, believer, recipient), adjectival nominalizations (ability, bitterness), and 16 other classes such as relational (father, president) and partitive nouns (set, variety). NOMLEX-PLUS helped us break down  ments may appear, e.g., the ARG2 (indirect object) to Dorothy can be left out of the phrase Glinda's gift of the slippers. the nouns into classes, which in turn helped us gain an understanding of the dif culty of the task and the manpower needed to complete the task.</Paragraph>
    <Paragraph position="1"> We used a combination of NOMLEX-PLUS and PropBank's lexical entries (or frames) to produce automatic approximations of noun frames for NomBank. These entries specify the inventory of argument roles for the annotators. For nominalizations of verbs that were covered in PropBank, we used straightforward procedures to convert existing PropBank lexical entries to nominal ones.</Paragraph>
    <Paragraph position="2"> However, other entries needed to be created by automatic means, by hand or by a combination of the two. Figure 2 compares the PropBank lexical entry for the verb claim with the NomBank entry for the noun claim. The noun claim and the verb claim share both the ASSERT sense and the SEIZE sense, permitting the same set of argument roles for those senses. However, only the ASSERT sense is actually attested in the sample PropBank corpus that was available when we began working on NomBank.</Paragraph>
    <Paragraph position="3"> Thus we added the SEIZE sense to both the noun and verb entries. The noun claim also has a LAWSUIT sense which bears an entry similar to the verb sue. Thus our initial entry for the noun claim was a copy of the verb entry at that time. An annotator edited the frames to re ect noun usage she added the second and third senses to the noun frame and updated the verb frame to include the second sense.</Paragraph>
    <Paragraph position="4"> In NOMLEX-PLUS, we marked anniversary and advantage as cousins of nominalizations indicating that their lexical entries should be modeled respectively on the verbs commemorate and exploit, although both entries needed to be modi ed in some respect. We use the term cousins of nominalizations to refer to those nouns which take argument structure similar to some verb (or adjective), but which are not morphologically related to that word. Examples are provided in Figure 3 and 4. For adjective nominalizations, we began with simple procedures which created frames based on NOMLEX-PLUS entries (which include whether the subject is +/-sentient). The entry for accuracy (the nominalization of the adjective accurate) plus a simple example is provided in gure 5 the ATTRIBUTE-LIKE frame is one of the most common frames for adjective nominalizations. To cover the remaining nouns in the corpus, we created classes of lexical items and manually constructed one frame for each class. Each member of a class was was given the corresponding frame. Figure 6 provides a sample of these classes, along with descriptions of their frames. As with the nominalization cousins, annotators sometimes had to adjust these frames for particular words.</Paragraph>
  </Section>
  <Section position="5" start_page="0" end_page="0" type="metho">
    <SectionTitle>
4 A Merged Representation
</SectionTitle>
    <Paragraph position="0"> Beginning with the PropBank and NomBank propositions in Figure 7, it is straight-forward to derive the  Roles: ARG0 = agent, ARG1 = thing remembered, ARG2 = times celebrated Noun Example: Investors celebrated the second anniversary of Black Monday.</Paragraph>
    <Paragraph position="2"/>
  </Section>
  <Section position="6" start_page="0" end_page="0" type="metho">
    <SectionTitle>
1. EXPLOIT
</SectionTitle>
    <Paragraph position="0"> Roles: ARG0 = exploiter, ARG1 = entity exploited Noun Example: Investors took advantage of Tuesday 's stock rally.</Paragraph>
    <Paragraph position="1"> REL = advantage, SUPPORT = took, ARG0 = Investors, ARG1 = of Tuesday's stock rally  combined PropBank/NomBank graphical representation in Figure 8 in which each role corresponds to an arc label. For this example, think of the argument structure of the noun ovation as analogous to the verb applaud. According to our analysis, they are both the givers and the applauders and the chefs are both the recipients of something given and the ones who are applauded. Gave and ovation have two distinct directional relations: a standing ovation is something that is given and gave serves as a link between ovation and its two arguments. This diagram demonstrates how NomBank is being designed for easy integration with PropBank. We believe that this is the sort of predicate argument representation that will be needed to easily merge this work with other annotation efforts.</Paragraph>
  </Section>
  <Section position="7" start_page="0" end_page="0" type="metho">
    <SectionTitle>
5 Analysis of the Task
</SectionTitle>
    <Paragraph position="0"> As of this writing we have created the various lexicons associated with NomBank. This has allowed us to break down the task as follows: a5 There are approximately 240,000 instances of common nouns in the PTB (approximately one out of every 5 words).</Paragraph>
    <Paragraph position="1"> a5 At least 36,000 of these are nouns that cannot take arguments and therefore need not be looked at by an annotator.</Paragraph>
    <Paragraph position="2"> a5 There are approximately 99,000 instances of verbal nominalizations or related items (e.g., cousins) a5 There are approximately 34,000 partitives (including 6,000 instances of the percent sign), 18,000 sub-ject nominalizations, 14,000 environmental nouns, 14,000 relational nouns and fewer instances of the various other classes.</Paragraph>
    <Paragraph position="3"> a5 Approximately 1/6 of the cases are instances of nouns which occur in multiple classes.5 The dif culty of the annotation runs the gamut from nominalization instances which include the most arguments, the most adjuncts and the most instances of support to the partitives, which have the simplest and most predictable structure.</Paragraph>
  </Section>
  <Section position="8" start_page="0" end_page="0" type="metho">
    <SectionTitle>
6 Error Analysis and Error Detection
</SectionTitle>
    <Paragraph position="0"> We have conducted some preliminary consistency tests for about 500 instances of verbal nominalizations during the training phases of NomBank. These tests yielded inter-annotator agreement rates of about 85% for argument roles and lower for adjunct roles. We are currently engaging in an effort to improve these results.6 We have identi ed certain main areas of disagreement including: disagreements concerning SUPPORT verbs and the shared arguments that go with them; disagreements about role assignment to prenominals; and differences between annotators caused by errors (typos, slips of the mouse, ill-formed output, etc.) In addition to improving our speci cations and annotator help texts, we are beginning to employ some automatic means for error detection.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
6.1 Support
</SectionTitle>
      <Paragraph position="0"> For inconsistencies with SUPPORT, our main line of attack has been to outline problems and solutions in our speci cations. We do not have any automatic system in effect yet, although we may in the near future.</Paragraph>
      <Paragraph position="1"> SUPPORT verbs (Gross, 1981; Gross, 1982; Mel' cuk, 1988; Mel' cuk, 1996; Fontenelle, 1997) are verbs which 5When a noun ts into multiple categories, those categories may predict multiple senses, but not necessarily. For example, drive has a nominalization sense (He went for a drive) and an attribute sense (She has a lot of drive). Thus the lexical entry for drive includes both senses. In constrast, teacher in the math teacher has the same analysis regardless of whether one thinks of it as the nominalization of teach or as a relational (ACTREL) noun.</Paragraph>
      <Paragraph position="2"> 6Consistency is the average precision and recall against a gold standard. The preliminary tests were conducted during training, and only on verbal nominalizations.</Paragraph>
      <Paragraph position="3"> connect nouns to one (or more) of their arguments via argument sharing. For example, in John took a walk, the verb took shares its subject with the noun walk. SUPPORT verbs can be problematic for a number of reasons.</Paragraph>
      <Paragraph position="4"> First of all the concept of argument sharing is not black and white. To illustrate these shades of gray, compare the relation of Mary to attack in: Mary's attack against the alligator, Mary launched an attack against the alligator, Mary participated in an attack against the alligator, Mary planned an attack against the alligator and Mary considered an attack against the alligator. In each subsequent example, Mary's level of agency decreases with respect to the noun attack. However, in each case Mary may still be viewed as some sort of potential attacker. It turned out that the most consistent position for us to take was to assume all degrees of argument-hood (in this case subject-hood) were valid. So, we would mark Mary as the ARG0 of attack in all these instances. This is consistent with the way control and raising structures are marked for verbs, e.g., John is the subject of leave and do in John did not seem to leave and John helped do the project under most accounts of verbal argument structure that take argument sharing (control, raising, etc.) into account.</Paragraph>
      <Paragraph position="5"> Of course a liberal view of SUPPORT has the danger of overgeneration. Consider for example, Market conditions led to the cancellation of the planned exchange.</Paragraph>
      <Paragraph position="6"> The unwary annotator might assume that market conditions is the ARG0 (or subject) of cancellation. In fact, the combination lead to and cancellation do not have any of the typical features of SUPPORT described in gure 9.</Paragraph>
      <Paragraph position="7"> However, the nal piece of evidence is that market conditions violate the selection restrictions of cancellation. Thus the following paraphrase is ill-formed *Market conditions canceled the planned exchange. This suggests that market conditions is the subject of lead and not the subject of cancellation. Therefore, this is not an instance of support in spite of the apparent similarity.</Paragraph>
      <Paragraph position="8"> We require that the SUPPORT relation be lexical. In other words, there must be something special about a SUPPORT verb or the combination of the SUPPORT verb and the noun to license the argument sharing relation. In addition to SUPPORT, we have cataloged several argument sharing phenomena which are markable. For example, consider the sentence, President Bush arrived for a celebration. Clearly, President Bush is the ARG0 of celebration (one of the people celebrating). However, arrive is not a SUPPORT verb. The phrase for a celebration is a subject-oriented adverbial, similar to adverbs like willingly, which takes the subject of the sentence as an argument. Thus President Bush could also be the sub-ject of celebration in President Bush waddled into town for the celebration and many similar sentences that contain this PP.</Paragraph>
      <Paragraph position="9"> Finally, there are cases where argument sharing may a5 Support verb/noun pairs can be idiosyncratically connected to the point that some researchers would call them idioms or phrasal verbs, e.g., take a walk, keep tabs on.</Paragraph>
      <Paragraph position="10"> a5 The verb can be essentially empty , e.g., make an attack, have a visit.</Paragraph>
      <Paragraph position="11"> a5 The verb/noun combination may take a different set of arguments than either does alone, e.g., take advantage of.</Paragraph>
      <Paragraph position="12"> a5 Some support verbs share the subject of almost any nominalization in a particular argument slot. For example attempt shares its subject with most following nominalizations, e.g., He attempted an attack.</Paragraph>
      <Paragraph position="13"> These are the a lot like raising/control predicates.</Paragraph>
      <Paragraph position="14"> a5 In some cases, the support verb and noun are from similar semantic classes, making argument sharing very likely, e.g., ght a battle.</Paragraph>
      <Paragraph position="15"> Figure 9: Possible Features of Support be implied by discourse processes, but which we do not mark (as we are only handling sentence-level phenomena). For example, the words proponent and rival strongly imply that certain arguments appear in the discourse, but not necessarily in the same sentence. For example in They didn't want the company to fall into the hands of a rival, there is an implication that the company is an ARG1 of rival, i.e., a rival should be interpreted as a rival of the company.7 The connection between a rival and the company is called a bridging relation (a process akin to coreference, cf. (Poesio and Vieira, 1998)) In other words, fall into the hands of does not link rival with the company by means of SUPPORT. The fact that a discourse relation is responsible for this connection becomes evident when you see that the link between rival and company can cross sentence boundaries, e.g., The company was losing money. This was because a rival had come up with a really clever marketing strategy.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
6.2 Prenominal Adjectives and Error Detection
</SectionTitle>
      <Paragraph position="0"> ARGM is the annotation tag used for nonarguments, also known as adjuncts. For nouns, it was decided to only tag such types of adjuncts as are also found with verbs, e.g., temporal, locative, manner, etc. The rationale for this included: (1) only the argument-taking common nouns are being annotated and other sorts of adjuncts occur with common nouns in general; (2) narrowing the list of potential labels helped keep the labeling consistent; and (3) this was the minimum set of adjuncts that would keep the 7The noun rival is a subject nominalization of the verb rival.</Paragraph>
      <Paragraph position="1"> noun annotation consistent with the verb annotation.</Paragraph>
      <Paragraph position="2"> Unfortunately, it was not always clear whether a prenominal modi er (particularly an adjective) fell into one of our classes or not. If an annotator felt that a modier was somehow important , there was a temptation to push it into one of the modi er classes even if it was not a perfect t. Furthermore, some annotators had a broader view than others as to the sorts of semantic relationships that fell within particular classes of adjuncts, particularly locative (LOC), manner (MNR) and extent (EXT). Unlike the SUPPORT verbs, which are often idiosyncratic to particular nominal predicates, adjunct prenominal modiers usually behave the same way regardless of the noun with which they occur.</Paragraph>
      <Paragraph position="3"> In order to identify these lexical properties of prenominals, we created a list of all time nouns from COMLEX Syntax (ntime1 and ntime2) and we created a specialized dictionary of adjectives with adverbial properties which we call ADJADV. The list of adjective/adverb pairs in ADJADV came from two sources: (1) a list of adjectives that are morphologically linked to -ly adverbs created using some string matching techniques; and (2) adjective/adverb pairs from CATVAR (Habash and Dorr, 2003). We pruned this list to only include adjectives found in the Penn Treebank and then edited out inappropriate word pairs. We completed the dictionary by transferring portions of the COMLEX Syntax adverb entries to the corresponding adjectives.</Paragraph>
      <Paragraph position="4"> We now use ADJADV and our list of temporal nouns to evaluate NOMBANK annotation of modi ers. Each annotated left modi er is compared against our dictionaries. If a modi er is a temporal noun, it can bear the ARGM-TMP role (temporal adjunct role), e.g., the temporal noun morning can ll the ARGM-TMP slot in the morning broadcast. Most other common nouns are compatible with argument role slots (ARG0, ARG1, etc.), e.g., the noun news can ll the ARG1 slot in the news broadcast. Finally, roles associated with adjectives depend on their ADJADV entry, e.g., possible can be an ARGM-ADV in possible broadcasts due to the epistemic feature encoded in the lexical entry for possible (derived from the corresponding adjverb possibly). Discrepancies between these procedures and the annotator are resolved on a case by case basis. If the dictionary is wrong, the dictionary should be changed, e.g., root, as in root cause was added to the dictionary as a potential MNR adjective with a meaning like the adverb basically. However, if the annotator is wrong, the annotation should be changed, e.g., if an annotator marked slow as a ARGM-TMP, the program would let them know that it should be a ARGM-MNR. This process both helps with annotation accuracy and enriches our lexical database.</Paragraph>
    </Section>
    <Section position="3" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
6.3 Other Automatically Detected Errors
</SectionTitle>
      <Paragraph position="0"> We used other procedures to detect errors including: Nom-type Argument nominalizations are nominalizations that play the role of one of the arguments in the ROLESET. Thus the word acquirer should be assigned the ARG0 role in the following example because acquirer is a subject nominalization: a possible acquirer of Manville REL = acquirer, ARG0 = acquirer, ARG1 = of Manville, ARGM-ADV = possible A procedure can compare the NOMLEX-PLUS entry for each noun to each annotated instance of that noun to check for incompatibilities.</Paragraph>
      <Paragraph position="1"> Illformedness Impossible instances are ruled out. Checks are made to make sure obligatory labels (REL) are present and illegal labels are not. Similarly, procedures make sure that in nitive arguments are marked with the -PRD function tag (a PropBank convention).</Paragraph>
      <Paragraph position="2"> Probable Illformedness Certain con gurations of role labels are possible, but very unlikely. For example, the same argument role should not appear more than once (the stratal uniqueness condition in Relational Grammar or the theta criterion in Principles and parameters, etc.). Furthermore, it is unlikely for the rst word of a sentence to be an argument unless the main predicate is nearby (within three words) or unless there is a nearby support verb. Finally, it is unlikely that there is an empty category that is an argument of a predicate noun unless the empty category is linked to some real NP.8 WRONG-POS We use procedures that are part of our systems for generating GLARF, a predicate argument framework discussed in (Meyers et al., 2001a; Meyers et al., 2001b), to detect incorrect parts of speech in the Penn Treebank. If an instance is predicted to be a part of speech other than a common noun, but it is still tagged, that instance is agged. For example, if a word tagged as a singular common noun is the rst word in a VP, it is probably tagged with the wrong part of speech.</Paragraph>
    </Section>
    <Section position="4" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
6.4 The Results of Error Detection
</SectionTitle>
      <Paragraph position="0"> The processes described in the previous subsections are used to create a list of annotation instances to check along with short standardized descriptions of what was wrong, e.g., wrong-pos, non-functional (if there were two identical argument roles), etc. Annotators do a second pass 8Empty categories mark invisible constituents in the Treebank, e.g., the subject of want in Johna6 wanted ea6 to leave.  on just these instances (currently about 5 to 10% of the total). We will conduct a formal evaluation of this procedure over the next month.</Paragraph>
    </Section>
  </Section>
  <Section position="9" start_page="0" end_page="0" type="metho">
    <SectionTitle>
7 Future Research: Automatic Annotation
</SectionTitle>
    <Paragraph position="0"> We are just starting a new phase in this project: the creation of an automatic annotator. Using techniques similar to those described in (Meyers et al., 1998) in combination with our work on GLARF (Meyers et al., 2001a; Meyers et al., 2001b), we expect to build a hand-coded PROPBANKER a program designed to produce a Prop-Bank/NomBank style analysis from Penn Treebank style input. Although the PropBanker should work with input in the form of either treebank annotation or treebank-based parser output, this project only requires application to the Penn Treebank itself. While previous programs with similar goals (Gildea and Jurafsky, 2002) were statistics-based, this tool will be based completely on hand-coded rules and lexical resources.</Paragraph>
    <Paragraph position="1"> Depending on its accuracy, automatically produced annotation should be useful as either a preprocessor or as an error detector. We expect high precision for very simple frames, e.g., nouns like lot as in gure 10. Annotators will have the opportunity to judge whether particular automatic annotation is good enough to serve as a preprocessor. We hypothesize that a comparison of automatic annotation that fails this level of accuracy against the hand annotation will still be useful for detecting errors. Comparisons between the hand annotated data and the automatically annotated data will yield a set of instances that warrant further checking along the same lines as our previously described error checking mechanisms.</Paragraph>
  </Section>
class="xml-element"></Paper>