<?xml version="1.0" standalone="yes"?>
<Paper uid="C04-1204">
  <Title>Deep Linguistic Analysis for the Accurate Identification of Predicate-Argument Relations</Title>
  <Section position="3" start_page="0" end_page="0" type="metho">
    <SectionTitle>
2 Deep linguistic analysis and
</SectionTitle>
    <Paragraph position="0"> semantically annotated corpora Riezler et al. (2002) reported the successful application of a hand-crafted LFG (Bresnan, 1982) grammar to the parsing of the Penn Treebank (Marcus et al., 1994) by exploiting various techniques for robust parsing. The study was impressive because most researchers had believed that deep linguistic analysis of real-world text was impossible. Their success owed much to a consistent effort to maintain a wide-coverage LFG grammar, as well as var- null ious techniques for robust parsing.</Paragraph>
    <Paragraph position="1"> However, the manual development of wide-coverage linguistic grammars is still a difficult task. Recent progress in deep linguistic analysis has mainly depended on the acquisition of lexicalized grammars from annotated corpora (Xia, 1999; Chen and Vijay-Shanker, 2000; Chiang, 2000; Hockenmaier and Steedman, 2002a; Cahill et al., 2002; Frank et al., 2003; Miyao et al., 2004). This approach not only allows for the low-cost development of wide-coverage grammars, but also provides the training data for statistical modeling as a byproduct. Thus, we now have a basis for integrating statistical language modeling with deep linguistic analysis. To date, accurate parsers have been developed for LTAG (Chiang, 2000), CCG (Hockenmaier and Steedman, 2002b; Clark et al., 2002; Hockenmaier, 2003), and LFG (Cahill et al., 2002; Burke et al., 2004). Those studies have opened up the application of deep linguistic analysis to practical use. However, the accuracy of those parsers was still below PCFG parsers (Collins, 1999; Charniak, 2000) in terms of the PARSEVAL score, i.e., labeled bracketing accuracy of CFG-style parse trees. Since one advantage of deep parsers is that they can output a sort of semantic representation, e.g. predicate-argument structures, several studies have reported the accuracy of predicate-argument relations (Hockenmaier and Steedman, 2002b; Clark et al., 2002; Hockenmaier, 2003; Miyao et al., 2003). However, their evaluation employed a treebank developed for a specific grammar formalism. Hence, those results cannot be compared fairly with parsers based on other formalisms including PCFG parsers.</Paragraph>
    <Paragraph position="2"> At the same time, following the great success of machine learning approaches in NLP, many research efforts are being devoted to developing various annotated corpora. Notably, several projects are underway to annotate large corpora with semantic information such as semantic relations of words and coreferences.</Paragraph>
    <Paragraph position="3"> PropBank (Kingsbury and Palmer, 2002) and FrameNet (Baker et al., 1998) are large English corpora annotated with the semantic relations of words in a sentence. Figure 1 shows an example of the annotation of the PropBank. As the target text of the PropBank is the same as the Penn Treebank, a syntactic structure is given by the Penn Treebank.</Paragraph>
    <Paragraph position="4"> The PropBank includes additional annotations representing a predicate and its semantic arguments in a syntactic tree. For example, in Figure 1, REL denotes a predicate, &amp;quot;choose&amp;quot;, and ARGa0 represents its semantic arguments: &amp;quot;they&amp;quot; for the 0th argument (i.e., subject) and &amp;quot;this particular moment&amp;quot; for the 1st argument (i.e., object).</Paragraph>
    <Paragraph position="5"> Existing studies applied statistical classifiers to the identification of the PropBank or FrameNet annotations. Similar to many methods of applying machine learning to NLP tasks, they first formulated the task as identifying in a sentence each argument of a given predicate. Then, parameters of the identifier were learned from the annotated corpus. Features of a statistical model were defined as a pattern on a partial structure of the syntactic tree output by an automatic parser (Gildea and Palmer, 2002; Gildea and Jurafsky, 2002).</Paragraph>
    <Paragraph position="6"> Several studies proposed the use of deep linguistic features, such as predicate-argument relations output by a CCG parser (Gildea and Hockenmaier, 2003) and derivation trees output by an LTAG parser (Chen and Rambow, 2003). Both studies reported that the identification accuracy improved by introducing such deep linguistic features. Although deep analysis has not outperformed PCFG parsers in terms of the accuracy of surface structure, these results are implicitly supporting the necessity of deep linguistic analysis for the recognition of semantic relations.</Paragraph>
    <Paragraph position="7"> However, these results do not directly reflect the performance of deep parsers. Since these corpora provide deeper structure of a sentence than surface parse trees, they would be suitable for the evaluation of deep parsers. In Section 4, we explore the possibility of using the PropBank for the evaluation of an HPSG parser.</Paragraph>
  </Section>
  <Section position="4" start_page="0" end_page="2" type="metho">
    <SectionTitle>
3 Implementation of an HPSG parser
</SectionTitle>
    <Paragraph position="0"> This study evaluates the accuracy of a general-purpose HPSG parser that outputs predicate argument structures. While details have been explained in other papers (Miyao et al., 2003; Miyao et al., 2004), in the remainder of this section, we briefly review the grammar and the disambiguation model of our HPSG parser.</Paragraph>
    <Paragraph position="1">  Penn Treebank-style parse tree</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
3.1 Grammar
</SectionTitle>
      <Paragraph position="0"> The grammar used in this paper follows the theory of HPSG (Pollard and Sag, 1994), and is extracted from the Penn Treebank (Miyao et al., 2004). In this approach, a treebank is annotated with partially specified HPSG derivations using heuristic rules.</Paragraph>
      <Paragraph position="1"> By inversely applying schemata to the derivations, partially specified constraints are percolated and integrated into lexical entries, and a large HPSG-style lexicon is extracted from the treebank.</Paragraph>
      <Paragraph position="2"> Figure 2 shows an example of extracting HPSG lexical entries from a Penn Treebank-style parse tree. Firstly, given a parse tree (the top of the figure), we annotate partial specifications on an HPSG derivation (the middle). Then, HPSG schemata are applied to each branching in the derivation. Finally,  we get lexical entries for all of the words in the tree (the bottom).</Paragraph>
      <Paragraph position="3"> As shown in the figure, we can also obtain complete HPSG derivation trees, i.e., an HPSG treebank. It is available for the machine learning of disambiguation models, and can also be used for the evaluation of HPSG parsing.</Paragraph>
      <Paragraph position="4"> In an HPSG grammar, syntax-to-semantics mappings are implemented in lexical entries. For example, when we have a lexical entries for &amp;quot;choose&amp;quot; as shown in Figure 3, the lexical entry includes mappings from syntactic arguments (SUBJ and COMPS features) into a predicate-argument structure (ARG0 and ARG1 features). Argument labels in a predicate-argument structure are basically defined in a left-to-right order of syntactic realizations, while if we had a cue for a movement in the Penn Treebank, arguments are put in its canonical position in a predicate-argument structure.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="2" type="sub_section">
      <SectionTitle>
3.2 Disambiguation model
</SectionTitle>
      <Paragraph position="0"> By grammar extraction, we are able to obtain a large lexicon together with complete derivation trees of HPSG, i.e, an HPSG treebank. The HPSG treebank can then be used as training data for the machine learning of the disambiguation model.</Paragraph>
      <Paragraph position="1"> Following recent research about disambiguation models on linguistic grammars (Abney, 1997; Johnson et al., 1999; Riezler et al., 2002; Clark and Curran, 2003; Miyao et al., 2003; Malouf and van Noord, 2004), we apply a log-linear model or maximum entropy model (Berger et al., 1996) on HPSG derivations. We represent an HPSG sign as a tuple a1a3a2a5a4a7a6a9a8a7a10a11a8a13a12a15a14 , where a6 is a lexical sign of the head word, a10 is a part-of-speech, and a12 is a symbol representing the structure of the sign (mostly corresponding to nonterminal symbols of the Penn Treebank). Given an HPSG schema a16 and the distance a17 between the head words of the head/nonhead daughter constituents, each (binary) branching of an HPSG derivation is represented as a tuple</Paragraph>
      <Paragraph position="3"> . We divided the probability intoa10a15a14a24a23a25a17a19a13a21a20 and a52a53a14a16a12a18a17a23 a8a10a13a21a20 in order to accelerate the estimation of the probability model by usinga10a15a14a24a23a50a17a19a13a21a20 as a reference distribution (Miyao et al., 2003), because the direct estimation of a52a53a14a16a12a18a17a19a13a21a20 was computationally expensive.</Paragraph>
      <Paragraph position="4"> Feature function a45 a41 returns 1 when a certain part of tuple a18 is observed. Table 1 lists templates of feature functions used in the disambiguation model, where a check means that the corresponding element in the tuple is seen. For example, when we have a branching a4 head compa8a55a54 a8 transa8 VBa8 VPa8 nouna8 NNSa8 NPa14 ,  the following feature functions return 1, while all  model, lexical entry templates are more fine-grained (as shown in Section 5, a grammar has more than 1,000 templates), while we used a simple example here.</Paragraph>
      <Paragraph position="5">  are efficiently estimated using a dynamic programming algorithm for maximum entropy estimation (Miyao and Tsujii, 2002; Geman and Johnson, 2002).</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="2" end_page="2" type="metho">
    <SectionTitle>
4 Evaluating HPSG parsing with
</SectionTitle>
    <Paragraph position="0"> semantically annotated corpora Our study aims toward the fair evaluation of deep linguistic parsers, thus we want to directly compare the output of HPSG parsing with hand-annotated test data. However, disagreements between the output of HPSG parser and the PropBank prevents us from a direct comparison.</Paragraph>
    <Paragraph position="1"> In the PropBank annotation, semantic arguments can occur in multiple syntactic realizations, as in the following example (Figure 4).</Paragraph>
    <Paragraph position="2">  1. He broke the window.</Paragraph>
    <Paragraph position="3"> 2. The window broke.</Paragraph>
    <Paragraph position="4">  In the first example, a semantic object appears in a syntactic object position, while in the second sentence it becomes the syntactic subject. This alternation is caused by two reasons: syntactic alternations such as passive constructions and long-distance dependencies, and lexical alternations such as ergative verbs. It should also be noted that the assignment of argument labels have some arbitrariness. For example, Figure 5 shows the PropBank annotation for &amp;quot;The window broke into a million pieces.&amp;quot;, where a phrase &amp;quot;a million pieces&amp;quot; is annotated with ARG3, not with ARG2. This is because ARG2 is reserved for an instrument argument (e.g. &amp;quot;with a rock&amp;quot;). However, the choice of selecting ARG2 or ARG3 for &amp;quot;a million pieces&amp;quot; is arbitrary. Existing studies exploited statistical methods to mend these alternations and arbitrariness.</Paragraph>
    <Paragraph position="5"> Basically, deep linguistic parsers derived from the Penn Treebank can handle syntactic alternations owing to trace annotation in the treebank. However, lexical alternations and arbitrariness of assignments of argument labels will be a problem when we directly compare the output of an HPSG parser with the PropBank.</Paragraph>
    <Paragraph position="6"> However, we can see that the remaining disagreements are about the labels of argument labels. In general, we can assume that argument labels can be uniquely determined if a syntactic class of the predicate is given.3 In the example given in Section 2, &amp;quot;the window&amp;quot; always occurs in the object position when &amp;quot;broke&amp;quot; is transitive, while it appears in the subject position when it is intransitive. Since syntactic classes are expressed by lexical entries in HPSG, this indicates that we can establish a unique mapping from an HPSG lexical entry into PropBank semantic roles.</Paragraph>
    <Paragraph position="7"> Following this idea, we developed a mapping from HPSG argument labels into PropBank argument labels. This mapping was developed with a very simple algorithm as follows. We first computed predicate-argument structures from an HPSG treebank. We then compared the obtained predicate-argument structures with the PropBank annotations, and for each pair of a surface form of a word and its syntactic class, the mapping from argument labels of a predicate-argument structure into those of Prop-Bank was registered. When we found a conflict, that is, multiple mappings were found for a pair, a mapping found later was simply discarded.</Paragraph>
    <Paragraph position="8"> Our method is much simpler than existing studies, and it should be noted that PropBank was not used for training the probabilistic model or statistical identifier. This might be a handicap for our evaluation, but this method can clearly show the lower bound of the accuracy that has been attained by HPSG parsing.</Paragraph>
  </Section>
class="xml-element"></Paper>