<?xml version="1.0" standalone="yes"?>
<Paper uid="P06-1105">
  <Title>Japanese Dependency Parsing Using Co-occurrence Information and a Combination of Case Elements</Title>
  <Section position="7" start_page="835" end_page="839" type="evalu">
    <SectionTitle>
5 Experiments
</SectionTitle>
    <Paragraph position="0"> We evaluated the effectiveness of our model experimentally. Since our model treats only the de-</Paragraph>
    <Paragraph position="2"> ties of particle sets pendency relations between a noun and a verb, we cannot determine all the dependency relations in a sentence. We therefore use one of the currently available dependency analyzers to generate an ordered list of n-best possible parses for the sentence and then use our proposed model to rerank them and select the best parse.</Paragraph>
    <Section position="1" start_page="835" end_page="835" type="sub_section">
      <SectionTitle>
5.1 Dependency analyzer for outputting
</SectionTitle>
      <Paragraph position="0"> n-best parses We generated the n-best parses by using the &amp;quot;posterior context model&amp;quot; (Uchimoto et al., 2000). The features we used were those in (Uchimoto et al., 1999) and their combinations. We also added our original features and their combinations, with reference to (Sassano, 2004; Kudo and Matsumoto, 2002), but we removed the features that had a frequency of less than 30 in our training data. The total number of features is thus 105,608.</Paragraph>
    </Section>
    <Section position="2" start_page="835" end_page="836" type="sub_section">
      <SectionTitle>
5.2 Reranking method
</SectionTitle>
      <Paragraph position="0"> Because our model considers only the dependency relations between a noun and a verb, and thus cannot determine all the dependency relations in a sentence, we restricted the possible parses for  reranking as illustrated in Figure 2. The possibleparsesforrerankingwerethefirst-rankedparse null and those of the next-best parses in which the verb to modify was different from that in the firstranked one. For example, parses 1 and 3 in Figure 2 are the only candidates for reranking. In our experiments, n is set to 50.</Paragraph>
      <Paragraph position="1"> The score we used for reranking the parses was the product of the probability of the posterior context model and the probability of our proposed model:</Paragraph>
      <Paragraph position="3"> where Pcontext(T) is the probability of the posterior context model. The a here is a parameter with which we can adjust the balance of the two probabilities, and is fixed to the best value by considering development data (different from the training  Many methods for reranking the parsing of English sentences have been proposed (Charniak and Johnson, 2005; Collins and Koo, 2005; Henderson and Titov, 2005), all of which are discriminative methods which learn the difference between the best parse and next-best parses. While our reranking model using generation probability is quite simple, we can easily verify our hypothesis that the two proposed probabilities have an effect on improving the parsing accuracy. We can also verify that the parsing accuracy improves by using imprecise information obtained from an automatically parsed corpus.</Paragraph>
      <Paragraph position="4"> Klein and Manning proposed a generative model in which syntactic (PCFG) and semantic (lexical dependency) structures are scored with separate models (Klein and Manning, 2002), but 1In our experiments, a is set to 2.0 using development data.</Paragraph>
      <Paragraph position="5"> they do not take into account the combination of dependencies. Shirai et al. also proposed a statistical model of Japanese language which integrates lexical association statistics with syntactic preference (Shirai et al., 1998). Our proposed model differs from their method in that it explicitly uses the combination of multiple cases.</Paragraph>
    </Section>
    <Section position="3" start_page="836" end_page="837" type="sub_section">
      <SectionTitle>
5.3 Estimation of co-occurrence probability
</SectionTitle>
      <Paragraph position="0"> We estimated the co-occurrence probability of the particle set and the co-occurrence probability of the case element set used in our model by analyzing a large-scale corpus. We collected a 30-year newspaper corpus2, applied the morphological analyzer JUMAN (Kurohashi and Nagao, 1998b), and then applied the dependency analyzer with a posterior context model3. To ensure that we collected reliable co-occurrence information, we removed the information for the bunsetsus with punctuation4.</Paragraph>
      <Paragraph position="1"> Like (Torisawa, 2001), we estimated the co-occurrence probability P(&lt;n,r,v&gt; ) of the case  element set (noun n, particle r, and verb v) by using probabilistic latent semantic indexing (PLSI) (Hofmann, 1999)5. If &lt;n,r,v&gt; is the co-occurrence of n and &lt;r,v&gt; , we can calculate</Paragraph>
      <Paragraph position="3"> where z indicates a latent semantic class of co-occurrence (hidden class). Probabilistic parameters P(n|z), P(&lt;r,v&gt; |z), and P(z) in Equation (9) can be estimated by using the EM algorithm. In ourexperiments, thedimensionof thehiddenclass z was set to 300. As a result, the collected &lt;n,r,v&gt; total 102,581,924 pairs. The number of n and v is 57,315 and 15,098, respectively.</Paragraph>
      <Paragraph position="4"> The particles for which the co-occurrence probabilitywasestimatedwerethesetofcaseparticles, null the &amp;quot;ha&amp;quot; case particle, and a class of &amp;quot;fukujoshi&amp;quot;  particles. Therefore, the total number of particles was 10.</Paragraph>
      <Paragraph position="5"> Wealsoestimatedtheco-occurrenceprobability of the particle set P(rs|syn,v) by using PLSI. We regarded the triple &lt;rs,syn,v&gt; (the co-occurrence of particle set rs, verb v, and the syntactic prop-erty syn) as the co-occurrence of rs and &lt;syn,v&gt; . The dimension of the hidden class was 100. The total number of &lt;rs,syn,v&gt; pairs was 1,016,508, v was 18,423, and rs was 1,490. The particle set should be treated not as a non-ordered set but as an occurrence ordered set. However, we think correct probability estimation using an occurrence ordered set is difficult, because it gives rise to an explosion in the number of combination,</Paragraph>
    </Section>
    <Section position="4" start_page="837" end_page="837" type="sub_section">
      <SectionTitle>
5.4 Experimental environment
</SectionTitle>
      <Paragraph position="0"> The evaluation data we used was Kyodai Corpus 3.0, a corpus manually annotated with dependency relations (Kurohashi and Nagao, 1998a).</Paragraph>
      <Paragraph position="1"> The statistics of the data are as follows:  accuracy (the percentage of bunsetsu for which the correct modifyee was identified) and sentence accuracy (the percentage of sentences for which the correct dependency structure was identified).</Paragraph>
    </Section>
    <Section position="5" start_page="837" end_page="838" type="sub_section">
      <SectionTitle>
5.5 Experimental results
5.5.1 Evaluation of our model
</SectionTitle>
      <Paragraph position="0"> Our first experiment evaluated the effectiveness of reranking with our proposed model. Bunsetsu Our reranking model  correct bunsetsu (posterior context model x our model) and sentence accuracies before and after reranking, for the entire set of test data as well as for only those sentences whose parse was actually reranked, are listed in Table 3.</Paragraph>
      <Paragraph position="1"> The results showed that the accuracy could be improved by using our proposed model to rerank the results obtained with the posterior context model. McNemar testing showed that the null hypothesis that there is no difference between the accuracy of the results obtained with the posterior context model and those obtained with our model could be rejected with a p value &lt; 0.01. The difference in accuracy is therefore significant.  We next experimentally compare the following variations of the proposed model:  (a) one in which the case element set is assumed to be independent [Equation (7)] (b) one using the co-occurrence probability of the particle set, P(rs|syn,v), in our model (c) one using only the co-occurrence probability of the case element, P(n|r,v), in our model (d) one not taking into account the syntactic property of a verb (i,e. a model in which the co-occurrence probability is defined as P(r|v), without the syntactic property syn) (e) one in which the co-occurrence probability of the case element, P(n|r,v), is simply added  to a feature set used in the posterior context model (f) one using only our proposed probabilities without the probability of the posterior context model The accuracies obtained with each of these models are listed in Table 5, from which we can conclude that it is effective to take into account the dependencybetweencaseelementsbecausemodel (a) is less accurate than our model.</Paragraph>
      <Paragraph position="2"> Since the accuracy of model (d) is comparable to that of our model, we can conclude that the consideration of the syntactic property of a verb does not necessarily improve dependency analysis. The accuracy of model (e), which uses the co-occurrence probability of the case element set as features in the posterior context model, is comparable to that of the posterior context model. This result is similar to the one obtained by (Kehler et al., 2004), where the task was anaphora resolution. Although we think the co-occurrence probability is useful information for dependency analysis, this result shows that simply adding it as a feature does not improve the accuracy.</Paragraph>
      <Paragraph position="3"> 5.5.3 Changing the amount of training data Changing the size of the training data set, we investigated whether the degree of accuracy improvement due to reranking depends on the accuracy of the existing dependency analyzer. Figure 3 shows that the accuracy improvement is constant even if the accuracy of the dependency analyzer is varied.</Paragraph>
    </Section>
    <Section position="6" start_page="838" end_page="839" type="sub_section">
      <SectionTitle>
5.6 Discussion
</SectionTitle>
      <Paragraph position="0"> The score used in reranking is the product of the probability of the posterior context model and the  training data is changed probability of our proposed model. The results in Table 5 show that the parsing accuracy of model (f), whichusesonlytheprobabilitiesobtainedwith our proposed model, is quite low. We think the reason for this is that our two co-occurrence probabilities cannot take account of syntactic properties, such as punctuation and the distance between two bunsetsus, which improve dependency analysis. null Furthermore, when the sentence has multiple verbs and case elements, the constraint of our proposed model tends to distribute case elements to each verb equally. To investigate such bias, we calculated the variance of the number of case elements per verb.</Paragraph>
      <Paragraph position="1"> Table6showsthatthevarianceforourproposed model (Equation [5]) is the lowest, and this model distributes case elements to each verb equally. The variance of the posterior context model is higher than that of the test data, probably because the syntactic constraint in this model affects parsing too much. Therefore the variance of the reranking model (Equation [8]), which is the combination of our proposed model and the posterior context model, is close to that of the test data.</Paragraph>
      <Paragraph position="2"> The best parse which uses this data set is (Kudo and Matsumoto, 2005), and their parsing accuracy is 91.37%. The features and the parsing method used by their model are almost equal to the posterior context model, but they use a different method of probability estimation. If their model could generate n-best parsing and attach some kind of score to each parse tree, we would combine their model in place of the posterior context model.</Paragraph>
      <Paragraph position="3"> At the stage of incorporating the proposed approach to a parser, the consistency with other pos- null context model test data Equation [8] Equation [5] variance (s2) 0.724 0.702 0.696 0.666 *The average number of elements per verb is 1.078.  sible methods that deal with other relations should be taken into account. This will be one of our future tasks.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>