<?xml version="1.0" standalone="yes"?>
<Paper uid="I05-4002">
  <Title>Evaluation of a Japanese CFG Derived from a Syntactically Annotated Corpus with Respect to Dependency Measures</Title>
  <Section position="6" start_page="12" end_page="45" type="evalu">
    <SectionTitle>
4 Results
</SectionTitle>
    <Paragraph position="0"> Table 1 shows the results when D2 BP BD, which means the top parse result of each sentence is used for evaluation. In this case, &amp;quot;NEAREST&amp;quot; means only PGLR model was used for disambiguation without any other information (e.g. lexical infor- null mation, semantic information, etc.) On the other hand, &amp;quot;BEST&amp;quot; means only disambiguation of adnominal phrase attachment was done in the subsequent processing. Results by KNP and CaboCha are shown in the same table for comparison.</Paragraph>
    <Paragraph position="1"> As seen from Table 1, accuracy is still lower than KNP and CaboCha even if disambiguation of adnominal phrase attachment was done correctly in the subsequent processing. However, in this case, we do not use any information but PGLR model for disambiguation of any relations except adnominal phrase attachment (i.e. adverbial phrase attachment).</Paragraph>
    <Paragraph position="2"> Next, assuming that disambiguation of other relations, we carried out another evaluation changing D2 from 1 to 100. The result is shown in Figure 6. Dependency accuracy could achieve about 95.24% for &amp;quot;BEST&amp;quot;, which exceeds the dependency accuracy by KNP and CaboCha, if choosing the best result among top-100 parse results ranked by PGLR model would be done correctly in the subsequent processing 4. From the results, we can conclude the accuracy will increase as soon as lexical and semantic information is incorporated in the subsequent processing 5.</Paragraph>
    <Paragraph position="3"> However, segmentation accuracy is still significantly lower. The main reasons are as follows: POS Conversion Error: As mentioned previously, we converted POS tags automatically since the POS system of the Kyoto corpus is  some lexical information is useful for disambiguation, and it is necessary to consider what kind of lexical information could improve the accuracy.</Paragraph>
    <Paragraph position="4">  different from that of the RWC corpus. However, accuracy of the conversion is not high (about 80%). Since we used only POS information and did not use any word information for parsing, the result can be easily affected by the conversion error. Segmentation accuracy by CaboCha is also a little lower than accuracy by KNP. Since POS tags were converted in the same way, we think the reason is same. However, the difference between the accuracy by KNP and CaboCha is smaller since CaboCha uses not only POS information but also word information.</Paragraph>
    <Paragraph position="5"> Difference in Segmentation Policy: There is difference in bunsetsu segmentation policy between the Kyoto corpus and our corpus.</Paragraph>
    <Paragraph position="6"> For example: 1. 3 gatsu 31 nichi gogo 9 ji 43 fun goro, jishin ga atta (An earthquake occurred at around 9:43 p.m., March 1st.) 2. gezan suru no wo miokutta (We gave up going down the mountain.) null In the former case, the underlined part is segmented into 5 bunsetsu (&amp;quot;3 gatsu&amp;quot;, &amp;quot;31 nichi&amp;quot;, &amp;quot;gogo&amp;quot;, &amp;quot;9 ji&amp;quot;, and &amp;quot;43 fun goro,&amp;quot;) in the Kyoto corpus, while it is not segmented in our corpus. On the other hand, in the latter case, the underlined part is segmented into 2 bunsetsu (&amp;quot;gezan suru&amp;quot; and &amp;quot;no wo&amp;quot;) in our corpus, while it is not segmented in the Kyoto corpus. By correction of these two types of error, segmentation accuracy improved by 4.35% (76.53%AX80.88%) and dependency accuracy improved by 0.61% (95.24% AX 95.85%).</Paragraph>
  </Section>
class="xml-element"></Paper>