<?xml version="1.0" standalone="yes"?>
<Paper uid="C04-1159">
  <Title>Dependency Structure Analysis and Sentence Boundary Detection in Spontaneous Japanese</Title>
  <Section position="4" start_page="0" end_page="0" type="concl">
    <SectionTitle>
4 Experiments and Discussion
</SectionTitle>
    <Paragraph position="0"> In our experiments, we used the transcriptions of 188 talks in the CSJ. We used 10 talks for testing. Dependency structure analysis results were evaluated for closed- and open-test data in terms of accuracy, which was defined as the percentage of correct dependencies out of all dependencies. In Tables 1 to 3, we use words &amp;quot;closed&amp;quot; and &amp;quot;open&amp;quot; to describe the results obtained for closed- and open-test data, respectively. Sentence boundary detection results were evaluated in terms of F-measure.</Paragraph>
    <Paragraph position="1"> First, we show the baseline accuracy of dependency structure analysis and sentence boundary detection. The method described in Section 3.2 was used as a baseline method for sentence boundary detection (Process 1 in Figure 1). To train the language model represented by P(Y ), we used the transcriptions of 178 talks excluding the test data. The method described in Section 3.1 was used as a baseline method for dependency structure analysis. (Process 3 in Figure 1) As sentence boundaries, we used the results of the baseline method for sentence boundary detection. We obtained an F-measure of 75.6, a recall of 64.5%, and a precision of 94.2% for the sentence boundary detection in our experiments. The dependency structure analysis accuracy was 75.2% for the open data and 80.7% for the closed data.</Paragraph>
    <Paragraph position="2"> The dependency probability of the rightmost bunsetsus in a given sentence was not calculated in our model. So, we assumed that the right-most bunsetsus depended on the next bunsetsu and that the dependency probability was 0.5 when we used dependency information in the experiments described in the following sections.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.1 Sentence Boundary Detection
Results Obtained by Method 1
</SectionTitle>
      <Paragraph position="0"> We evaluated the results obtained by the method described in Section 3.3. The results of baseline dependency structure analysis were used as dependency information (Process 5 in Figure 1).</Paragraph>
      <Paragraph position="1"> First, we investigated the optimal values of parameters p and q described in Section 3.3 by using held-out data, which differed from the test data and consisted of 15 talks. The optimal values of p and q were, respectively, 0 and 0.9 for the open-test data, and 0 and 0.8 for the closed-test data. These values were used in the following experiments. The value of p was 0, and these results show that bunsetsus that depended on a bunsetsu beyond 50 bunsetsus were ignored as described in assumption (2) in Section 3.3.</Paragraph>
      <Paragraph position="2"> The obtained results are shown in Table 1.</Paragraph>
      <Paragraph position="3"> When dependency information was used, the F-measure increased by approximately 1.4 for the open-test data and by 2.0 for the closed test data, respectively. Although the accuracy of dependency structure analysis for closed test data was about 5.5% higher than that for the open-test data, the difference between the accuracies of sentence boundary detection for the closedand open-test data was only about 0.6%. These results indicate that equivalent accuracies can be obtained for both open- and closed-test data in detecting dependencies related to sentence boundaries.</Paragraph>
      <Paragraph position="4"> When all the extracted candidates were considered as sentence boundaries without using language models, the accuracy of sentence boundary detection obtained by using the base-line method was 68.2%(769/1,127) in recall and 81.5%(769/943) in precision, and that obtained by using Method 1 was 87.2%(983/1,127) in recall and 27.7%(983/3,544) in precision. The results show that additional 214 sentence boundary candidates were correctly extracted by using dependency information. However, only 108 sentence boundaries were chosen out of the 214 candidates when language models were used. We investigated in detail the points that were not chosen and found errors in noun-final clauses, clauses where the rightmost constituents were adjectives or verbs such as &amp;quot;q O(it to-omou, think)&amp;quot; or &amp;quot;x`M(it wamuzukashii, difficult)&amp;quot;, and clauses where the rightmost constituents were &amp;quot;qMOwx(it to- null iu-no-wa, because)&amp;quot; and &amp;quot;q`ox(it to-si-tewa, as)&amp;quot;, and so on. Some errors, except for those in noun-final clauses, could have been correctly detected if we had had more training data.</Paragraph>
      <Paragraph position="5"> We also found that periods were sometimes erroneously inserted when preceding expressions were &amp;quot;U(ga, but)&amp;quot;, &amp;quot;`o(mashite, and)&amp;quot;, and &amp;quot;Zr(keredomo, but)&amp;quot;, which are typically the rightmost constituents of a sentence, as weel as &amp;quot;o(te, and)&amp;quot;, which is not, typically, the rightmost constituent of a sentence. The language models were not good at discriminating between subtle differences.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.2 Sentence Boundary Detection
Results Obtained by Method 2
</SectionTitle>
      <Paragraph position="0"> We evaluated the results obtained by the method described in Section 3.4 (Process 6 in Figure 1). For training, we used 178 talks excluding test data.</Paragraph>
      <Paragraph position="1"> The results are shown in Table 2. The F-measure was about 6.9 points higher than that described in Section 4.1. The results show that the approach based on machine learning is more effective than that based on statistical machine translation. The results also show that the accuracy of sentence boundary detection can be increased by using dependency information in Method 2. However, we found that the amount of accuracy improvement achieved by using dependency information depended on the method used. This may be because other features used in SVM may provide information similar to dependency information. For example, Feature 1 described in Section 3.4 might provide information similar to that in Features 4 and 5. Although in our experiments we used only three words to the right and three words to the left of the target word, the degradation in accuracy without dependency information was slight. This may be because long-distance dependencies may not be related to sentence boundaries, or because Feature 5 does not contribute to increasing the accuracy because the accuracy of dependency structure analysis in detecting long-distance dependencies is not high.</Paragraph>
    </Section>
    <Section position="3" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.3 Dependency Structure Analysis
Results
</SectionTitle>
      <Paragraph position="0"> We evaluated the results of dependency structure analysis obtained when sentence boundaries detected automatically by the two methods described above were used as inputs (Process 7 in Figure 1). The results are shown in Table 3. The accuracy of dependency structure analysis improved by about 2% when the most accurate and automatically detected sentence boundaries were used as inputs. This is because more sentence boundaries were detected correctly, and the number of bunsetsusthatdepended on those in other sentences decreased. We investigated the accuracy of dependency structure analysis when 100% accurate sentence boundaries were used as inputs. The accuracy was 80.1% for the open-test data, and 86.1% for the closed-test data. Even when the sentence boundary detection was perfect, the error rate was approximately 14% even for the closed-test data. The accuracy of dependency structure analysis for spoken text was about 8% lower than that for written text (newspapers).</Paragraph>
      <Paragraph position="1"> We speculate that this is because spoken text has no punctuation marks and many bunsetsus depend on others far from them because of insertion structures. These problems need to be addressed in future studies.</Paragraph>
      <Paragraph position="2"> 5Conclusion This paper described a project to detect dependencies between bunsetsus and sentence boundaries in a spontaneous speech corpus. It is more difficult to detect dependency structures inspontaneousspokenspeechthaninwritten text. The biggest problem is that sentence boundaries are ambiguous. We proposed two methods for improving the accuracy of sentence boundary detection in spontaneous Japanese speech. Using these methods, we obtained an F-measure of 84.9 for the accuracy of sentence boundary detection. The accuracy of dependency structure analysis was also improved from 75.2% to 77.2% by using automatically detected sentence boundaries. The accuracy of dependency structure analysis and that of sentence boundary detection were improved by interactively using automatically detected dependency information and sentence boundaries.</Paragraph>
      <Paragraph position="3"> There are several future directions. In the future, we would like to solve the problems that we found in our experiments. In particular, we want to reduce the number of errors due to inserted structures and solve other problems described in Section 2.1.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>