File Information
File: 05-lr/acl_arc_1_sum/cleansed_text/xml_by_section/evalu/06/p06-1145_evalu.xml
Size: 8,174 bytes
Last Modified: 2025-10-06 13:59:43
<?xml version="1.0" standalone="yes"?> <Paper uid="P06-1145"> <Title>Sydney, July 2006. c(c)2006 Association for Computational Linguistics Time Period Identification of Events in Text Taichi Noro + Takashi Inui ++ Hiroya Takamura ++</Title> <Section position="7" start_page="1156" end_page="1159" type="evalu"> <SectionTitle> 5 Experimental Results and Discussion </SectionTitle> <Paragraph position="0"/> <Section position="1" start_page="1156" end_page="1158" type="sub_section"> <SectionTitle> 5.1 Time-Slot Classifier with Time- Associated Words </SectionTitle> <Paragraph position="0"> We used 11.668 positive (time-unknown) samples and 2,552 negative (morning ~ night) samples. We conducted a classification experiment by Support Vector Machines with 10-fold cross validation. We used TinySVM software package for implementation. The soft margin parameter is automatically estimated by 10-fold cross validation with training data. The result is shown in Table 2.</Paragraph> <Paragraph position="1"> Table 2 clarified that the &quot;time-unknown&quot; filter achieved good performance; F-measure of 0.899. In addition, since we obtained a high recall (0.969), many of the noisy sentences will be filtered out at this step and the classifier of the second step is likely to perform well.</Paragraph> <Paragraph position="2"> the time-unknown filter.</Paragraph> <Paragraph position="3"> In step 2, we used &quot;time-known&quot; sentences classified by the unknown filter as test data. We conducted a classification experiment by Naive Bayes classifier + the EM algorithm with 10-fold cross validation. For unlabeled data, we used 64,782 sentences, which have no intersection with the labeled data. The parameters, l and b , are automatically estimated by 10-fold cross validation with training data. The result is shown in Table 3.</Paragraph> <Paragraph position="4"> In Table 3, &quot;Explicit&quot; indicates the result by a simple classifier based on regular expressions including explicit temporal expressions. The baseline method classifies all sentences into night because the number of night sentences is the largest. The &quot;CONTEXT&quot; column shows the results obtained by classifiers learned with the features in CONTEXT in addition to the features For example, we classify sentences matching following regular expressions into morning class:</Paragraph> <Paragraph position="6"> [456789(10)][(am)(AM)].</Paragraph> <Paragraph position="7"> ( &quot;gozen&quot;, &quot;gozen- no&quot; means before noon. &quot;asa&quot;, &quot;asa-no&quot; means morning. &quot;ji&quot; means o'clock.) in NORMAL. The accuracy of the Explicit method is lower than the baseline. This means existing methods based on explicit temporal expressions cannot work well in blog text. The accuracy of the method 'NB' exceeds that of the baseline by 16%. Furthermore, the accuracy of the proposed method 'NB+EM' exceeds that of the 'NB' by 11%. Thus, we figure out that using unlabeled data improves the performance of our time-slot classification.</Paragraph> <Paragraph position="8"> In this experiment, unfortunately, CONTEXT only deteriorated the accuracy. The time-slot tags of the sentences preceding or following the target sentence may still provide information to improve the accuracy. Thus, we tried a sequential tagging method for sentences, in which tags are predicted in the order of their occurrence. The predicted tags are used as features in the prediction of the next tag. This type of sequential tagging method regard as a chunking procedure (Kudo and Matsumoto, 2000) at sentence level.</Paragraph> <Paragraph position="9"> We conducted time-slot (five classes) classification experiment, and tried forward tagging and backward tagging, with several window sizes.</Paragraph> <Paragraph position="10"> We used YamCha , the multi-purpose text chunker using Support Vector Machines, as an experimental tool. However, any tagging direction and window sizes did not improve the performance of classification. Although a chunking method has possibility of correctly classifying a sequence of text units, it can be adversely biased by the preceding or the following tag. The sentences in blog used in our experiments would not have a very clear tendency in order of tags. This is why the chunking-method failed to improve the performance in this task. We would like to try other bias-free methods such as Conditional Random Fields (Lafferty et al., 2001) for future work.</Paragraph> <Paragraph position="11"> Finally, we show an accuracy of the 2-step classifier (Method A) and compare it with those of other classifiers in Table 6. The accuracies are calculated with the equation: .</Paragraph> <Paragraph position="12"> In Table 6, the baseline method classifies all sentences into time-unknown because the number of time-unknown sentences is the largest. Accuracy of Method A (proposed method) is higher than that of Method B (4.1% over). These results show that time-unknown sentences adversely affect the classifier learning, and 2-step classification is an effective method.</Paragraph> <Paragraph position="13"> Table 4 shows the confusion matrix corresponding to the Method A (NORMAL). From this table, we can see Method A works well for classification of morning, daytime, evening, and night, but has some difficulty in associated words: &quot;Explicit&quot; indicates the number of sentences including explicit temporal expressions, &quot;NE-TIME&quot; indicates the number of sentences including NE-TIME tag.</Paragraph> <Paragraph position="14"> classification of time-unknown. The 11.7% of samples were wrongly classified into &quot;night&quot; or &quot;unknown&quot;.</Paragraph> <Paragraph position="15"> We briefly describe an error analysis. We found that our classifier tends to wrongly classify samples in which two or more events are written in a sentence. The followings are examples: Example 3 a. I attended a party last night, and I got back on the first train in this morning because the party was running over.</Paragraph> <Paragraph position="16"> b. I bought a cake this morning, and ate it after the dinner.</Paragraph> </Section> <Section position="2" start_page="1158" end_page="1159" type="sub_section"> <SectionTitle> 5.2 Examples of Time-Associated Words </SectionTitle> <Paragraph position="0"> Table 5 shows some time-associated words obtained by the proposed method. The words are sorted in the descending order of the value of ( )wcP |. Although some consist of two or three words, their original forms in Japanese consist of one word. There are some expressions appearing more than once, such as &quot;dinner&quot;. Actually these expressions have different forms in Japanese.</Paragraph> <Paragraph position="1"> phological analysis error are presented as the symbol &quot;---&quot;. We obtained a lot of interesting time-associated words, such as &quot;commute (morning)&quot;, &quot;fireworks (night)&quot;, and &quot;cocktail (night)&quot;. Most words obtained are significantly different from explicit temporal expressions and NE-TIME expressions.</Paragraph> <Paragraph position="2"> Figure 2 shows the number of sentences including time-associated words in blog text. The horizontal axis represents the number of time-associated words. We sort the words in the descending order of and selected the top N words. The vertical axis represents the number of sentences including any N-best time-associated words. We also show the number of sentences including explicit temporal expressions, and the number of sentences including NE-TIME tag (Sekine and Isahara, 1999) for comparison. The set of explicit temporal expressions was extracted by the method described in Section 5.1.2. We used a Japanese linguistic analyzer &quot;Cabo- null &quot; to obtain NE-TIME information. From this graph, we can confirm that the number of target sentences of our proposed method is larger than that of existing methods.</Paragraph> </Section> </Section> class="xml-element"></Paper>