<?xml version="1.0" standalone="yes"?>
<Paper uid="W05-0306">
  <Title>Investigating the Characteristics of Causal Relations in Japanese Text</Title>
  <Section position="4" start_page="37" end_page="39" type="metho">
    <SectionTitle>
3 Annotated information
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="37" end_page="38" type="sub_section">
      <SectionTitle>
3.1 Causal relation tags
</SectionTitle>
      <Paragraph position="0"> We use three tags head, mod, and causal rel to represent the basic causal relation information. Our annotation scheme for events is similar to that of the PropBank (Palmer et al., 2005). An event is regarded as consisting of a head element and some modi ers. The tags head and mod are used to represent an event which forms one part of the two events held in a causal relation. The tag causal rel is used to represent a causal relation between two annotated events.</Paragraph>
      <Paragraph position="1"> Figure 1 shows an example of attaching the causal relation information to the sentence (2a), in which a causal relation is held between two events indicated  (2b) and (2c) . Hereafter, we denote the former (cause) part of event as e1 and the latter (effect) part of event as e2.</Paragraph>
      <Paragraph position="2"> (2) a. a21a23a22a25a24a27a26a14a28a30a29a32a31a34a33a25a35a37a36a39a38a41a40a43a42a45a44a25a46a48a47a37a49a23a50a51a24a27a52a54a53a43a55a14a56 (As the Golden Week holidays come, the number of sightseers from all over begins to increase.)  The annotation process is executed as follows.</Paragraph>
      <Paragraph position="3"> First, each sentence in the text is split to some bunsetsu-phrase chunks1, as shown in Figure 1 ( / indicates a bunsetsu-phrase chunk boundary). Second, for each bunsetsu-phrase, an annotator nds the seg- null Japanese, which consists of a content word (noun, verb, adjective, etc.) accompanied by some function words (particles, auxiliaries, etc.).</Paragraph>
      <Paragraph position="4">  and he/she adds the head tag to the segment (see also head1 and head2 in Figure 1). If the event has any other elements in addition to head element, the annotator also adds the mod tags to the segments representing modi ers to the head element (mod1 and mod2 in Figure 1). The elements marked with any tags which have a common suf x number are constituents of the same event: that is, the elements marked with head1 and mod1 tags are constituents of e1 and the elements marked with head2 and mod2 are constituents of e2. Finally, the annotator adds the causal rel tag between head1 and head2 as link information which indicates that the corresponding two events are held in a causal relation.</Paragraph>
      <Paragraph position="5"> When there are any cue phrase markers helpful in recognizing causal relations such as a122a124a123 (because) in (1a) , the annotator also adds the marker tag to their segments.</Paragraph>
    </Section>
    <Section position="2" start_page="38" end_page="39" type="sub_section">
      <SectionTitle>
3.2 Annotation criteria
</SectionTitle>
      <Paragraph position="0"> To judge whether two events represented in text are held in a causal relation or not, we apply new criteria based on linguistic test.</Paragraph>
      <Paragraph position="1"> The linguistic test is a method for judging whether target linguistic expressions conforms to a given set of rules. In our cases, the target expressions are two sets of bunsetsu-phrase chunks. Each set represents as a whole an event which can be an argument in a causal relation, such as in (2b) and (2c) . The rules are realized as linguistic templates which are linguistic expressions including several slots.</Paragraph>
      <Paragraph position="2"> In practice, a linguistic test is usually applied using the following steps:  1. Preparing a template.</Paragraph>
      <Paragraph position="3"> 2. Embedding the target expressions in the slots of the template to form a candidate sentence.</Paragraph>
      <Paragraph position="4"> 3. If the candidate sentence is syntactically and semantically correct, the target expressions are  judged to conform to the rules. If the candidate sentence is incorrect, the targets are judged non-conforming.</Paragraph>
      <Paragraph position="5"> In this work, we prepared eighteen linguistic templates such as in Figure 2. The square brackets indicate the slots. The symbol &lt;adv&gt; is replaced by one of three adverbs a125a23a126a127a125a23a126 (often),a128a130a129 (usually), or  We embed two target expressions representing events in the slots of the template to form a candidate sentence. Then, if an annotator can recognize that the candidate sentence is syntactically and semantically correct, the causal relation is supposed to hold between two events. In contrast, if recognized that the candidate sentence is incorrect, this template is rejected, and the other template is tried. If all eighteen templates are rejected by the annotator, it is supposed that there is no causal relations between these two events. Note that the annotator's recognition of whether the candidate sentence is correct or incorrect, in other words, whether a causal relation is held between the two events embedded in the candidate sentence or not, is not really relevant to the author's intention.</Paragraph>
      <Paragraph position="6"> The fundamental idea of our criteria based on linguistic test is similar to that of the criteria for annotation of implicit connectives adopted in PDTB corpus2. In the annotation process of the PDTB corpus, an annotator judges whether or not the explicit connective, for example, because , relates two linguistic expressions representing events. This process is essentially the same as ours.</Paragraph>
      <Paragraph position="7"> Three adverbs in the linguistic templates, a125a34a126a150a125</Paragraph>
      <Paragraph position="9"> dicate a pragmatic constraint on the necessity of the relationship between any two events; the relations indicated by these words usually have a high degree of necessity. With this pragmatic constraint, we introduce an attribute to the causal rel tags about the degree of necessity. For each of eighteen templates, if one judges the two target expressions as holding a causal relation by using the template with one of three adverbs, the necessity attribute value is added to the relation instance. If one judges the two target expressions as holding a causal relation by using the template deleting &lt;adv&gt; , three adverbs, the chance  attribute value is added.</Paragraph>
      <Paragraph position="10"> We assume that a target expression embedded in the slot is represented by a single sentence. If an event is represented by noun phrase (NP), the following rewriting rules are applied before embedded to the slot to transform the NP into a single sentence.  If a head element of a target expression representing an event is conjugated, the head element is replaced by its base form before embedded to the slot.</Paragraph>
    </Section>
    <Section position="3" start_page="39" end_page="39" type="sub_section">
      <SectionTitle>
3.3 Annotation ranges
</SectionTitle>
      <Paragraph position="0"> Ideally, we should try to judge for tagging of the causal relation tags over all any event pairs in text.</Paragraph>
      <Paragraph position="1"> However, it seems that the more the distance between two events represented in text, the smaller the probability of holding a causal relation between them. Thus, we set a constraint on the ranges of judgements. If both two events are represented in the same sentence or two sentences adjacent to each other, we try judgements, if not, skip judgements. This constraint is applied only when tagging the head tag. A modi er and its head element are sometimes located in different sentences overtly in Japanese text when anaphora or ellipsis phenomenon occurs. In such cases, we add mod tags to the text segments anywhere in the text.</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="39" end_page="39" type="metho">
    <SectionTitle>
4 Data
</SectionTitle>
    <Paragraph position="0"> We selected as text for annotation Mainichi Shimbun newspaper articles (Mainichi, 1995). In particular, we used only articles included on the social aspect domain. When adding the causal relation tags to the text, it is preferable that each annotator can understand the whole contents of the articles. The contents of social aspect domain articles seems to be familiar to everybody and are easier to understand than the contents of articles included on politics, economy domain, etc.</Paragraph>
    <Paragraph position="1"> Furthermore, in our previous examination, it is found that as the length of articles gets longer, it is getting hard to judge which bunsetsu-phrase chunks represent as a whole an event. This is because as described in Section 3.3, annotators sometimes need to search several sentences for modi ers of the head element in order to add mod tags precisely. Therefore, we focus on social aspect domain articles which consists of less than or equal to 10 sentences. After all, we extracted 750 articles (3912 sentences) for our annotation work with above conditions.</Paragraph>
  </Section>
  <Section position="6" start_page="39" end_page="40" type="metho">
    <SectionTitle>
5 Annotation work ow
</SectionTitle>
    <Paragraph position="0"> Three annotators have been employed. Each annotator has added tags to the same 750 document articles independently. Two annotators of the three are linguists, and the last one is the author of this paper. We denote each annotator under anonymity, A, B and C. After training phase for annotators, we spent approximately one month to create a corpus annotated with causal relation information. The annotation work ow is executed ef ciently using an annotation interface. Using the interface, all of annotators can add tags through only simple keyboard and mouse operations. The annotation work ow is as follows.</Paragraph>
    <Paragraph position="1"> I. Annotation phase: A document article is displayed to each annotator. The sentences in the document are automatically split to bunsetsu-phrases by preprocessing. Some kinds of words such as connectives and verbs are highlighted to draw annotators' attention to the text segments which could represent elements in causal relation instances. The annotator nds text segments which represent causal relation instances, and then he/she adds the causal relation tags to their segments as described in Section 3.</Paragraph>
    <Paragraph position="2"> II. Modi cation phase: After each annotator nished the annotation phase for a xed number of document articles (in this work, 30 document articles), he/she moves to a modi cation phase. In this phase, rst, only the segments with causal relation tags are extracted from the documents such as instances in Table 1. Then,  the same annotator who adds tags to the extracted segments, checks their extracted causal relation instances with attention. Since the extraction is done automatically, each annotator can check all the segments to be checked. When wrong tagged instances are found, they are corrected on the moment. After checking and correcting for all the extracted instances, the annotator moves back to the annotation phase in order to annotate a new 30 document articles set.</Paragraph>
  </Section>
  <Section position="7" start_page="40" end_page="41" type="metho">
    <SectionTitle>
6 Results
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="40" end_page="40" type="sub_section">
      <SectionTitle>
6.1 Total number of tagged instances
</SectionTitle>
      <Paragraph position="0"> 2014 instances were tagged by the annotator A, 1587 instances by B, 1048 instances by C. Some examples of tagged instances are shown in Table 1.</Paragraph>
      <Paragraph position="1"> The total numbers of tagged instances of the three annotators are quite different. Although all annotators tagged under the same annotation criteria, the annotator A tagged to twice as many segments as the annotator C did. Though this difference may be caused by some factors, we assume that the difference is mainly caused by missing judgements, since the annotators added tags to a variety of linguistic expressions, especially expressions without cue phrases.</Paragraph>
      <Paragraph position="2"> To verify the above assumption, we again asked each annotator to judge whether or not a pair of linguistic expressions representing events is holding a causal relation. In this additional work, in order to prevent the annotators from skipping judgement itself, we present beforehand to the annotators the pairs of linguistic expressions to be judged. We presented a set of 600 pairs of linguistic expressions to each of the three annotators. All of these pairs are  the causal relation instances already tagged by one or more annotators in the main work described in the previous sections.</Paragraph>
      <Paragraph position="3"> From the comparison between the results of the additional work and those of the main work, we found that if causal relation instances are expressed without explicit cues in text, they tend to be more frequently missed than those with explicit cues. The missing judgements on expressions without explicit cues are an important issue in the realization of more sophisticated analyses.</Paragraph>
    </Section>
    <Section position="2" start_page="40" end_page="41" type="sub_section">
      <SectionTitle>
6.2 Inter-annotator agreement
</SectionTitle>
      <Paragraph position="0"> We examined inter-annotator agreement. First, we de ne an agreement measure between two relation instances. Let x and y be causal relation instances tagged by two different annotators. The instance x consists of e1x and e2x, and y consists of e1y and e2y. The event e1x has head1x as its head element. Similarly, head2x, head1y and head2y are the head elements corresponding respectively to events e2x, e1y and e2y. Then, we regard two instances x and y as the same instance, when head1x and head1y are located in the same bunsetsu-phrase and head2x and head2y are also located in the same bunsetsuphrase. Using the above de ned agreement measure,  we counted the number of instances tagged by the different annotators.</Paragraph>
      <Paragraph position="1"> Table 2 shows the results. The symbol 1 in the left-hand side of Table 2 indicates that the corresponding annotator tagged to instances, and the 0 indicates not tagged. For example, the fourth row ( 110 ) indicates that both A and B tagged to instances but C did not.</Paragraph>
      <Paragraph position="2"> Let Smixed denote a set of all tagged instances, Sn denote a set of all tagged instances with the necessity attribute value, and Sc denote a set of all tagged instances with the chance attribute value.</Paragraph>
      <Paragraph position="3"> First, we focus on the relation instances in the set Smixed. The 1233 (= 372 + 133 + 140 + 588) instances are tagged by more than one annotator, and the 588 instances are tagged by all three annotators. Next, we focus on the two different contrastive sets of instances, Sn and Sc. The ratio of the instances tagged by more than one annotator is small in Sc.</Paragraph>
      <Paragraph position="4"> This becomes clear when we look at the bottom row ( 111 ). While the 270 instances are tagged by all three annotators in Sn, only the 64 instances are tagged by all three annotators in Sc.</Paragraph>
      <Paragraph position="5"> To statistically con rm this difference, we applied the hypothesis test of the differences in population rates. The null hypothesis is that the difference of population rate is d %. As a result, the null hypothesis was rejected at 0.01 signi cance level when d was equal or less than 7 (p-value was equal or less than 0.00805). In general, it can be assumed that if a causal relation instance is recognized by many annotators, the instance is much reliable. Based on this assumption and the results in Table 2, reliable instances are more concentrated on the set of instances with the necessity attribute value than those with the chance attribute value.</Paragraph>
    </Section>
  </Section>
  <Section position="8" start_page="41" end_page="43" type="metho">
    <SectionTitle>
7 Discussion
</SectionTitle>
    <Paragraph position="0"> In this section, we discuss some characteristics of in-text causal relations and suggest some points for developing the knowledge acquisition methods for causal relations. Here, to guarantee the reliability of the data used for the discussion, we focus on the 699 (= 230 +92 + 107 + 270) instances marked by more than one annotator with the necessity attribute value. We examined the following three parts: (i) cue phrase markers, (ii) the parts-of-speech of head elements, and (iii) the positions of head elements.</Paragraph>
    <Section position="1" start_page="41" end_page="41" type="sub_section">
      <SectionTitle>
7.1 Cue phrase markers
</SectionTitle>
      <Paragraph position="0"> While annotating the document articles with our causal relation tags, head, mod, and causal rel, the annotators also marked the cue phrase markers for causal relations with the marker tag at the same time. We investigated a proportion of instances attached with the marker tag.</Paragraph>
      <Paragraph position="1"> The result is shown in Table 3. Table 4 shows the cue phrase markers actually marked by at least one annotator 3.</Paragraph>
      <Paragraph position="2"> It has been supposed that causal relation instances are sometimes represented with no explicit cue phrase marker. We empirically con rmed the supposition. In our case, only 30% of our 699 instances have one of cue phrase markers shown in Table 4, though this value can be dependent of the data.</Paragraph>
      <Paragraph position="3"> This result suggests that in order to develop knowledge acquisition methods for causal relations with high coverage, we must deal with linguistic expressions with no explicit cue phrase markers as well as those with cue phrase markers.</Paragraph>
    </Section>
    <Section position="2" start_page="41" end_page="42" type="sub_section">
      <SectionTitle>
7.2 The parts-of-speech of head elements
</SectionTitle>
      <Paragraph position="0"> Next, we classi ed the events included in the 699 instances into two syntactic categories: the verb phrase (VP) and the noun phrase (NP). To do this, we used morphological information of their head elements. If the part-of-speech of a head is verb or adjective, the event is classi ed as a verb phrase. If  the part-of-speech of a head is noun (including general noun and verbal noun), the event is classi ed as a noun phrase. We used ChaSen 4 to get part-of-speech information.</Paragraph>
      <Paragraph position="1"> The result is shown in Table 5. More than half events are classi ed as the VP. This matches our intuition. However, the number of events classi ed as the NP is comparable to the number of events classi ed as the VP; 322 events of e1 are represented as noun phrases, and 269 events of e2 are also represented as noun phrases.</Paragraph>
      <Paragraph position="2"> This result is quite suggestive. To promote the current methods for knowledge acquisition to further stage, we should develop a knowledge acquisition framework applicable both to the verb phrases and to the noun phrases.</Paragraph>
    </Section>
    <Section position="3" start_page="42" end_page="43" type="sub_section">
      <SectionTitle>
7.3 The positions of head elements
</SectionTitle>
      <Paragraph position="0"> For each e1 and e2 included in the 699 instances, we examined the positions of their head elements in the sentences.</Paragraph>
      <Paragraph position="1"> We consider dependency structures between bunsetsu-phrases in the original sentences from which causal relation instances are extracted. The dependency structures form tree structures. The bunsetsu-phrase located in the end of the sentence is the root node of the tree. We focus on the depth of the head element from the root node. We used CaboCha5 to get dependency structure information between bunsetsu-phrases. null The results are shown in Figure 3 and Figure 4.</Paragraph>
      <Paragraph position="2"> Figure 3 is the result for the head elements of e1, and Figure 4 is the result for the head elements of e2. The letter f in Figure 3 and Figure 4 indicates frequency at each position. Similarly, the letter c  indicates cumulative frequency.</Paragraph>
      <Paragraph position="3"> In Figure 4, the 198 head elements of the events represented as a verb phrase are located in the end of the sentences, namely depth = 0. The 190 of the 269 events represented as a noun phrase are located in depth = 1. For events represented as either a verb phrase or a noun phrase, over 80% of head elements of the events are located within depth &lt; 3. In Figure 3, similarly, over 80% of head elements of the events are located within depth &lt; 4.</Paragraph>
      <Paragraph position="4"> These ndings suggest that the most of the events are able to be found simply by searching the bunsetsu-phrases located in the shallow position at the phase of causal knowledge acquisition.</Paragraph>
      <Paragraph position="5"> 7.4 Relative positions of two head elements Finally, we examined relative positions between head elements of e1 and e2 where these two events are held in a causal relation. In Section 7.3, we discussed each absolute position for e1 and e2 by means of the notion of depth in sentences. Here, we focus on the difference (D) of the depth values between e1 and e2.</Paragraph>
      <Paragraph position="6"> The result is shown in Table 6. The symbol e1= e2 in Table 6 indicates the case where the head element of e1 is located nearer to the beginning of the</Paragraph>
      <Paragraph position="8"> = 2 152 23 &gt; 2 33 4 no dep 72 inter-sentential 141 sentence than that of e2. The e2= e1 indicates the opposite case. The symbol no dep indicates the case where neither the condition a nor b is satis ed: a. the head element of e2 is an ancestor of the head element of e1.</Paragraph>
      <Paragraph position="9"> b. the head element of e2 is a descendant of the head element of e1.</Paragraph>
      <Paragraph position="10"> The symbol inter-sentential indicates the case where two head elements appear in different sentences. null The most instances a214 259 instancesa215 are categorized into D = 1 on e1= e2, that is, the head element of e1 directly depends on the head element of e2. This result matches our intuition. However, there are several other cases. For example, 152 instances are categorized into D = 2 on e1= e2, 72 instances are categorized into no dep . Most of the instances extracted from sentences including any parallel relations are categorized into no dep . In this study, we consider causal relation instances as binary relation. To deal with instances categorized into no dep adequately, we should extend our framework to the more complex structure.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>