<?xml version="1.0" standalone="yes"?>
<Paper uid="P06-2046">
  <Title>Japanese Idiom Recognition: Drawing a Line between Literal and Idiomatic Meanings</Title>
  <Section position="6" start_page="356" end_page="358" type="evalu">
    <SectionTitle>
4 Evaluation
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="356" end_page="357" type="sub_section">
      <SectionTitle>
4.1 Experiment Condition
</SectionTitle>
      <Paragraph position="0"> We conducted an experiment to see the effectiveness of the lexical knowledge we proposed.</Paragraph>
      <Paragraph position="1"> As an evaluation corpus, we collected 300 example sentences of the 100 idioms from Mainichi newspaper of '95: three sentences for each idiom. Then we added another nine sentences for three idioms that are orthographic variants of one of the 100 idioms. Among the three idioms, one belonged to Class B and the other two belonged to Class C. Thus, 67 out of the 103 idioms were Class B and the other 36 were Class C. After all, 309  We found that the most frequently used 100 idioms in Kindaichi and Ikeda (1989) cover as many as 53.49% of all tokens in Mainichi newspaper of 10 years. This implies that our dictionary accounts for approximately half of all idiom tokens in a corpus.</Paragraph>
      <Paragraph position="2">  sentences were prepared. Table 2 shows the breakdown of them. &amp;quot;Positive&amp;quot; indicates sentences in- null cluding a true idiom, while &amp;quot;Negative&amp;quot; indicates those including a literal-usage &amp;quot;idiom.&amp;quot; A baseline system was prepared to see the effect of the disambiguation knowledge. The base-line system was the same as the recognizer except that it exploited no disambiguation knowledge.</Paragraph>
    </Section>
    <Section position="2" start_page="357" end_page="357" type="sub_section">
      <SectionTitle>
4.2 Result
</SectionTitle>
      <Paragraph position="0"> The result is shown in Table 3. The left side shows the performances of the recognizer, while the right side shows that of the baseline. Differences of performances between the two systems are marked with bold. Recall, Precision, and F-Measure, are calculated using the following equations.</Paragraph>
      <Paragraph position="1">  As a result, more than 90% of the idioms can be recognized with 90% accuracy. Note that the recognizer made fewer errors due to the employment of the disambiguation knowledge.</Paragraph>
      <Paragraph position="2"> The result shows the high performances. However, there turns out to be a long way to go to solve the most difficult problem of idiom recognition: drawing a line between literal and idiomatic meanings. In fact, the precision of recognizing idioms of Class C remains less than 70% as in Table 3.</Paragraph>
      <Paragraph position="3"> Besides, the recognizer successfully rejected only 15 out of 42 negative sentences. That is, its success rate of rejecting negative ones is only 35.71%</Paragraph>
    </Section>
    <Section position="3" start_page="357" end_page="358" type="sub_section">
      <SectionTitle>
4.3 Discussion of the Disambiguation
Knowledge
</SectionTitle>
      <Paragraph position="0"> First of all, positive sentences, i.e., sentences containing true idioms, are in the blank region of Figure 2, while negative ones, i.e., those containing literal phrases, are in both regions. Accordingly, the disambiguation amounts to i) rejecting negative ones in the shaded region, ii) rejecting negative ones in the blank region, or iii) accepting positive ones in the blank region. i) is relatively easy since there are visible evidences in a sentence that tell us that it is NOT an idiom. However, ii) and iii) are difficult due to the absence of visible evidences. Our method is intended to perform i), and thus has an obvious limitation.</Paragraph>
      <Paragraph position="1"> Next, we look cloosely at cases of success or failure of rejecting negative sentences. There were 15 cases where rejection succeeded, which correspond to i). The disambiguation knowledge that contributed to rejection and the number of sentences it rejects are as follows.</Paragraph>
      <Paragraph position="2">  1. Genitive Phrase Prohibition (6aII) .......6 2. Relative Clause Prohibition (6aI) ........5 3. Detachment Constraint (6e) .............2 4. Negation Prohibition (6dI) ..............1  This shows that the Adnominal Modification Constraints, 1. and 2. above, are the most effective. There were 27 cases where rejection failed.</Paragraph>
      <Paragraph position="3"> These are classified into two types:  There was one case where rejection succeeded due to the dependency analysis error.</Paragraph>
      <Paragraph position="4">  1. Those that could have been rejected by the Selectional Restriction (6f) ..............5 2. Those that might be beyond the current technology ...............................22 1. and 2. correspond to i) and ii), respectively.  We see that the Selectional Restriction would have been as effective as the Adnominal Modification Constraints. A part of a sentence that the knowledge could have rejected is below.</Paragraph>
      <Paragraph position="5">  &amp;quot;The bus floated in midair.&amp;quot; An idiom, tyuu-ni uku (midair-DAT float) &amp;quot;remain to be decided,&amp;quot; takes as its argument something that can be decided, i.e., &lt;1000:abstract&gt; rather than &lt;2:concrete&gt; in the sense of the Goi-Taikei ontology (Ikehara et al., 1997). Thus, (10) has no idiomatic sense.</Paragraph>
      <Paragraph position="6"> A simplified example of 2. is illustrated in (11).  &amp;quot;It makes more sense to be naked than wearing clothes in a sweat.&amp;quot; The phrase ase-o nagasu (sweat-ACC shed) could have been an idiom meaning &amp;quot;work hard.&amp;quot; It is contextual knowledge that prevented it from being the idiom. Clearly, our technique is unable to handle such a case, which belongs to ii), since no visible evidence is available. Dealing with that might require some sort of machine learning technique that exploits contextual information. Exploring that possibility is one of our future works.</Paragraph>
      <Paragraph position="7"> Finally, the 42 negative sentences consist of 15 sentences, which we could disambiguate, 5 sentences, which Selectional Restriction could have disambiguated, and 22, which belong to ii) and are beyond the current technique. Thus, the real challenge lies in 7% (  ) of all idiom occurrences.</Paragraph>
    </Section>
    <Section position="4" start_page="358" end_page="358" type="sub_section">
      <SectionTitle>
4.4 Discussion of the Dependency Knowledge
</SectionTitle>
      <Paragraph position="0"> The dependency knowledge failed in only five cases. Three of them were due to the defect of dealing with case particles' change like omission. The other two cases were due to the noun constituent's incorporation into a compound noun.</Paragraph>
      <Paragraph position="1">  &amp;quot;(Economics) get back on a recovery track.&amp;quot; The idiom, kidou-ni noru (orbit-DAT ride) &amp;quot;get on track,&amp;quot; has a constituent, kidou, which is incorporated into a compound noun kaihuku-kidou &amp;quot;recovery track.&amp;quot; This is unexpected and cannot be handled by the current machinery.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>