<?xml version="1.0" standalone="yes"?>
<Paper uid="I05-3013">
  <Title>NIL Is Not Nothing: Recognition of Chinese Network Informal Language Expressions</Title>
  <Section position="7" start_page="99" end_page="101" type="evalu">
    <SectionTitle>
6 Experiments
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="99" end_page="99" type="sub_section">
      <SectionTitle>
6.1 Experiment Description
</SectionTitle>
      <Paragraph position="0"> We conduct experiments to evaluate the two methods in performing the task of NIL expression recognition. In training phase we use NIL corpus to construct NIL dictionary and pattern set for PM method, and generate statistical NIL features, support vectors and parameters for SVM methods. To observe how performance is influenced by the volume of training data, we create five NIL corpora, i.e. C#1~C#5, with five numbers of NIL sentences, i.e. 10,000, 13,000, 16,000, 19,000 and 22,432, by randomly selecting sentence from NIL corpus described in Section 4.1.</Paragraph>
      <Paragraph position="1"> To generate test set, we download 5,690 sentences from YESKY system which cover BBS text in March 2005. We identify and annotate NIL expressions within these sentences manually and consider the annotation results as gold standard.</Paragraph>
      <Paragraph position="2"> We first train the system with the five corpora to produce five versions of NIL dictionary, pattern set, statistical NIL feature set and SVM model. We then run the two methods with each version of the above knowledge over the test set to produce recognition results automatically. We compare these results against the gold stand and present experimental results with criteria including precision, recall and F1-measure.</Paragraph>
    </Section>
    <Section position="2" start_page="99" end_page="99" type="sub_section">
      <SectionTitle>
6.2 Experimental Results
</SectionTitle>
      <Paragraph position="0"> We present experimental results of the two methods on the five corpora in Table 3.</Paragraph>
    </Section>
    <Section position="3" start_page="99" end_page="100" type="sub_section">
      <SectionTitle>
6.3 Discussion I: The Two Methods
</SectionTitle>
      <Paragraph position="0"> To compare performance of the two methods, we present the experimental results with smoothed curves for precision, recall and F1-Mesure in Figure 3, Figure 4 and Figure 5 respectively.</Paragraph>
      <Paragraph position="1">  higher precision, i.e. 91.5%, and SVM produces higher recall, i.e. 79.3%, and higher F1-Measure, i.e. 87.1%, with corpus C#5. It can be inferred that PM method is self-restrained. In other words, if a NIL expression is identified with this method, it is very likely that the decision is right. However, the weakness is that more NIL expressions are neglected. On the other hand, SVM method outper- null forms PM method regarding overall capability, i.e. F1-Measure, according to Figure 5.</Paragraph>
      <Paragraph position="2">  We argue that each method holds strength and weakness. Different methods should be adopted to cater to different application demands. For example, in CRM text processing, we might favor precision. So PM method may be the better choice. On the other hand, to perform the task of chat room security monitoring, recall is more important. Then SVM method becomes the better option. We claim that there exists an optimized approach which combines the two methods and yields higher precision and better robustness at the same time.</Paragraph>
    </Section>
    <Section position="4" start_page="100" end_page="100" type="sub_section">
      <SectionTitle>
6.4 Discussion II: How Volume Influences Per-
</SectionTitle>
      <Paragraph position="0"> formance To observe how training corpus influences performance in the two methods regarding volume, we present experimental results with smoothed quality curves for the two method in Figure 6 and  training data leads to better processing quality. Meanwhile, the improvement tends to decrease along with increasing of volume. It thus predicts that there exists a corpus with a certain volume that produces the best quality according to the tendency. Although current corpus is not big enough to prove the optimal volume, the tendency revealed by the curves is obvious.</Paragraph>
    </Section>
    <Section position="5" start_page="100" end_page="101" type="sub_section">
      <SectionTitle>
6.5 Error Analysis
</SectionTitle>
      <Paragraph position="0"> We present two examples to analyze errors occur within our experiments.</Paragraph>
      <Paragraph position="1">  (mi3)&amp;quot; succeeds &amp;quot;8(ba1)&amp;quot; in the word segments, i.e. &amp;quot; |E |8|2G |,Q &amp;quot;, and it can be used as a unit word, PM method therefore refuses to identify &amp;quot;8(ba)&amp;quot; as a NIL expression according to the pattern described in Section 5.2.1. In fact, &amp;quot;2G,Q &amp;quot; is an unseen NIL expression. SVM method successfully recognizes &amp;quot;2G,Q &amp;quot; to be &amp;quot;2G (mi3 you3)&amp;quot;, thus recognizes &amp;quot;8&amp;quot;. In our experiments 56 errors in PM method suffer the same failure, while SVM method identifies 48 of them. This demonstrates that PM method is self-restrained and SVM method is relatively scalable in process- null Actually, there is no NIL expression in example 2. But because of a same 1-gram with &amp;quot;4D&amp;quot;, i.e. &amp;quot;4&amp;quot;, SVM outputs &amp;quot;4U&amp;quot; as a NIL expression. In fact, it is the name for a mobile dealer. There are 78 same errors in SVM method in our experiments, which reveals that SVM method is sometimes over-predicting. In other words, some NIL expressions are recognized with SVM method by mistake, which results in lower precision.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>