<?xml version="1.0" standalone="yes"?>
<Paper uid="I05-2003">
  <Title>A Hybrid Chinese Language Model based on a Combination of Ontology with Statistical Method</Title>
  <Section position="5" start_page="14" end_page="17" type="evalu">
    <SectionTitle>
3 Evaluation of language model
</SectionTitle>
    <Paragraph position="0"> We completed two groups of experiments on text re-ranking for information retrieval, text similarity computing to verify the performance of lingual ontology knowledge.</Paragraph>
    <Section position="1" start_page="14" end_page="16" type="sub_section">
      <SectionTitle>
3.1 Texts reordering
</SectionTitle>
      <Paragraph position="0"> Information retrieval is used to retrieve relevant documents from a large document set for a user query, where the user query can be a simple description by natural. As a general rule, users hope more to acquire relevant information from the top ranking documents, so they concern more on the precision of top ranking documents than the recall.</Paragraph>
      <Paragraph position="1"> We use the Chinese document set CIRB011 (132,173 documents) and CIRB020 (249,508 documents) from NTCIR3 CLIR dataset and select 36 topics from 50 search topics (see http://research.nii.ac.jp/ntcir-ws3/work-en.html for more information) to evaluate our method.</Paragraph>
      <Paragraph position="2"> We use the same method to retrieve documents mentioned by Yang LingpengP [12] P, i.e. we use vector space model to retrieve documents, use cosine to calculate the similarity between document and user query. We respectively use bi-grams and words as indexing unitsP [13,14] P, the average precision of top N ranking documents acts as the normal results. In this paper, we used a Chinese dictionary that contains about 85,000 items to segment Chinese document and query. To measure the effectiveness of information retrieval, we use the same two kinds of relevant measures: relax-relevant and rigidrelevantP null [14,15] P. A document is rigid-relevant if it's highly relevant or relevant with user query, and a document is relax-relevant if it is high relevant or relevant or partially relevant with user query. We also use PreAt10 and PreAt100 to represent the precision of top 10 ranking documents and top 100 ranking documents.</Paragraph>
      <Paragraph position="3">  First, we get some keywords to every topic by query description. For example, Title: Ke Long Zhi Dan Sheng (The birth of a cloned calf) Description: Cha Xun Yu Shi Yong Bei Cheng Wei Ti Xi Bao Yi Zhi De Ji Zhu Chuang Zao Ke Long Niu Xiang Guan De Wen Zhang (Find Articles relating to the birth of cloned calves using the technique called somatic cell nuclear transfer) We extract &amp;quot;Ke Long , Ti Xi Bao , Yi Zhi , Wu Xing Fan Zhi &amp;quot; as feature word in this topic.</Paragraph>
      <Paragraph position="4"> Second, acquire lingual ontology knowledge every topic by their feature words. In this proposal, we arrange 300 Chinese texts of this topic as learning corpus to get lingual ontology knowledge bank.</Paragraph>
      <Paragraph position="5"> Third, get the evaluation value of every text about this topic, i.e. respectively add up all the average co-occurrence distance l C to the same semantic relation pairs in every text from lingual ontology knowledge bank.</Paragraph>
      <Paragraph position="6"> If a text has several keywords, repeat step3 to acquire every evaluation value to these keywords, and then, add up each evaluation value to act as the text evaluation value.</Paragraph>
      <Paragraph position="7"> Final, we reorder the initial retrieval texts according to the every text evaluation value of every topic.</Paragraph>
      <Paragraph position="8">  We calculate the evaluation value of every text in each topic to reorder the initial relevant documents.</Paragraph>
      <Paragraph position="9"> Table 3 lists the normal results and our results based on bi-gram indexing, our results are acquired based on Chinese lingual ontology knowledge to enhance the effectiveness.</Paragraph>
      <Paragraph position="10"> PreAt10 is the average precision of 36 topics in precision of top 10 ranking documents, while PreAt100 is top 100 ranking documents.</Paragraph>
      <Paragraph position="11"> Table 4 lists the normal results and our results based on word indexing. Ratio displays an increase ratio of our result compared with normal result.</Paragraph>
      <Paragraph position="12">  In table 3, it is shown that compared with bi-grams as indexing units, our method respectively increases 18.49% in relax relevant measure and 17.45% in rigid in PreAt10. In PreAt100 level, our method respectively increases 15.35% in relax relevant and 12.65% in rigid relevant measure. Figure 3 displays the PreAt10 values of each topic in relax relevant measure based on bi-gram indexing where one denotes the precision enhanced with our method, another denotes the normal precision. It is shown the precision of  In table 4, using words as indexing units, our method respectively increases 17.03% in relax relevant measure and 15.45% in rigid in PreAt10. In PreAt100 level, our method respectively increases 14.05% in relax relevant measure and 11.96% in rigid.</Paragraph>
      <Paragraph position="13"> In our experiments, compared with the two Chinese indexing units: bi-gram and words, our method increases the average precision of all queries in top 10 and top 100 measure levels for about 17.1% and 13.5%. What lies behind our method is that for each topic, we manually select some Chinese corpus to acquire the lingual ontology knowledge, and can help us to focus on relevant documents. Our experiment also shows improper extract and corpus may decrease the precision of top documents. So our method depends on right keywords in texts, queries and the corpus.</Paragraph>
    </Section>
    <Section position="2" start_page="16" end_page="17" type="sub_section">
      <SectionTitle>
3.2 Text similarity computing
</SectionTitle>
      <Paragraph position="0"> Text similarity is a measure for the matching degree between two or more texts, the more high the similarity degree is, the more the meaning of text expressing is closer, vice versa. Some proposal methods include Vector Space ModelP  B, we respectively extract k same feature words, if the same feature words in the two texts is less than k, we don't compare their similarity.</Paragraph>
      <Paragraph position="1"> Second, acquire lingual ontology knowledge every text by their feature words. Third, get the evaluation value of every text, i.e. respectively add up all the average co-occurrence distance l C to the same semantic relation pairs in two texts.</Paragraph>
      <Paragraph position="2"> Final, compute the similarity ratio of every  We download four classes of text for testing from Sina, Yahoo, Sohu and Tom, which include 71 current affairs news, 68 sports news, 69 IT news, 74 education news.</Paragraph>
      <Paragraph position="3"> For the test of current affairs texts, according to the strategy of similarity computation, we choose five words as feature word. They are &amp;quot;Mao Yi , Xie Yi , Tan Pan , Zhong Guo , Mei Guo &amp;quot;. In the texts, the word &amp;quot;Jing Mao , Shang Mao &amp;quot; are all replaced by word &amp;quot;Mao Yi &amp;quot; and other classes are similar. The testing result is shown in table 5.</Paragraph>
      <Paragraph position="4">  We analyzed all the experimental results to find that the results for current affairs texts are the best, while the sports texts are lower than others. We think it is mainly because some sports terms are unprofessional for the lower sports texts recognition, such as &amp;quot;Yi Jia Jun , Jiu Zhu , Hao Dong &amp;quot;. Other feature words are more fixed and more concentrated.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>