<?xml version="1.0" standalone="yes"?>
<Paper uid="C02-1151">
  <Title>Probabilistic Reasoning for Entity &amp; Relation Recognition/</Title>
  <Section position="5" start_page="13" end_page="13" type="evalu">
    <SectionTitle>
4 Experiments
</SectionTitle>
    <Paragraph position="0"> The following subsections describe the data preparation process, the approaches tested in the experiments, and the experimental results.</Paragraph>
    <Section position="1" start_page="13" end_page="13" type="sub_section">
      <SectionTitle>
4.1 Data Preparation
</SectionTitle>
      <Paragraph position="0"> In order to build different datasets, we first collected sentences from TREC documents, which are mostly daily news such as Wall Street Journal, Associated Press, and San Jose Mercury News. Among the collected sentences, 245 sentences contain relation kill (i.e. two entities that have the murder-victim relation). 179 sentences contain relation born in (i.e. a pair of entities where the second is the birthplace of the first). In addition to the above sentences, we also collected 502 sentences that contain no relations.2  ple rule: consecutive proper nouns and commas are combined and treated as an entity. Predefined entity class labels include other ent, person, and location. Moreover, relations are defined by every pair of entities in a sentence, and the relation class labels defined are other rel, kill, and birthplace.</Paragraph>
      <Paragraph position="1"> Three datasets are constructed using the collected sentences. Dataset &amp;quot;kill&amp;quot; has all the 245 sentences of relation kill. Dataset &amp;quot;born in&amp;quot; has all the 179 sentences of relation born in. The third dataset &amp;quot;all&amp;quot; mixes all the sentences.</Paragraph>
    </Section>
    <Section position="2" start_page="13" end_page="13" type="sub_section">
      <SectionTitle>
4.2 Tested Approaches
</SectionTitle>
      <Paragraph position="0"> We compare three approaches in the experiments: basic, omniscient, and BN. The first approach, basic, tests our baseline - the performance of the basic classifiers. As described in Section 3.1, these classifiers are learned independently using local features and make predictions on entities and relations separately. Without taking global interactions into account, the features extracted are described as follows. For the entity classifier, features from the words around each entity are: words, tags, conjunctions of words and tags, bigram and trigram of words and tags. Features from the entity itself include the number of words it contains, bigrams of words in it, and some attributes of the words inside such as the prefix and suffix. In addition, whether the entity has some strings that match the names of famous people and places is also used as a feature. For the relation classifier, features are extracted from words around and between the two entity arguments. The types of features include bigrams, trigrams, words, tags, and words related to &amp;quot;kill&amp;quot; and &amp;quot;birth&amp;quot; retrieved from WordNet.</Paragraph>
      <Paragraph position="1"> The second approach, omniscient, is similar to basic.</Paragraph>
      <Paragraph position="2"> The only difference here is the labels of entities are revealed to the R classifier and vice versa. It is certainly impossible to know the true entity and relation labels in advance. However, this experiment may give us some ideas about how much the performance of the entity classifier can be enhanced by knowing whether the target is involved in some relations, and also how much the relation classifier can be benefited from knowing the entity labels of its arguments. In addition, it also provides a comparison to see how well the belief network inference model can improve the results.</Paragraph>
      <Paragraph position="3"> The third approach, BN, tests the ability of making global inferences in our framework. We use the Bayes Net Toolbox for Matlab by Murphy 3 to implement the network and set the maximum number of the iteration of belief propagation algorithm as 20. Given the probabilities estimated by basic classifiers, the network infers the labels of the entities and relations globally in a sentence.</Paragraph>
      <Paragraph position="4"> Compared to the first two approaches, where some predictions may violate the constraints, the belief network model incorporates the constraints between entities and  domly separated into 5 disjoint subsets, and experiments are done 5 times by iteratively using 4 of them as training data and the rest as testing.</Paragraph>
    </Section>
    <Section position="3" start_page="13" end_page="13" type="sub_section">
      <SectionTitle>
4.3 Results
</SectionTitle>
      <Paragraph position="0"> The experimental results in terms of recall, precision,and Ffl=1 for datasets &amp;quot;kill&amp;quot;, &amp;quot;born in&amp;quot;, and &amp;quot;all&amp;quot; are given in Table 1, Table 2, and Table 3 respectively. We discuss two interesting facts of the results as follows.</Paragraph>
      <Paragraph position="1"> First, the belief network approach tends to decrease recall in a small degree but increase precision significantly.</Paragraph>
      <Paragraph position="2"> This phenomenon is especially clear on the classification results of some relations. As a result, the F1 value of the relation classification results is still enhanced to the extent that is near or even higher than the results of the Omniscient approach. This may be explained by the fact that if the label of a relation is predicted as positive (i.e.</Paragraph>
      <Paragraph position="3"> not other rel), the types of its entity arguments must satisfy the constraints. This inference process reduces the number of false positive, thus enhance the precision.</Paragraph>
      <Paragraph position="4"> Second, knowing the class labels of relations does not seem to help the entity classifier much. In all three datasets, the difference of Basic and Omniscient approaches is usually less than 3% in terms of F1, which is not very significant given the size of our datasets. This phenomenon may be due to the fact that only a few of entities in a sentence are involved in some relations. Therefore, it is unlikely that the entity classifier can use the relation information to correct its prediction.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>