<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-1656">
  <Title>Boosting Unsupervised Relation Extraction by Using NER</Title>
  <Section position="8" start_page="477" end_page="480" type="evalu">
    <SectionTitle>
4 Experimental Evaluation
</SectionTitle>
    <Paragraph position="0"> Our experiments aim to answer three questions:  1. Can we train URES's classifier once, and then use the results on all other relations? 2. What boost will we get by introducing a simple NER into the classification scheme of URES? 3. How does URES's performance compare with KnowItAll and KnowItAll-PL?  sense that the order of its attributes does not matter. Acquisition is antisymmetric, and the other three are tested as bound in the first  attribute. For the bound predicates, we are only interested in the instances with particular prespecified values of the first attribute. The Invention attribute of the InventorOf predicate is of type CommonNP. All other attributes are of type ProperName.</Paragraph>
    <Paragraph position="1"> The data for the experiments were collected by the KnowItAll crawler. The data for the Acquisition and Merger predicates consist of about 900,000 sentences for each of the two predicates, where each sentence contains at least one predicate keyword. The data for the bounded predicates consist of sentences that contain a predicate keyword and one of a hundred values of the first (bound) attribute. Half of the hundred are frequent entities (&gt;100,000 search engine hits), and another half are rare (&lt;10,000 hits).</Paragraph>
    <Paragraph position="2"> The pattern learning for each of the predicates was performed using the whole corpus of sentences for the predicate. For testing the precision of each of the predicates in each of the systems we manually evaluated sets of 200 instances that were randomly selected out of the full set of instances extracted from the whole corpus.</Paragraph>
    <Paragraph position="3"> In the first experiment, we test the performance of the classification component using different predicates for building the model. In the second experiment we evaluate the full system over the whole dataset.</Paragraph>
    <Section position="1" start_page="478" end_page="479" type="sub_section">
      <SectionTitle>
4.1 Cross-Predicate Classification
Performance
</SectionTitle>
      <Paragraph position="0"> In this experiment we test whether the choice of the model predicate for training the classifier is significant.</Paragraph>
      <Paragraph position="1"> The pattern learning for each of the predicates was performed using the whole corpus of sentences for the predicate. For testing we used a small random selection of sentences, run the Instance Extractor over them, and manually evaluated each extracted instance. The results of the evaluation for Acquisition, CEO_Of, and Merger are summarized in Figure 2. As can be seen, using any of the predicates as the model produces similar results. The graphs for the other two predicates are similar. We have used only the first 15 features, as the NER-based feature (f  using the five different model predicates. As can be seen, the curves on each graph are very similar.</Paragraph>
    </Section>
    <Section position="2" start_page="479" end_page="480" type="sub_section">
      <SectionTitle>
4.2 Performance of the whole system
</SectionTitle>
      <Paragraph position="0"> In this experiment we compare the performance of URES with classification to the performance of KnowItAll. To carry out the experiments, we used extraction data kindly provided by the KnowItAll group. They provided us with the extractions obtained by the KnowItAll system and by its pattern learning component (KnowItAll-PL). Both are sketched in Section 2.1 and are described in detail in (Etzioni, Cafarella et al. 2005).</Paragraph>
      <Paragraph position="1"> In this experiment we used Acquisition as the model predicate for testing all other predicates except itself. For testing Acquisition we used CEO_Of as the model predicate. The results are summarized in the five graphs in the Figure 3.</Paragraph>
      <Paragraph position="2"> For three relations (Acquisition, Merger, and InventorOf) URES clearly outperforms KnowItAll. Yet for the other two (CEO_Of and MayorOf), the simpler method of KnowItAll-PL or even the KnowItAll-baseline do as well as URES. Close inspection reveals that the key difference is the amount of redundancy of instances of those relations in the data. Instances of CEO_Of and MayorOf are mentioned frequently in a wide variety of sentences whereas instances of the other relations are relatively infrequent.</Paragraph>
      <Paragraph position="3"> KnowItAll extraction works well when redundancy is high and most instances have a good chance of appearing in simple forms that KnowItAll is able to recognize. The additional machinery in URES is necessary when redundancy is low. Specifically, URES is more effective in identifying low-frequency instances, due to its more expressive rule representation, and its classifier that inhibits those rules from overgeneralizing.</Paragraph>
      <Paragraph position="4"> In the same graphs we can see that URES-NER outperforms URES by 5-15% in recall for similar precision levels. We can also see that for Person-based predicates the improvement is much more pronounced, because Person is a much simpler entity to recognize. Since in the InventorOf predicate the 2 nd attribute is of type CommonNP, the NER component adds no value and URES-NER and URES results are identical for this predicate.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>