<?xml version="1.0" standalone="yes"?>
<Paper uid="P99-1009">
  <Title>Man* vs. Machine: A Case Study in Base Noun Phrase Learning</Title>
  <Section position="3" start_page="0" end_page="65" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> Machine learning has been very successful at solving many problems in the field of natural language processing. It has been amply demonstrated that a wide assortment of machine learning algorithms are quite effective at extracting linguistic information from manually annotated corpora.</Paragraph>
    <Paragraph position="1"> Among the machine learning algorithms studied, rule based systems have proven effective on many natural language processing tasks, including part-of-speech tagging (Brill, 1995; Ramshaw and Marcus, 1994), spelling correction (Mangu and Brill, 1997), word-sense disambiguation (Gale et al., 1992), message understanding (Day et al., 1997), discourse tagging (Samuel et al., 1998), accent restoration (Yarowsky, 1994), prepositional-phrase attachment (Brill and Resnik, 1994) and base noun phrase identification (Ramshaw and Marcus, In Press; Cardie and Pierce, 1998; Veenstra, 1998; Argamon et al., 1998). Many of these rule based systems learn a short list of simple rules (typically on the order of 50-300) which are easily understood by humans.</Paragraph>
    <Paragraph position="2"> Since these rule-based systems achieve good performance while learning a small list of simple rules, it raises the question of whether peo*and Woman.</Paragraph>
    <Paragraph position="3">  ple could also derive an effective rule list manually from an annotated corpus. In this paper we explore how quickly and effectively relatively untrained people can extract linguistic generalities from a corpus as compared to a machine. There are a number of reasons for doing this. We would like to understand the relative strengths and weaknesses of humans versus machines in hopes of marrying their con~plementary strengths to create even more accurate systems. Also, since people can use their meta-knowledge to generalize from a small number of examples, it is possible that a person could derive effective linguistic knowledge from a much smaller training corpus than that needed by a machine. A person could also potentially learn more powerful representations than a machine, thereby achieving higher accuracy.</Paragraph>
    <Paragraph position="4"> In this paper we describe experiments we performed to ascertain how well humans, given an annotated training set, can generate rules for base noun phrase chunking. Much previous work has been done on this problem and many different methods have been used: Church's PARTS (1988) program uses a Markov model; Bourigault (1992) uses heuristics along with a grammar; Voutilainen's NPTool (1993) uses a lexicon combined with a constraint grammar; Juteson and Katz (1995) use repeated phrases;  We used the base noun phrase system of Ramshaw and Marcus (R&amp;M) as the machine learning system with which to compare the human learners. It is difficult to compare different machine learning approaches to base NP annotation, since different definitions of base NP are used in many of the papers, but the R&amp;M system is the best of those that have been tested on the Penn Treebank. 1 To train their system, R&amp;M used a 200k-word chunk of the Penn Treebank Parsed Wall Street Journal (Marcus et al., 1993) tagged using a transformation-based tagger (Brill, 1995) and extracted base noun phrases from its parses by selecting noun phrases that contained no nested noun phrases and further processing the data with some heuristics (like treating the possessive marker as the first word of a new base noun phrase) to flatten the recursive structure of the parse. They cast the problem as a transformation-based tagging problem, where each word is to be labelled with a chunk structure tag from the set {I, O, B}, where words marked 'T' are inside some base NP chunk, those marked &amp;quot;O&amp;quot; are not part of any base NP, and those marked &amp;quot;B&amp;quot; denote the first word of a base NP which immediately succeeds another base NP. The training corpus is first run through a part-of-speech tagger. Then, as a baseline annotation, each word is labelled with the most common chunk structure tag for its part-of-speech tag.</Paragraph>
    <Paragraph position="5"> After the baseline is achieved, transformation rules fitting a set of rule templates are then learned to improve the &amp;quot;tagging accuracy&amp;quot; of the training set. These templates take into consideration the word, part-of-speech tag and chunk structure tag of the current word and all words within a window of 3 to either side of it.</Paragraph>
    <Paragraph position="6"> Applying a rule to a word changes the chunk structure tag of a word and in effect alters the boundaries of the base NP chunks in the sentence. null An example of a rule learned by the R&amp;M system is: change a chunk structure tag of a word from I to B if the word is a determiner, the next word ks a noun, and the two previous words both have chunk structure tags of I. In other words, a determiner in this context is likely to begin a noun phrase. The R&amp;M system learns a total 1We would like to thank Lance Ramshaw for providing us with the base-NP-annotated training and test corpora that were used in the R&amp;M system, as well as the rules learned by this system.</Paragraph>
    <Paragraph position="7"> of 500 rules.</Paragraph>
  </Section>
class="xml-element"></Paper>