<?xml version="1.0" standalone="yes"?>
<Paper uid="W94-0112">
  <Title>Bootstrapping Statistical Processing into a Rule-based Natural Language Parser</Title>
  <Section position="5" start_page="99" end_page="364" type="evalu">
    <SectionTitle>
4 Results
</SectionTitle>
    <Paragraph position="0"> We have used our parser to compute both simple and conditioned probabilities, as described above, during the parsing of the ! million word Brown corpus. In round numbers, this process took about 34 hours on a 486/66 PC, for an average of 2.5 seconds per sentence. There are about 55,000 sentences in the Brown corpus, averaging 18 words  in length, but those over 35 words in length (more than 7,000) were not parsed, for the reasons given earlier.</Paragraph>
    <Paragraph position="1"> The probabilities thus computed were incorporated for use by the probabilistic algorithm of the parser, and the parser was then applied to two sets of selected sentences in order to evaluate the anticipated improvements in parsing speed and accuracy. The first set contained 500 sentences, averaging 17 words in length, and randomly selected from different sources, including articles from Time magazine and the Wall Street Journal, linguistic textbook examples, and correspondence. The efficiency of the parser in processing these sentences, both with and without probabilities, is documented in Table 1.</Paragraph>
    <Paragraph position="2">  Useful measures of parsing efficiency include the total number of records in the chart when a parse is obtained or the parser otherwise stops, the number of rules aRempted for a given input string and, of course, the time required to parse a string (assuming a dedicated, non-multi-tasking computer system). On average, using the conditioned probabilities resulted in half as many records being placed in the chart during the processing of a sentence and a corresponding speed-up by a factor of 2. Rule attempts decreased by more than a factor of 5. A large number of sentences parsed many times faster than with the non-probabilistic algorithm, but this was tempered in the averaging process by a number of long sentences that parsed in nearly the same time, and on very rare occasions, slightly slower. 2 In the probabilistic algorithm used in this evaluation, we also implemented a low-probability cutoff to stop the parser from continuing to apply rules after a certain number of rules (whose probability is less than the average probability of all the rules) had been attempted. This number is multiplied by the number of words in a sentence (to adjust for the obvious fact that more rule applications are needed for longer sentences) and has been determined experimentally by running the parser on sets of sentences and examining how often a well-formed (in contrast to &amp;quot;fiRed&amp;quot;) parse is actually obtained after a certain number of lessthan-average rules have been attempted. The parser currently produces a fired parse for just over 20% of the sentences in the first set described above. In practice, using this low-probability cutoff rarely increases the number of fitted parses obtained, and then only slightly (perhaps a percentage point or so). This is more than offset by the use of the probabilities which, due tO their positive effect on parsing efficiency, allow for the successful parsing of much longer and more complicated sentences without exhausting computational resources such as available computer memory.</Paragraph>
    <Paragraph position="3"> The second set of sentences on which the parser was evaluated contained 100 sentences, roughly half being randomly selected from a linguistic textbook and the other half from some Time magazine articles. Although the former half were fairly short (10 words/sentence), they exhibited a variety of linguistic structures, in contrast to the somewhat more straightforward, but longer (17 words/sentence), sentences from the latter half.</Paragraph>
    <Paragraph position="4"> All the sentences in this set shared the characteristic that the parser produced two or more parses for each of them. The parse trees produced by the parser for these sentences were examined and it was determined whether the correct parse 2 Slower parsing is actually possible, when the probabilities turn out to be useless for a given sentence, because of the overhead of maintaining and accessing the PLIST described in Figure 3.</Paragraph>
    <Paragraph position="5">  was produced first by the probabilistic algorithm, using both simple and conditioned probabilities.</Paragraph>
    <Paragraph position="6"> For the non-probabilistic algorithm, the parse trees were ordered according to the degree of fight attachment they exhibited (i.e., deepest structures first). As shown in Table 2, the algorithm using conditioned probabilities selected the correct parse more than twice as often as simple right attachment. It is interesting to note that while the probabilistic algorithm performed somewhat better on the shorter textbook sentences than on the longer magazine sentences, fight attachment performed worse. This is most likely due to the wide variety of (not simple fight-branching) linguistic structures in the textbook sentences.</Paragraph>
    <Paragraph position="7">  over 100 sentences for which multiple parses are produced</Paragraph>
  </Section>
class="xml-element"></Paper>