<?xml version="1.0" standalone="yes"?>
<Paper uid="P81-1001">
  <Title>A Practical Comparison of Parsing Strategies</Title>
  <Section position="1" start_page="0" end_page="0" type="intro">
    <SectionTitle>
INTRODUCTION
</SectionTitle>
    <Paragraph position="0"> Although the literature dealing with formal and natural languages abounds with theoretical arguments of worst-case performance by various parsing strategies \[e.g., Griffiths &amp; Petrick, 1965; Aho &amp; Ullman, 1972; Graham, Harrison &amp; Ruzzo, Ig80\], there is little discussion of comparative performance based on actual practice in understanding natural language. Yet important practical considerations do arise when writing programs to understand one aspect or another of natural language utterances. Where, for example, a theorist will characterize a parsing strategy according to its space and/or time requirements in attempting to analyze the worst possible input acc3rding to ~n arbitrary grammar strictly limited in expressive power, the researcher studying Natural Language Processing can be justified in concerning himself more with issues of practical performance in parsing sentences encountered in language as humans Actually use it using a grammar expressed in a form corve~ie: to the human linguist who is writing it.</Paragraph>
    <Paragraph position="1"> Moreover, ~ry occasional poor performance may be quite acceptabl:, particularly if real-time considerations are not invo~ed, e.g., if a human querant is not waiting for the answer to his question), provided the overall average performance is superior. One example of such a situation is off-line Machine Translation.</Paragraph>
    <Paragraph position="2"> This paper has two purposes. One is to report an evaluation of the performance of several parsing strategies in a real-world setting, pointing out practical problems in making the attempt, indicating which of the strategies is superior to the others in which situations, and most of all determining the reasons why the best strategy outclasses its competition in order to stimulate and direct the design of improvements. The other, more important purpose is to assist in establishing such evaluation as a meaningful and valuable enterprise that contributes to the evolution of Natural Language PrcJessing from an art form into an empirical science.</Paragraph>
    <Paragraph position="3"> T~t is, our concern for parsing efficiency transcends the issue of mere practicality. At slow-to-average parsing rates, the cost of verifying linguistic theories on a large, general sample of natural language can still be prohibitive. The author's experience in MT has demonstrated the enormous impetus to linguistic theory formulation and refinement that a suitably fast parser will impart: when a linguist can formalize and encode a theory, then within an hour test it on a few thousand words of natural text, he will be able to reject inadequate ideas at a fairly high rate. This argument may even be applied to the production of the semantic theory we all hope for: it is not likely that its early formulations will be adequate, and unless they can be explored inexpensively on significant language samples they may hardly be explored at all, perhaps to the extent that the theory's qualities remain undiscovered.</Paragraph>
    <Paragraph position="4"> The search for an optimal natural language parsing technique, then, can be seen as the search for an instrument to assist in extending the theoretical frontiers of the science of Natural Language Processing.</Paragraph>
    <Paragraph position="5"> Following an outline below of some of the historical circumstances that led the author to design and conduct the parsing experiments, we will detail our experimental setting and approach, present the results, discuss the implications of those results, and conclude with some remarks on what has been l~rned.</Paragraph>
    <Paragraph position="6"> The SRI Connection At SRI International the~thor was responsible for the development of the English front-end for the LADDER system \[Hendrix etal., 1978\]. LADDER was developed as a prototype system for understanding questions posed in English about a naval domain; it translated each English question into one or more relational database queries, prosecuted the queries on a remote computer, and responded with the requested information in a readable format tailored to the characteristics of the answer.</Paragraph>
    <Paragraph position="7"> The basis for the development of the NLP component of the LADDER system was the LIFER parser, which interpreted sentences according to a 'semantic grammar' \[Burton, 1976\] whose rules were carefully ordered to produce the most plausible interpretation first.</Paragraph>
    <Paragraph position="8"> After more than two years of intensive development, the human costs of extending the coverage began to mount significantly. The semantic grammar interpreted by LIFER had become large and unwieldy. Any change, however small, had the potential to produce &amp;quot;ripple effects&amp;quot; which eroded the integrity of the system. A more linguistically motivated grammar was required. The question arose, &amp;quot;Is LIFER as suited to more traditional grammars as it is to semantic grammars?&amp;quot; At the time, there were available at SRI three production-quality parsers: LIFER; DIAMOND, an implementation of the Cocke-Kasami~nger parsing algorithm programmed by William Paxton of SRI; and CKY, an implementation of the identical algorithm programmed initially by Prof. Daniel Chester at the University of Texas. In this environment, experiments comparing various aspects of performance were inevitable.</Paragraph>
    <Paragraph position="9"> The LRC Connection In 1979 the author began research in Machine Translation at the Linguistics Research Center of the University of Texas. The LRC environment stimulated the design of a new strategy variation, though in retrospect it is obviously applicable to any parser supporting a facility for testing right-hand-side rule constituents. It also stimulated the production of another parser. (These will be defined and discussed later.) To test the effects of various strategies on the two LRC parsers, an experiment was designed to determine whether they interact with the different parsers and/or each other, whether any gains are offset by introduced overhead, and whether the source and precise effects of any overhead could be identified and explained.</Paragraph>
  </Section>
class="xml-element"></Paper>