<?xml version="1.0" standalone="yes"?>
<Paper uid="J98-2004">
  <Title>New Figures of Merit for Best-First Probabilistic Chart Parsing</Title>
  <Section position="2" start_page="0" end_page="276" type="abstr">
    <SectionTitle>
1. Introduction
</SectionTitle>
    <Paragraph position="0"> Chart parsing is a commonly used algorithm for parsing natural language texts. The chart is a data structure that contains all of the constituents for which subtrees have been found, that is, constituents for which a derivation has been found and which may therefore appear in some complete parse of the sentence. The agenda is a structure that stores a list of constituents for which a derivation has been found but which have not yet been combined with other constituents. Initially, the agenda contains the terminal symbols from the sentence to be parsed. A constituent is removed from the agenda and added to the chart, and the system considers how this constituent can be used to extend its current structural hypothesis by combining with other constituents in the chart according to the grammar rules. (We will often refer to these expansions of rules as &amp;quot;edges&amp;quot;.) In general this can lead to the creation of new, more encompassing constituents, which themselves are then added to the agenda. When one constituent has been processed, a new one is chosen to be removed from the agenda, and so on. Traditionally, the agenda is represented as a stack, so that the last item added to the agenda is the next one removed. Chart parsing is described extensively in the literature; for one such discussion see 'Section 1.4 of Charniak (1993).</Paragraph>
    <Paragraph position="1"> Best-first probabilistic chart parsing is a variation of chart parsing that attempts to find the most likely parses first, by adding constituents to the chart in order of the likelihood that they will appear in a correct parse, rather than simply popping constituents off of a stack. Some probabilistic figure of merit is assigned to the constituents on the agenda, and the constituent maximizing this value is the next to be added to the chart.</Paragraph>
    <Paragraph position="2"> In this paper we consider probabilities primarily based on probabilistic context-free grammars, though in principle, other, more complicated schemes could be used. The purpose of this work is to compare how well several figures of merit select</Paragraph>
    <Paragraph position="4"> constituents to be moved from the agenda to the chart. Ideally, we would like to use as our figure of merit the conditional probability of that constituent, given the entire sentence, in order to choose a constituent that not only appears likely in isolation, but is most likely given the sentence as a whole; that is, we would like to pick the constituent that maximizes the following quantity: P(N~,k I to,n) where t0,n is the sequence of the n tags, or parts of speech, in the sentence (numbered to ..... tn-1), and N~, k is a nonterminal of type i covering terms tj... tk-1. (See Figure 1.) In our experiments, we use only tag sequences (as given in the test data) for parsing. More accurate probability estimates should be attainable using lexical information in future experiments, as more detail usually leads to better statistics, but lexicalized figures of merit are beyond the scope of the research described here.</Paragraph>
    <Paragraph position="5"> Note that our &amp;quot;ideal&amp;quot; figure is simply a heuristic, since there is no guarantee that a constituent that scores well on this measure will appear in the correct parse of a sentence. For example, there may be a very large number of low-probability derivations of N~, k, which are combined here to give a high value, but a parse of the sentence can only include one of these derivations, making it unlikely that N~, k appears in the most probable parse of the sentence. On the other hand, there is no reason to believe that such cases are common in practice.</Paragraph>
    <Paragraph position="6"> We cannot calculate p(N~, k \[ t0,n), since in order to do so, we would need to completely parse the sentence. In this paper, we examine the performance of several proposed figures of merit that approximate it in one way or another, using two different grammars. We identify a figure of merit that gives superior results on all of our performance measures and on both grammars.</Paragraph>
    <Paragraph position="7"> Section 2 of this paper describes the method we used to determine the effectiveness of figures of merit, that is, to compare how well they choose constituents to be moved from the agenda to the chart. Section 2.1 explains the experiment, Section 2.2 describes the measures we used to compare the performance of the figures of merit, and Section 2.3 describes a model we used to represent the performance of a traditional parser using a simple stack as an agenda.</Paragraph>
    <Paragraph position="8"> In Section 3, we describe and compare three simple and easily computable figures of merit based on inside probability. Sections 3.1 through 3.3 describe each figure in detail, and Section 3.4 presents the results of an experiment comparing these three figures. Sections 4 and 5 have a similar structure to Section 3, with Section 4 evaluating two figures of merit using statistics on the left-side context of the constituent, and  Caraballo and Charniak Figures of Merit Section 5 evaluating three additional figures of merit using statistics on the context on both sides of the constituent. Section 6 contains a table summarizing the results from Sections 3, 4, and 5.</Paragraph>
    <Paragraph position="9"> In Section 7, we use another grammar in the experiment, to verify that our results are not an artifact of the grammar used for parsing. Section 8 describes previous work in this area, and Section 9 presents our conclusions and recommendations.</Paragraph>
    <Paragraph position="10"> There are also three appendices to this paper. Appendix A gives our method for computing inside probability estimates while maintaining parser speed. Appendix B explains how we obtained our boundary statistics used in Section 5. Appendix C presents data comparing the parsing accuracy obtained by each of our parsers as the number of edges they create increases.</Paragraph>
  </Section>
class="xml-element"></Paper>