File Information
File: 05-lr/acl_arc_1_sum/cleansed_text/xml_by_section/metho/00/a00-1029_metho.xml
Size: 19,803 bytes
Last Modified: 2025-10-06 14:07:02
<?xml version="1.0" standalone="yes"?> <Paper uid="A00-1029"> <Title>A Tool for Automated Revision of Grammars for NLP Systems</Title> <Section position="3" start_page="210" end_page="213" type="metho"> <SectionTitle> 2 Automated Grammar Revision by rule </SectionTitle> <Paragraph position="0"> modification In this section, we describe an algorithm (see Figure 1) for revising grammars that directly modifies the rules of the grammar to disallow counter-examples. For each counter-example 2, we generate the parse tree (representation of all the grammar rules needed to generate the sentence or set of sentences) and the grammar modifier modifies the production rules of the grammar to invalidate the counter-example. This process is repeated for each counter-example using the revised grammar from the previous iteration for generating the parse tree for the current counter-example. If a counter-example generates multiple parse trees, the above algorithm is repeated for each parse tree in turn.</Paragraph> <Section position="1" start_page="210" end_page="212" type="sub_section"> <SectionTitle> 2.1 Grammar modification algorithm </SectionTitle> <Paragraph position="0"> We present the grammar modification algorithm below. For, we assume that the parse-tree(s) of the counter-example contain no recursion (i.e.</Paragraph> <Paragraph position="1"> the same production rule does not occur twice in any of the parse trees). In section 2.4, we present an approach for using the algorithm even when the parse-trees contain recursion. Thus, the algorithm is applicable for any context-free grammar. The grammar modification algorithm a Note that a counter-example can be a sentence such as &quot;move to operator&quot; or a set of sentences such as &quot;transfer <AMOUNT> to online account&quot;. The latter is specified using non-terminals interspersed with words.</Paragraph> <Paragraph position="2"> for modifying the rules of a grammar to disallow a counter-example c (identified by a grammar developer) using a parse-tree for e proceeds as follows : 1. For each non-terminal <N> in the parse tree, except the <<START>> symbol, a. Add a rule to define a new non-terminal <N'> such that <N'> generates all phrases that <N> generates except for the phrase in the counter-example that <N> generates.</Paragraph> <Paragraph position="3"> b. Add a rule to define a new non-terminal <No> such that <No> generates only the phrase(s) in the counter-example that <N> generates.</Paragraph> <Paragraph position="4"> 2. Modify the rule that contains the <<START>> symbol in the parse tree, such that the <<START>> symbol no longer generates the given counterexample. null We illustrate the algorithm with an example. Figure 2(a) shows a simple grammar. Suppose the sentence &quot;move to operator&quot; is a counter-example for an application. Figure 2(b) shows the parse-tree for &quot;move to operator&quot;. Since the parse tree contains the rule: <V> ::= &quot;move&quot;, new rules are added to define non-terminals <V'> and <Vo>, where <V'> does not generate &quot;move&quot; and <Vo> generates only &quot;move&quot;. Similarly, since the parse tree contains the rule: <N>::= &quot;operator&quot;, the new rules: <N'>::= &quot;checking&quot; I &quot;savings&quot; I &quot;money&quot;; and <No>::= &quot;operator&quot;, are added. For the non-terminal <PP>, the new rules: <PP'>::= &quot;to&quot; <N'>; and <PPo>::= &quot;to&quot; <No>, are added. Note that since <No> only generates the phrase &quot;operator&quot; which is part of the counter-example, <PPo> only generates the phrase &quot;to operator&quot; which is part of the counter-example. Also, <PP'> generates all phrases that <PP> generates except for the phrase &quot;to operator&quot;. Finally, the rule: <<START>>::= <V> <PP> is modified using the newly created non-terminals <V'>, <Vo>, <PP'> and <PPo> such that the only sentences which are accepted by the grammar and begin with the phrase &quot;move&quot; do not end with the phrase &quot;to operator&quot;, and also, the only sentences which are accepted by the grammar and end with the phrase &quot;to operator&quot; do not begin with the phrase &quot;move&quot;. Figure 3 shows the final modified grammar that accepts all the sentences that the grammar in Figure 2(a) accepts except for the sentence &quot;move to</Paragraph> <Paragraph position="6"> operator&quot;. In Figure 3, all the grammar rules that are new or modified are shown in bold and italics.</Paragraph> <Paragraph position="7"> The above algorithm for grammar modification has a time complexity of O(m*2 k) rule creation (or modification) steps for removing a counterexample, where m is the number of production rules in the parse tree of the counter-example and k is the largest number of non-terminals on the right hand side of any of these production rules. Since grammars used for real applications rarely have more than a handful of non-terminals on the right hand side of production rules, this complexity is quite manageable.</Paragraph> </Section> <Section position="2" start_page="212" end_page="213" type="sub_section"> <SectionTitle> 2.2 Automated grammar compaction using </SectionTitle> <Paragraph position="0"> MDL based grammar induction As seen in the example described above, the size of the grammar (number of production rules) can increase greatly by applying our algorithm successively for a number of counter-examples.</Paragraph> <Paragraph position="1"> However, we can remedy this by applying grammar induction algorithms based on minimum description length (MDL) (e.g.</Paragraph> <Paragraph position="2"> Grunwald (1996) and Zadrozny (1997)) to combine rules and create a compact grammar that accepts the same language.</Paragraph> <Paragraph position="3"> The MDL principle (Rissanen (1982)) selects that description (theory) of data, which minimizes the sum of the length, in bits, of the description of the theory, and the length, in bits, of data when encoded using the theory. In our case, the data is the set of possible word combinations and the theory is the grammar that specifies it. We are primarily interested in using the MDL principle to obtain (select) a compact grammar (the theory) from among a set of equivalent grammars. Since the set of possible word combinations (data) is the same for all grammars in consideration, we focus on the description length of the grammars itself, which we approximate by using a set of heuristics described in step 1 below.</Paragraph> <Paragraph position="4"> We use the following modified version of Zadrozny's (1997) algorithm to generate a more compact grammar from the revised grammar using the MDL principle: 1. Compute the description length of the grammar, i.e. the total number of symbols needed to specify the grammar, where each non-terminal, &quot;::=&quot;, and &quot;1&quot; are counted as one symbol.</Paragraph> <Paragraph position="5"> 2. Modify the current grammar by concatenating all possible pairs of nonterminals, and compute the description length of each such resultant grammar.</Paragraph> <Paragraph position="6"> For concatenating <NI> and <N2>, introduce the rule <N3>::= <NI> <N2>, search all other rules for consecutive occurrences of <NI> and <N2>, and replace such occurrences with <N3>.</Paragraph> <Paragraph position="7"> Note that this change results in an equivalent grammar (that accepts the same set of sentences as the original grammar).</Paragraph> <Paragraph position="8"> 3. Modify the current grammar by merging all possible pairs of non-terminals, and compute the description length of each such resultant grammar. For merging <N4> and <N5>, introduce the rule: <N6>::= <N4> \[ <N5>, search for pairs of rules which differ only in one position such that for one of the rules, <N4> occurs in that position and the other rule, the <N5> occurs in the same position. Replace the pair of rules with a new rule that is exactly the same as either of the pairs of rules, except for the use of <N6> instead of <N3> or <N4>.</Paragraph> <Paragraph position="9"> Note that this change results in an equivalent grammar (that accepts the same set of sentences as the original grammar).</Paragraph> <Paragraph position="10"> 4. Compute a table of description lengths of the grammars obtained by concatenating or merging all possible pairs of non- terminals of the initial grammar, as described above. Select the pair of non-terminals (if any) together with the action (concatenate or merge) that results in the least description length and execute the corresponding action.</Paragraph> <Paragraph position="11"> 5. Iterate steps 2, 3, and 4 until the description length does not decrease.</Paragraph> <Paragraph position="12"> No further modification is performed if the base description length of the grammar is lower than that resulting from merging or concatenating any pair of non- terminals.</Paragraph> <Paragraph position="13"> In variations of this algorithm, the selection of the pairs of non-terminals to concatenate or merge, can be based on; the syntactic categories of the corresponding terminals, the semantic categories of the corresponding terminals, and the frequency of occurrence of the nonterminals. null Using the algorithm described above in conjunction with the algorithm in section 2.1, we can obtain a compact grammar that is guaranteed to disallow the counter-examples.</Paragraph> </Section> </Section> <Section position="4" start_page="213" end_page="214" type="metho"> <SectionTitle> 2.3 Results for grammar revision for speech </SectionTitle> <Paragraph position="0"> understanding systems We have built a graphical tool for revising grammars for NLP systems based on the algorithm described in sections 2.1 and 2.2 above. The tool takes as input an existing grammar and can randomly generate sentences accepted by the grammar including non-terminal strings and strings containing terminals and non-terminals (e.g. both &quot;move to operator&quot; and &quot;transfer <AMOUNT> to online account&quot; would be generated if they were accepted by the grammar). A grammar developer (a human) interacts with the tool and either inputs counter-examples selected from speech recognition error logs or selects counter-examples like the ones listed above. The grammar developer can then revise the grammar to disallow the counter-examples by pressing a button and then reduce the size of the resulting grammar using the algorithm in section 2.2 by pressing another button to obtain a compact grammar that does not accept any of the identified counterexamples. Typically, the grammar developer repeats the above cycle several times to obtain a tightly constrained grammar.</Paragraph> <Paragraph position="1"> We have successfully used the tool described above to greatly constrain overgeneralizing grammars for speech understanding systems that we built for telephony banking, stock trading and directory assistance (Zadrozny et al, 1998).</Paragraph> <Paragraph position="2"> The speech recognition grammars for these systems accepted around fifty million sentences each. We successfully used the reviser tool to constrain these grammars by eliminating thousands of sentences and obtained around 20-30% improvement in sentence recognition accuracy. We conducted two user studies of our telephony banking system at different stages of development. The user studies were conducted eight months apart. During these eight months, we used a multi-pronged strategy of constraining grammars using the grammar revision algorithms described in this paper, improving the pronunciation models of some words and redesigning the prompts of the system to enable fast and easy error recovery by users. The combination of all these techniques resulted in improving the 'successful transaction in first try '3 from 43% to 71deg/0, an improvement of 65%. The average number of wrong tries (turns of conversation) to get a successful answer was reduced from 2.1 to 0.5 tries. We did not conduct experiments to isolate the contribution of each factor towards this improvement in system performance.</Paragraph> <Paragraph position="3"> It is important to note here that we would probably have obtained this improvement in recognition accuracy even with a manual revision of the grammars. However, the main advantage in using our tool is the tremendous simplification of the whole process of revision for a grammar developer who now selects counter-examples with an interactive tool instead of manually revising the grammars.</Paragraph> <Section position="1" start_page="213" end_page="214" type="sub_section"> <SectionTitle> 2.4 Handling recursion in grammars </SectionTitle> <Paragraph position="0"> We now describe an extension of the algorithm in section 2.1 that can modify grammars with recursion to disallow a finite set of counterexamples. The example grammars shown above are regular grammars (i.e. equivalent finite state automatons exist). For regular grammars (and only for regular grammars), an alternative approach for eliminating counter-examples using standard automata theory is&quot; * Compute the finite state automaton (FSA) G corresponding to the original grammar.</Paragraph> <Paragraph position="1"> * Compute the FSA C corresponding to the set of counter-examples.</Paragraph> <Paragraph position="2"> * Compute C', the complement of C with respect to the given alphabet.</Paragraph> <Paragraph position="3"> * Compute G', the intersection of G and C'. The FSA G' is equivalent to a revised grammar which disallows the counterexamples. null 3 We measured the number of times the user's transactional intent (e.g. checking balance, last five transactions etc.) was recognized and acted upon correctly by the system in the first try, even when the actual utterance may not have been recognized correctly word for word.</Paragraph> <Paragraph position="4"> 914.</Paragraph> <Paragraph position="5"> The time complexity of the algorithm is O(n*m), where n and m are the number of states in the finite state automatons G and C respectively. This is comparable to the quadratic time complexity of our grammar revision algorithm presented in Section 3.1.</Paragraph> <Paragraph position="6"> However, the above algorithm for eliminating counter-examples only works for regular grammars. This is because context-free grammars are not closed under complementation and intersection. However we can use our algorithm for grammar modification (section 2.1) to handle any context-free grammar as follows: 1) As before, generate parse tree p for counter-example c for an initial grammar G.</Paragraph> <Paragraph position="7"> 2) If p contains a recursion (two or more repetitions of any production rule in the same parse tree), rewrite the initial grammar G as the equivalent grammar G', where the recursion is &quot;unrolled&quot; sufficiently many times (at least one more time than the number of repetitions of the recursive production rule in the parse tree). We explain the unrolling of recursion in greater detail below.</Paragraph> <Paragraph position="8"> If p does not contain any recursion, go to step 4.</Paragraph> <Paragraph position="9"> 3) Generate parse tree p' for the counter-example c for the rewritten grammar G'. Note that p' will no longer contain a recursive application of any production rules, though G' itself will still have recursion.</Paragraph> <Paragraph position="10"> 4) Use the algorithm described in section 2.1 to modify the grammar G' to eliminate the counter-example c using the parse tree p'.</Paragraph> <Paragraph position="11"> We illustrate the above algorithm with an example. Figure 4(a) shows a context free grammar which accepts all strings of the form a&quot;b&quot;, for any n greater than 0. Note that this is not a regular language. Suppose we wish to eliminate the counter-example aaabbb from the initial grammar. The parse treep for the counter-example aaabbb is shown in Figure 4(b). The grammar in 4(a) can be rewritten as the equivalent grammar 4(c), where the recursion of (S->aSb) is unrolled three times. The parse tree p' for the counter-example aaabbb with respect to grammar in 4(c) is shown in Figure 4(d). Note that p' does not contain any recursion, though the rewritten grammar does. We revised the grammar in 4(c) to eliminate the counter-example aaabbb using the parse tree in Figure 4(d). The revised grammar is shown in Figure 4(e). Note that here we are assuming that a mechanism exists for rewriting the rules of a grammar with recursion to unroll the recursion (if it exists) a finite number of times. Such an unrolling is readily accomplished by introducing a set of new non-terminars, one for each iteration of unrolling as shown in Figure 4(c).</Paragraph> </Section> </Section> <Section position="5" start_page="214" end_page="214" type="metho"> <SectionTitle> 3 Automated revision of attribute-value </SectionTitle> <Paragraph position="0"> grammars In this section, we delineate an approach for automatically modifying attribute value grammars using counter-examples. We first convert an attribute value grammar into an equivalent non-attributed grammar by creating new non-terminals and encoding the attributes in the names of the new non-terminals (see Manaster Ramer and Zadrozny (1990) and Pollard and Sag (1994)).</Paragraph> <Paragraph position="1"> For example, suppose the grammar in Figure 2(a) is an attribute value grammar with an</Paragraph> <Paragraph position="3"> attribute 'account', which encodes information about the type of account specified, e.g.</Paragraph> <Paragraph position="4"> 'account' might have the values, SAVINGS, CHECKING and UNSPECIFIED. Figure 5 shows an equivalent non-attributed grammar, where the value of the attribute 'account' has been encoded in the names of the non-terminals. Note that such an encoding can potentially create a very large number of non-terminals.</Paragraph> <Paragraph position="5"> Also, the specific coding used needs to be such that the attributes can be easily recovered from the non-terminal names later on.</Paragraph> <Paragraph position="6"> We can now use our modification algorithms (Section 2.1 and 2.2) to eliminate counter-examples from the non-attributed grammar. For instance, suppose we wish to eliminate 'move to operator' from the attributed grammar based on Figure 2(a), as discussed above. We apply our algorithm (Section 2.1) to the grammar in Figure 5 and obtain the grammar shown in Figure 6.</Paragraph> <Paragraph position="7"> Note that we name any new non-terminals created during the grammar modification in such a way as to leave the encoding of the attribute values in the non-terminal names intact.</Paragraph> <Paragraph position="8"> After applying the grammar revision algorithm, we can extract the attribute values from the encoding in the non-terminal names. For instance, in the example outlined above, we might systematically check for suffixes of a certain type and recover the attributes and their values. Also, as described earlier, we can use the algorithm described in section 2.2 to make the resulting grammar compact again by using MDL based grammar induction algorithms.</Paragraph> </Section> class="xml-element"></Paper>