GRAMMAR SPECIALIZATION THROUGH 
ENTROPY THRESHOLDS 
Christer Samuelsson 
Swedish Institute of Computer Science 
Box 1263 S-164 28 Kista, Sweden 
Internet: christer@sics, se 
Abstract 
Explanation-based generalization is used to extract a 
specialized grammar from the original one using a train- 
ing corpus of parse trees. This allows very much faster 
parsing and gives a lower error rate, at the price of a 
small loss in coverage. Previously, it has been necessary 
to specify the tree-cutting criteria (or operationality cri- 
teria) manually; here they are derived automatically 
from the training set and the desired coverage of the 
specialized grammar. This is done by assigning an en- 
tropy value to each node in the parse trees and cutting 
in the nodes with sufficiently high entropy values. 
BACKGROUND 
Previous work by Manny Rayner and the author, see 
\[Samuelsson &~ Rayner 1991\] attempts to tailor an ex- 
isting natural-language system to a specific application 
domain by extracting a specialized grammar from the 
original one using a large set of training examples. The 
training set is a treebank consisting of implicit parse 
trees that each specify a verified analysis of an input 
sentence. The parse trees are implicit in the sense that 
each node in the tree is the (mnemonic) name of the 
grammar rule resolved on at that point, rather than the 
syntactic category of the LHS of the grammar rule as is 
the case in an ordinary parse tree. Figure 1 shows five 
examples of implicit parse trees. The analyses are ver- 
ified in the sense that each analysis has been judged to 
be the preferred one for that input sentence by a human 
evaluator using a semi-automatic evaluation method. 
A new grammar is created by cutting up each implicit 
parse tree in the treebank at appropriate points, creat- 
ing a set of new rules that consist of chunks of original 
grammar rules. The LHS of each new rule will be the 
LHS phrase of the original grammar rule at the root of 
the tree chunk and the RHS will be the RHS phrases of 
the rules in the leaves of the tree chunk. For example, 
cutting up the first parse tree of Figure 1 at the NP of 
the rule vp_v_np yields rules 2 and 3 of Figure 3. 
The idea behind this is to create a specialized gram- 
mar that retains a high coverage but allows very much 
faster parsing. This has turned out to be possible -- 
speedups compared to using the original grammar of 
in median 60 times were achieved at a cost in cover- 
age of about ten percent, see \[Samuelsson 1994a\].1 An- 
other benefit from the method is a decreased error rate 
when the system is required to select a preferred ana- 
lysis. In these experiments the scheme was applied to 
the grammar of a version of the SRI Core Language 
Engine \[Alshawi ed. 1992\] adapted to the Atis domain 
for a speech-translation task \[Rayner el al 1993\] and 
large corpora of real user data collected using Wizard- 
of-Oz simulation. The resulting specialized gram- 
mar was compiled into LR parsing tables, and a spe- 
cial LR parser exploited their special properties, see 
\[Samuelsson 1994b\]. 
The technical vehicle previously used to extract the 
specialized grammar is explanation-based generaliza- 
tion (EBG), see e.g. \[Mitchell et al 1986\]. Very briefly, 
this consists of redoing the derivation of each train- 
ing example top-down by letting the implicit parse tree 
drive a rule expansion process, and aborting the expan- 
sion of the specialized rule currently being extracted if 
the current node of the implicit parse tree meets a set 
of tree-cutting criteria 2. In this case the extraction pro- 
cess is invoked recursively to extract subrules rooted in 
the current node. The tree-cutting criteria can be local 
("The LHS of the original grammar rule is an NP,") or 
dependent on the rest of the parse tree ("that doesn't 
dominate the empty string only,") and previous choices 
of nodes to cut at ("and there is no cut above the cur- 
rent node that is also labelled NP."). 
A problem not fully explored yet is how to arrive 
at an optimal choice of tree-cutting criteria. In the 
previous scheme, these must be specified manually, and 
the choice is left to the designer's intuitions. This article 
addresses the problem of automating this process and 
presents a method where the nodes to cut at are selected 
automatically using the information-theoretical concept 
of entropy. Entropy is well-known from physics, but the 
concept of perplexity is perhaps better known in the 
speech-recognition and natural-language communities. 
1Other more easily obtainable publications about this are 
in preparation. 
2These are usually referred to as "operationality criteria" 
in the EBG literature. 
188 
For this reason, we will review the concept of entropy 
at this point, and discuss its relation to perplexity. 
Entropy 
Entropy is a measure of disorder. Assume for exam- 
ple that a physical system can be in any of N states, 
and that it will be in state si with probability Pi. The 
entropy S of that system is then 
N 
S -= ~ -- Pi " In Pi 
i=1 
If each state has equal probability, i.e. if Pi ~- 1 for all 
i, then 
N 1 1 
S=Z-~--ln ~-=lnN 
i=1 
In this case the entropy is simply the logarithm of the 
number of states the system can be in. 
To take a linguistic example, assume that we are try- 
ing to predict the next word in a word string from the 
previous ones. Let the next word be wk and the pre- 
vious word string wl,...,wk-1. Assume further that 
we have a language model that estimates the proba- 
bility of each possible next word (conditional on the 
previous word string). Let these probabilities be Pi 
for i = 1 .... ,N for the N possible next words w~, 
i.e. Pi = p(wik I Wl, ..., wk-a). The entropy is then a 
measure of how hard this prediction problem is: 
S(L/)I,...,Wk-x) : 
N 
-- P(Wik I w,, ..., Wk-i) . In p(w~ I Wl, '", Wk-i) 
i=1 
If all words have equal probability, the entropy is the 
logarithm of the branching factor at this point in the 
input string. 
Perplexity 
Perplexity is related to entropy as follows. The observed 
perplexity Po of a language model with respect to an 
(imaginary) infinite test sequence wl, w2, ... is defined 
through the formula (see \[Jelinek 1990\]) 
In Po = lim --lln p(wi, ..., wn) rl--* OO n 
Here p(wl,..., Wn) denotes the probability of the word 
string Wl, ..., W n. 
Since we cannot experimentally measure infinite lim- 
its, we terminate after a finite test string wl, ...,WM, 
arriving at the measured perplexity Pro: 
Pm = -- --~--ln p(wl,..., WM) In 
Rewriting p(wl,...,wk) as p(wk \[ wl,...,wk-1) • 
p(wl, ..., wk-1) gives us 
M 1 
In Pm ----- ~ ~ -In p(wk I wl, ..., w~-l) 
k=l 
Let us call the exponential of the expectation value of 
-In p(w \[ String) the local perplexity P~(String), which 
can be used as a measure of the information content of 
the initial String. 
In Pt(wl, ..., wk-1) = E(-ln P(~k I wl,..., wk-i)) = 
N 
-- p(wik I wl, ..., wk-1)" In p(wik I wl .... , wk-i) 
i=1 
Here E(q) is the expectation value of q and the sum- 
mation is carried out over all N possible next words w~. 
Comparing this with the last equation of the previous 
section, we see that this is precisely the entropy S at 
point k in the input string. Thus, the entropy is the 
logarithm of the local perplexity at a given point in the 
word string. If all words are equally probable, then the 
local perplexity is simply the branching factor at this 
point. If the probabilities differ, the local perplexity 
can be viewed as a generalized branching factor that 
takes this into account. 
Tree entropy 
We now turn to the task of calculating the entropy of a 
node in a parse tree. This can be done in many different 
ways; we will only describe two different ones here. 
Consider the small test and training sets of Figure 1. 
Assume that we wish to calculate the entropy of the 
phrases of the rule PP --* Prep NP, which is named 
pp_prep_np. In the training set, the LHS PP is at- 
tached to the RHS PP of the rule np_np_pp in two 
cases and to the RHS PP of the rule vp_vp_pp in one 
case, giving it tile entropy -2ln~ - ½1n½ ,~ 0.64. The 
RHS preposition Prep is always a lexical lookup, and 
the entropy is thus zero 3, while the RHS NP in one case 
attaches to the LHS of rule np_det_np, in one case to 
the LHS of rule np_num, and in one case is a lexical 
lookup, and the resulting entropy is thus -ln½ ~ 1.10. 
The complete table is given here: 
Rule LHS 1st RHS 2nd RHS 
s_np_vp 0.00 
np_np_pp 0.00 
np_det_n 1.33 
np_pron 0.00 
np_num 0.00 
vp_vp_pp 0.00 
vp_v_np 0.00 
vp_v 0.00 
pp_prep_np 0.64 
0.56 0.56 
0.00 0.00 
0.00 0.00 
0.00 
0.00 
0.00 0.00 
0.00 0.64 
0.00 
0.00 1.10 
If we want to calculate the entropy of a particular 
node in a parse tree, we can either simply use the phrase 
3Since there is only one alternative, namely a lexiea\] 
lookup. In fact, the scheme could easily be extended to en- 
compass including lexical lookups of particular words into 
the specialized rules by distinguishing lexical lookups of dif- 
ferent words; the entropy would then determine whether or 
not to cut in a node corresponding to a lookup, just as for 
any other node, as is described in the following. 
189 
Training examples: 
s_np_vp 
/k 
np_pron vp_v_np 
I /\ 
lex lex np_det_n 
I I /\ 
I want lex lex 
I I 
a ticket 
s_np_vp /\ 
np_pron vp_v_np 
I /\ 
lex / \ 
I / \ 
I lex np_np_pp 
I /\ 
need np_det_n pp_prep_np /\ /\ 
lex lex lex lex 
I I I I 
a flight to Boston 
s_np_vp /\ 
/ \ 
s_np_vp np_det_n vp_vp_pp 
/\ /\ /\ 
np_pron vp_v_np lex lex vp_v pp_prep_np 
I /\ I I I /\ 
lex / \ The flight lex lex 
I / \ I I 
We lex np_np_pp departs at \] /\ 
have / \ 
np_det_n pp_prep_np /\ /\ 
lex lex lex np_det_n 
I I I /\ 
a departure I lex lex 
in I I 
the morning 
np_nuR 
I 
lex 
I 
ten 
Test example: 
s_np_vp 
/\ 
np_pron vp_v_np 
I /\ 
lex / \ 
I / \ 
He lex np_np_pp 
I /\ 
booked / \ 
np_det_n pp_prep_np /\ /\ 
lex lex / \ 
I I / \ 
a ticket lex np_np_pp 
I /\ 
\] np_det_n pp_prep_np 
for /\ /\ 
lex lex lex lex 
I I I I 
a flight to Dallas 
Figure 1: A tiny training set 
entropy of the RttS node, or take the sum of the en- 
tropies of the two phrases that are unified in this node. 
For example, the entropy when the RHS NP of the 
rule pp_prep_np is unified with the LHS of the rule 
np_det n will in the former case be 1.10 and in the 
latter case be 1.10 + 1.33 = 2.43. 
SCHEME OVERVIEW 
In the following scheme, the desired coverage of the spe- 
cialized grammar is prescribed, and the parse trees are 
cut up at appropriate places without having to specify 
the tree-cutting criteria manually: 
1. Index the treebank in an and-or tree where the or- 
nodes correspond to alternative choices of grammar 
rules to expand with and the and-nodes correspond 
to the RHS phrases of each grammar rule. Cutting 
up the parse trees will involve selecting a set of or- 
nodes in the and-or tree. Let us call these nodes 
"cutnodes". 
2. Calculate the entropy of each or-node. We will cut at 
each node whose entropy exceeds a threshold value. 
The rationale for this is that we wish to cut up the 
parse trees where we can expect a lot of variation 
i.e. where it is difficult to predict which rule will be 
resolved on next. This corresponds exactly to the 
nodes in the and-or tree that exhibit high entropy 
values. 
3. The nodes of the and-or tree must be partitioned 
into equivalence classes dependent on the choice of 
cutnodes in order to avoid redundant derivations at 
parse time. 4 Thus, selecting some particular node as 
a cutnode may cause other nodes to also become cut- 
nodes, even though their entropies are not above the 
threshold. 
4. Determine a threshold entropy that yields the desired 
coverage. This can be done using for example interval 
bisection. 
5. Cut up the training examples by matching them 
against the and-or tree and cutting at the determined 
cutnodes. 
It is interesting to note that a textbook method 
for conslructing decision trees for classification from 
attribute-value pairs is to minimize the (weighted aver- 
age of the) remaining entropy 5 over all possible choices 
of root attribute, see \[Quinlan 1986\]. 
4This can most easily be seen as follows: Imagine two 
identical, but different portions of the and-or tree. If the 
roots and leaves of these portions are all selected as cut- 
nodes, but the distribution of cutnodes within them differ, 
then we will introduce multiple ways of deriving the portions 
of the parse trees that match any of these two portions of 
the and-or tree. 
5Defined slightly differently, as described below. 
190 
DETAILED SCHEME 
First, the treebank is partitioned into a training set and 
a test set. The training set will be indexed in an and- 
or tree and used to extract the specialized rules. The 
test set will be used to check the coverage of the set of 
extracted rules. 
Indexing the treebank 
Then, the set of implicit parse trees is stored in an and- 
or tree. The parse trees have the general form of a rule 
identifier Id dominating a list of subtrees or a word of 
the training sentence. From the current or-node of the 
and-or tree there will be arcs labelled with rule iden- 
tifiers corresponding to previously stored parse trees. 
From this or-node we follow an arc labelled Id, or add 
a new one if there is none. We then reach (or add) 
an and-node indicating the RHS phrases of the gram- 
mar rule named Id. Here we follow each arc leading 
out from this and-node in turn to accommodate all the 
subtrees in the list. Each such arc leads to an or-node. 
We have now reached a point of recursion and can index 
the corresponding subtree. The recursion terminates if 
Id is the special rule identifier lex and thus dominates 
a word of the training sentence, rather than a list of 
subtrees. 
Indexing the four training examples of Figure 1 will 
result in the and-or tree of Figure 2. 
Finding the cutnodes 
Next, we find the set of nodes whose entropies exceed a 
threshold value. First we need to calculate the entropy 
of each or-node. We will here describe three different 
ways of doing this, but there are many others. Before 
doing this, though, we will discuss the question of re- 
dundancy in the resulting set of specialized rules. 
We must equate the cutnodes that correspond to the 
same type of phrase. This means that if we cut at a 
node corresponding to e.g. an NP, i.e. where the arcs 
incident from it are labelled with grammar rules whose 
left-hand-sides are NPs, we must allow all specialized 
NP rules to be potentially applicable at this point, not 
just the ones that are rooted in this node. This requires 
that we by transitivity equate the nodes that are dom- 
inated by a cutnode in a structurally equivalent way; if 
there is a path from a cutnode cl to a node nl and a 
path from a cutnode c2 to a node n2 with an identical 
sequence of labels, the two nodes nl and n2 must be 
equated. Now if nl is a cutnode, then n2 must also 
be a cutnode even if it has a low entropy value. The 
following iterative scheme accomplishes this: 
Function N* (N °) 
1. i:=0; 
2. Repeat i := i + 1; N i := N(NI-1); 
3. Until N i = N i-1 
4. Return N~; 
root 
I s_np_vp 
/\ 
/ k / \ 
1/ \2 / X 
/ X 
/ X 
ni(0.89) n2(0.56) 
IX IX 
np_pronl \np_det_n I \ / \ / \ 
11 1/\2 / \ 
n n n I \ 
lex I lex I I lex / \ / \ 
vp_v_np/ \vp_vp_pp / \ 
/ \ 
/ \ 
/\ /k 
il X2 il \2 / X / \ 
n n3(1.08) (O.O0)n7 n8(0.64) 
lex I /\ vp_v I I pp_prep_np 
/ \ II 11 \2 
np_det_n/ \np_np_pp n n n9(l. I0) 
/ \ lex~ lex~ I np_num 
I \ ll 
ilk2 I\ n 
n n / \ llex 
lexl Ilex 1/ \2 / \ 
/ \ 
(1.33)n4 
np_de¢_n I 
11 \2 
n 
lexl 
n5(0.64) 
Ipp_prep_np /\ 
n 11 \2 
flex / \ 
n n6(1.76) 
lexl /\ 
lex/ \np_det_n 
/ \ 
1/\2 
nn 
lexl Jlex 
Figure 2: The resulting and-or tree 
191 
Here N(N j) is the set of cutnodes NJ augmented with 
those induced in one step by selecting N~ as the set of 
cutnodes. In ~ practice this was accomplished by compil- 
ing an and-or graph from the and-or tree and the set 
of selected cutnodes, where each set of equated nodes 
constituted a vertex of the graph, and traversing it. 
In the simplest scheme for calculating the entropy of 
an or-node, only the RHS phrase of the parent rule, 
i.e. the dominating and-node, contributes to the en- 
tropy, and there is in fact no need to employ an and-or 
tree at all, since the tree-cutting criterion becomes local 
to the parse tree being cut up. 
In a slightly more elaborate scheme, we sum over the 
entropies of the nodes of the parse trees that match this 
node of the and-or tree. However, instead of letting each 
daughter node contribute with the full entropy of the 
LHS phrase of the corresponding grammar rule, these 
entropies are weighted with the relative frequency of 
use of each alternative choice of grammar rule. 
For example, the entropy of node n3 of the and- 
or tree of Figure 2 will be calculated as follows: The 
mother rule vp_v_np will contribute the entropy asso- 
ciated with the RHS NP, which is, referring to the table 
above, 0.64. There are 2 choices of rules to resolve on, 
namely np_det_n and np_np_pp with relative frequen- 
cies ½ and ~ respectively. Again referring to the entropy 
table above, we find that the LHS phrases of these rules 
have entropy 1.33 and 0.00 respectively. This results in 
the following entropy for node n3: 
1 2 S(n3) = 0.64+ ~- 1.33+ ~-0.00 = 1.08 
The following function determines the set of cutnodes 
N that either exceed the entropy threshold, or are in- 
duced by structural equivalence: 
Function N ( Smin ) 
1. N :: {n: S(n) > S, ni,-,}; 
2. Return N*(N); 
Here S(n) is the entropy of node n. 
In a third version of the scheme, the relative frequen- 
cies of the daughters of the or-nodes are used directly 
to calculate the node entropy: 
S(n) = ~., - p(niln). In p(n, lu ) 
ni:(n,ni)EA 
Here A is the set of arcs, and {n, ni) is an arc from n to 
hi. This is basically the entropy used in \[Quinlan 1986\]. 
Unfortunately, this tends to promote daughters of cut- 
nodes to in turn become cutnodes, and also results in a 
problem with instability, especially in conjunction with 
the additional constraints discussed in a later section, 
since the entropy of each node is now dependent on the 
choice of cutnodes. We must redefine the function N(S) 
accordingly: 
Function N(Smin) 
1. N O := 0; 
2. Repeat i := i+ 1; 
N := {n: S(nlg '-1) > S,~i,~}; g i := N*(N); 
3. Until N*" = N i-1 
4. Return N i; 
Here S(n\]N j) is the entropy of node n given that the 
set of cutnodes is NJ. Convergence can be ensured 6 by 
modifying the termination criterion to be 
3. Until 3j e \[0, i- 1\] : p(Ni,Y j) < 6(Yi,N j) 
for some appropriate set metric p(N1, N2) (e.g. the size 
of the symmetric difference) and norm-like function 
6(N1,N2) (e.g. ten percent of the sum of the sizes), 
but this is to little avail, since we are not interested in 
solutions far away from the initial assignment of cut- 
nodes. 
Finding the threshold 
We will use a simple interval-bisection technique for 
finding the appropriate threshold value. We operate 
with a range where the lower bound gives at least the 
desired coverage, but where the higher bound doesn't. 
We will take the midpoint of the range, find the cut- 
nodes corresponding to this value of the threshold, and 
check if this gives us the desired coverage. If it does, 
this becomes the new lower bound, otherwise it becomes 
the new upper bound. If the lower and upper bounds 
are close to each other, we stop and return the nodes 
corresponding to the lower bound. This termination cri- 
terion can of course be replaced with something more 
elaborate. This can be implemented as follows: 
Function N(Co) 
1. Stow := 0; Shigh := largenumber; Nc := N(0); 
2. If Shigh - Sto~o < 6s 
then goto 6 
Sto,,, + Sh i h . else Staid := 2 ' 
3. N := N(Smla); 
4. If c(g) < Co 
then Shiflh :: Srnid 
else Sio~, := Smld; N¢ := N; 
5. Goto 2; 
6. Return Arc; 
Here C(N) is the coverage on the test set of the spe- 
cialized grammar determined by the set of cutnodes N. 
Actually, we also need to handle the boundary case 
where no assignment of cutnodes gives the required cov- 
erage. Likewise, the coverages of the upper and lower 
bound may be far apart even though the entropy dif- 
ference is small, and vice versa. These problems can 
readily be taken care of by modifying the termination 
criterion, but the solutions have been omitted for the 
sake of clarity. 
6albeit in exponential time 
192 
1) "S => Det N V Prep ~IP" 
s_np_vp /\ 
/ \ 
np_det_n vp_vp_pp 
/\ /\ 
lex lex vp_v pp_prep_np 
I /\ 
lex lex NP 
2) "S => Pron V NP" 
s_npvp /\ 
np_pron vp_v_np 
I /\ 
lex lex NP 
3) "NP => Det N" 
npdet_n 
/\ 
lex lex 
4) "NP => NP Prep NP" 
np_np_pp 
/\ 
NP pp_prep_np 
/\ 
lex NP 
5) "NP => Nu~" 
np _hUm 
I 
lex 
Figure 3: The specialized rules 
In the running example, using the weighted sum of 
the phrase entropies as the node entropy, if any thresh- 
old value less than 1.08 is chosen, this will yield any 
desired coverage, since the single test example of Fig- 
ure 1 is then covered. 
Retrieving the specialized rules 
When retrieving the specialized rules, we will match 
each training example against the and-or tree. If the 
current node is a cutnode, we will cut at this point in 
the training example. The resulting rules will be the 
set of cut-up training examples. A threshold value of 
say 1.00 in our example will yield the set of cutnodes 
{u3, n4, n6, ng} and result in the set of specialized rules 
of Figure 3. 
If we simply let the and-or tree determine the set 
of specialized rules, instead of using it to cut up the 
training examples, we will in general arrive at a larger 
number of rules, since some combinations of choices in 
6) "S => Det N V NP" 
s_np_vp 
/\ 
np_det_n vp_vnp 
/\ /\ 
lex lex lex NP 
7) "S => Pron Y Prep NP" 
s_np_vp /\ 
np_pren vpvp_pp 
I /\ 
lex vp_v pp_prep_np 
I /\ 
lex lex NP 
Figure 4: Additional specialized rules 
the and-or tree may not correspond to any training ex- 
ample. If this latter strategy is used in our example, 
this will give us the two extra rules of Figure 4. Note 
that they not correspond to any training example. 
ADDITIONAL CONSTRAINTS 
As mentioned at the beginning, the specialized gram- 
mar is compiled into LR parsing tables. Just finding 
any set of cutnodes that yields the desired coverage 
will not necessarily result in a grammar that is well 
suited for LP~ parsing. In particular, LR parsers, like 
any other parsers employing a bottom-up parsing strat- 
egy, do not blend well with empty productions. This is 
because without top-down filtering, any empty produc- 
tion is applicable at any point in the input string, and a 
naive bottom-up parser will loop indefinitely. The LR 
parsing tables constitute a type of top-down filtering, 
but this may not be sufficient to guarantee termination, 
and in any case, a lot of spurious applications of empty 
productions will most likely take place, degrading per- 
formance. For these reasons we will not allow learned 
rules whose RHSs are empty, but simply refrain from 
cutting in nodes of the parse trees that do not dominate 
at least one lexical lookup. 
Even so, the scheme described this far is not totally 
successful, the performance is not as good as using 
hand-coded tree-cutting criteria. This is conjectured 
to be an effect of the reduction lengths being far too 
short. The first reason for this is that for any spurious 
rule reduction to take place, the corresponding RHS 
phrases must be on the stack. The likelihood for this to 
happen by chance decreases drastically with increased 
rule length. A second reason for this is that the number 
of states visited will decrease with increasing reduction 
length. This can most easily be seen by noting that the 
number of states visited by a deterministic LR parser 
equals the number of shift actions plus the number of 
reductions, and equals the number of nodes in the cot- 
193 
responding parse tree, and the longer the reductions, 
the more shallow the parse tree. 
The hand-coded operationality criteria result in an 
average rule length of four, and a distribution of reduc- 
tion lengths that is such that only 17 percent are of 
length one and 11 percent are of length two. This is in 
sharp contrast to what the above scheme accomplishes; 
the corresponding figures are about 20 or 30 percent 
each for lengths one and two. 
An attempted solution to this problem is to impose 
restrictions on neighbouring cutnodes. This can be 
done in several ways; one that has been tested is to 
select for each rule the RHS phrase with the least en- 
tropy, and prescribe that if a node corresponding to the 
LHS of the rule is chosen as a cutnode, then no node 
corresponding to this RHS phrase may be chosen as a 
cutnode, and vice versa. In case of such a conflict, the 
node (class) with the lowest entropy is removed from 
the set of cutnodes. 
We modify the function N* to handle this: 
2. Repeat i := i+ 1; N i := N(N i-1) \ B(Ni-1); 
Here B(NJ) is the set of nodes in NJ that should be re- 
moved to avoid violating the constraints on neighbour- 
ing cutnodes. It is also necessary to modify the termi- 
nation criterion as was done for the function N(S,,~in) 
above. Now we can no longer safely assume that the 
coverage increases with decreased entropy, and we must 
also modify the interval-bisection scheme to handle this. 
It has proved reasonable to assume that the coverage 
is monotone on both sides of some maximum, which 
simplifies this task considerably. 
EXPERIMENTAL RESULTS 
A module realizing this scheme has been implemented 
and applied to the very setup used for the previous ex- 
periments with the hand-coded tree-cutting criteria, see 
\[Samuelsson 1994a\]. 2100 of the verified parse trees con- 
stituted the training set, while 230 of them were used 
for the test set. The table below summarizes the re- 
sults for some grammars of different coverage extracted 
using: 
1. Hand-coded tree-cutting criteria. 
2. Induced tree-cutting criteria where the node entropy 
was taken to be the phrase entropy of the RHIS phrase 
of the dominating grammar rule. 
3. Induced tree-cutting criteria where the node entropy 
was the sum of the phrase entropy of the RHS phrase 
of the dominating grammar rule and the weighted 
sum of the phrase entropies of the LHSs of the alter- 
native choices of grammar rules to resolve on. 
In the latter two cases experiments were carried out 
both with and without the restrictions on neighbouring 
cutnodes discussed in the previous section. 
Coverage 
90.2 % 
Hand-coded tree-cutting criteria 
Reduction lengths (%) Times (ms) 
1 2 3 > 4 Ave. Med. 
17.3 11.3 21.6 49.8 72.6 48.0 
RHS phrase entropy. Neighbour restrictions 
Coverage Reduction lengths (%) Times (ms) 
1 2 3 > 4 Ave. Med. 
75.8% 11.8 26.1 17.7 44.4 128 38.5 
80.5% 11.5 27.4 20.0 41.1 133 47.2 
85.3% 14.0 37.3 24.3 24.4 241 70.5 
RI-IS phrase entropy. 
Coverage Reduction 
1 2 
75.8 % 8.3 12.4 
79.7 % 9.0 16.2 
85.3 9{ 8.4 17.3 
90.9 % 18.2 27.5 
No neighbour restrictions 
lengths (%) Times (ms) 
3 > 4 Ave. Med. 
25.6 53.7 76.7 37.0 
26.9 47.9 99.1 49.4 
31.1 43.2 186 74.0 
21.7 32.6 469 126 
Mixed phrase entropies. Neighbour restrictions 
Coverage Reduction lengths (%) Times (ms) 
1 2 3 > 4 Ave. Med. 
75.3 % 6.1 11.7 30.8 51.4 115.4 37.5 
Mixed phrase entropies. No neighbour restrictions 
Coverage Reduction lengths (%) Times (ms) 
1 2 3 > 4 Ave. Med. 
75 % 16.1 13.8 19.8 50.3 700 92.0 
80 % 18.3 16.3 20.1 45.3 842 108 
With the mixed entropy scheme it seems important 
to include the restrictions on neighbouring cutnodes, 
while this does not seem to be the case with the RHS 
phrase entropy scheme. A potential explanation for the 
significantly higher average parsing times for all gram- 
mars extracted using the induced tree-cutting criteria 
is that these are in general recursive, while the hand- 
coded criteria do not allow recursion, and thus only 
produce grammars that generate finite languages. 
Although the hand-coded tree-cutting criteria are 
substantially better than the induced ones, we must 
remember that the former produce a grammar that in 
median allows 60 times faster processing than the orig- 
inal grammar and parser do. This means that even if 
the induced criteria produce grammars that are a fac- 
tor two or three slower than this, they are still approx- 
imately one and a half order of magnitude faster than 
the original setup. Also, this is by no means a closed 
research issue, but merely a first attempt to realize the 
scheme, and there is no doubt in my mind that it can 
be improved on most substantially. 
SUMMARY 
This article proposes a method for automatically find- 
ing the appropriate tree-cutting criteria in the EBG 
scheme, rather than having to hand-code them. The 
EBG scheme has previously proved most successful for 
194 
tuning a natural-language grammar to a specific ap- 
plication domain and thereby achieve very much faster 
parsing, at the cost of a small reduction in coverage. 
Instruments have been developed and tested for con- 
trolling the coverage and for avoiding a large number 
of short reductions, which is argued to be the main 
source to poor parser performance. Although these 
instruments are currently slightly too blunt to enable 
producing grammars with the same high performance 
as the hand-coded tree-cutting criteria, they can most 
probably be sharpened by future research, and in par- 
ticular refined to achieve the delicate balance between 
high coverage and a distribution of reduction lengths 
that is sufficiently biased towards long reductions. Also, 
banning recursion by category specialization, i.e. by for 
example distinguishing NPs that dominate other NPs 
from those that do not, will he investigated, since this is 
believed to be an important ingredient in the version of 
the scheme employing hand-coded tree-cutting criteria. 
ACKNOWLEDGEMENTS 
This research was made possible by the basic research 
programme at the Swedish Institute of Computer Sci- 
ence (SICS). I wish to thank Manny Rayner of SRI 
International, Cambridge, for help and support in mat- 
ters pertaining to the treebank, and for enlightening 
discussions of the scheme as a whole. I also wish to 
thank the NLP group at SICS for contributing to a 
very conductive atmosphere to work in, and in particu- 
lar Ivan Bretan for valuable comments on draft versions 
of this article. Finally, I wish to thank the anonymous 
reviewers for their comments. 

References 
\[Alshawi ed. 1992\] Hiyan Alshawi, editor. The Core 
Language Engine, MIT Press 1992. 
\[Jelinek 1990\] Fred Jelinek. "Self-Organizing Language 
Models for Speech Recognition", in Readings in 
Speech Recognition, pp. 450-506, Morgan Kauf- 
mann 1990. 
\[Mitchell el al 1986\] 
Tom M. Mitchell, Richard M. Keller and Smadar 
T. Kedar-Cabelli. "Explanation-Based Generaliza- 
tion: A Unifying View", in Machine Learning 1, 
No. l, pp. 47-80, 1986. 
\[Quinlan 1986\] J. Ross Quinlan. "Induction of Decision 
Trees", in Machine Learning 1, No. 1, pp. 81-107, 
1986. 
\[Rayner et al 1993\] M. Rayner, H. Alshawi, I. Bretan, 
D. Carter, V. Digalakis, B. Gamb£ck, J. Kaja, 
J. Karlgren, B. Lyberg, P. Price, S. Pulman and 
C. Samuelsson. "A Speech to Speech Transla- 
tion System Built From Standard Components", 
in Procs. ARPA Workshop on Human Language 
Technology, Princeton, NJ 1993. 
\[Samuelsson 1994a\] Christer Samuelsson. Fast Natural- 
Language Parsing Using Explanation-Based Learn- 
ing, PhD thesis, Royal Institute of Technology, 
Stockholm, Sweden 1994. 
\[Samuelsson 1994b\] Christer Samuelsson. "Notes on 
LR Parser Design" to appear in Procs. 15th In- 
ternational Conference on Computational Linguis- 
tics, Kyoto, Japan 1994. 
\[Samuelsson   Rayner 1991\] Christer Samuelsson and 
Manny Rayner. "Quantitative Evaluation of Ex- 
planation-Based Learning as an Optimization Tool 
for a Large-ScMe Natural Language System", in 
Procs. 12th International Joint Conference on Ar- 
tificial Intelligence, pp. 609-615, Sydney, Australia 
1991. 
