Memory-Based Learning: Using Similarity for Smoothing 
Jakub Zavrel and Walter Daelemans 
Computational Linguistics 
Tilburg University 
PO Box 90153 
5000 LE Tilburg 
The Netherlands 
{zavrel, walt er}@kub, nl 
Abstract 
This paper analyses the relation between 
the use of similarity in Memory-Based 
Learning and the notion of backed-off 
smoothing in statistical language model- 
ing. We show that the two approaches are 
closely related, and we argue that feature 
weighting methods in the Memory-Based 
paradigm can offer the advantage of au- 
tomatically specifying a suitable domain- 
specific hierarchy between most specific 
and most general conditioning information 
without the need for a large number of pa- 
rameters. We report two applications of 
this approach: PP-attachment and POS- 
tagging. Our method achieves state-of-the- 
art performance in both domains, and al- 
lows the easy integration of diverse infor- 
mation sources, such as rich lexical repre- 
sentations. 
1 Introduction 
Statistical approaches to disambiguation offer the 
advantage of making the most likely decision on the 
basis of available evidence. For this purpose a large 
number of probabilities has to be estimated from a 
training corpus. However, many possible condition- 
ing events are not present in the training data, yield- 
ing zero Maximum Likelihood (ML) estimates. This 
motivates the need for smoothing methods, which re- 
estimate the probabilities of low-count events from 
more reliable estimates. 
Inductive generalization from observed to new 
data lies at the heart of machine-learning approaches 
to disambiguation. In Memory-Based Learning 1 
(MBL) induction is based on the use of similarity 
(Stanfill & Waltz, 1986; Aha et al., 1991; Cardie, 
1994; Daelemans, 1995). In this paper we describe 
how the use of similarity between patterns embod- 
ies a solution to the sparse data problem, how it 
1The Approach is also referred to as Case-based, 
Instance-based or Exemplar-based. 
relates to backed-off smoothing methods and what 
advantages it offers when combining diverse and rich 
information sources. 
We illustrate the analysis by applying MBL to 
two tasks where combination of information sources 
promises to bring improved performance: PP- 
attachment disambiguation and Part of Speech tag- 
ging. 
2 Memory-Based Language 
Processing 
The basic idea in Memory-Based language process- 
ing is that processing and learning are fundamen- 
tally interwoven. Each language experience leaves a 
memory trace which can be used to guide later pro- 
cessing. When a new instance of a task is processed, 
a set of relevant instances is selected from memory, 
and the output is produced by analogy to that set. 
The techniques that are used are variants and 
extensions of the classic k-nearest neighbor (k- 
NN) classifier algorithm. The instances of a task 
are stored in a table as patterns of feature-value 
pairs, together with the associated "correct" out- 
put. When a new pattern is processed, the k nearest 
neighbors of the pattern are retrieved from memory 
using some similarity metric. The output is then de- 
termined by extrapolation from the k nearest neigh- 
bors, i.e. the output is chosen that has the highest 
relative frequency among the nearest neighbors. 
Note that no abstractions, such as grammatical 
rules, stochastic automata, or decision trees are ex- 
tracted from the examples. Rule-like behavior re- 
sults from the linguistic regularities that are present 
in the patterns of usage in memory in combination 
with the use of an appropriate similarity metric. 
It is our experience that even limited forms of ab- 
straction can harm performance on linguistic tasks, 
which often contain many subregularities and excep- 
tions (Daelemans, 1996). 
2.1 Similarity metrics 
The most basic metric for patterns with symbolic 
features is the Overlap metric given in equations 1 
436 
and 2; where A(X, Y) is the distance between pat- 
terns X and Y, represented by n features, wi is a 
weight for feature i, and 5 is the distance per fea- 
ture. The k-NN algorithm with this metric, and 
equal weighting for all features is called IB1 (Aha 
et al., 1991). Usually k is set to 1. 
where: 
A(X, Y) = ~ wi 6(xi, yi) (1) 
i=l 
tf(xi,yi) = 0 if xi = yi, else 1 (2) 
This metric simply counts the number of 
(mis)matching feature values in both patterns. If 
we do not have information about the importance 
of features, this is a reasonable choice. But if we 
do have some information about feature relevance 
one possibility would be to add linguistic bias to 
weight or select different features (Cardie, 1996). An 
alternative--more empiricist--approach, is to look 
at the behavior of features in the set of examples 
used for training. We can compute statistics about 
the relevance of features by looking at which fea- 
tures are good predictors of the class labels. Infor- 
mation Theory gives us a useful tool for measuring 
feature relevance in this way (Quinlan, 1986; Quin- 
lan, 1993). 
Information Gain (IG) weighting looks at each 
feature in isolation, and measures how much infor- 
mation it contributes to our knowledge of the cor- 
rect class label. The Information Gain of feature f 
is measured by computing the difference in uncer- 
tainty (i.e. entropy) between the situations with- 
out and with knowledge of the value of that feature 
(Equation 3). 
w\] = H(C) - ~-\]~ev, P(v) x H(Clv ) 
si(f) (3) 
si(f) = - Z P(v)log 2 P(v) (4) 
vEVs 
Where C is the set of class labels, V f is 
the set of values for feature f, and H(C) = 
- ~cec P(c) log 2 P(e) is the entropy of the class la- 
bels. The probabilities are estimated from relative 
frequencies in the training set. The normalizing fac- 
tor si(f) (split info) is included to avoid a bias in 
favor of features with more values. It represents the 
amount of information needed to represent all val- 
ues of the feature (Equation 4). The resulting IG 
values can then be used as weights in equation 1. 
The k-NN algorithm with this metric is called ml- 
IG (Daelemans & Van den Bosch, 1992). 
The possibility of automatically determining the 
relevance of features implies that many different and 
possibly irrelevant features can be added to the fea- 
ture set. This is a very convenient methodology if 
theory does not constrain the choice enough before- 
hand, or if we wish to measure the importance of 
various information sources experimentally. 
Finally, it should be mentioned that MB- 
classifiers, despite their description as table-lookup 
algorithms here, can be implemented to work 
fast, using e.g. tree-based indexing into the case- 
base (Daelemans et al., 1997). 
3 Smoothing of Estimates 
The commonly used method for probabilistic clas- 
sification (the Bayesian classifier) chooses a class 
for a pattern X by picking the class that has the 
maximum conditional probability P(classlX ). This 
probability is estimated from the data set by looking 
at the relative joint frequency of occurrence of the 
classes and pattern X. If pattern X is described by 
a number of feature-values Xl,..., xn, we can write 
the conditional probability as P(classlxl,... , xn). If 
a particular pattern x~,..., x" is not literally present 
among the examples, all classes have zero ML prob- 
ability estimates. Smoothing methods are needed to 
avoid zeroes on events that could occur in the test 
material. 
There are two main approaches to smoothing: 
count re-estimation smoothing such as the Add-One 
or Good-Turing methods (Church & Gale, 1991), 
and Back-off type methods (Bahl et al., 1983; Katz, 
1987; Chen & Goodman, 1996; Samuelsson, 1996). 
We will focus here on a comparison with Back-off 
type methods, because an experimental comparison 
in Chen & Goodman (1996) shows the superiority 
of Back-off based methods over count re-estimation 
smoothing methods. With the Back-off method the 
probabilities of complex conditioning events are ap- 
proximated by (a linear interpolation of) the proba- 
bilities of more general events: 
/5(ctasslX) = ~x/3(clas~lX) + ~x,/3(d~sslX') 
+... + ),x,~(dasslX n) (5) 
Where/5 stands for the smoothed estimate,/3 for 
the relative frequency estimate, A are interpolation 
weights, ~-'\]i~0"kx' = 1, and X -< X i for all i, 
where -< is a (partial) ordering from most specific 
to most general feature-sets 2 (e.g the probabilities 
of trigrams (X) can be approximated by bigrams 
(X') and unigrams (X")). The weights of the lin- 
ear interpolation are estimated by maximizing the 
probability of held-out data (deleted interpolation) 
with the forward-backward algorithm. An alterna- 
tive method to determine the interpolation weights 
without iterative training on held-out data is given 
in Samuelsson (1996). 
2X -< X' can be read as X is more specific than X'. 
437 
We can assume for simplicity's sake that the Ax, 
do not depend on the value of X i, but only on i. In 
this case, if F is the number of features, there are 
2 F - 1 more general terms, and we need to estimate 
A~'s for all of these. In most applications the inter- 
polation method is used for tasks with clear order- 
ings of feature-sets (e.g. n-gram language modeling) 
so that many of the 2 F -- 1 terms can be omitted 
beforehand. More recently, the integration of infor- 
mation sources, and the modeling of more complex 
language processing tasks in the statistical frame- 
work has increased the interest in smoothing meth- 
ods (Collins ~z Brooks, 1995; Ratnaparkhi, 1996; 
Magerman, 1994; Ng & Lee, 1996; Collins, 1996). 
For such applications with a diverse set of features 
it is not necessarily the case that terms can be ex- 
cluded beforehand. 
If we let the Axe depend on the value of X ~, the 
number of parameters explodes even faster. A prac- 
tical solution for this is to make a smaller number 
of buckets for the X i, e.g. by clustering (see e.g. 
Magerman (1994)): 
Note that linear interpolation (equation 5) actu- 
ally performs two functions. In the first place, if the 
most specific terms have non-zero frequency, it still 
interpolates them with the more general terms. Be- 
cause the more general terms should never overrule 
the more specific ones, the Ax e for the more general 
terms should be quite small. Therefore the inter- 
polation effect is usually small or negligible. The 
second function is the pure back-off function: if the 
more specific terms have zero frequency, the proba- 
bilities of the more general terms are used instead. 
Only if terms are of a similar specificity, the A's truly 
serve to weight relevance of the interpolation terms. 
If we isolate the pure back-off function of the in- 
terpolation equation we get an algorithm similar to 
the one used in Collins & Brooks (1995). It is given 
in a schematic form in Table 1. Each step consists 
of a back-off to a lower level of specificity. There 
are as many steps as features, and there are a total 
of 2 F terms, divided over all the steps. Because all 
features are considered of equal importance, we call 
this the Naive Back-off algorithm. 
Usually, not all features x are equally important, 
so that not all back-off terms are equally relevant 
for the re-estimation. Hence, the problem of fitting 
the Axe parameters is replaced by a term selection 
task. To optimize the term selection, an evaluation 
of the up to 2 F terms on held-out data is still neces- 
sary. In summary, the Back-off method does not pro- 
vide a principled and practical domain-independent 
method to adapt to the structure of a particular do- 
main by determining a suitable ordering -< between 
events. In the next section, we will argue that a for- 
mal operationalization of similarity between events, 
as provided by MBL, can be used for this purpose. 
In MBL the similarity metric and feature weighting 
scheme automatically determine the implicit back- 
If f(xl,..., xn) > 0: 
#(clzl ...,xn) = f(c,~l ..... ~.) ' f(~l ..... ~.) 
Else if f(xl, ...,Xn-1, *) "4- ... A- f(*,x2, ...,Xn) > O: 
~(clzl,...,zn) = f(c,~l ..... ~,-1,,)+...+f(c,*,~2 ..... ~) f(zl,...,z.-1,*)+...+/(*,z2 ..... z.) 
Else if ... : 
~(clzl, ..., z,~) = 
Else if f(xl, *, ..., *) + ... + f(*, ..., *, x,~) > O: 
~(clzl,...,x~) = f(c'~l'* ...... )++f(c'*' ........ ) f(zl,*,...,*)+...+/(*,...,*,z~) 
Table 1: The Naive Back-off smoothing algorithm. 
f(X) stands for the frequency of pattern X in the 
training set. An asterix (*) stands for a wildcard in 
a pattern. The terms at a higher level in the back-off 
sequence are more specific (-<) than the lower levels. 
off ordering using a domain independent heuristic, 
with only a few parameters, in which there is no 
need for held-out data. 
4 A Comparison 
If we classify pattern X by looking at its nearest 
neighbors, we are in fact estimating the probabil- 
ity P(classlX), by looking at the relative frequency 
of the class in the set defined by simk(X), where 
slink(X) is a function from X to the set of most sim- 
ilar patterns present in the training data 3. Although 
the name "k-nearest neighbor" might mislead us by 
suggesting that classification is based on exactly k 
training patterns, the sima(X) fimction given by the 
Overlap metric groups varying numbers of patterns 
into buckets of equal similarity. A bucket is defined 
by a particular number of mismatches with respect 
to pattern X. Each bucket can further be decom- 
posed into a number of schemata characterized by 
the position of a wildcard (i.e. a mismatch). Thus 
simk(X) specifies a ~ ordering in a Collins 8z Brooks 
style back-off sequence, where each bucket is a step 
in the sequence, and each schema is a term in the 
estimation formula at that step. In fact, the un- 
weighted overlap metric specifies exactly the same 
ordering as the Naive Back-off algorithm (table 1). 
In Figure 1 this is shown for a four-featured pat- 
tern. The most specific schema is the schema with 
zero mismatches, which corresponds to the retrieval 
of an identical pattern from memory, the most gen- 
eral schema (not shown in the Figure) has a mis- 
match on every feature, which corresponds to the 
3Note that MBL is not limited to choosing the best 
class. It can also return the conditional distribution of 
all the classes. 
438 
Overlap 
exact match 
IIIII 
Overlap IG 
IIitl 
1 mismatch 2 mismatches 3 mismatches 
X X 
X X 
xx XXX XX × XX 
x ~ X ×× X ×xx 
× X x X 
IX\] I I I 1><\]><3 I I ~ I 1><3 I I IX\] I I><:\] 
I I I IX\] I IXI IX\] 
Figure 1: An analysis of nearest neighbor sets into buckets (from left to right) and schemata (stacked). IG 
weights reorder the schemata. The grey schemata are not used if the third feature has a very high weight 
(see section 5.1). 
entire memory being best neighbor. 
If Information Gain weights are used in combina- 
tion with the Overlap metric, individual schemata 
instead of buckets become the steps of the back-off 
sequence 4. The -~ ordering becomes slightly more 
complicated now, as it depends on the number of 
wildcards and on the magnitude of the weights at- 
tached to those wildcards. Let S be the most specific 
(zero mismatches) schema. We can then define the 
ordering between schemata in the following equa- 
tion, where A(X,Y) is the distance as defined in 
equation 1. 
s' -< s" ~ ~,(s, s') < a(s, s") (6) 
Note that this approach represents a type of im- 
plicit parallelism. The importance of the 2~back-off 
terms is specified using only F parameters--the IG 
weights-, where F is the number of features. This 
advantage is not restricted to the use of IG weights; 
many other weighting schemes exist in the machine 
learning literature (see Wettschereck et aL (1997) 
for an overview). 
Using the IG weights causes the algorithm to rely 
on the most specific schema only. Although in most 
applications this leads to a higher accuracy, because 
it rejects schemata which do not match the most 
important features, sometimes this constraint needs 
4Unless two schemata are exactly tied in their IG 
values. 
to be weakened. This is desirable when: (i) there 
are a number of schemata which are almost equally 
relevant, (ii) the top ranked schema selects too few 
cases to make a reliable estimate, or (iii) the chance 
that the few items instantiating the schema are mis- 
labeled in the training material is high. In such 
cases we wish to include some of the lower-ranked 
schemata. For case (i) this can be done by discretiz- 
ing the IG weights into bins, so that minor differ- 
ences will lose their significance, in effect merging 
some schemata back into buckets. For (ii) and (iii), 
and for continuous metrics (Stanfill & Waltz, 1986; 
Cost & Salzberg, 1993) which extrapolate from ex- 
actly k neighbors 5, it might be necessary to choose a 
k parameter larger than 1. This introduces one addi- 
tional parameter, which has to be tuned on held-out 
data. We can then use the distance between a pat- 
tern and a schema to weight its vote in the nearest 
neighbor extrapolation. This results in a back-off 
sequence in which the terms at each step in the se- 
quence are weighted with respect to each other, but 
without the introduction of any additional weight- 
ing parameters. A weighted voting function that 
was found to work well is due to Dudani (1976): the 
nearest neighbor schema receives a weight of 1.0, the 
furthest schema a weight of 0.0, and the other neigh- 
bors are scaled linearly to the line between these two 
points. 
5Note that the schema analysis does not apply to 
these metrics. 
439 
Method 
IB1 (=Naive Back-off) 
IBI-IG 
LexSpace IG 
Back-off model (Collins & Brooks) 
C4.5 (Ratnaparkhi et al.) 
Max Entropy (Ratnaparkhi et al.) 
Brill's rules (Collins 8z Brooks) 
% Accuracy 
83.7 % 
84.1% 
84.4 % 
84.1% 
79.9 % 
81.6 % 
81.9 % 
Table 2: Accuracy on the PP-attachment test set. 
5 Applications 
5.1 PP-attachment 
In this section we describe experiments with MBL 
on a data-set of Prepositional Phrase (PP) attach- 
ment disambiguation cases. The problem in this 
data-set is to disambiguate whether a PP attaches 
to the verb (as in I ate pizza with a fork) or to the 
noun (as in I ate pizza with cheese). This is a dif- 
ficult and important problem, because the semantic 
knowledge needed to solve the problem is very diffi- 
cult to model, and the ambiguity can lead to a very 
large number of interpretations for sentences. 
We used a data-set extracted from the Penn 
Treebank WSJ corpus by Ratnaparkhi et al. (1994). 
It consists of sentences containing the possibly 
ambiguous sequence verb noun-phrase PP. Cases 
were constructed from these sentences by record- 
ing the features: verb, head noun of the first noun 
phrase, preposition, and head noun of the noun 
phrase contained in the PP. The cases were la- 
beled with the attachment decision as made by 
the parse annotator of the corpus. So, for the 
two example sentences given above we would get 
the feature vectors ate,pizza,with,fork,V, and 
ate,pizza, with, cheese, N. The data-set contains 
20801 training cases and 3097 separate test cases, 
and was also used in Collins & Brooks (1995). 
The IG weights for the four features (V,N,P,N) 
were respectively 0.03, 0.03, 0.10, 0.03. This identi- 
fies the preposition as the most important feature: 
its weight is higher than the sum of the other three 
weights. The composition of the back-off sequence 
following from this can be seen in the lower part 
of Figure 1. The grey-colored schemata were effec- 
tively left out, because they include a mismatch on 
the preposition. 
Table 2 shows a comparison of accuracy on the 
test-set of 3097 cases. We can see that Isl, which 
implicitly uses the same specificity ordering as the 
Naive Back-off algorithm already performs quite well 
in relation to other methods used in the literature. 
Collins & Brooks' (1995) Back-off model is more so- 
phisticated than the naive model, because they per- 
formed a number of validation experiments on held- 
out data to determine which terms to include and, 
more importantly, which to exclude from the back- 
off sequence. They excluded all terms which did 
not match in the preposition! Not surprisingly, the 
84.1% accuracy they achieve is matched by the per- 
formance of IBI-IG. The two methods exactly mimic 
each others behavior, in spite of their huge differ- 
ence in design. It should however be noted that the 
computation of IG-weights is many orders of mag- 
nitude faster than the laborious evaluation of terms 
on held-out data. 
We also experimented with rich lexical represen- 
tations obtained in an unsupervised way from word 
co-occurrences in raw WSJ text (Zavrel & Veenstra, 
1995; Schiitze, 1994). We call these representations 
Lexical Space vectors. Each word has a numeric 25 
dimensional vector representation. Using these vec- 
tors, in combination with the IG weights mentioned 
above and a cosine metric, we got even slightly bet- 
ter results. Because the cosine metric fails to group 
the patterns into discrete schemata, it is necessary 
to use a larger number of neighbors (k = 50). The 
result in Table 2 is obtained using Dudani's weighted 
voting method. 
Note that to devise a back-off scheme on the basis 
of these high-dimensional representations (each pat- 
tern has 4 x 25 features) one would need to consider 
up to 2 l°° smoothing terms. The MBL framework 
is a convenient way to further experiment with even 
more complex conditioning events, e.g. with seman- 
tic labels added as features. 
5.2 POS-tagging 
Another NLP problem where combination of differ- 
ent sources of statistical information is an impor- 
tant issue, is POS-tagging, especially for the guess- 
ing of the POS-tag of words not present in the lex- 
icon. Relevant information for guessing the tag of 
an unknown word includes contextual information 
(the words and tags in the context of the word), and 
word form information (prefixes and suffixes, first 
and last letters of the word as an approximation of 
affix information, presence or absence of capitaliza- 
tion, numbers, special characters etc.). There is a 
large number of potentially informative features that 
could play a role in correctly predicting the tag of 
an unknown word (Ratnaparkhi, 1996; Weischedel 
et al., 1993; Daelemans et al., 1996). A priori, it 
is not clear what the relative importance is of these 
features. 
We compared Naive Back-off estimation and MBL 
with two sets of features: 
• PDASS: the first letter of the unknown word (p), 
the tag of the word to the left of the unknown 
word (d), a tag representing the set of possible 
lexical categories of the word to the right of the 
unknown word (a), and the two last letters (s). 
The first letter provides information about cap- 
italisation and the prefix, the two last letters 
440 
about suffixes. 
• PDDDAAASSS: more left and right context fea- 
tures, and more suffix information. 
The data set consisted of 100,000 feature value 
patterns taken from the Wall Street Journal corpus. 
Only open-class words were used during construc- 
tion of the training set. For both IBI-IG and Naive 
Back-off, a 10-fold cross-validation experiment was 
run using both PDASS and PDDDAAASSS patterns. 
The results are in Table 3. The IG values for the 
features are given in Figure 2. 
The results show that for Naive Back-off (and ml) 
the addition of more, possibly irrelevant, features 
quickly becomes detrimental (decrease from 88.5 to 
85.9), even if these added features do make a gener- 
alisation performance increase possible (witness the 
increase with IBI-IG from 88.3 to 89.8). Notice that 
we did not actually compute the 21° terms of Naive 
Back-off in the PDDDAAASSS condition, as IB1 is 
guaranteed to provide statistically the same results. 
Contrary to Naive Back-off and IB1, memory-based 
learning with feature weighting (ml-IG) manages 
to integrate diverse information sources by differ- 
entially assigning relevance to the different features. 
Since noisy features will receive low IG weights, this 
also implies that it is much more noise-tolerant. 
0.30 - 
0.25 - 
0.20 - 
F 
o~ "3 
~_ 0.15- 
0.10 , 
0.05 - 
0.0- 
p d d d a a a s s s 
feature 
Figure 2: IG values for features used in predicting 
the tag of unknown words. 
IB1, Naive Back-off IBI-IG 
PDASS 88.5 (0.4) 88.3 (0.4) 
PDDDAAASSS 85.9 (0.4) 89.8 (0.4) 
Table 3: Comparison of generalization accuracy of 
Back-off and Memory-Based Learning on prediction 
of category of unknown words. All differences are 
statistically significant (two-tailed paired t-test, p < 
0.05). Standard deviations on the 10 experiments 
are between brackets. 
6 Conclusion 
We have analysed the relationship between Back- 
off smoothing and Memory-Based Learning and es- 
tablished a close correspondence between these two 
frameworks which were hitherto mostly seen as un- 
related. An exception is the use of similarity for al- 
leviating the sparse data problem in language mod- 
eling (Essen & Steinbiss, 1992; Brown et al., 1992; 
Dagan et al., 1994). However, these works differ in 
their focus from our analysis in that the emphasis 
is put on similarity between values of a feature (e.g. 
words), instead of similarity between patterns that 
are a (possibly complex) combination of many fea- 
tures. 
The comparison of MBL and Back-off shows that 
the two approaches perform smoothing in a very sim- 
ilar way, i.e. by using estimates from more general 
patterns if specific patterns are absent in the train- 
ing data. The analysis shows that MBL and Back-off 
use exactly the same type of data and counts, and 
this implies that MBL can safely be incorporated 
into a system that is explicitly probabilistic. Since 
the underlying k-NN classifier is a method that does 
not necessitate any of the common independence or 
distribution assumptions, this promises to be a fruit- 
ful approach. 
A serious advantage of the described approach, 
is that in MBL the back-off sequence is specified 
by the used similarity metric, without manual in- 
tervention or the estimation of smoothing parame- 
ters on held-out data, and requires only one param- 
eter for each feature instead of an exponential num- 
ber of parameters. With a feature-weighting met- 
ric such as Information Gain, MBL is particularly 
at an advantage for NLP tasks where conditioning 
events are complex, where they consist of the fusion 
of different information sources, or when the data is 
noisy. This was illustrated by the experiments on 
PP-attachment and POS-tagging data-sets. 
441 
Acknowledgements 
This research was done in the context of the "Induc- 
tion of Linguistic Knowledge" research programme, 
partially supported by the Foundation for Lan- 
guage Speech and Logic (TSL), which is funded by 
the Netherlands Organization for Scientific Research 
(NWO). We would like to thank Peter Berck and 
Anders Green for their help with software for the 
experiments. 

References 
D. Aha, D. Kibler, and M. Albert. 1991. Instance- 
based Learning Algorithms. Machine Learning, 
Vol. 6, pp. 37-66. 
L.R. Bahl, F. Jelinek and R.L. Mercer. 1983. 
A Maximum Likelihood Approach to Continu- 
ous Speech Recognition. IEEE Transactions on 
Pattern Analysis and Machine Intelligence, Vol. 
PAMI-5 (2), pp. 179-190. 
Peter F. Brown, Vincent J. Della Pietra, Peter 
V. deSouza, Jennifer C. Lai, and Robert L. Mer- 
cer. 1992. Class-based N-gram Models of Natural 
Language. Computational Linguistics, Vol. 18(4), 
pp. 467-479. 
Claire Cardie. 1994. Domain Specific Knowl- 
edge Acquisition for Conceptual Sentence Anal- 
ysis, PhD Thesis, University of Massachusets, 
Amherst, MA. 
Claire Cardie. 1996. Automatic Feature Set Selec- 
tion for Case-Based Learning of Linguistic Knowl- 
edge. In Proc. of the Conference on Empirical 
Methods in Natural Language Processing, May 17- 
18, 1996, University of Pennsylvania. 
Stanley F.Chen and Joshua Goodman. 1996. An 
Empirical Study of Smoothing Techniques for 
Language Modelling. In Proc. of the 34th Annual 
Meeting of the ACL, June 1996, Santa Cruz, CA, 
ACL. 
Kenneth W. Church and William A. Gale. 1991. 
A comparison of the enhanced Good-Turing and 
deleted estimation methods for estimating proba- 
bilities of English bigrams. Computer Speech and 
Language, Vol 19(5), pp. 19-54. 
M. Collins. 1996. A New Statistical Parser Based on 
Bigram Lexical Dependencies. In Proc. of the 34th 
Annual Meeting of the ACL, June 1996, Santa 
Cruz, CA, ACL. 
M. Collins and J. Brooks. 1995. Prepositional 
Phrase Attachment through a Backed-Off Model. 
In Proceedings of the Third Workshop on Very 
Large Corpora, Cambridge, MA. 
S. Cost and S. Salzberg. 1993. A weighted near- 
est neighbour algorithm for learning with symbolic 
features. Machine Learning, Vol. 10, pp. 57-78. 
Walter Daelemans and Antal van den Bosch. 
1992. Generalisation Performance of Backprop- 
agation Learning on a Syllabification Task. In 
M. F. J. Drossaers & A. Nijholt (eds.), TWLT3: 
Connectionism and Natural Language Processing. 
Enschede: Twente University. pp. 27-37. 
Walter Daelemans. 1995. Memory-based lexical 
acquisition and processing. In P. Steffens (ed.), 
Machine Translation and the Lexicon. Springer 
Lecture Notes in Artificial Intelligence, no. 898. 
Berlin: Springer Verlag. pp. 85-98 
Walter Da~lemans. 1996. Abstraction Considered 
Harmful: Lazy Learning of Language Process- 
ing. In J. van den Herik and T. Weijters (eds.), 
Benelearn-96. Proceedings of the 6th Belgian- 
Dutch Conference on Machine Learning. MA- 
TRIKS: Maastricht, The Netherlands, pp. 3-12. 
Walter Daelemans, Jakub Zavrel, Peter Berck, and 
Steven Gillis. 1996. MBT: A Memory-Based Part 
of Speech Tagger Generator. In E. Ejerhed and 
I. Dagan (eds.) Proc. of the Fourth Workshop on 
Very Large Corpora, Copenhagen: ACL SIGDAT, 
pp. 14-27. 
Walter Daelemans, Antal van den Bosch, and Ton 
Weijters. 1997. IGTree: Using Trees for Com- 
pression and Classification in Lazy Learning Al- 
gorithms. In D. Aha (ed.) Artificial Intelligence 
Review, special issue on Lazy Learning, Vol. 11(1- 
5). 
Ido Dagan, Fernando Pereira, and Lillian Lee. 1994. 
Similarity-Based Estimation of Word Cooccur- 
fence Probabilities. In Proc. of the 32nd Annual 
Meeting of the ACL, June 1994, Las Cruces, New 
Mexico, ACL. 
S.A. Dudani. 1981 The Distance-Weighted k- 
Nearest Neighbor Rule IEEE Transactions on 
Systems, Man, and Cybernetics, Vol. SMC-6, pp. 
325-327. 
Ute Essen, and Volker Steinbiss. 1992. Coocurrence 
Smoothing for Stochastic Language Modeling. In 
Proc. ofICASSP, Vol. 1, pp. 161-164, IEEE. 
Slava M. Katz. 1987. Estimation of Probabilities 
from Sparse Data for the Language Model Com- 
ponent of a Speech Recognizer IEEE Transac- 
tions on Acoustics, Speech and Signal Processing, 
Vol. ASSP-35, pp. 400-401, March 1987. 
David M. Magerman. 1994. Natural Language Pars- 
ing as Statistical Pattern Recognition, PhD The- 
sis, Department of Computer Science, Stanford 
University. 
Hwee Tou Ng and Hian Beng Lee. 1996. Integrat- 
ing Multiple Knowledge Sources to Disambiguate 
Word Sense: An Exemplar-Based Approach. In 
Proc. of the 3~th Annual Meeting of the ACL, 
June 1996, Santa Cruz, CA, ACL. 
J..R. Quinlan. 1986. Induction of Decision Trees. 
Machine Learning, Vol. 1, pp. 81-206. 
J..R. Quinlan. 1993. ¢4.5: Programs for Machine 
Learning. San Mateo, CA: Morgan Kaufmann. 
Adwait Ratnaparkhi. 1996. A Maximum Entropy 
Part-Of-Speech Tagger. In Proc. of the Confer- 
ence on Empirical Methods in Natural Language 
Processing, May 17-18, 1996, University of Penn- 
sylvania. 
A. Ratnaparkhi, J. Reynar and S. Roukos. 1994. A 
maximum entropy model for Prepositional Phrase 
Attachment. In ARPA ~,Vorkshop on Human Lan- 
guage Technology, Plainsboro, NJ. 
Christer Samuelsson. 1996. Handling Sparse Data 
by Successive Abstraction In Proc. of the Interna- 
tional Confercncc on Computational Linguistics 
(COLING'96), August 1996, Copenhagen, Den- 
mark. 
Hinrich Sch/itze. 1994. Distributional Part-of- 
Speech Tagging. In Proc. of the 7th Conference of 
the European Chapter of the Association for Com- 
putational Linguistics (EACL'95), Dublin, Ire- 
land. 
C. Stanfill and D. Waltz. 1986. Toward memory- 
based reasoning. Communications of the ACM, 
Vol. 29, pp. 1213-1228. 
Ralph Weischedel, Marie Meteer, Richard Schwartz, 
Lance Ramshaw, and Jeff Palmucci. 1993. Cop- 
ing with Ambiguity and Unknown Words through 
Probabilistic Models. Computational Linguistics, 
Vol. 19(2). pp. 359-382. 
D. Wettschereck, D. W. Aha, and T. Mohri. 
1997. A Review and Comparative Evaluation of 
Feature-Weighting Methods for Lazy Learning Al- 
gorithms. In D. Aha (ed.) Artificial Intelligence 
Review, special issue on Lazy Learning, Vol. 11(1- 5). 
Jakub Zavrel and Jorn B. Veenstra. 1995. The Lan- 
guage Environment and Syntactic Word-Class Ac- 
quisition. In C.Koster and F.Wijnen (eds.) Proc. 
of the Groningen Assembly on Language Acquisi- 
tion (GALA95). Center for Language and Cogni- 
tion, Groningen, pp. 365-374. 
