Fast Methods for Kernel-based Text Analysis
Taku Kudo and Yuji Matsumoto
Graduate School of Information Science,
Nara Institute of Science and Technology
ftaku-ku,matsug@is.aist-nara.ac.jp
Abstract
Kernel-based learning (e.g., Support Vec-
tor Machines) has been successfully ap-
plied to many hard problems in Natural
Language Processing (NLP). In NLP, al-
though feature combinations are crucial to
improving performance, they are heuris-
tically selected. Kernel methods change
this situation. The merit of the kernel
methods is that effective feature combina-
tion is implicitly expanded without loss
of generality and increasing the compu-
tational costs. Kernel-based text analysis
shows an excellent performance in terms
in accuracy; however, these methods are
usually too slow to apply to large-scale
text analysis. In this paper, we extend
a Basket Mining algorithm to convert a
kernel-based classifier into a simple and
fast linear classifier. Experimental results
on English BaseNP Chunking, Japanese
Word Segmentation and Japanese Depen-
dency Parsing show that our new classi-
fiers are about 30 to 300 times faster than
the standard kernel-based classifiers.
1 Introduction
Kernel methods (e.g., Support Vector Machines
(Vapnik, 1995)) attract a great deal of attention re-
cently. In the field of Natural Language Process-
ing, many successes have been reported. Examples
include Part-of-Speech tagging (Nakagawa et al.,
2002) Text Chunking (Kudo and Matsumoto, 2001),
Named Entity Recognition (Isozaki and Kazawa,
2002), and Japanese Dependency Parsing (Kudo and
Matsumoto, 2000; Kudo and Matsumoto, 2002).
It is known in NLP that combination of features
contributes to a significant improvement in accuracy.
For instance, in the task of dependency parsing, it
would be hard to confirm a correct dependency re-
lation with only a single set of features from either
a head or its modifier. Rather, dependency relations
should be determined by at least information from
both of two phrases. In previous research, feature
combination has been selected manually, and the
performance significantly depended on these selec-
tions. This is not the case with kernel-based method-
ology. For instance, if we use a polynomial ker-
nel, all feature combinations are implicitly expanded
without loss of generality and increasing the compu-
tational costs. Although the mapped feature space
is quite large, the maximal margin strategy (Vapnik,
1995) of SVMs gives us a good generalization per-
formance compared to the previous manual feature
selection. This is the main reason why kernel-based
learning has delivered great results to the field of
NLP.
Kernel-based text analysis shows an excellent per-
formance in terms in accuracy; however, its inef-
ficiency in actual analysis limits practical applica-
tion. For example, an SVM-based NE-chunker runs
at a rate of only 85 byte/sec, while previous rule-
based system can process several kilobytes per sec-
ond (Isozaki and Kazawa, 2002). Such slow exe-
cution time is inadequate for Information Retrieval,
Question Answering, or Text Mining, where fast
analysis of large quantities of text is indispensable.
This paper presents two novel methods that make
the kernel-based text analyzers substantially faster.
These methods are applicable not only to the NLP
tasks but also to general machine learning tasks
where training and test examples are represented in
a binary vector.
More specifically, we focus on a Polynomial Ker-
nel of degree d, which can attain feature combina-
tions that are crucial to improving the performance
of tasks in NLP. Second, we introduce two fast clas-
sification algorithms for this kernel. One is PKI
(Polynomial Kernel Inverted), which is an exten-
sion of Inverted Index in Information Retrieval. The
other is PKE (Polynomial Kernel Expanded), where
all feature combinations are explicitly expanded. By
applying PKE, we can convert a kernel-based clas-
sifier into a simple and fast liner classifier. In order
to build PKE, we extend the PrefixSpan (Pei et al.,
2001), an efficient Basket Mining algorithm, to enu-
merate effective feature combinations from a set of
support examples.
Experiments on English BaseNP Chunking,
Japanese Word Segmentation and Japanese Depen-
dency Parsing show that PKI and PKE perform re-
spectively 2 to 13 times and 30 to 300 times faster
than standard kernel-based systems, without a dis-
cernible change in accuracy.
2 Kernel Method and Support Vector
Machines
Suppose we have a set of training data for a binary
classification problem:
(x1;y1);:::;(xL;yL) xj 2<N; yj 2f+1;¡1g;
where xj is a feature vector of the j-th training sam-
ple, and yj is the class label associated with this
training sample. The decision function of SVMs is
defined by
y(x) = sgn
‡ X
j2SV
yjfij`(xj)¢`(x)+b
·
; (1)
where: (A) ` is a non-liner mapping function from
<N to <H (N ¿ H). (B) fij;b 2<; fij ‚ 0.
The mapping function ` should be designed such
that all training examples are linearly separable in
<H space. Since H is much larger than N, it re-
quires heavy computation to evaluate the dot prod-
ucts `(xi)¢`(x) in an explicit form. This problem
can be overcome by noticing that both construction
of optimal parameter fii (we will omit the details
of this construction here) and the calculation of the
decision function only require the evaluation of dot
products `(xi)¢`(x). This is critical, since, in some
cases, the dot products can be evaluated by a simple
Kernel Function: K(x1;x2) = `(x1)¢`(x2). Sub-
stituting kernel function into (1), we have the fol-
lowing decision function.
y(x) = sgn
‡ X
j2SV
yjfijK(xj;x)+b
·
(2)
One of the advantages of kernels is that they are not
limited to vectorial object x, but that they are appli-
cable to any kind of object representation, just given
the dot products.
3 Polynomial Kernel of degree d
For many tasks in NLP, the training and test ex-
amples are represented in binary vectors; or sets,
since examples in NLP are usually represented in so-
called Feature Structures. Here, we focus on such
cases 1.
Suppose a feature set F = f1;2;:::;Ng and
training examples Xj(j = 1;2;:::;L), all of
which are subsets of F (i.e., Xj  F). In this
case, Xj can be regarded as a binary vector xj =
(xj1;xj2;:::;xjN) where xji = 1 if i 2 Xj,
xji = 0 otherwise. The dot product of x1 and x2
is given by x1 ¢x2 = jX1 \X2j.
Definition 1 Polynomial Kernel of degree d
Given sets X and Y , corresponding to binary fea-
ture vectors x and y, Polynomial Kernel of degree d
Kd(X;Y) is given by
Kd(x;y) = Kd(X;Y) = (1+jX \Yj)d; (3)
where d = 1;2;3;:::.
In this paper, (3) will be referred to as an implicit
form of the Polynomial Kernel.
1In the Maximum Entropy model widely applied in NLP, we
usually suppose binary feature functions fi(Xj) 2f0;1g. This
formalization is exactly same as representing an example Xj in
a set fkjfk(Xj) = 1g.
It is known in NLP that a combination of features,
a subset of feature set F in general, contributes to
overall accuracy. In previous research, feature com-
bination has been selected manually. The use of
a polynomial kernel allows such feature expansion
without loss of generality or an increase in compu-
tational costs, since the Polynomial Kernel of degree
d implicitly maps the original feature space F into
Fd space. (i.e., ` : F ! Fd). This property is
critical and some reports say that, in NLP, the poly-
nomial kernel outperforms the simple linear kernel
(Kudo and Matsumoto, 2000; Isozaki and Kazawa,
2002).
Here, we will give an explicit form of the Polyno-
mial Kernel to show the mapping function `(¢).
Lemma 1 Explicit form of Polynomial Kernel.
The Polynomial Kernel of degree d can be rewritten
as
Kd(X;Y) =
dX
r=0
cd(r)¢jPr(X \Y)j; (4)
where
† Pr(X) is a set of all subsets of X with exactly
r elements in it,
† cd(r) = Pdl=r¡dl¢
‡Pr
m=0(¡1)r¡m ¢ml
¡r
m
¢·.
Proof See Appendix A.
cd(r) will be referred as a subset weight of the Poly-
nomial Kernel of degree d. This function gives a
prior weight to the subset s, where jsj = r.
Example 1 Quadratic and Cubic Kernel
Given sets X = fa;b;c;dg and Y = fa;b;d;eg,
the Quadratic Kernel K2(X;Y) and the Cubic Ker-
nel K3(X;Y) can be calculated in an implicit form
as:
K2(X;Y) = (1+jX \Yj)2 = (1+3)2 = 16,
K3(X;Y) = (1+jX \Yj)3 = (1+3)3 = 64.
Using Lemma 1, the subset weights of the
Quadratic Kernel and the Cubic Kernel can be cal-
culated as c2(0) = 1; c2(1) = 3; c2(2) = 2 and
c3(0)=1; c3(1)=7; c3(2)=12; c3(3)=6.
In addition, subsets Pr(X\Y) (r = 0;1;2;3)
are given as follows: P0(X \ Y) =
f`g; P1(X\Y)=ffag;fbg;fdgg; P2(X\Y)=
ffa;bg;fa;dg;fb;dgg; P3(X\Y) = ffa;b;dgg.
K2(X;Y) and K3(X;Y) can similarly be calcu-
lated in an explicit form as:
function PKI classify (X)
r = 0# an array, initialized as 0
foreach i2X
foreach j 2h(i)
rj = rj +1
endend
result = 0foreach j 2 SV
result = result+yjfij ¢(1+rj)d
endreturn sgn(result+b)
end
Figure 1: Pseudo code for PKI
K2(X;Y) = 1¢1+3¢3+2¢3 = 16,
K3(X;Y) = 1¢1+7¢3+12¢3+6¢1 = 64.
4 Fast Classifiers for Polynomial Kernel
In this section, we introduce two fast classification
algorithms for the Polynomial Kernel of degree d.
Before describing them, we give the baseline clas-
sifier (PKB):
y(X) = sgn
‡ X
j2SV
yjfij ¢(1+jXj \Xj)d +b
·
: (5)
The complexity of PKB is O(jXj ¢ jSVj), since it
takes O(jXj) to calculate (1+jXj \Xj)d and there
are a total of jSVj support examples.
4.1 PKI (Inverted Representation)
Given an item i 2 F, if we know in advance the
set of support examples which contain item i 2 F,
we do not need to calculate jXj \Xj for all support
examples. This is a naive extension of Inverted In-
dexing in Information Retrieval. Figure 1 shows the
pseudo code of the algorithm PKI. The function h(i)
is a pre-compiled table and returns a set of support
examples which contain item i.
The complexity of the PKI is O(jXj¢B +jSVj),
where B is an average of jh(i)j over all item i 2 F.
The PKI can make the classification speed drasti-
cally faster when B is small, in other words, when
feature space is relatively sparse (i.e., B ¿ jSVj).
The feature space is often sparse in many tasks in
NLP, since lexical entries are used as features.
The algorithm PKI does not change the final ac-
curacy of the classification.
4.2 PKE (Expanded Representation)
4.2.1 Basic Idea of PKE
Using Lemma 1, we can represent the decision
function (5) in an explicit form:
y(X) = sgn
‡X
j2SV
yjfij¡
dX
r=0
cd(r)¢jPr(Xj \X)j¢+b
·
: (6)
If we, in advance, calculate
w(s) = X
j2SV
yjfijcd(jsj)I(s 2 Pjsj(Xj))
(where I(t) is an indicator function 2) for all subsets
s 2Sdr=0 Pr(F), (6) can be written as the following
simple linear form:
y(X) = sgn
‡ X
s2Γd(X)
w(s)+b
·
: (7)
where Γd(X) = Sdr=0 Pr(X).
The classification algorithm given by (7) will be
referred to as PKE. The complexity of PKE is
O(jΓd(X)j) = O(jXjd), independent on the num-
ber of support examples jSVj.
4.2.2 Mining Approach to PKE
To apply the PKE, we first calculate jΓd(F)j de-
gree of vectors
w = (w(s1);w(s2);:::;w(sjΓd(F)j)):
This calculation is trivial only when we use a
Quadratic Kernel, since we just project the origi-
nal feature space F into F £ F space, which is
small enough to be calculated by a naive exhaustive
method. However, if we, for instance, use a poly-
nomial kernel of degree 3 or higher, this calculation
becomes not trivial, since the size of feature space
exponentially increases. Here we take the following
strategy:
1. Instead of using the original vector w, we use
w0, an approximation of w.
2. We apply the Subset Mining algorithm to cal-
culate w0 efficiently.
2I(t) returns 1 if t is true,returns 0 otherwise.
Definition 2 w0: An approximation of w
An approximation of w is given by w0 =
(w0(s1);w0(s2);:::;w0(sjΓd(F)j)), where w0(s) is
set to 0 if w(s) is trivially close to 0. (i.e.,  neg <
w(s) <  pos ( neg < 0;  pos > 0), where  pos and
 neg are predefined thresholds).
The algorithm PKE is an approximation of the
PKB, and changes the final accuracy according to
the selection of thresholds  pos and  neg. The cal-
culation of w0 is formulated as the following mining
problem:
Definition 3 Feature Combination Mining
Given a set of support examples and subset weight
cd(r), extract all subsets s and their weights w(s) if
w(s) holds w(s) ‚  pos or w(s) •  neg .
In this paper, we apply a Sub-Structure Mining
algorithm to the feature combination mining prob-
lem. Generally speaking, sub-structures mining al-
gorithms efficiently extract frequent sub-structures
(e.g., subsets, sub-sequences, sub-trees, or sub-
graphs) from a large database (set of transactions).
In this context, frequent means that there are no less
than » transactions which contain a sub-structure.
The parameter » is usually referred to as the Mini-
mum Support. Since we must enumerate all subsets
of F, we can apply subset mining algorithm, in some
times called as Basket Mining algorithm, to our task.
There are many subset mining algorithms pro-
posed, however, we will focus on the PrefixSpan al-
gorithm, which is an efficient algorithm for sequen-
tial pattern mining, originally proposed by (Pei et
al., 2001). The PrefixSpan was originally designed
to extract frequent sub-sequence (not subset) pat-
terns, however, it is a trivial difference since a set
can be seen as a special case of sequences (i.e., by
sorting items in a set by lexicographic order, the set
becomes a sequence). The basic idea of the PrefixS-
pan is to divide the database by frequent sub-patterns
(prefix) and to grow the prefix-spanning pattern in a
depth-first search fashion.
We now modify the PrefixSpan to suit to our fea-
ture combination mining.
† size constraint
We only enumerate up to subsets of size d.
when we plan to apply the Polynomial Kernel
of degree d.
† Subset weight cd(r)
In the original PrefixSpan, the frequency of
each subset does not change by its size. How-
ever, in our mining task, it changes (i.e., the
frequency of subset s is weighted by cd(jsj)).
Here, we process the mining algorithm by
assuming that each transaction (support ex-
ample Xj) has its frequency Cdyjfij, where
Cd = max(cd(1);cd(2);:::;cd(d)). The
weight w(s) is calculated by w(s) = !(s) £
cd(jsj)=Cd, where !(s) is a frequency of s,
given by the original PrefixSpan.
† Positive/Negative support examples
We first divide the support examples into posi-
tive (yi > 0) and negative (yi < 0) examples,
and process mining independently. The result
can be obtained by merging these two results.
† Minimum Supports  pos; neg
In the original PrefixSpan, minimum support is
an integer. In our mining task, we can give a
real number to minimum support, since each
transaction (support example Xj) has possibly
non-integer frequency Cdyjfij. Minimum sup-
ports  pos and  neg control the rate of approx-
imation. For the sake of convenience, we just
give one parameter  , and calculate  pos and
 neg as follows
 pos =  ¢
‡#of positive examples
#of support examples
·
;
 neg = ¡ ¢
‡#of negative examples
#of support examples
·
:
After the process of mining, a set of tuples Ω =
fhs;w(s)ig is obtained, where s is a frequent subset
and w(s) is its weight. We use a TRIE to efficiently
store the set Ω. The example of such TRIE compres-
sion is shown in Figure 2. Although there are many
implementations for TRIE, we use a Double-Array
(Aoe, 1989) in our task. The actual classification of
PKE can be examined by traversing the TRIE for all
subsets s 2 Γd(X).
5 Experiments
To demonstrate performances of PKI and PKE, we
examined three NLP tasks: English BaseNP Chunk-
ing (EBC), Japanese Word Segmentation (JWS) and
a0a2a1a4a3
a0a2a5a6a3
a0a2a1a8a7a9a4a3
a0a2a1a8a7a10a11a3
a0a4a9a8a7a10a11a3
a0a4a9a8a7a5a6a3
a0a2a10a11a7a5a6a3
a0a4a9a8a7a10a12a7a5a6a3
a13a12a14a16a15a17
a18
a13a11a14a16a15a17
a13a11a19
a13a11a19
a18
a13a11a19
a18
a13a12a20
a18
a19a2a21
a18
a13a11a19
a22a24a23a26a25a4a27
a28 a29
a30
a30 a31a32a31a29
a31
a29
a29
a33a35a34a36a34a38a37
a39a26a40a35a41a42
a39a26a43a44a39a45a43
a46
a39a45a40a35a41a42
a46
a43a45a47
a46
a39a26a48
a46
a39a45a43
a46
a39a45a43
sa49
Figure 2: Ω in TRIE representation
Japanese Dependency Parsing (JDP). A more de-
tailed description of each task, training and test data,
the system parameters, and feature sets are presented
in the following subsections. Table 1 summarizes
the detail information of support examples (e.g., size
of SVs, size of feature set etc.).
Our preliminary experiments show that a
Quadratic Kernel performs the best in EBC, and a
Cubic Kernel performs the best in JWS and JDP.
The experiments using a Cubic Kernel are suitable
to evaluate the effectiveness of the basket mining
approach applied in the PKE, since a Cubic Kernel
projects the original feature space F into F3 space,
which is too large to be handled only using a naive
exhaustive method.
All experiments were conducted under Linux us-
ing XEON 2.4 Ghz dual processors and 3.5 Gbyte of
main memory. All systems are implemented in C++.
5.1 English BaseNP Chunking (EBC)
Text Chunking is a fundamental task in NLP – divid-
ing sentences into non-overlapping phrases. BaseNP
chunking deals with a part of this task and recog-
nizes the chunks that form noun phrases. Here is an
example sentence:
[He] reckons [the current account deficit]
will narrow to [only $ 1.8 billion] .
A BaseNP chunk is represented as sequence of
words between square brackets. BaseNP chunking
task is usually formulated as a simple tagging task,
where we represent chunks with three types of tags:
B: beginning of a chunk. I: non-initial word. O:
outside of the chunk. In our experiments, we used
the same settings as (Kudo and Matsumoto, 2002).
We use a standard data set (Ramshaw and Marcus,
1995) consisting of sections 15-19 of the WSJ cor-
pus as training and section 20 as testing.
5.2 Japanese Word Segmentation (JWS)
Since there are no explicit spaces between words in
Japanese sentences, we must first identify the word
boundaries before analyzing deep structure of a sen-
tence. Japanese word segmentation is formalized as
a simple classification task.
Let s = c1c2¢¢¢cm be a sequence of Japanese
characters, t = t1t2¢¢¢tm be a sequence of Japanese
character types 3 associated with each character,
and yi 2 f+1;¡1g; (i = (1;2;:::;m¡1)) be a
boundary marker. If there is a boundary between ci
and ci+1, yi = 1, otherwise yi = ¡1. The feature
set of example xi is given by all characters as well
as character types in some constant window (e.g., 5):
fci¡2;ci¡1;¢¢¢;ci+2;ci+3;ti¡2;ti¡1;¢¢¢;ti+2;ti+3g.
Note that we distinguish the relative position of
each character and character type. We use the Kyoto
University Corpus (Kurohashi and Nagao, 1997),
7,958 sentences in the articles on January 1st to
January 7th are used as training data, and 1,246
sentences in the articles on January 9th are used as
the test data.
5.3 Japanese Dependency Parsing (JDP)
The task of Japanese dependency parsing is to iden-
tify a correct dependency of each Bunsetsu (base
phrase in Japanese). In previous research, we pre-
sented a state-of-the-art SVMs-based Japanese de-
pendency parser (Kudo and Matsumoto, 2002). We
combined SVMs into an efficient parsing algorithm,
Cascaded Chunking Model, which parses a sentence
deterministically only by deciding whether the cur-
rent chunk modifies the chunk on its immediate right
hand side. The input for this algorithm consists of
a set of the linguistic features related to the head
and modifier (e.g., word, part-of-speech, and inflec-
tions), and the output from the algorithm is either of
the value +1 (dependent) or -1 (independent). We
use a standard data set, which is the same corpus de-
scribed in the Japanese Word Segmentation.
3Usually, in Japanese, word boundaries are highly con-
strained by character types, such as hiragana and katakana
(both are phonetic characters in Japanese), Chinese characters,
English alphabets and numbers.
5.4 Results
Tables 2, 3 and 4 show the execution time, accu-
racy4, and jΩj (size of extracted subsets), by chang-
ing  from 0.01 to 0.0005.
The PKI leads to about 2 to 12 times improve-
ments over the PKB. In JDP, the improvement is sig-
nificant. This is because B, the average of h(i) over
all items i 2 F, is relatively small in JDP. The im-
provement significantly depends on the sparsity of
the given support examples.
The improvements of the PKE are more signifi-
cant than the PKI. The running time of the PKE is
30 to 300 times faster than the PKB, when we set an
appropriate  , (e.g.,  = 0:005 for EBC and JWS,
 = 0:0005 for JDP). In these settings, we could
preserve the final accuracies for test data.
5.5 Frequency-based Pruning
The PKE with a Cubic Kernel tends to make Ω large
(e.g., jΩj = 2:32 million for JWS, jΩj = 8:26 mil-
lion for JDP).
To reduce the size of Ω, we examined sim-
ple frequency-based pruning experiments. Our ex-
tension is to simply give a prior threshold »(=
1;2;3;4:::), and erase all subsets which occur in
less than » support examples. The calculation of fre-
quency can be similarly conducted by the PrefixS-
pan algorithm. Tables 5 and 6 show the results of
frequency-based pruning, when we fix  =0:005 for
JWS, and  =0:0005 for JDP.
In JDP, we can make the size of set Ω about one
third of the original size. This reduction gives us
not only a slight speed increase but an improvement
of accuracy (89.29%!89.34%). Frequency-based
pruning allows us to remove subsets that have large
weight and small frequency. Such subsets may be
generated from errors or special outliers in the train-
ing examples, which sometimes cause an overfitting
in training.
In JWS, the frequency-based pruning does not
work well. Although we can reduce the size
of Ω by half, the accuracy is also reduced
(97.94%!97.83%). It implies that, in JWS, features
even with frequency of one contribute to the final de-
cision hyperplane.
4In EBC, accuracy is evaluated using F measure, harmonic
mean between precision and recall.
Table 1: Details of Data Set
Data Set EBC JWS JDP
# of examples 135,692 265,413 110,355
jSVj # of SVs 11,690 57,672 34,996
# of positive SVs 5,637 28,440 17,528
# of negative SVs 6,053 29,232 17,468
jFj (size of feature) 17,470 11,643 28,157
Avg. of jXjj 11.90 11.73 17.63
B (Avg. of jh(i)j)) 7.74 58.13 21.92
(Note: In EBC, to handle K-class problems, we use a pairwise
classification; building K£(K¡1)=2 classifiers considering all
pairs of classes, and final class decision was given by majority
voting. The values in this column are averages over all pairwise
classifiers.)
6 Discussion
There have been several studies for efficient classi-
fication of SVMs. Isozaki et al. propose an XQK
(eXpand the Quadratic Kernel) which can make their
Named-Entity recognizer drastically fast (Isozaki
and Kazawa, 2002). XQK can be subsumed into
PKE. Both XQK and PKE share the basic idea; all
feature combinations are explicitly expanded and we
convert the kernel-based classifier into a simple lin-
ear classifier.
The explicit difference between XQK and PKE is
that XQK is designed only for Quadratic Kernel. It
implies that XQK can only deal with feature com-
bination of size up to two. On the other hand, PKE
is more general and can also be applied not only to
the Quadratic Kernel but also to the general-style of
polynomial kernels (1 + jX \ Yj)d. In PKE, there
are no theoretical constrains to limit the size of com-
binations.
In addition, Isozaki et al. did not mention how to
expand the feature combinations. They seem to use
a naive exhaustive method to expand them, which is
not always scalable and efficient for extracting three
or more feature combinations. PKE takes a basket
mining approach to enumerating effective feature
combinations more efficiently than their exhaustive
method.
7 Conclusion and Future Works
We focused on a Polynomial Kernel of degree d,
which has been widely applied in many tasks in NLP
Table 2: Results of EBC
PKE Time Speedup F1 jΩj
 (sec./sent.) Ratio (£ 1000)
0.01 0.0016 105.2 93.79 518
0.005 0.0016 101.3 93.85 668
0.001 0.0017 97.7 93.84 858
0.0005 0.0017 96.8 93.84 889
PKI 0.020 8.3 93.84
PKB 0.164 1.0 93.84
Table 3: Results of JWS
PKE Time Speedup Acc.(%) jΩj
 (sec./sent.) Ratio (£ 1000)
0.01 0.0024 358.2 97.93 1,228
0.005 0.0028 300.1 97.95 2,327
0.001 0.0034 242.6 97.94 4,392
0.0005 0.0035 238.8 97.94 4,820
PKI 0.4989 1.7 97.94
PKB 0.8535 1.0 97.94
Table 4: Results of JDP
PKE Time Speedup Acc.(%) jΩj
 (sec./sent.) Ratio (£ 1000)
0.01 0.0042 66.8 88.91 73
0.005 0.0060 47.8 89.05 1,924
0.001 0.0086 33.3 89.26 6,686
0.0005 0.0090 31.8 89.29 8,262
PKI 0.0226 12.6 89.29
PKB 0.2848 1.0 89.29
Table 5: Frequency-based pruning (JWS)
PKE time Speedup Acc.(%) jΩj
» (sec./sent.) Ratio (£ 1000)
1 0.0028 300.1 97.95 2,327
2 0.0025 337.3 97.83 954
3 0.0023 367.0 97.83 591
PKB 0.8535 1.0 97.94
Table 6: Frequency-based pruning (JDP)
PKE time Speedup Acc.(%) jΩj
» (sec./sent.) Ratio (£ 1000)
1 0.0090 31.8 89.29 8,262
2 0.0072 39.3 89.34 2,450
3 0.0068 41.8 89.31 1,360
PKB 0.2848 1.0 89.29
and can attain feature combination that is crucial to
improving the performance of tasks in NLP. Then,
we introduced two fast classification algorithms for
this kernel. One is PKI (Polynomial Kernel In-
verted), which is an extension of Inverted Index. The
other is PKE (Polynomial Kernel Expanded), where
all feature combinations are explicitly expanded.
The concept in PKE can also be applicable to ker-
nels for discrete data structures, such as String Ker-
nel (Lodhi et al., 2002) and Tree Kernel (Kashima
and Koyanagi, 2002; Collins and Duffy, 2001).
For instance, Tree Kernel gives a dot product of
an ordered-tree, and maps the original ordered-tree
onto its all sub-tree space. To apply the PKE, we
must efficiently enumerate the effective sub-trees
from a set of support examples. We can similarly
apply a sub-tree mining algorithm (Zaki, 2002) to
this problem.
Appendix A.: Lemma 1 and its proof
cd(r) =
dX
l=r
 d
l
¶‡ rX
m=0
(¡1)r¡m ¢ml
 r
m
¶·
:
Proof.
Let X;Y be subsets of F = f1;2;:::;Ng. In this case, jX \
Yj is same as the dot product of vector x;y, where
x = fx1;x2;:::;xNg; y = fy1;y2;:::;yNg
(xj;yj 2f0;1g)
xj = 1 if j 2 X, xj = 0 otherwise.
(1+jX \Yj)d = (1+x¢y)d can be expanded as follows
(1+x¢y)d =
dX
l=0
 d
l
¶‡ NX
j=1
xjyj
·l
=
dX
l=0
 d
l
¶
¢¿(l)
where
¿(l) =
k1+:::+kN=lX
kn‚0
l!
k1!:::kN!(x1y1)
k1 :::(xNyN)kN:
Note that xkjj is binary (i:e:;xkjj 2 f0;1g), the num-
ber of r-size subsets can be given by a coefficient of
(x1y1x2y2 :::xryr). Thus,
cd(r) =
dX
l=r
 d
l
¶ k1+:::+kr=lX
kn‚1;n=1;2;:::;r
l!
k1!:::kr!
¶
=
dX
l=r
 d
l
¶ 
rl¡
 r
1
¶
(r¡1)l+
 r
2
¶
(r¡2)l ¡:::
¶
=
dX
l=r
 d
l
¶‡ rX
m=0
(¡1)r¡m ¢ml
 r
m
¶·
: 2
References
Junichi Aoe. 1989. An efficient digital search algorithm by us-
ing a double-array structure. IEEE Transactions on Software
Engineering, 15(9).
Michael Collins and Nigel Duffy. 2001. Convolution kernels
for natural language. In Advances in Neural Information
Processing Systems 14, Vol.1 (NIPS 2001), pages 625–632.
Hideki Isozaki and Hideto Kazawa. 2002. Efficient support
vector classifiers for named entity recognition. In Proceed-
ings of the COLING-2002, pages 390–396.
Hisashi Kashima and Teruo Koyanagi. 2002. Svm kernels
for semi-structured data. In Proceedings of the ICML-2002,
pages 291–298.
Taku Kudo and Yuji Matsumoto. 2000. Japanese Dependency
Structure Analysis based on Support Vector Machines. In
Proceedings of the EMNLP/VLC-2000, pages 18–25.
Taku Kudo and Yuji Matsumoto. 2001. Chunking with support
vector machines. In Proceedings of the the NAACL, pages
192–199.
Taku Kudo and Yuji Matsumoto. 2002. Japanese dependency
analyisis using cascaded chunking. In Proceedings of the
CoNLL-2002, pages 63–69.
Sadao Kurohashi and Makoto Nagao. 1997. Kyoto University
text corpus project. In Proceedings of the ANLP-1997, pages
115–118.
Huma Lodhi, Craig Saunders, John Shawe-Taylor, Nello Cris-
tianini, and Chris Watkins. 2002. Text classification using
string kernels. Journal of Machine Learning Research, 2.
Tetsuji Nakagawa, Taku Kudo, and Yuji Matsumoto. 2002. Re-
vision learning and its application to part-of-speech tagging.
In Proceedings of the ACL 2002, pages 497–504.
Jian Pei, Jiawei Han, and et al. 2001. Prefixspan: Mining
sequential patterns by prefix-projected growth. In Proc. of
International Conference of Data Engineering, pages 215–
224.
Lance A. Ramshaw and Mitchell P. Marcus. 1995. Text chunk-
ing using transformation-based learning. In Proceedings of
the VLC, pages 88–94.
Vladimir N. Vapnik. 1995. The Nature of Statistical Learning
Theory. Springer.
Mohammed Zaki. 2002. Efficiently mining frequent trees in a
forest. In Proceedings of the 8th International Conference
on Knowledge Discovery and Data Mining KDD, pages 71–
80.
