Dialogue Helpsystem based on Flexible Matching of 
User Query with Natural Language Knowledge Base 
Sadao Ku_rohashi and Wataru Higasa 
Graduate School of Informatics 
Kyoto University 
Yoshida-honmachi, Sakyo, Kyoto, 606-8501 Japan 
kuro@i, kyoto-u, ac. jp, higasa@pine, kuee. kyoto-u, ac. jp 
Abstract 
This paper describes a dialog help- 
system which advises users in us- 
ing computer facilities and software 
applications provided by the Cen- 
ter for Information and Multime- 
dia Studies, Kyoto University. The 
system employs a knowledge base 
written in natural  and re- 
trieves a proper knowledge unit by 
flexible matching of user query with 
the knowledge base. The system 
is running since July 1999, received 
about 2,000 queries for the first 
seven months, and answered about 
40~ of them satisfactory. 
1 Introduction 
One of the ultimate goals of Natural Lan- 
guage Processing is to realize a dialogue sys- 
tem which can communicate with human be- 
ings in a natural way (Wilks, 1999). How- 
ever, no effective real world dialogue applica- 
tion exists so far, not only in spoken  
for which speech recognition is still a big ob- 
stacle, but also in typing . 
The most serious problem is knowledge. It 
is obvious that without sufficient knowledge a 
dialogue system cannot talk with people sat- 
isfactorily. Classical dialogue systems like UC 
(Wilensky et al., 1984) utilized a formal lan- 
guage to represent knowledge, which requires 
the heavy cost of construction and mainte- 
nance and makes the scaling up quite difficult. 
In contrast, along with the improvement 
of NLP, research activities which utilize nat- 
ural  text as a knowledge base be- 
come popular, such as START (Katz, 1990), 
FAQ Finder (Cooper, 1996), and QA Track 
in TREC 8 (NIST and DARPA, 2000). These 
systems, however, basically produce one time 
response, and do not have a conversation with 
users. 
This paper proposes a dialogue helpsystem 
in which natural  knowledge base is 
not only used for one time response, but also 
for conducting a conversation. To put it con- 
cretely, the system can 
• ask the user back if there is an unknown 
word in the user utterance, 
• interpret the user utterance contextually, 
and 
• ask the user back when the user utter- 
ance is not clear. 
In order to achieve these functions, a mild 
restriction is imposed on the knowledge base; 
it is compiled in a dictionary-like structure 
(entry and its description) and the descrip- 
tion is written in an if-then format (though 
several expressions can be used to show if- 
then relation). 
The dialogue helpsystem reported in this 
paper advises faculties and students in us- 
ing the computer facilities and software ap- 
plications of a university. The computer 
facilities of Kyoto University, about 1,200 
computers (PCs and WSs) in several rooms 
and modem connections are administrated 
by the Center for Information and Multime- 
dia Studies (CIMS; http://www.media.kyoto- 
u.ac.jp/). The situation at CIMS is that 
many people are using computers, teaching 
141 
assistants (TAs) help the users, but are not al- 
ways available, and certain fundamental ques- 
tions are asked frequently. Considering this, 
we assumed that it is a very promising domain 
for an experimental dialogue system. 
The system started its service on July 1999, 
received about 2,000 queries for the first seven 
months, and answered about 40% of them sat- 
isfactorily. 
2 Examples 
The following examples (real logs of the ser- 
vice) are illustrative of the breadth of dia- 
logues in which the system aims to partic- 
ipate. The system is capable of generating 
the answers shown below in response to most 
forms of the request users might supply 1. 
<1999 Jul 08 19:31:49> 
Ui: Method to invoke a screen save in 
UNIX? 
Si: What is "screen save"? 
U2: screen saver. 
$2: Type "xlock" in the command line. 
<1999 Aug 19 21:20:52> 
UI: Please tell me how can \] read emails in 
Emacs. 
S l: Select mail software from "mh-e", 
,,mew ,~" 
U2: I try mh-e. 
$2: Type "M-x mh-rmail" on Emacs. Type 
'T' to get unread emails and type "" 
after you move the cursor on the line 
you want to read. 
<1999 Sep 16 22:32:20> 
UI: How can I use Mew in UNIX? 
SI: After invoking Emacs, type "M-x 
mew". See http://www.mew.org/ for 
det~.~l.~. 
U2: How can I delete emails? 
$2: Type "d" in the Summary Mode and 
then type "x" for execution. 
1Although our system is a Japanese dialogue sys- 
tem, in this paper we use their English translations 
for the explanation. 
<1999 Oct 25 12:42:28> 
UI: Tell me how to use Netscape. 
SI: On the \[Start Menu\], click \[program\] 
and then click \[Netscape Navigator\]. 
U2: What's start menu? 
$2: \[Start Menu\] is a pull-down menu sum- 
marizing Windows applications, which 
appears when you chck \[Start\] icon on 
the left bottom of the desktop screen. 
U3: What is click? 
$3: I cannot explain "click" in detail. 
3 Types of user utterances 
We observed the conversations of users and 
TAs in the CIMS computer rooms by record- 
ing and transcription (20 hours observation; 
1.5 hours recording). 
From this study, we discovered that user 
utterances can be classified as follows: 
What type: The user asks some fact. 
ex. What is Emacs? 
Explain Emacs. 
How type: The user asks how to do some- 
thing. 
ex. How can I input Japanese characters 
in Emacs? 
I want to input Japanese characters 
in Emacs. 
Symptom type: The user shows some 
symptom since he/she want to know 
how to cope with it. 
ex. I cannot access my private folder. 
The screen is white. 
Request type: The user requests something 
to C1MS. 
ex. Please install Visual-C. 
Please make my disk quota bigger. 
Addition type: The user adds or modifies 
the previous question. 
ex. How about WindowsNT? 
In the case of reply? 
142 
ut : How can I send emails by Mew? 
User 
User Interface (WWW browser) 
(KNP) 
Utterance content 
Dialogue 
Manager 
Knowledge 
Base 
Flexible Mathing 
1 <TITLE>Mcw</TITLE> 
b <SYN>mew, MEW, MIYU</SYN> 
<BT>Mail sofiwarc<\]BT> 
<DEF>A kind of mail sofficwam working on 
"~cbest <DESCRIPTION> .......... • <KU>In order to invoke Mew, 
i Woe "M-x mew" on Emacs.</KU> :::::::::::::::::::::::::: 
• "-<KU>Ifyou want to receive emails by Mew on Emacs, 
type "i'.</KU> 
• .-<KU>If you want to reply emails by Mew on Emacs, 
........ </KU> 
</DESCRIPTION> 
<TITLE>Mh-e</TITLE> 
<SYN>mh-e</SYN> 
<BT>Mail software</BT> 
Figure 1: The outline of the helpsystem. 
Answer type: The user answers the system 
question. 
ex. WINDOWS. 
The compression type is zip. 
The helpsystem reported in the paper an- 
swers what, how and symptom questions. In 
addition, it can interpret addition and answer 
utterances contextually. The request utter- 
ances are out of the system scope currently. 
4 Outline of the helpsystem 
The system is comprised of the following com- 
ponents (Figure 1): 
User Interface: Users access to the helpsys- 
tern via a WWW browser by using CGI 
based HTML forms. The helpsystem is 
actually running on a workstation in our 
lab. 
Input Analyzer: The user utterance is 
transformed into a dependency structure 
by a robust parser, KNP (Kurohashi 
and Nagao, 1994; Kurohashi and Na- 
gao, 1998), and utterance-pattern rules 
are applied to extract the utterance type 
and the utterance content. 
Japanese is head-final and the final ex- 
pression shows an utterance type. There- 
fore, the longest matching of utterance- 
pattern rules form the end of the utter° 
ance can detect the utterance type in 
most cases. For example, if the final ex- 
pression is "niha doushitara ii desu ka 
(How can I -- -)", how type is assigned; if 
"no baai ha (In case...)", addition type. 
143 
Knowledge base: The knowledge base is 
written in a natural , in a 
dictionary-like structure. 
Dialogue Manager: The core process of 
the dialogue manager is to match the 
user utterance with the hmowledge base 
in order to find the most appropriate de- 
scription. It also handles contextual in- 
terpretation of the user utterance and 
question to the user. 
Eraall Sender: The user can send his/her 
input-log to the CIMS staff via email if 
the automatic response is not satisfac- 
tory. So, the user does not have to in- 
put his questions a second time. This 
option surely contributes to the popular- 
ity of the system. 
In the following sections, we discuss the 
knowledge base and the dialogue manager, 
since we consider these components as the 
core of the system. 
5 Knowledge base 
5.1 The outline 
The knowledge base has a dictionary-like 
structure, in which each entry describes a 
concept/issue in the domain. It was com- 
piled manually by referring to the real QAs in 
CIMS rooms, the FAQ page of CIMS (about 
100 items), and question emails sent to CIMS 
(about 150 emails). Currently, it contains 
about 250 entries. 
Each entry consists of a headword 
(<TITLE> tag), synonyms(<SYN> tag), 
an upper word (<BT> tag), a definition 
of the headword (<DEF> tag) and sev- 
eral descriptions concerning the headword 
(<DESCRIPTION> tag; see Figure 1). All 
content words in the knowledge base were 
registered to the system database, which is 
used to see whether a user input word is 
known or unknown to the system (Section 
6.1). 
In addition, by collecting the headword 
and its upper word pairs from the knowl- 
edge base, the domain ontology (concept tax- 
onomy) is constructed automatically. The 
top categories of the current ontology are 
software, hardware, computer term (different 
to soft/hardware), action term, and general 
term. The domain ontology is used by the di- 
alogue manager in several ways (Section 6.2, 
6.3). 
5.2 Natural  representation 
In the knowledge base, the definition and sev- 
eral descriptions for the headword axe written 
in natural , which provides both high 
power of expression and high extensibility. 
The definition of the headword is used for 
what questions; the descriptions are used for 
how and symptom questions. Each descrip- 
tion, called knowledge unit (abbreviated to 
KU), is written in the following style: 
<KU> if a case, then what/how to do. </KU> 
In Japanese, there are many sentential pat- 
terns to express if-then relation. Authors of 
the knowledge base can use several expres- 
sions like "--- deareba... (if-..,-.-)", "..- no 
baai ha... (in case that .-., ..-)". 
The basic form of the how and symp- 
tom question is "in some case, what/how 
can I do?". Therefore, the system can an- 
swer the question by finding the most similar 
KU case part and showing the corresponding 
what/how to do part to the user (see Figure 
1). 
5.3 Matching of user question and 
knowledge unit 
Matching of the user question and a knowl- 
edge unit (KU) is done by comparing their 
dependency trees whose nodes are phrases. 
Their similarity is calculated as follows (Fig- 
ure 2): 
. For each phrase in the user question, the 
most similar phrase in the KU case part 
is looked for based on the following cri- 
teria: 
• Matching of content words : 3 points 
• The second or more matching 
of content words (when the 
phrase contains two or more 
content words) : 1 point 
144 
3+0+1+1=--5 
3+0+1+1=5 
The user question 
(The maximum matching score : 15) 
A knowledge uait 
(The maximum matching score : 20) 
The c~tainty score (5+5+5: 15 x 20 x 100 = 75 (%) 
Figure 2: Matching of the user question and a knowledge unit. 
• Matching of the depth of the 
phrases in parse trees : 1 point 
• Matching of the type of the 
phrases (phrase types differ 
depending on surface cases 
and verb conjugations, etc) : 1 point 
2. The similarity scores of phrases in the 
user question are summed up and nor- 
malized by the maximum matching score 
(MMS) as follows (the MMS is the simi- 
larity score with the same sentence): 
The sum of scores of~ 2 
phrase similarities \] 
The MMS of ~ (The MMS of~ 
the user question\] × \the KU case\] 
The above score is given to the KU as its 
certainty score. 
The above algorithm cares for the struc- 
tures of sentences to some extent by giving 
phrase depth scores and phrase type scores, 
but not in a strict way. This leads to a flex- 
ible matching for handling a variety of natu- 
ral  sentences and some parse errors. 
For the present, the parameters were given 
empirically. 
6 Dialogue manager 
Figure 1 showed the simplest case of a QA. In 
some cases, however, the user and the system 
have to take more turns until the user obtains 
a satisfactory answer. Such a turn-taking is 
an essential point of a conversation. 
To conduct a conversation, that is, to per- 
form a proper turn-taking, the participant 
have to be able to do the following functions 
at least: 
* ask the opponent back if there is an un- 
known word in the opponent's utterance, 
• interpret the opponent's utterance con- 
textually, and 
• ask the opponent ba~k when the oppo- 
nent's utterance is not clear. 
Our dialogue helpsystem can perform the 
basic level of the above functions by referring 
to natural  knowledge base. In the 
following subsections, we explain each of these 
functions in detail. 
6.1 Asking back of an unknown word 
Given the user utterance, the system first 
checks whether each content word in it is reg- 
istered in the system database or not. If the 
word is not registered, it means that the word 
is an unknown word to the system. An un- 
known word appears in the following cases: 
. Technical term not covered by the knowl- 
edge base. 
ex. shell script~ clone 
. Technical term whose synonym or related 
term is covered by the knowledge base. 
ex. Mozaic, Mozilla 
3. Misspell of the user. 
ex. Internetmai, Windo 
145 
Table 1: Patterns of the system responses. 
The best ~ of the candidate KUs 
cert~.~nty score one many 
100--60% 
60-30% 
30--0% 
<what/how to do> 
(of the KU) 
(one difference) 
Select <upper concept> from <list 
of the difference>. 
(two or more differences) 
Your question is not clear. Select 
<list of the candidate KU cases>. 
Your question is not clear. Select 
<list of the candidate KU cases> 
I cannot answer your question. 
4. General term. 
ex. name, summer vacation 
The system decision, whether the unknown 
word is general term or not, is taken accord- 
ing to whether it is an entry of a children's 
dictionary or not (Tadika, 199'7). 
If the unknown word is not a general term, 
the system asks the user back in the form 
"what is 'unknown word'?". If the system 
asks "what is Internetmai?', the user prob- 
ably notices his/her misspell and re-input 
it correctly. If the system asks "what is 
Mozilla?", the user might paraphrase it like 
"It means Netscape'. 
If the unknown word is a general word, it 
does not make sense to ask the user back, like 
''what is name?". Therefore, the system just 
overlooks it. 
6.2 Contextual interpretation of user 
questions 
The user question may be related to its pre- 
ceding utterances, modifying or supplement- 
ing them. As an example, consider the fol- 
lowing dialogue: 
UI: How can I send emails by Mew? 
S1: Type "C-c C-c" when you finish writing 
a mail. 
U2: In case to reply. 
$2: Move the cursor on the message to 
which you want to reply and type "A". 
In this dialogue, the user utterance U2 is a 
modification of U1, indicating "How can I re- 
ply emails by Mew?". 
In order to interpret such a context depen- 
dent utterance properly, the dialogue man- 
ager attempt to merge the user's new utter- 
mace onto the previous one. For each word of 
the user's new utterance, Wnew, if the previ- 
ous utterance contains the word of the same 
category, Wold, w,~ew is overwritten on wol d. 
If not, Wne w is added to the previous utter- 
ance. Two words are considered to be in the 
same category if they belong to the same top 
category of the domain ontology described in 
Section 4.1. Then, the system looks up the 
knowledge base by the merged utterance. 
In the above example, "reply" of U2 is over- 
written on "send" of U1, since they belong 
to the same category, action term. Then the 
system attempts to match the combined ut- 
terance "How can I reply emails by Mew?" 
with the knowledge base, and as a final re- 
sult, it can response as $2 2 
In the above example, since U2 is an addi- 
tion utterance, the system does not need to 
interpret U2 as a new, context independent 
question. However, if the user utterance has 
a different type, it is not possible to decide 
whether it is context dependent or indepen- 
2Note that the system keeps the resultant interpre- 
tation of the user query, which means that the system 
can keep more than one user utterances practically. 
For example, if the user asks "In case to forward" af- 
ter $2 in the above example, the system can interpret 
it as "How can I forward emails by Mew". 
146 
liiiiiiii~ii~!~iiiiiii~t -\] 
UI: I would like to uncompress a file. ° ............... "~ ¢ 
S1 : Select compressed file from "lzh file" 66°/°\[ ~ 
and "zip file". "~ ~, ~ 
Iby LHA U~iity ~ |by'LHA U~I,~ U2: lzh me 
$2 : Invoke LHA Utility 32 and select \ ...................................... uncompress wizard from 
\[File\] menu. I u oo p o , I I I 
• Candidate KUs 
J 
....... Difference : Izh file, zip file 
Upper concept : compressed file 
J 
Figure 3: An example of handling many candidate KUs. 
dent by seeing the new utterance alone. 
To decide this, the dialogue manager re- 
gard s the certainty score between the utter- 
ance and the most similar KU as an appropri- 
ateness measure of the interpretation. That 
is, we calculate both the certainty score be- 
tween the merged utterance (contextual inter- 
pretation) and the knowledge base, and that 
between the new utterance as it is and the 
knowledge base. Then; we choose the inter- 
pretation which obtained the KU with the 
bigger certainty score. 
For example, in the following dialogue, U2 
is interpreted as "How to invoke Mew?", since 
this combined interpretation has the bigger 
certainty score rather than U2 itself. 
Ui: What is Mew? 
SI: A kind of mail software working on 
Emacs. 
U2: How to invoke? 
$2: Type 'M-x mew" on Emacs. 
On the other hand, in the next example, U2 
is interpreted as it is, since U2 has the big- 
ger certainty score rather than the combined, 
somehow strange interpretation of "How to 
uncompress zip files by Outlook on Unix?". 
UI: I want to send an email by Outlook. 
$1: Invoke OutlookExpress, write your 
email, and select "Send/Get" button. 
U2: How to uncompress zip files on UNIX. 
$2: Type "unzip \[zip filename\]" in the com- 
mand line. 
6.3 Asking back for an unclear 
question 
As mentioned so far, the system retrieves 
proper KUs which are similar to the user 
question. The KU with the best certainty 
score and the KUs with the 90% or larger cer- 
tainty score are called candidate KUs. The di- 
alogue manager performs differently depend- 
ing on the best certainty score and the num- 
ber of candidate KUs, as shown in Table 1. 
If the certainty score is 60% or higher and 
the number of the candidate KUs are two or 
more, the dialogue manager detects the dif- 
ference between their cases and ask the user 
to make his/her question more specific. 
Figure 3 shows an example of such a dia- 
logue. The two candidate KUs are detected 
for U1 and their cases have only one differ- 
ence: "lzh file" and "zip file". Then, the sys- 
tem detects their common upper concept in 
the domain ontology and ask the user in the 
form "Select <upper concept> from <list of 
the difference >" as shown in Figure 3. 
If the candidate KUs contain two or more 
differences, it is hard to edit them in a neat 
way. Therefore, the system shows the candi- 
date KUs' cases as they are. If the certainty 
score is less than 60% and larger than 30%, 
the system responses in the same way. 
147 
oo ~ "~ = ....... ,,,, ~7~ ~o 7 
Figure 4: Evaluation of the helpsystem. 
7 Evaluation 
The helpsystem started its service on July 
1999 as part of the CIMS web service. All 
conversation logs between users and the sys- 
tem have been stored as a dialogue database. 
In the dialogue database, each dialogue be- 
tween a user and the system is segmented into 
task units manually. We call this unit a ses- 
sion. Figure 4 shows the number of sessions 
and their evaluation of each week from July 
5th to January 30th. On average, there are 
70 sessions in a week; a dialogue with a user 
and the system consists of 2.1 sessions, which 
means a user asks 2.1 topics in one dialogue; 
one session consists of 3.2 turns. 
The evaluation of sessions is based on the 
following criteria. 
Success: The system could return a satisfac- 
tory answer. 
Failure:Input Analyzer: 'The system 
could not response properly because of 
the input analysis error, mostly the lack 
of utterance-pattern rules. Utterance~ 
pattern rules are added whenever the 
lack is found. 
Failure:Dialog Manager: The system 
could not response properly because 
of the dialogue manager error. Dia- 
logue manager error comes both from 
simple bugs of the system and from 
unnoticed patterns of the user response. 
For example, when the system asks 
"select from A and B" expecting the 
answer "A" or "B", a user might answer 
"the latter". The system is modified 
whenever necessary. 
Failure:Knowledge: The system could not 
answer the question because of the lack 
of knowledge. This is the major rea- 
son of the failure as shown in Figure 4. 
Though the knowledge base is being ex- 
tended step by step, the range of the user 
148 
query is unlimited, including troubles in 
using PCs and advanced settings of soft- 
ware/hardware. 
Failure:Difilcult: Current system architec- 
ture could not handle the question. For 
example, a user sometimes asks "what is 
the difference between A and B", or when 
the system asks "select from A and B", a 
user answers "I don't know". In order to 
handle such utterances, we are planning 
to improve the system to exploit defini- 
tions of "A" and "B". 
Out of scope: Out of the system domain, 
such as questions about telephone 
charges in using PPP or the Y2K prob- 
lem. 
Miscellaneous: Such as "hello", "this is a 
test" or just a simple typo like "a". 
The success ratio, that is, the ratio of Suc- 
cess over Success plus Failure, of the whole 
period is 37%. The system became stable 
around October 1999, and the success ratio 
after that (14 weeks) is 39%. Considering rel- 
atively wide domain the system have to cover, 
we feel the success ratio is reasonable, and the 
system is contributing to CIMS to some ex- 
tent by handling simple FAQs like "how to 
change my password". 
8 Conclusion 
This paper described the dialogue helpsys- 
tern, which has been working in practice with 
real users. 
Construction of natural  knowl- 
edge base needs some cost, though it is much 
easier than that of formal  knowl- 
edge base. However, providing a high-quality 
service needs cost; good manuals and FAQs 
are important for any products, and a large 
amount of materials are prepared for cus- 
tomer service operators. With that in mind, 
we can say preparing a good document is a 
universal problem, not just to a dialogue sys- 
tem. 
By rlmning the system, the real dialogue 
database can be accumulated. Based on this 
database, we would like to study the phenom- 
ena of man-machine conversation and to ex- 
tend our work to user modeling, user intention 
estimation, and other interesting dialogue re- 
search areas. 
The system is designed to be domain- 
independent and can be ported to a new do- 
main by preparing a domain knowledge base. 
Exploiting this merit, we are planning to con- 
struct the automatic reference service system 
of Kyoto University Library, which certainly 
provides us with a wider breadth of dialogue 
data. 

References 
Edwin Cooper. 1996. Improving FAQ Finder's 
performance: Setting parameters by getetic 
programming. In Working Notes of the AAAI 
Spring Symposium on Machine Learning in In- 
formation Access. 
B Katz. 1990. Using english for indexing and 
retrieving. In Artificial Intelligence at MIT. 
Vol.1, MIT Press, pages 134-165. 
Sadao Kurohashi and Makoto Nagao. 1994. A 
syntactic analysis method of long Japanese sen- 
tences based on the detection of conjunctive 
structures. Computational Linguistics, 20(4). 
Sadao Kurohashi and Makoto Nagao. 1998. 
Building a Japanese parsed corpus while im- 
proving the parsing system. In Proceedings of 
the First International Conference on Language 
Resources ~ Evaluation, pages 719-724. 
NIST and DARPA. 2000. The Eighth Text RE- 
trieval Conference (TREC-8). NIST Special 
Publication. 
Jyunichi Tadika, editor. 1997. Reika Shougaku 
Kokugojiten (Japanese dictionary for children). 
Sanseido. 
Robert Wilensky, Yigal Arens, and David Chin. 
1984. Talking to unix in English: An oveview of 
UC. Communications of the ACM, 27(6):574- 
593. 
Yorick Wilks, editor. 1999. Machine Conversa- 
tions. Kluwer Academic Publishers. 
