Building Dialogue Corpora for Nursing Activity Analysis
Hiromi itoh Ozaku, Akinori Abe,
Noriaki Kuwahara, Futoshi Naya,
Kiyoshi Kogure
ATR Intelligent Robotics and Communication Labs
Hikaridai 2-2-2, Keihannna Science City,
Kyoto, 619-0288
{romi,ave,kuwahara,naya,kogure}@atr.jp
Kaoru Sagara
Seinan Jo Gakuin University
Ihori 1-3-5, Kokura Kita-ku, Kitakyusyu City,
Fukuoka, 803-0835
sagara@seinan-jo.ac.jp
Abstract
In this paper, we introduce our corpora
under development, which are recorded
in a real environment. These corpora
comprise dialogues collected in hospi-
tals with the aim of developing a nurs-
ing service support system through a
comprehensive understanding of nurs-
ing activities. We use the corpora to
analyze how nurses perform their nurs-
ing duties and how they express the per-
formance of their tasks. To understand
nursing activities, we investigated nurs-
ing services and the relevant medical
charts by using the corpora. In the pa-
per, we show features and promising
applications of the corpora.
1 Introduction
Recently, medical malpractice has become a seri-
ous social problem(Kohn, Corrlgan and Donald-
son, 1999). The Japanese Ministry of Health, La-
bor and Welfare has reported that nursing teams
are most frequently involved in medical accidents
in hospitals(Healthcare Safety Promotion Net-
work Project, 2001). The Japanese Nursing Asso-
ciation also states in its guidelines that nurses are
encouraged to make nursing reports and to ana-
lyze the cause of accidents, which is helpful to
prevent their recurrence. However, it is very dif-
ficult for nurses to make a detailed record during
their working hours.
We have been developing a nursing service
support system based on nurses using wearable
computers. The system is designed to record
nursing activities and to give warnings when nec-
essary. We will analyze the collected data to an-
alyze the sequence of their tasks and to quantify
their workload for the purpose of preventing med-
ical accidents. To create such a support system,
we need to acquire in-depth knowledge of nursing
activities closely. As a first step, we are collecting
nurses’ dialogues in hospitals, building dialogue
corpora, and analyzing the terms in the corpora
used to carry out nursing work.
We have already collected data on nursing tasks
in a specific hospital by using special devices to
record voice data. As a next step, we transcribed
conversation recorded by the devices. Since then,
we have been building dialogue corpora in actual
work sites. The corpora include various conversa-
tions with doctors, nurses, patients, and so on. We
generally exchange information, update informa-
tion, and share knowledge by conversation. We
believe that some medical accidents might occur
due to miscommunication. That is, nurses typi-
cally exchange patients’ information during con-
versation in clinical meetings, while at the same
time taking care of patients and other nursing ac-
tivities.
On the other hand, huge corpora have been
41
built from various voice data and text data(Kyoto
Text Corpus, ; K. Maekawa, 2003). Furthermore,
many types of tags have been developed for effec-
tive using of huge corpora(H. Koiso et al., 2000;
M Araki, et al., 1999). However, since the tradi-
tional dialogue corpora were mostly recorded on
a trial basis, their topics were usually fixed in the
corpora, and word usage and meaning of terms
were only defined clearly in one of them. In the
real field, word usage, topics, and the meaning of
terms could not easily be fixed. Furthermore, the
corpora have different features from those of the
real field data. For example, actual conversations
include many types of miscommunications, mis-
understanding, a resolution of the misunderstand-
ing, and so on. Consequently, corpora built from
such actual conversations can be reflected by mis-
communication or misunderstanding. Therefore,
it is difficult to analyze the real field data by rules
built from the traditional corpora.
In this paper, we focus on the corpora of the
voice data recorded by the special devices, and to
analyze the voice data to understand nursing ac-
tivities. For developing a nursing service support
system, we checked the corpora and other infor-
mation such as medical charts, describe features
of the corpora and their availability.
2 Corpora Collection
2.1 E-nightingale Project
In medical sites, higher levels of knowledge and
wider experience are needed for using sophisti-
cated medical devices and, in turn, for accom-
modating a diverse aging society. It is said that
insufficient knowledge and experience both in-
fluence the performance of medical professionals
and causes malpractice. On the other hand, by
promoting the use of wearable sensors and envi-
ronment sensors, it is possible to collect a huge
amount of data on actions in the real world. Ac-
cordingly, some attempts have been made to uti-
lize knowledge obtained by analyzing daily ac-
tions in the course of developing sensors for edu-
cation and accident prevention.
If important information could be obtained
by analyzing nursing activities automatically
recorded by wearable computers, it would be pos-
sible to automatically and more correctly make
nursing reports. Furthermore, if the nursing re-
ports could be objectively analyzed, the cost of in-
vestigating nursing practices could be kept down
and more effective survey research on nursing
activities could be conducted. Furthermore, it
would be possible to protect nurses from malprac-
tice claims by distinguishing between nursing ac-
tivities during normal situations and during emer-
gencies.
To exploit this potential, we launched the “E-
nightingale project” to share and construct knowl-
edge obtained by analyzing the daily actions and
experiences of nurses. This project aims to de-
velop the following technologies:
1. To observe and understand daily nursing
activities.
2. To build a knowledge base through under-
standing these activities.
3. To utilized the necessary knowledge when
it’s actually needed.
2.2 Obtained Voice Data of Nurses’
Activities
We have recorded various action data by wearable
computers designed to analyze nursing activities
in hospitals(N. Kuwahara, et al., 2003; N. Kuwa-
hara, et al., 2004). Wearable computers can col-
lect various types of data such as location, pas-
someter results, and posture inclination. In our
previous research, we used passometer, location,
and cue words to grasp nursing activities. We
could understand the nursing activities of a partic-
ular nurse but could not understand the relations
and information flow among staff in a hospital.
Dialogues between nurses are very important
in the performance of nursing activities and main-
taining staff relations. Dialogues between nurses
42
and other people, such as patients, doctors, and
patients’ family, are also important for obtaining a
variety of conversation behaviors more efficiently.
We believe that we can understand not only nurs-
ing activities but also conversation mechanisms
by developing and analysing corpora of nurses’
dialogues.
Therefore, we have conducted experiments to
collect voice data of nurses talking during their
activities. Our reasons and aims for focusing on
voice data are as follows:
• To more effectively make lengthy record-
ings than sensors with consumer model IC
recorders
• To obtain many data, such as medical charts,
in addition to voice data
• To collect and unify terms of nursing activi-
ties for developing a support system
• To collect natural dialogue data for clarifica-
tion of conversation mechanisms and infor-
mation flows
• To examine the appropriate sensors for un-
derstanding voice data
• To comprehend problems in actual field
recording
In the following section, features of voice data
collected in experiments are explained in detail.
2.2.1 Voice Data during Clinical Meetings
During nurses’ shift changes, they hold clin-
ical meetings to discuss patient information, in
the process modifying and confirming this infor-
mation. Furthermore, if problems occur in their
work area, they hold brief conferences to solve
the problems by discussing their experiences.
We have recorded nurses dialogues in clinical
meetings and brief conferences by using special
devices. At the same time, we have obtained
such information as who participated in the clini-
cal meetings and brief conferences.
In the hospital where we carried out our exper-
iments, clinical meetings are held in the morning
and the evening time for about 20 minutes to one
hour. The brief conferences are held at lunch time
for about 30 minutes. Collected data were 80 tri-
als for a one-week experiment. The entire record-
ing time was about 20 hours. We made transcrip-
tions of the collected data. Some sample data are
shown in table 11
The transcription was made by four staff mem-
bers, including an experienced specialist in mak-
ing transcriptions, a nurse who had worked in
hospitals for three years or longer, a pharmacist,
and a part-time employee, and it was assumed that
the latter three had no experience working in the
field of transcription.
2.2.2 Event-driven Voice Data
To understand nursing activities, a one-day
schedule of nursing activities should be recorded
and analyzed. However, it is difficult to transcribe
all such recorded data. Therefore, we recorded
the voice data of nursing activities along with
clues annotated by nurses using the special de-
vices.
To obtain voice data in hospitals, we devel-
oped an IC-recorder, a microphone with an event
button, and an intermediate control box with a
buzzer. The event button is used for explicit voice
annotation when nurses start or complete a task.
W hen the button is pushed, the buzzer sounds
once and its sound is recorded, and then nurses
record their tasks at the moment by speaking short
sentences. The buzzer is also set to sound periodi-
cally (every 10 minutes) to prompt nurses to make
voice input about their ongoing tasks. Simple
signal processing can extract and classify event-
driven and periodic voice records, as well as nurse
call rings.
Event-driven voice data recording is very use-
1Time indicated elapsed time from the time when the ex-
periment starts. NurseID indicates the nurses participating
in the conversation. In this table, there are two nurses in the
conversation. Utterance is the transcription of voice data.
43
Table. 1: Dialogue Transcription Example
Time Nurse ID Utterance
00:57:19 A The drip infusion is still being given, isn’t it? Not yet?
00:57:20 B Yes. It’s still dripping now.
00:57:22 A Ah, until when?
00:57:24 B Until tomorrow. Until 6 o’clock.
ful for accurately recording nursing activities. In
addition, this approach can record nursing activi-
ties at the desired time. As a result, time informa-
tion can be easily observed, and we can collect
objective data on nursing activities. Therefore,
our approach improves the efficiency of research
on time utilization.
In one department of a hospital, we have con-
ducted experiments on collecting data of nurs-
ing work through voice annotations. The nurses
work in three shifts, assigned to primary patients
of each ward. All nurses were given instructions
on the usage of our devices. The entire recording
time was about 500 hours. Collected data were
gathered from 39 trials for a one-week experiment
involving 15 nurses using our devices. We also
made transcriptions of the collected data. Some
sample transcriptions are shown in Table.2. The
transcription was also made by four staff mem-
bers as mentioned above in 2.2.1.
3 Features of the Corpora
As discussed in the previous sections, we focus on
dialogue data such as natural conversations col-
lected in daily life or work. We can observe the
following features in the corpora.
• People in the conversations come from vari-
ous levels of social status.
• Conversations are conducted in various
places or situations.
• There are certain relationships or dependen-
cies between the current conversation and
the previous conversations.
• Conversations on similar topics are repeated.
For example, nurses engage in conversation not
only with other nurses but also with patients, fam-
ilies of patients, and doctors. Naturally, they be-
have according to their social role. As a result,
features of conversation will be different. Exam-
ples of such differences are shown as follows:
• Conversations between doctors and nurses
include formal and many medical terms.
• Among nurses at meetings, formal and many
abbreviated expressions of medical terms.
• Among nurses, informal and many abbrevi-
ated expressions of medical terms.
• With patients, families of patients and
nurses, formal and many words expressing
medical terms in simple words.
Nurses exchange and share information on
their patients depending on their work situations.
The conversations are made in nurse stations,
beside a patient’s bed, in a clinic, and so on.
Sometimes conversation begins from necessity
and other times it begin when nurses happen to
encounter.
Information on patients should smoothly be
transferred from one shift nurse to the next shift
nurse. As a result, conversations dealing with the
same or similar matter should have certain rela-
tionships and dependencies. For example, topics
related to the same patient and the same opera-
tion patterns of a certain disease repeatedly ap-
peared in our corpora. For example, “to test blood
sugar levels”, in Table 2, appeared whenever the
patients finished a meal.
44
Table. 2: Transcription Example for Nursing Activities
Time Utterance
11:28:11 I’m going to prepare a set of drip infusion for Abe-san.
11:32:01 I’ve finished preparing the drip for Abe-san.
11:32:04 I’m going to test blood sugar levels of Naya-san and Kuwahara-san.
11:33:48 I’ve finished testing blood sugar levels of Naya-san and Kuwahara-san.
4 Applicability of the Corpora as
Spoken Language Corpora
In this section, we discuss the applicability of the
obtained corpora as spoken language corpora and
a strategy to build a set of ontologies from them.
We have already manually generated around 30
hours of corpora for event-driven data and 4 hours
of dialogue corpora at clinical meetings.
To understand nursing activities from event-
driven data, we used tags to understand situations.
We assumed that key information to evaluate a sit-
uation included time, place, medication, disease
and the person’s name. Therefore, we extracted
the person’s name, medication and disease from
our corpora by using a named entity recognition
tool.
Named entity recognition is useful for extract-
ing such information as the person’s name, com-
pany name, date, and place name. Named en-
tity recognition tools have been developed in re-
search on information extraction, machine trans-
lation, and so on. There are two types of named
entity recognition: rule-based recognition and
statistics-based recognition. We believe that some
connections with personal name, for example “
^�,^�,X�(Mr/Ms)”, are determined at
some level, and our target is data that can be ex-
ploited to make hand-crafted rules such as med-
ical charts, attendance sheets and so on. There-
fore, we studied how to extract personal name,
medication and disease using a rule-based named
entity recognition tool. NExT is a well-known
tool in Japan for rule-based named entity recog-
nition. We used the NExT version 0.82 (NExT a
Named Entity Extraction Tool, 2002), which can
be download from the Mie University web site.
The NExT tool utilizes Chasen version 2.3.3 for
morphological analysis and ipadic version 2.7.0
as a dictionary(Morphological Analysis System
ChaSen, 2003). The dictionary includes many
medicine names, diseases, personal names, and
place names.
4.1 Tags in the Corpora
Next, we extracted personal names, medication,
and diseases from the corpora and tagged the cor-
pora to better understand situations. For example,
the W HO tag means a speaker, in this case a nurse
as a subject using our special device, the W HOM
tag means a hearer, the PT tag means the person
or patient name mentioned, and the PLACE tag
means the place where the communication is con-
ducted. The TIME tag indicates the absolute time
calculated from the IC recorder’s elaped time and
the experiment’s start time. The MED tag means
medication, and the DIS tag indicates the disease
name. Table 3 shows a sample of the corpora.
We manually extracted the person name for
W HO, W HOM, and PT from the transcription
with medical charts. Person names were also
extracted by a named entity recognition tool to
check the NExT accuracy. Also, we evaluated
PLACE by ambience sounds of voice data and
schedule of nursing activities. In addition, we
manually extracted medication and disease for
MED and DIS.
It should be noted that we are analyzing words
specialized to nursing activities in the primitive
corpora. Nursing terms include daily used terms
for expressing work that supports patients’ daily
45
Table. 3: Corpora Example
Elapsed Time TIME PLACE Utterance
11:28:11 18:28:11 Nurse Station�W HO=nurseID�I�/W HO�’m going to prepare
a set of drip�MED�infusion�/MED�for�PT�Abe-san�/PT�.
11:32:01 18:32:01 Nurse Station�W HO=nurseID�I�/W HO�’ve finished preparing
the drip for�PT�Abe-san�/PT�.
11:32:04 18:32:04 Room 401�W HO=nurseID�I�/W HO�’m going to test blood sugar
levels of�PT�Naya-san�/PT�and�PT�Kuwahara-san�/PT�.
11:33:48 18:33:48 Room 401�W HO=nurseID�I�/W HO�’ve finished testing blood sugar
levels of�PT�Naya-san�/PT�and�PT�Kuwahara-san�/PT�.
lives as well as medical terms specialized for the
particular medical situation. Furthermore, some
medicine and disease names are different from the
expressions used in from dictionaries, since our
corpora include colloquial expressions. More-
over, it is difficult to tag some expressions that
have the same meaning.
4.2 Nursing Terms Featured in the Corpora
In this section, we describe some expressions that
express controversial features for tagging.
For instance, in general “d�U�(sengan)”
means “
�%(washing face),” but if it is narrated
before surgery in ophthalmology, it means “
�
(washing eye).” “�qb�(jiritsu-suru)” means
“standing walk” if it is narrated in a conversation
among nurses, but it means “to earn one’s living
by oneself” in general, as when it is narrated in
a conversation between a nurse and a patient’s
family. Thus some words have multiple mean-
ings according to the situation. As a result, some-
times nurses, especially novice nurses, experi-
ence misunderstanding in their communications.
Another example is “���O(horyu)”, which
means “suspend giving medicine” or “keep a sting
stung”. Therefore, we think we will be able to
gather many words that have multiple meanings
when we check the collected corpora. This will
allow us to build or extend the contents of a dic-
tionary of multiple-meaning expressions.
In our studies. we also observed the following
cases:
• multiple expressions
– “�(seki)” and “�I�gaisou�” mean
tussis and coughing
– “�(nyou)” and “���(harun)”, and “
S`l\�oshikko)”ymean urine and
pee
– “v�(tounyou)” and “����(di-
abe)” and “DM”ymean sugar diabetes
(diabetic mellitus)
These types of expressions are used to hide
the real meaning from patients, and they’re
sometimes used because the expression is
currently fashionable.
• abbreviated expressions
– “���an�”ymeans ampule
– “���miri)” means milliliter or mil-
ligram
– “
�y(bun-ni)” means half or two
times
– “�����(nobo-aaru)” means No-
volin R, which is a medication for dia-
betes
These types of expressions seem to be used
for quick communication. In particular,
medications are sometime shortened into
shorter words, such as Novolin R into R
only.
These expressions are relatively frequently
used and sometimes seem to cause nursing acci-
dents or incidents due to miscommunication.
We think we will be able to obtain many con-
cepts that have multiple expressions and abbre-
46
viated expressions by checking the collected cor-
pora. Then we could build or extend contents of
a multiple-expression concept dictionary and an
abbreviated-expression dictionary.
To develop a support system for nursing ac-
tivities, we should standardize technical nursing
terms. However, there are many types of terms
used in typical nurses’ dialogues. For example,
nurses uses
• admission to a hospital expressed as “AD”
• “AD” originates from ”admission” in En-
glish
• discharge from hospital expressed as “��
�(ento)”
• “���(ento)” originates from entlassen in
German
Furthermore, there are many new medicines
used in hospitals. It is difficult to uniformly man-
age these new words. The standardization of
nursing terms thus has many problems.
We think that even ambiguous words can be
understood if we recognize nurses’ working sit-
uations by using the information of many types
of sensors. This would require using a position
sensor to identify nurses’ working locations as
well as who is participating in conversations. We
could supplement missing information and com-
plete or correct ambiguous information by refer-
ring to such information from sensors.
4.3 Feature as Spoken Language Corpora
A chronological relationship or dependency map
among nurses can be accurately obtained by refer-
ring to nursing records and medical charts. Con-
ventional dialogue corpora can only offer one-to-
one conversations, permitting only a simple anal-
ysis. In addition, it is easy to make a wrong anal-
ysis.
On the other hand, if we analyze our corpora
with a chronological relationship or dependency
among nurses, we can discover true features of
conversation phenomena. For instance, person A
exchanges information with person B, but person
B misunderstands person A’s information. Person
B conveys the information to person C. Then C re-
alizes B’s misunderstanding and informs B so that
B can correct the information. Thus we can obtain
communication patterns between more than two
persons who do not know each other’s situation.
To utilize the dictionaries obtained from our
corpora and sensor information, we can build a
set of ontologies for conversations between mul-
tiple persons. In building a set of ontologies, the
mechanism of conversations can be clarified, and
a method for finding points of mistakes given am-
biguous expressions can be examined.
In this paper, we focus on nursing vocabularies
and nursing activities. Of course they are slightly
different from general terms and general situa-
tions, but these corpora and dictionary-building
techniques can be applied to general terms and
situations betweenyspoken language and writ-
ten language.
In finishing the transcription of the clinical
meetings and event-driven data, we are building
the corpora. Here we note that it is difficult to tag
MED and DIS because there are multiple expres-
sions and abbreviated expressions. We need to
develop a way to designate standard expressions
from the multiple and abbreviated expressions.
5 Conclusion
In this paper, we introduced our E-nightingale
project. We also showed voice data collection in
a real workplace such as a hospital and discussed
the importance and potential of generating cor-
pora from this data.
In the next step, all of the collected voice
data will be made into corpora, and tags will be
made to construct knowledge based on experi-
ences from conversation corpora. Furthermore,
we will develop methods to build corpora from
collected data automatically. From this corpora,
we will develop a system to analyze actual dia-
47
logues and activities in a real nursing situations
by utilizing multi-sensor information.
Acknowledgements
We would like to thank the nurses involved in
this study for their cooperation in our experiment.
This research is funded by the National Institute
of Information and Communications Technology.
References
L. T. Kohn, J. M. Corrlgan and M. S. Donaldson.
1999. To Err Is Human:Building a Safer Health
System. The National Academic Press, Nov. 1999.
Healthcare Safety Promotion Network Project.
http://www.mhlw.go.jp/topics/2001/0110/tp1030-
1.html#2-1. (in Japanese).
Kyoto Text Corpus. http://www.kc.t.u-tokyo.ac.jp/nl-
resource/corpus-e.html.
K. Maekawa. 2003. Corpus of Spontaneous
Japanese: Its design and evaluation. Proceedings
of the ISCA & IEEE workshop on Spontaneous
Speech Processing and Recognition. SSPR2003.
H. Koiso, N. Tsuchiya, Y. Mabuchi, M. Saito, T.
Kagomiya, H. Kikuchi, and K. Maekawa. 2000.
Transcription Criteria for the Corpus of Sponta-
neous Japanese. Special Interest Group of Sponken
Language Processing. Vol. 34-30, pp. 173-178. in
Japanese.
M. Araki, T Itoh, T. Kumagai, and M. Ishizaki. 1999.
Proposal of a Standard Utterance-Unit Tagging
Scheme. Transcription of the Japanese Society for
Artificial Intelligence Vol. 14, No. 2, pp. 251-260.
in Japanese.
N. Kuwahara, H. Noma, N. Tetsutani, N. Hagita, K.
Kogure, and H. Iseki. 2003 Auto-event-recording
forNursingOperationsbyUsingWearableSensors.
Information Processing Society of Japan Vol. 44,
No. 11, pp. 2638-2548. in Japanese.
N. Kuwahara, K. Kogure, N. Hagita, and H. Iseki
2004. Ubiquitous and Wearable Sensing for Mon-
itoring Nurses’ Activities. Transcription of the
Japanese Society for Aritificial Intelligence Proc.
of SCI2004, pp. 281-285.
NExT — Named Entity Extraction Tool.
http://www.ai.info.mie-u.ac.jp/˜next/next en.html.
ChaSen — Morphological Analysis System.
http://chasen.naist.jp/hiki/ChaSen/
48
