﻿AMulti lingualDecisionSupportPrototypefortheMedicalDomain
DavidDinh DennisChan JackChen 
osTechnologyPty.Ltd. PSTResearchGroup PSTResearchGroup
HealthTechnologySolutions VoiceSolutionsDeveloper HealthSolutionsD eveloper
david.dinh@ostechnology.com.au dennis.chan@pstresearch.info jack.chen@pstresearch.info 
Abstract 
Inthispaper,weareproposingamulti 
lingualprototypethatcanefectivelycol
lect,recordanddocumentmedicaldatain
adomainspecificenvironment.Theaim
ofthisprojectistodevelopanelectronic
supportsystemthatcanbeusedtoassist
asth mamanagementinanemergencyde
partment. 
1 Introduction 
Speechtechnologyhastheabilitytogenerater e
sourceandtimesavingswithinahospitalenviron
ment.Recordingandmanagingpatientdatafrom
nonEnglishbackgroundscanbeachievedsuccess
fullythro ughtheimplementationofamultilingual
voicesystemandastandardisedelectronicmedical
decisionsupportsystemsuchasACAFE(ACAFE
2006)describedinSection5.3.Byimplementing
theACAFEsta ndardizedprotocolstogetherwitha
voicesystem,wearea bletoassistinthefirststage
oftheclinicalpathwayinthetreatmentandma n
agementofAsthma(seillustrationofStage1in
fig ure3).
Inthisdemonstrationdescription,wearepr o
posingamulti lingualvoicesystembasedona
standardizedpatient managementsystemcalled
ACAFEthatcanefectivelycollectpatientdatain
electronicformat.Thecombinationofthetwosy s
temswouldmakeiteasiertoassistintherecording
anddocumentationofvastamountsofinfo rmation
whilstovercomingcommunicationandeficiency
bariers.Thisdatacanthenbeaggregatedandan a
lyzedaftertheeventtoassistwithclinicalandpe r
for mancemeasures.Thismakesefectiveuseof
emer gencydepartmentresourceswhileproviding
theemergencystafwithimmediateacesstoim
portantpatientinform ation. 
2 Objectives 
Toshowhowqualityhealthcarecanbedelivered
inacomplexmultilingualhospitalenvironment
withtheaidofanelectronicdecisionsupportsy s
temsuchasACAFE. 
3 DemoDescription 
Ourdemoprototypeintegr atesavoicerecogni tion
systemtogetherwiththeACAFEsystemdescribed
inmoredetailinsection5.3.Ourvoicerecognition
prototypereliesondataextractedfromthesta n
dardizedtreatmentprot ocolsthathavebenbased
onresearchby ACAFE(ACAFEeta l.,2006).
Thesestandardizedprotocolsformthebasisofour
system patientinter actiontothemedicalsub
domain(Sta rlanderetal.,2005).
SinceoursystemisheavilydrivenbyACAFE,
wehavebenabletominimizetherequirementfor
anopenrangeof questionsthatr equiretranslation.
Asaresult,weonlyrequiretheuseofthegra m
mar basedlanguagemodel(GLM)thathasben
implementedusingNuance’sspechrecognizer
(Nuance2005),andnotastatisticalla nguage
model(SLM).
Thestandardizedproto colsrequirenomanipul a
tionorchangesintenseastheACAFEsystemis
essentiallyadecisionsupporttool. Theflex ibility
ofthedecisionsupporttoolallowstheclini cianto
makethefinaldecisionandvaryanyresponsesor
inputs. Hencetherangeofq uestionsourmultilin
gualsystemposestothepatientisalsostandar d
izedandlimited.Withthe smallersetofquestions
itisfeasiblefortran slationtooccurviadirect
ACAFEto'target language'mappings (subjectla n
guagetomanyvariationsofatarge tlanguage).
TheuseofGLMsoverSLMsformedical
speechtranslationhasbenproventoprovide
highertranslationacuracy(Rayneretal.,2004,
Rayneretal.,2005).Weexpectthatbycombining
thehighera curacylevelsofrecognitionthrough
theuse ofGLMswithalimitedsetofpossible
questionsforapar ticularmedicalsubdomain,we
canachieveanimprovedtran slationsuccessrate.
Currently,oursystemrequirestheOverser
(suchasanurse)tospecifythepa tient’snative
language(inourexamp leChineseMandarin)and
problemsub domain(inourexampleasthma).
Fromthere,theOversercaneitherspeakaque s
tionasde finedintheprotocolscontainedwithin
theACAFEsystem(usingEnglish),orselectone
usingtheterminal.Thequestionisthen rendered
usingr ecordedaudio(TSisusedasafallback
strategy)andplayedtothepatient.Oncethepa
tientrespondsve rballyorphysically(e.g.nodof
thehead),theOverserisrequiredtoenterthat
responseintothesystem.
TheOverseriscapabl eofviewingreportsthat
detailaparticularpatient’sresponsespriortofur
theranalysis/treatment,ortheycanviewsta tistical
reports.Asaprofofconcept,theOversercan
generateastatisti calreportthatdetailspatient
backgroundprecipitating factors(numbersofre s
piratorytractinfections,coldweather,exe rciseand
dust/pollens) 
4 SuggestedScenario 
Thetriagenursewillidentifythepatient’snative
languagetoenablethecorectvoicesystemtran s
lator.Thevoicesystemwilltranslatethe standar d
izedasthmamanagementplanquestionsintothe
patient’snativela nguage.
Patientwillanswereachquestionintheirnative
tongue.Thevoicesystemwillconvertthisinfo r
mationintotheACAFEsystemformat.Wheneach
questionhasbeenanswered, theACAFEsystem
willstorethea nswersandthevoicesystemwill
thenfollowthroughtothenextACAFEquestion.
UponcompletionofthesetofACAFEbased
questionsthevoicesystemwillthenprovidear e
viewofthequestionswithanswersintheACAFE
systemineitherEng lishorthenativelanguage.A
voicerecordingwillalsobestoredtoplaybackfor
futurereference.
Triagereferstotheanswersthathavebenco l
latedintheACAFEsystemviatheassistanceof
thevoicesystem.Thisinformationcan beunder 
stoodbyalleme rgencyteamstafasthevoicesys
temhastran slatedtheanswersofthepatientinto
Englishacordingtothestandardizedmanagement
answers.
TheEmergencyDepartmentnowhasapre 
compiledlistofpatientinformationcompliant with
Stage1oftheclinicalpathwaycontainedinthe
ACAFEsystemtohelpassistinthetreatmentof
asthma,withouthavingtoworyaboutcommuni
cationdificultiesb etweenpatientandmedical
staf. 
4.1 Demoscript
TriageNurse:  “Hello,whatpainsordificulties
areyouexperiencing?” 
Patient: “UnderstandEnglishnogood,asthma…” 
TriageNurse:  “Canyouconfirmyourlanguage,
MandarinorCantonese?” 
Patient: “Chinese,mandarin.” 
TriageNurse:  “OK,whatIwilldonowisusea
specialmachinetoaskafewsimplequestions,you
canjustansweryesorno,itwillasktheque stions
inmandarinsoyoucanunderstandbetter.OK,
herewego…“
Triagenursethenactivatesthevoicesystemwhich
goesthroughthesetofACAFEbasedquestionsin
mandarin.
Patient
ACAFE
NursingTriage 
Figure1:High levelviewofuserACAFEinte r
action 
5 SystemArchitecture
5.1 Overview 
Figure2illustratesacomponentviewofthede sign
forourprototypesystem.TheOverseractsasan
overidingauthorityfortheACAFEDecisionSup
port component,providinginterpretationsofthe
Pa tient’snativelanguage,medicalproblemsub
domain,andasafailover,thePatient’sresponses
(bothverbalandphysical)totheque stionsasked.
Records
Overseer Patient
Multilingual
Recogntion AudioOutput
ACAFE
questionsresponses
Reports
Language/
Problem/
Responses
MultilanguageMappings
question
Language/
Problem/
Responses 
Figure2: ComponentoverviewoftheSystem
Architecture 
5.2 SystemComponents 
Thefollowingsectionoutlineseachcomponent
shownintheOverviewdiagram(Figure2). 
AudioOutput  – Rendersquestions(asrequired
bytheDecisionSupport)inthePatient’snative
languageusingrecordedspech,orTexttoSpeech
(TTS)ifther ecordedspechisnotavailable. 
MultilingualRecognition  – Themajorityof
questionsposedtothePa tientareintheformof
yes/noquestions.Assuch,therecognitionofthe
Patient’sutteranceneedsonlytorecogniz ebasic
responsesinthePatient’sselectednativela nguage. 
ACAFE  – Providedwiththemedicalsub
domain(e.g.asthma/breathingdificulties),spec i
fiesquestionsacordingtoastandardsetofdia g
nosisque stions. 
Records  – RecordsPatientresponsestoQue s
tions(bothtextualandaudiorepr esentations),final
outcome,andstatisticsthatareusedforbothindi
vidualPatientreportingandstatisticalrepor ting. 
Reports  – ProvidesindividualPatientreporting
(i.e.nativelanguage,medicalsub domain,r e
sponsestoquestions,andfinaloutcome)andstati s
ticalreportingfortheuseofmeasuringthe
relationshipbetweenasthmaandthepr ecipitating
factors. 
5.3 AsthmaDecisionSuport 
ACAFEisanelectronicinterfacefortheEmer
gencyDepartmentthatprovidesclini cianswitha
decisionsupporttooltoassistinthemanagement
andtreatmentofasthma.Thesystemincorporates
clinicalde cisionsupportbasedoncurrentevidence
andguidelinesthatissimpletoacess,adaptableto
theneedsoftheclinicianswor kingintheERandis
capableofbeingintegratedwithexistingmedical
databases.
Thesystem’scorefocusliesinclinicalpathways
forthetreatmentofasthma.ThisisshowninFig
ure3below.Aclinicalpathwayinthemedical
senseisadecisiontrebased onclinicalassess
mentthatguidesthemanagementandfurtherin
vestigationofapatientwithaparticularclinical
problem.Thisdecisiontrehasbenbasedoncon
sensusguidelinesandinstitutionalprotocolsbased
onthebestavailableevidenceforthe management
ofasthma.
STAGE 1 – PatientHistory 
Presentingproblem
Historyofpresentingproblem
Specificasthmariskhistory
Medication , Alergy 
STAGE 2  Examination 
GeneralApearance
VitalSigns
RespiratoryExamination 
STAGE 5 – FinalAsesment
STAGE 3  Diagnosis 
WorkingDiagnosis
DiferentialDiagnosis
ConfoundingFactors 
STAGE 4 – ElectronicDecisionSuport 
Figure3: TheACAFEclinicalpathway
IntheACAFEsystemtheclinicalpathwayis
representedbytheinformationrequiredtoasce r
tainthes everityofasthmatodecideonalistof
furtherinvestigations,consultationsandmedica
tiono rders.Theclinicalpathwayoutlinesthe
meansthroughwhichthesystemcanadvisethe
doctorontheoptimalasthmamanagementcare
plan.
Atthisstage,ourvoicesystemwillbeintegra ted
withstage1ofACAFE’sclinical pathway,inpa r
ticularthehistory/informationcollectionsideof
things. 
6 Conclusion 
WehaveshownthattheACAFEsystemwiththe
assistanceofourvoicesystemcancapturethein
formationrequiredtoassistcliniciansbetterma n
agethetreatmentofasthmainanemergency
department.Incapturingthisdata,theACAFEand
voicesystemincorporatestheclinicalpathways
anddecisionsupportintheworkflowofthedo ctor.
Inthisdemonstratorpaper,weproposedasy stem
that:
ReliesonACAFEbyprovidingan electronic
standardizedprotocolforthetreatmentof
asthma.
Allowsmulti lingualsupporttherebyincreasing
communicationbetweenmedicalstafand
patientsduringinformationcollectionand
follow upreviewafterthepatienthasbeen
discharged.
Increa seseficiencybyautomatinghowinfo r
mationiscollectedbyassistinginther e
cordinganddocumentationofvastamounts
ofinformationwhilealsostreamliningthe
updateofdataelectronicallyintothepa tient
medicalsystem. 
Acknowledgements 
WewouldliketothankosTechnologyfortheir
inputrelatingtotheACAFEsystemdevelopment.
Wewouldalsoliketothanktheemergencyde
partmentteamatCanterburyHospitalfortheira s
sistanceandexperta dviceinthefieldofasthma
patientcareandinformationc ollection. 

References
ACAFEResearchProjectandDevelopmentteam,
http:/ww.ostechnology.com.au/acafe/our_team.ht
ml.AsofJanuary2006
Nuance, htp:/ww.nuance.com .Asof8December
2005.
M.Rayner,P.Bouilon,N .Chatzichrisafis,B.A.
Hockey,M.Santaholma,M.Starlander,H.Isa hara,
K.Kanzaki,Y.Nakao(2005).  AMethodologyfor
ComparingGrammarBasedandR obustApproaches
toSpeechUnderstanding ,In  ProcedingsofEu
rospeechInterspeech,4 8,September,2005, Lisboa,
Portugal. 
M.Rayner,P.Builon,B.A.Hockey,N.Chat
zichrisafis,M.Starlander(2004).  ComparingRu le
BasedandStatisticalApproachestoSpechUnder
standinginaLimitedDomainSpechTranslation
System .In  ProcedingsofTMI2004,Baltimore, MD
UA,2004. 
M.Starlander,P.Bouilon,N.Chatzichrisafis,M.
Rayner,B.A.Hockey,H.Isahara,K.Kanzaki,Y.
Nakao,M.Santaholma(2005). BreakingtheLa n
guageBarrier:MachineAsistedDiagnosisusingthe
MedicalSpeechTranslator .In  Procedingsofthe
XIXInternationCongresoftheEuropeanFedera
tionforMedicalInformaticsMIE,28August 1Sep
tember,2005,Geneva,Switzerland.
