<?xml version="1.0" standalone="yes"?>
<Paper uid="W97-0117">
  <Title>A Natural Language Correction Model for Continuous Speech Recognition 1</Title>
  <Section position="6" start_page="171" end_page="173" type="evalu">
    <SectionTitle>
5. EXPERIMENTAL RESULTS
</SectionTitle>
    <Paragraph position="0"> The experimental data was obtained from the University of Maryland Medical Center in Baltimore, which is one of the clinical sites used in this project. The validated transcriptions have been extracted from the hospital database by the hospital personnel, and then sanitized to remove any patient information such as names, addresses, etc. At the time this report is written, we collected nearly 7000 transcribed dictations, all in the area of chest X-ray. Chest X-ray is the most prevalent form of radiology, and we decided to start with this sub-area because of its the largest potential practical significance.</Paragraph>
    <Paragraph position="1"> The sanitized reports were subsequently re-dictated through the automated speech recognition system in order to obtain parallel samples of automated transcription. The redictation was done over a period of several months by a final-year radiology resident at Albany Medical Center, a native North American English speaker. At the time this paper is prepared, some 1000 reports have been redictated. Clinical tests of the system equipped with the C-Box that are starting in early 1997 will provide additional speakers.</Paragraph>
    <Paragraph position="2">  Generally, we observed significant word error rates in automated speech recognition, in some cases as high as 38%, with the average of 14.3%. This is substantially higher than the advertised 5% error rate. Before starting re-dictation, the speaker underwent a few hours training session, learning how to use the system, and having his voice patterns incorporated into the language model (the system we use is speaker adaptable). The above numbers therefore represent an optimal performance of the system for this speaker, although there are some hard-to-measure mitigating considerations. For example, the radiology reports used in these experiments were read by an AMC resident, who while obviously familiar with the subject matter, also pointed out some fine vocabulary and style differences between AMC and UMMC Baltimore, where the reports were produced. This could potentially have an impact at SRS performance. It should be noted that a typical chest X-ray dictation report is quite short, from a few lines to a few paragraphs, and is dictated quite rapidly in anywhere from 15 seconds to a few minutes.</Paragraph>
    <Paragraph position="3"> Preliminary experiments with context-free rules have already shown interesting results: we noticed that the average word error rate decreased from 14.3% to 11.3% (a 21% reduction) on a test sample after running it through a C-Box equipped with only a few CF rules. This C-Box was trained on 800 reports (0.3 MByte) and tested on 200 reports (92 KBytes).</Paragraph>
    <Paragraph position="4"> Below is a sample radiology report, its automated transcription version, and the effect of a partial correction. Note that only context-free rules are used; a context-sensitive correction indication colon ~ indication : would fix the problem in the first line.</Paragraph>
    <Paragraph position="5"> Original ASR Transcription: (errors highlighted) indication colon and trachea to place.</Paragraph>
    <Paragraph position="6"> the endotracheal tube is in size factor position, there is and re-expansion of the fight upper lobe. mild changes of the 8th rds persist bilaterally.</Paragraph>
    <Paragraph position="7">  the endotracheal tube is in SATISFACTORY position, there HAS BEEN re-expansion of the right upper lobe. mild changes of the 8th rds persist bilaterally.</Paragraph>
    <Paragraph position="8"> Correction rules used: and trachea to ~ endotracheal tube, size factor ~ satisfactory, is and ~ has been.</Paragraph>
  </Section>
class="xml-element"></Paper>