A Prototype Reading Coach that Listens: 
Summary of Project LISTEN 
Alex Hauptmann, Jack Mostow, Steven F. Roth, Matthew Kane, and Adam Swift 
Project LISTEN 1, 215 Cyert Hall, 4910 Forbes Avenue 
Carnegie Mellon University Robotics Institute, Pittsburgh, PA 15213-3890 
What: Project LISTEN is developing a novel weapon against 
illiteracy: an automated reading coach that displays a story on a 
computer screen, listens to a child read it aloud, and helps where 1. 
needed. The coach provides a combination of reading and listen- 
ing, in which the child reads wherever possible, and the coach 
helps wherever necessary. We demonstrated a prototype of this 
coach at the ARPA Workshop on Human Language Technology 
in March 1994. A short video shows the coach in action \[1\]. 2. 
Who: The intended users of the coach include children in grades 
1-3, where oral reading is emphasized. Its developers include 3. 
experts on speech technology, reading, and human-computer in- 
teraction. Its testers include approximately 100 second graders in 
Pittsburgh public schools who have difficulty in reading. 4. 
Why: Illiteracy costs the United States over $225 billion dollars 
annually in corporate retraining, lost competitiveness, and in- 
dustrial accidents \[2\]. Individuals with low reading proficiency 
are much likelier to be unemployed, poor, or incarcerated \[3\]. 5. 
Automated literacy tutoring is an important, real, and challenging 
task requiring multimodal interaction and real-time response. 
How: Project LISTEN is made possible by years of government- 
funded research that produced CMU's Sphinx-I/ speech 
recognizer \[4\], which we adapted to detect errors in oral reading 6. 
of known text \[5, 6, 7\]. We modelled the coach after expert 
reading teachers and refined it based on experimental use \[8\]. 
Evaluation: Pilot experiments reported in \[7\] tested both the 
coach's accuracy in detecting reading errors, and its potential 
effectiveness in helping children read. 7. 
Future work: Our primary goal is to extend and deploy the 
coach in order to help children read better over time. Possible 
spinoff applications include adult literacy, English as a second 
language, foreign language learning, interactive entertainment, 8. 
and computer-assisted writing. We envision a new generation of 
intelligent tutoring systems that listen to their students, providing 
individualized attention that conventional classrooms cannot. 
REFERENCES 
J. Mostow, S. Roth, A. Hauptmann, M. Kane, A. Swift, L. Chase, 
and B. Weide, "A Reading Coach that Listens (6-minute video)", 
VMeo Track of the Twelfth National Conference on Artificial 
Intelligence (AAA194), American Association for Artificial Intel- 
ligence, Seattle, WA, August 1994. 
E. Herrick, "Literacy Questions and Answers." Pamphlet, Con- 
tact Center, Inc., 1990. 
National Center for Education Statistics, "Adult Literacy in 
America", Tech. report GPO 065-000-00588-3, U.S. Department 
of Education, September 1993. 
X. D. Huang, F. AUeva, H. W. Hon, M. Y. Hwang, K. F. Lee, and 
R. Rosenfeld, "The SPHINX-II speech recognition system: An 
overview", Computer Speech and Language, Vol. 7, No. 
2, April 1993, pp. 137-148. 
J. Mostow, A. G. Hauptmann, L. L. Chase, and S. Roth, 
"Towards a Reading Coach that Listens: Automated Detection of 
Oral Reading Errors", Proceedings of the Eleventh National 
Conference on Artificial Intelligence (AAAI93), American As- 
sociation for Artificial Intelligence, Washington, DC, July 1993, 
pp. 392-397. 
A. G. Hauptmann, L. L. Chase, and J. Mostow, "Speech Recog- 
nition Applied to Reading Assistance for Children: A Baseline 
Language Model", Proceedings of the 3rd European Conference 
on Speech Communication and Technology (EUROSPEECH93), 
Berlin, September 1993, pp. 2255-2258. 
J. Mostow, S. Roth, A. G. Hauptmann, and M. Kane, "A 
Prototy~ Reading Coach that Listens", Proceedings of the 
Twelfth National Conference on Artificial Intelligence (AAAI94 ), 
American Association for Artificial Intelligence, Seattle, WA, 
August 1994. 
J. Mostow, S. Roth, A. Hauptmann, M. Kane, A. Swift, L. Chase, 
and B. Weide, "Getting Computers to Listen to Children Read: A 
New Way to Combat Illiteracy (7-minute video)", Overview and 
research methodology of Project LISTEN as of July 1993. 
ffThis research was supported primarily by the National Science Foundation under Grant Number MDR-9154059 and by the Advanced 
Research Projects Agency, DoD, through DARPA Order 5167, monitored by the Air Force Avionics Laboratory under contract 
N00039-85-C-0163. The views and conclusions contained in this document are those of the authors and should not be interpreted as 
representing the official policies, either expressed or implied, of the sponsors or of the United States Government. 
We thank our principal reading consultant Leslie Thyberg; Raj Reddy and the rest of the CMU Speech Group (FiUeno AUeva, Bob 
Brennan, Lin Chase, Xuedong Huang, Mei-Yuh Hwang, Sunil Issar, Fu-hua Liu, Chenxiang Lu, Pedro Moreno, Ravi Mosur, Yoshiaki 
Ohshima, Paul Placeway, Roni Rosenfeld, Alex Rudnicky, Matt Siegler, Rich Stern, Eric Thayer, Wayne Ward, and Bob Weide) for 
Sphinx-H; Paige Angstadt, Morgan Hankins, and Cindy Neelan for transcription; Maxine Eskenazi for transcript analysis; Lee Ann Kane 
for her voice; Jim Kocher for video production; CTB Macmillan/McGraw-Hill for permission to use copyrighted reading materials from 
George Spache's Diagnostic Reading Scales; the students and educators at Colfax Elementary School, East Hills Elementary School, 
Turner School, and Winchester Thurston School for participating in our experiments; and many friends for their assistance. 
237 
