Advanced Human-Computer Interface and 
Voice Processing Applications in Space 
Julie Payette 
Canadian Space Agency 
Canadian Astronaut Program 
St-Hubert, Quebec 
J3Y 8Y9 
ABSTRACT 
Much interest already exists in the electronics research community 
for developing and integrating speech technology to a variety of ap- 
plications, ranging from voice-activated systems to automatic tele- 
phone transactions. This interest is particularly true in the field of 
aerospace where the training and operational demands on the crew 
have significantly increased with the proliferation of technology. In- 
deed, with advances in vehicule and robot automation, the role of the 
human operator has evolved from that of pilot/driver and manual con- 
troller to supervisor and decision maker. Lately, some effort has been 
expended to implement alternative modes of system control, but au- 
tomatic speech recognition (ASR) and human-computer interaction 
(HCI) research have only recently extended to civilian aviation and 
space applications. The purpose of this paper is to present the par- 
ticularities of operator-computer interaction in the unique conditions 
found in space. The potential for voice control applications inside 
spacecraft is outlined and methods of integrating spoken-language 
interfaces onto operational space systems are suggested. 
1. INTRODUCTION 
For more than three decades, space programs internationally have 
been synonymous with the frontier of technological developments. 
Since 1957, NASA alone has launched an impressive series of earth- 
orbiting satellites, exploration missions and manned vehicules. 
Mission complexity has increased tremendously as instrumentation 
and scientific objectives have become more sophisticated. Recent 
developments in robotics and machine intelligence have led to strik- 
ing changes in the way systems are monitored, controlled, and 
operated\[i\]. In the past, individual subsystems were managed by 
operators in complete supervisory and directing mode. Now the de- 
cision speed and complexity of many aerospace systems call for a 
new approach based on advanced computer and software technology. 
In this context, the importance of the human computer interface 
cannot be underestimated. Astronauts will come to depend on the 
system interface for all aspects of space life including the control 
of the onboard environment and life support system, the conduct 
of experiments, the communication among the crew and with the 
ground, and the execution of emergency procedures. 
One of the technology sought to help solve the human interface 
challenge in space is voice processing. Though Automatic Speech 
Recognition (ASR) and other forms of advanced voice I/0 techniques 
have only recently been experimented with simple avionic systems, 
much remains to be done to meet the ergonomic characteristics of 
the unique operational envkonment found in space. 
This paper presents the particularities of operator-computer interac- 
tion in space. As examples, the current envkonment of the Space 
Shuttle System is described and projected requirements for the in- 
ternational Space Station are examined. The paper also outlines the 
potential for voice control applications inside spacecraft and suggest 
methods of integrating spoken-language interfaces onto operational 
space systems. 
2. THE WORKPLACE: SPACE 
Any space flight represents some degree of risk and working in 
space, as in aviation, comports some hazards. Suddenly, at any time 
during a mission, a situation may occur that will threaten the life 
of the astronauts or radically alter the flight plan. Thus, critical to 
the success of the mission and security of the crew is the complex 
process of interaction between astronauts and their spacecraft, not 
only in routine operation, but also in unforseen, unplanned, and 
life-threatening situations. 
2.1. Environmental Factors 
The environment outside spacecraft is unforgiving. With surface 
temperatures ranging from -180 C in darkness and 440 C in sunlight, 
high radiation and no atmosphere, lower earth orbit is hostile to life. 
Yet, astronauts work in this environment, under high workload and 
high stress, sheltered inside protective vehicules or dressed in bulky 
spacesuits. 
To limit the risks of space walks, the ability to perform physical 
actions remotely is crucial. Aboard the shuttle, remote action is per- 
formed using of the Remote Manipulator System (RMS) 1 . And more 
than any other task performed in space, telerobotics introduces higher 
demands on the relationship between operators and machines\[2\]. 
Microgravity is another important environmental factors which con- 
siderably influence the interface configuration of aerospace systems. 
Space travel not only generates significant levels of stress in the 
human organism, but transforms the entire operational conditions. 
For instanc e , to perform their work in space, "weightless" astronauts 
must hold themselves "down" one way or another. They use straps 
and foot restraints, or simply grasp hand holders to maintain their 
position. 
All the above considerations impose restrictions and introduce severe 
design requirements as follows: 
• Safety: security is a paramount consideration aboard any 
1 The Remote Manipulator System of the Shuttle is a Canadian-built tele- 
operated robot arm that is used in a semi- autonomous mode during those 
space flights that require objects to be handled, captured, and released into 
space. 
416 
spacecraft. Every procedure and piece of equipment under- 
goes thorough review before being rated flight eligible. For 
example, all critical shuttle controls, such as an emergency 
stop switch, are required to meet very stringent layout require- 
ments. No floating object or particle may inadvertently activate 
. or damage a sensitive system. 
• Reliability/Accuracy/Redundancy: high tolerance to failure 
is a condition to safety. Operative systems in space must be 
at least two fault-tolerant, if not more in the ease of critical 
systems such as flight controls or envkonmental control and life 
support systems (ECLSS). Where applicable, error correction 
mechanisms must be implemented. 
• Accessibility: the crew's ability to execute tasks safely and 
efficiently is notably improved if controls are ergonomically 
placed, clearly marked, and readily available\[3\]. Indirect ac- 
cessibility is also crucial, particularly where overriding of au- 
tomated functions is required. 
• Feedback: in diffeult operational envkonments such as micro- 
gravity, precise system feedback becomes essential. Through 
visual, auditive and tactile means, feedback reinforces security 
procedures and lessen the monitoring workload, particularly 
for telerobotic tasks which must be performed with extreme 
caution. 
On the shuttle, for example, robot arm operations are executed 
by two astronauts, one manipulating the robot and the other 
assisting with secondary functions, camera controls and sta- 
res displays. Visual feedback, if not precisely obtained from 
camera views, is directly available from the four windows of 
the Shuttle's flight deck. On Space Station, dkect visual feed- 
back will only be available on rare occasions and thus other 
means of feedback will have to be developed and integrated 
with the HCI of the robotics control workstation to provide 
camera redundancy. 
• Commonality: system configuration consistent in type and 
quality for crew operations enhance efficacy and lowers train- 
ing demands. Operating Space Station with an international 
crew, in particular, will necessitate very high commonality of 
functions to ensure safety. 
2.2. Technology Proliferation 
Environmental constraints are only one of the many factors influ- 
encing the HCI problem in space. Technological diversification is 
another. Recent advances have considerably increased the process- 
ing and information handling capability of computer systems, thus 
bringing additional operative complexity that must be absorbed by 
operators. In aircraft and spacecraft, despite notable efforts to in- 
tegrate systems more efficiently, there is so much information, so 
many sources, datatypes, categories, variations, possibilities, lay- 
outs, scales, etc. that crew members no longer operate their system 
globally. Instead, they receive specific training or pair up to accom- 
plish their tasks. 
The impact of technology proliferation is clearly seen on the Space 
Shuttle. As described in a NASA technical report, it is clear "that 
the Shuttle cockpit contains the most complicated assortment of 
D&C (Displays and Controls) ever developed for an aerodynamic 
vehicule. For control, there are toggle, push button, thumbwheel, 
press-down and rotary switches; potienfiometers; keyboards; circuit 
breakers; and hand controllers. Display devices include circular and 
vertical meters, tape meters, mechanical talkbacks, annunciators, 
flight control meters, digital readouts and CRTs. There are more 
than 2100 D&C devices in the orbiter cockpit."\[3\] 
With the number and types of redundant subsystems continually 
increasing, the use of dedicated control devices is rapidly growing 
into a large, complex system difficult to update and interact with. 
These conditions have prompted reconsideration of the direction 
taken in aerospace system design. 
An obvious solution to the problem of the exploding cockpit and 
crew workload in a demanding environment is a greater level of 
automation of functions and the introduction of alternative interfaces. 
3. ADVANCED INTERFACES 
The computer and operational systems used in space function under 
either autonomous or human control. Much of the configuration 
complexity is kept as transparent as possible to the users, to allow 
them to concentrate on the purpose of the interaction, rather than 
system design details. The current design approach focuses on means 
of simplifying operations wherever possible and facilitating operator- 
machine communication. 
The concept of a more integrated human-computer system is clearly 
pertinent in space application. Astronauts are functional compo- 
nents of space systems, not only as operators and controllers, but 
as contributors to the overall performance of the system. On Space 
Station, where the network of computers will control and monitor 
thousands of automated systems as well as provide an interface to 
the crew, the need for performance will be heightened, necessitating 
increased automation and expansion of the supervisory role of the 
crew members. 
However, the decision to automate certain aspects of aerospace mis- 
sion operations demands a careful consideration of the potential 
human-computer relationship. The decision to use a machine for a 
particular set of functions will depend on many factors such as avail- 
ability, appropriateness, cost, compatibility with existing systems, 
and more importantly, safety and efficiency. 
Since few, ff any, external resources and development systems will 
be available on a permanent space platform such as Space Station, 
great selectivity and perspicacity must be exercised when designing 
and building the human computer interface. Hence, the interest in 
investigating new forms of interfaces and input/output devices, such 
as voice command and automatic speech recognition. 
4. AUTOMATIC SPEECH IN SPACE 
Automatic recognition and understanding of speech is one of the 
very promising application of advanced information technology. As 
the most natural communication means for humans, speech is often 
argued as being the ultimate medium for human-machine interaction. 
On the other hand, with its hesitations and complexity of intention, 
spoken language is often thought as being inadequate and unsafe 
for accurate control and time critical tasks\[4\]. Unconvinced of the 
reliability of speech processing as a control technology, pilots and 
astronauts have traditionally been reluctant to accept voice interfaces. 
Yet within a domain-limited command vocabulary, voice control has 
already been identified as a likely choice for controlling multifunc- 
tion systems, displays and control panels in a variety of environ- 
ments. Requiring minimal training, information transfer via voice 
control offers the basis for more effective information processing, 
417 
particularly in situations where speakers are already busy performing 
some other tasks. 
4.1. Benefits of Speech Technology 
Motivations for using ASR in space are numerous. Traditionally, 
space operations have been accomplished via hardware devices, 
dedicated system switches, keyboards and display interfaces. In 
such context, ASR is seen as a complement to existing controls that 
should be used in conjunction with other interaction devices akeady 
bounded in terms of previously defined needs and capabilities. 
It is foreseeable that voice control and synthesis could be used as an 
added I/O channel to use the crew more efficiently during peak work- 
load periods. In particular, ASR may serve to facilitate operations 
in such areas as simultaneous control and monitoring (when hands 
and eyes are busy), extravehicular activities (EVA) and information 
storage and retrieval. 
For some applications, voice commands combined with manual con- 
trois may allow more rapid task completion than would be possible 
with manual methods alone. As an example, a study conducted in 
the Manipulator Development Facility (MDF) of the NASA John- 
son Space Center showed that voice control could be effectively 
used to perform the many switching camera functions associated 
with the closed-circuit television system supporting the RMS robot 
arm\[3\]. The study also revealed that identical tasks (berthing and de- 
ployment) were completed in virtually identical times using manual 
switching and voice controlled switching having recognition accu- 
racy between 85 and 95 percent. Using more accurate, state-of-the- 
art ASR equipment should allow for marked improvement in the 
overall RMS operations. 
Interest in voice command and automatic speech recognition inter- 
faces for space stems from the benefits it may bring to the demanding 
operational environment: 
• hands free control 
• altemate control (redundancy) 
• extension capabilities 
• task adaptability 
• consistency of interface 
• commonafity of usage 
• generic input/output function without requiring diversion of 
visual attention from monitoring tasks. 
4.2. Disadvantages and Concerns 
As described in section 2, technical constraints and environmental 
factors impose significant implementation requirements on the use 
of ASR and voice technology in space. Other issues to be consid- 
ered range from the technical choices (isolated word vs continuous 
speech, single vs multiple speakers, word based vs phoneme based), 
the recognizer training update and maintenance requirements, the 
magnitude of changes in voice characteristics while in microgravity, 
and the effect of the space suit (0.3 atmosphere, pure oxygen) upon 
maintenance of highly accurate recognition. 
Without a doubt, ASR system will require a very high recognition 
accuracy rate, possibly 99evaluations performed at NASA that as- 
tronauts will switch to habitual controls if latency, reliability and 
efficiency criteria are not met\[5\]. Also, safety and requirements will 
necessitate a high level of recognition feedback to the users, with 
interactive error correction and user query functions. 
Finally, on the international Space Station, the diversity of languages 
and accents may make ASR an even more difficult challenge to meet. 
5. APPLYING VOICE IN SPACE 
Interest in voice technology for space appfications is not new. NASA 
is actively pursuing applications of voice recognition and synthesis 
for its spacecraft and ground operations\[5, 6, 7, 8\]. Several testbeds 
have incorporated voice into their commanding scheme, but only a 
few experiments have been performed in operational environments. 
These experiments are summarized below, followed by an outline of 
future applications. 
5.1. A Bit of History 
Ground Test 
On the Shuttle, most Extra-Vehicular Activities (EVA) are performed 
for a specific tasks and rehearsed many times before the mission. 
To aid in these operations, cuff-mounted checklists have served as 
a useful reminder of procedures to follow. The problem with cuff 
checklists is that the wrist is not always in the best position for reading 
and at least one hand is required to turn the pages. Furthermore, 
information is limited to 3.25x4.5 inch pages which require a restraint 
to keep in position. For the longer missions on Space Station where 
EVA tasks will be less predictable, cuff checklist will be inadequate. 
In 1986 and 1988, a voice-activated/voice-synthesis system was de- 
veloped in conjunction with a prototype space suit to provide an 
alternate information system. Equipped with a voice- controlled 
head-mounted display, the suit was evaluated on the ground in a 
series of neutral buoyancy tests\[9\]. 
The voice system was termed an improvement over the cuffchecklist, 
allowing both hands on the job while moving through procedures, 
but astronauts commented that the system created a lot of disruptive 
"chatter" on the channel and interfered with communications. 
Voice Recording Test 
In 1990, direct digital recordings of an astronaut's voice were per- 
formed on the ground before a mission, in flight during the mission 
and on the ground upon return. A selected vocabulary was used 
and templates were made. After analysis, significant acoustic dif- 
ferences were noted. No conclusions were drawn, however, as to 
whether microgravity was the cause of these changes in voice pro- 
duction, since the discrepancy was mostly blamed on a substantial 
difference between recording environments. 
Shuttle Flight Test 
To date, only one experiment using voice recognition technology 
has ever been used aboard a spacecraft. The voice command system 
(VCS) experiment flew on board Space Shuttle Discovery STS-41 
in October 1990 and allowed astronauts Bill Shepherd and Bruce 
Melnick to control the closed-circuit television (CCTV) cameras 
and monitors by voice inputs\[10\]. The voice command system had 
the capability to control the CCTV camera selection, and camera 
functions such as pan, tilt, focus, iris and zoom. The VCS paralleled 
the manual controls and provided both audible and visual feedback. 
The system was speaker dependent with templates of the voice of 
the two astronauts previously made on the ground. The recognizer 
418 
had limited continuous recognition and syntactic capabilities. 
The VCS intended to collect baseline data on the effect of micrograv- 
ity on speech production and recognition. In addition, the experiment 
was meant to show the operational effectiveness of controlling a 
spacecraft subsystem using voice input. Analysis of the data showed 
little variation between the microgravity and ground-based templates 
of the astronauts voices. According to the investigators, astronauts 
were pleased with the tests and stated that voice control was a useful 
tool for performing secondary tasks on the Shuttle. 
Recent evaluations of different modes of camera control performed 
at NASA and to which the author participated have shown however 
that non-hardware controls will only be considered as sufficiently 
safe to be used in space if reliability can be proven, redundancy 
possible and efficiency significantly optimized. Moreover, as men- 
tioned, experience has shown that crew members readily revert to 
the primary control system to which they are used to if an alternative 
system is not sufficiently accurate. 
5.2. Application Potential 
Spoken language communication with the control and monitoring 
subsystems onboard the Shuttle or Space Station is a convenience 
that could be provided through automatic speech recognition applica- 
tions. Although ASR could not be the primary means of controlling 
critical actinns\[7\], it could be used to backup the primary controller 
and as an alternative I/O medium for the crew. 
Speech could also be used to query the status of a particular subsys- 
tem or database. Reference manuals could be called up and paged 
through. ASR could also be used for "hands-free" maintenance re- 
porting, allowing the crew to attend to more important work and 
spend less time generating written reports. 
Other possible applications would be the use of speech to overcome 
reduced manual dexterity caused by astronauts having to wear bulky 
space suit and gloves during ascent and reentry. Voice interfaces 
could be used to allow a diversity in the number of tasks to be 
performed as flexible as the size of the recognizer's vocabulary. 
Of particular interest are the high workload and adverse condition 
situations (G-load, noise, stress) where dkect voice input could make 
a significant contribution to overall efficiency. 
Another promising application for voice technology in space is dur- 
ing EVA activities, where voice control would allow astronauts on 
space walks to perform interactive queries and/or remote manipula- 
tor control, while busy performing some other maintenance or repair 
task or even simply, busy holding themselves down. 
5.3. Space Station 
The proposed Space Station may also benefit from some form of 
automatic voice interaction to reduce transaction time between crew 
members and their multitasking, multi-panel workstations. Space- 
based crew are expected to interact with highly automated systems 
and to perform these interactions with often little prior training, or 
on an infrequent or sporadic basis. These activities will characterize 
a new role for astronauts, that of supervisory control. 
For instance, current plans for the Space Station involve the use of 
a significant robotic workforce for assembly, servicing and mainte- 
nance tasks. Generically referred to as the Mobile Servicing System, 
this workforce will be operated from a multi- purpose control work- 
station. Equipped with three display devices, the workstation will 
include one keyboard, one cursor control device and a dedicated 
hardware switching panel. The HCI aspects of the workstation are 
currently under designed in Canada and the proposed configura- 
tion has already raised several major issues centering around how 
crewmembers will interact with multiscreen systems. As there will 
be times in which users will be performing up to four simultaneous 
tasks using the robotics workstation, designers are now looking at al- 
ternative methods of interaction, including voice-activated features. 
Finally, as technology progresses, ASR might be used in conjunc- 
tion with voice synthesis and natural language techniques to provide 
technical advice or even language translation to assist with commu- 
nication between the international crew members. 
6. CONCLUSION 
Developing advanced human-computer interaction for space oper- 
ations is a challenging task that requires the coordinated effort of 
various fields of study. A potential avenue for solution is to consider 
voice technology and automatic speech recognition techniques as 
means of optimizing astronaut performance and helping reduce their 
workload in space. 
The voice experiment performed on the Space Shuttle mission STS- 
41 in 1990 has demonstrated that an advanced voice system for 
interacting with on-board subsystems may prove both useful and 
cost effective. 
Yet, as Dr. Vladimir Solov'yev, a veteran of two russian space mis- 
sions totalling more than 12 months points out, the human component 
of the human-machine system remains a key: 
Today, we have been accustomed to spacecraft launches 
and space flights are now perceived as something to be 
taken for granted. The experience of our cosmonauts is 
that there is no such thing as an easy space flight. A 
cosmonaut or ground control specialist is still a human 
being with its own set of capacities and problems. It 
means that it is a human being who takes a machine \[up 
in space.., and\] a human, surrounded by a the hostile 
environment of space, who makes decisions and interacts 
with a machine to attain a desired result\[11\]. 
Any future HCI proposal will only achieve its purpose flit is designed 
with these limits in mind. 
References 
1. Heer and Lum editors. Machine lntelligence and Autonomy for 
Aerospace Systems. Progress in Astronautics and Aeronautics, 
volume 115, AIAA, 1988. 
2. Smith T.J. and Stuart M.A. "Human Factors of Teleoperation 
in Space". In Proceedings of the Human Factors Society 34th 
Annual Meeting, 1990. 
3. Burghduff, R.D. and Lewis, J.R. "Man-machine Interface and 
Control of the Shuttle Digital Flight System". In Proceedings 
of the Space Shuttle Technical Conference, Houston TX, 1983. 
4. Howie, M.B., Woods, D.D. and Cook, R.I. "The Natural His- 
tory of Introducing New Information Technology into a High- 
risk Environment". In Proceedings of the Human Factors So- 
ciety 34th AnnuaI Meeting, 1990, pp. 429-433. 
419 
5. Coler, C.R. "Use of Automatic Speech Recognition Technol- 
ogy for Space Applications". In Proceedings of the Voice I/O 
Systems Applications Conference, Pale Alto CA, 1984. 
6. Parks, D.L. and White, R.W. Study to Determine Potential 
Flight Application and Human Factors Design Guidelines for 
Voice Recognition and Synthesis Systems. Boeing Commercial 
Air,plane Company, NAS 1-17367, 1985. 
7. Sommers, M3. "Capabilities and Cons~aints of Automatic 
Speech Recognition and its Use for the Space Station". In 
Proceedings of Speech Teeh, New York, 1987, pp. 318-319. 
8. Jordan, W.T. "Space Shuttle Voice Recognition System". In 
Proceedings of Speech Tech, New York, 1987, pp. 320-321. 
9. Griffin, B.N. "Zero-G Simulation Verifies EVA Servicing of 
Space Station Modules". A/AA Space Station in the 21st Cen- 
tury, AIAA paper 86-2312, Reno NV, September 1986. 
10. Sa]azar, G. "Voice Recognition Makes its Debut on the NASA 
STS-41 Mission". Speech Technology, Feb/March 1991, pp. 
86-92. 
11. So]ov'yev, V. "Problems of Human-machine Interaction on 
Space Flight". In Humans and Machines in Space, Proceedings 
of the 29th Goddard Memorial Symposium, Johnson, May & 
Kern editors, Univelt publishers, 1991, pp. 73-77. 
420 
