Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 105–108,
New York, June 2006. c©2006 Association for Computational Linguistics
Using Semantic Authoring for Blissymbols Communication Boards
Yael Netzer
Dept. of Computer Science
Ben Gurion University
Beer Sheva, Israel
yaeln@cs.bgu.ac.il
Michael Elhadad
Dept. of Computer Science
Ben Gurion University
Beer Sheva, Israel
elhadad@cs.bgu.ac.il
Abstract
Natural language generation (NLG) refers
to the process of producing text in a spo-
ken language, starting from an internal
knowledge representation structure. Aug-
mentative and Alternative Communica-
tion (AAC) deals with the development
of devices and tools to enable basic con-
versation for language-impaired people.
We present an applied prototype of an
AAC-NLG system generating written out-
put in English and Hebrew from a se-
quence of Bliss symbols. The system does
not “translate” the symbols sequence, but
instead, it dynamically changes the com-
munication board as the choice of sym-
bols proceeds according to the syntactic
and semantic content of selected symbols,
generating utterances in natural language
through a process of semantic authoring.
1 Introduction
People who suffer from severe language impair-
ments lack the ability to express themselves through
natural usage of language and cannot achieve var-
ious forms of communication. The field of Aug-
mentative and Alternative Communication (AAC) is
concerned with methods that can be added to the
natural communication. In the most common form,
iconic symbols are presented on a display (or a com-
munication board). Communication is conducted by
the sequential selection of symbols on the display
(with vocal output when available), which are then
interpreted by the partner in the interaction.
AAC devices are characterized by three aspects:
(i) Selection method i.e., the physical choice of sym-
bols on the communication board; (ii) input lan-
guage and (iii) output medium. In a computerized
system, as (McCoy and Hershberger, 1999) mention,
a processing method aspect is added to this list. This
method refers to the process which creates the out-
put once symbols are inserted.
We specifically study the set of symbols (as an in-
put language) called Blissymbolics (Bliss in short).
Bliss is a graphic meaning-referenced language, cre-
ated by Charles Bliss to be used as a written univer-
sal language (Bliss, 1965); since 1971, Blissymbols
are used for communication with severely language-
impaired children. Bliss is designed to be a written-
only language, with non-arbitrary symbols. Sym-
bols are constructed from a composition of atomic
icons. Because words are structured from seman-
tic components, the graphic representation by itself
provides information on words’ connectivity 1.
In the last decade, several systems that integrate
NLG techniques for AAC systems have been devel-
oped ((McCoy, 1997), (Vaillant, 1997) for example).
These systems share a common architecture: a tele-
graphic input sequence (words or symbols) is first
parsed, and then a grammatical sentence that repre-
sents the message is generated.
This paper presents an NLG-AAC system that
generates messages through a controlled process of
authoring, where each step in the selection of sym-
bols is controlled by the input specification defined
1See http://www.bci.org for reference on the language
105
for the linguistic realizer.
2 Generating Messages via Translation
A major difficulty when parsing a telegraphic se-
quence of words or symbols, is that many of the
hints that are used to capture the structure of the
text and, accordingly, the meaning of the utterance,
are missing. Moreover, as an AAC device is usu-
ally used for real-time conversation, the interpreta-
tion of utterances relies heavily on pragmatics – time
of mentioned events, reference to the immediate en-
vironment.
Previous works dealing with translating tele-
graphic text, such as (Grishman and Sterling, 1989),
(Lee et al., 1997) requires to identify dependency
relations among the tokens of the telegraphic input.
Rich lexical knowledge is needed to identify possi-
ble dependencies in a given utterance, i.e., to find
the predicate and to apply constraints, such as selec-
tional restrictions to recognize its arguments.
Similar methods were used for AAC applica-
tions, COMPANSION (McCoy, 1997) for example
– where the telegraphic text is expanded to full sen-
tences, using a word order parser, and a semantic
parser to build the case frame structure of the verb
in the utterance, filling the slots with the rest of the
content words given. The system uses the semantic
representation to re-generate fluent text, relying on
lexical resources and NLG techniques.
The main questions at stake in this approach are
how good can a semantic parser be, in order to re-
construct the full structure of the sentence from tele-
graphic input and are pragmatic gaps in the given
telegraphic utterances recoverable in general.
3 Generating Messages via Semantic
Authoring
Our approach differs from previous NLG-AAC sys-
tems in that, with the model of semantic authoring
(Biller et al., 2005), we intervene during the process
of composing the input sequence, and thus can pro-
vide early feedback (in the form of display composi-
tion and partial text feedback), while preventing the
need for parsing a telegraphic sequence.
Semantic parsing is avoided by constructing a se-
mantic structure explicitly while the user inputs the
sequence incrementally. It combines three aspects
into an integrated approach for the design of an AAC
system:
• Semantic authoring drives a natural language
realization system and provides rich semantic
input.
• A display is updated on the fly as the authoring
system requires the user to select options.
• Ready-made inputs, corresponding to prede-
fined pragmatic contexts are made available to
the user as semantic templates.
In this method, each step of input insertion is con-
trolled by a set of constraints and rules, which are
drawn from an ontology. The system offers, at each
step, only possible complements to a small set of
concepts. For example, if the previous symbol de-
notes a verb which requires an instrumental theme,
only symbols that can function as instruments are
presented on the current display. Other symbols are
accessible through navigation operations, which are
interpreted in the context of the current partial se-
mantic specification. The general context of each
utterance or conversation can be determined by the
user, therefore narrowing the number of symbols
displayed in the board.
The underlying process of message generation is
based on layered lexical knowledge bases (LKB)
and an ontology. The ontology serves as a basis
for the semantic authoring process; it includes a hi-
erarchy of concepts and relations, and the informa-
tion it encodes interacts with the conceptual graphs
processing performed as part of content determina-
tion and lexical choice. The ontology was acquired
with a semi-automatic tool, which relies on WordNet
(Miller, 1995) and VerbNet (Kipper et al., 2000).
We designed and implemented the Bliss lexicon
for both Hebrew and English. The lexicon can be
used either as a stand-alone lexicon or as part of an
application through an API. The design of the lexi-
con takes advantage of the unique properties of the
language. The Bliss lexicon provides the list of sym-
bols accessible to the user, along with their graphic
representation, semantic information, and the map-
ping of symbols to English and Hebrew words. The
lexicon can be searched by keyword (learn), or by
semantic/graphic component: searching all words in
the lexicon that contain both food and meat returns
the symbols hamburger, hot-dog, meatball etc. (see
106
Fig. 1). The lexicon currently includes 2,200 en-
tries.
Figure 1: A snapshot of the Bliss Lexicon Web Ap-
plication
The core of the processing machinery of the
AAC message generation system is based on SAUT
(Biller et al., 2005) – an authoring system for logical
forms encoded as conceptual graphs (CG). The sys-
tem belongs to the family of WYSIWYM (What You
See Is What You Mean) (Power and Scott, 1998) text
generation systems: logical forms are entered inter-
actively and the corresponding linguistic realization
of the expressions is generated in several languages.
The system maintains a model of the discourse con-
text corresponding to the authored documents to en-
able reference planning in the generation process.
Generating language from pictorial inputs, and
specifically from Bliss symbols using semantic au-
thoring in the WYSIWYM approach is not only a
pictorial application of the textual version, but it also
addresses specific needs of augmentative communi-
cation.
As was mentioned above, generating text from a
telegraphic message for AAC usage must take the
context of the conversation into account. We address
this problem in two manners: (1) adding pre-defined
inputs into the system (yet allowing accurate text
generation that considers syntactic variations), and
(2) enabling the assignment of default values to each
conversation (such as participants, tense, mood). We
also take advantage of the unique properties of the
Bliss symbols; the set of symbols that are offered
in each display can be filtered using their seman-
tic/graphical connectivity; the reduction of the num-
ber of possible choices that are to be made by the
user in each step of the message generation affects
the cognitive load and can affect the rate of commu-
nication.
4 Evaluation
We evaluate our system as an AAC application for
message generation from communication boards.
From an NLG evaluation perspective, this corre-
sponds to an intrinsic evaluation, i.e. judging quality
criteria of the generated text and its adequacy rela-
tive to the input (Bangalore et al., 1998). Since the
prototype of our system is not yet adjusted to inter-
act with alternative pointing devices, we could not
test it on actual Bliss users, and could not perform a
full extrinsic (task-based) evaluation.
However, as argued in (Higginbotham, 1995),
evaluations of AAC systems with nondisabled sub-
jects, when appropriately used, is easier to per-
form, and in some cases provide superior results.
Higginbotham’s claims rely on the observation that
the methods of message production are not unique
to AAC users and analogous communication situa-
tions exist both for disabled and nondisabled users.
Nondisabled subjects can contribute to the under-
standing of the cognitive processes underlying the
acquisition of symbol and device performance com-
petencies. We believe that the evaluation of effi-
ciency for non-AAC users should be served as base-
line.
The approach we offer for message generation re-
quires users to plan their sentences abstractly. (Mc-
Coy and Hershberger, 1999) points that novel sys-
tems may be found to slow communication but to in-
crease literacy skills. We therefore tested both speed
of message generation and semantic coverage (the
capability to generate a given message correctly).
The usage of semantic authoring was evaluated on
nondisabled subjects through a user study of 10 sub-
jects. This provides a reliable approximation of the
learning curve and usability of the system in general
(Biller et al., 2005).
In order to evaluate the keystroke savings of the
system we have collected a set of 19 sentences writ-
ten in Bliss and their full English correspondents.
We compared the number of the words in the Eng-
lish sentences with the number of choices needed
to generate the sentence with our system. The total
number of choice steps is 133, while the total num-
107
ber of words in the sentences is 122. This simple ra-
tio shows no improvement of keystrokes saving us-
ing our system. Savings, therefore, must be calcu-
lated in terms of narrowing the choice possibilities
in each step of the process.
However, counting the number of words does not
include morphology which in Bliss symbols requires
additional choices. We have counted the words
in the sentences considering morphology markers
of inflections as additional words, all summing to
138, as was suggested in (McCoy and Hershberger,
1999).
Assuming a display with 50 symbols (and addi-
tional keys for functions) – a vocabulary of requires
50 different screens. Assuming symbols are orga-
nized by frequencies (first screens present the most
frequently used words) or by semantic domain.
The overall number of selections is reduced using
our communication board since the selectional re-
strictions narrow the number of possible choices that
can be made at each step. The extent to which selec-
tion time can be reduced at each step depends on the
application domain and the ontology structure. We
cannot evaluate it in general, but expect that a well-
structured ontology could support efficient selection
mechanisms, by grouping semantically related sym-
bols in dedicated displays.
In addition, the semantic authoring approach can
generate fluent output in other languages (English
and Hebrew, beyond the Bliss sequence – without re-
quiring noisy translation). We also hypothesize that
ontologically motivated grouping of symbols could
speed up each selection step – but this claim must be
assessed empirically in a task-based extrinsic evalu-
ation, which remains to be done in the future.
We are now building the environment for AAC
users with cooperation with ISAAC-ISRAEL 2, in
order to make the system fully accessible and to be
tested by AAC-users. However, this work is still in
progress. Once this will be achieved, full evaluation
of the system will be plausible.
5 Conclusions and Future Work
This work offers a new approach for message gen-
eration in the context of AAC displays using seman-
2Israeli chapter of the International Society for Augmenta-
tive and Alternative Communication
tic authoring and preventing the need to parse and
re-generate. We have designed and implemented a
Bliss lexicon for both Hebrew and English, which
can either be used a stand-alone lexicon for refer-
ence usage or as a part of an application.
Future work includes an implementation of a sys-
tem with full access for alternative devices, expan-
sion of the underlying lexicon for Hebrew genera-
tion, and adding voice output.
References
Srinivas Bangalore, Anoop Sarkar, Christy Doran, and Beth-
Ann Hockey. 1998. Grammar and parser evaluation in the
XTAG project. In Proc. of Workshop on Evaluation of Pars-
ing Systems, Granada, Spain, May.
Ofer Biller, Michael Elhadad, and Yael Netzer. 2005. Interac-
tive authoring of logical forms for multilingual generation.
In Proc. of the 10th workshop of ENLG, Aberdeen, Scotland.
Charles K. Bliss. 1965. Semantography (Blissymbolics). Se-
mantography Press, Sidney.
Ralph Grishman and John Sterling. 1989. Analyzing tele-
graphic messages. In Proc. of DARPA Speech and Natural
Language Workshop, pages 204–208, Philadelphia, Febru-
ary.
D. Jeffery Higginbotham. 1995. Use of nondisabled subjects
in AAC research: Confessions of a research infidel. AAC
Augmentative and Alternative Communication, 11, March.
AAC Research forum.
K. Kipper, H. Trang Dang, and M. Palmer. 2000. Class-based
construction of a verb lexicon. In Proceeding of AAAI-2000.
Young-Suk Lee, Clifford Weinstein, Stephanie Seneff, and Di-
nesh Tummala. 1997. Ambiguity resolution for machine
translation of telegraphic messages. In Proc. of the 8th con-
ference on EACL, pages 120–127.
Kathleen F. McCoy and Dave Hershberger. 1999. The role
of evaluation in bringing NLP to AAC: A case to consider.
In Filip T. Loncke, John Clibbens, Helen H. Arvidson, and
Lyle L. Lloyd, editors, AAC: New Directions in Research and
Practice, pages 105–122. Whurr Publishers, London.
Kathleen F. McCoy. 1997. Simple NLP techiques for expand-
ing telegraphic sentences. In Proc. of workshop on NLP for
Communication Aids, Madrid, July. ACL/EACL.
George A. Miller. 1995. WORDNET: a lexical database for
English. Commun. ACM, 38(11):39–41.
Roger Power and Donia Scott. 1998. Multilingual authoring
using feedback texts. In Proc. of COLING-ACL 98, Mon-
treal, Canada.
Pascal Vaillant. 1997. A semantic-based communication sys-
tem for dysphasic subjects. In Proc. of the 6th conference
on AI in Medicine Europe (AIME’97), Grenoble, France,
March.
108
