An MAT Tool and Its Effectiveness 
Robert Frederking, Dean Grannes, Peter Cousseau, Sergei Nirenburg 
Carnegie Mellon University 
Center for Machine Translation 
Pittsburgh, PA 15213 
ABSTRACT 
Although automatic machine translation (MT) of unconstrained text 
is beyond the state of the art today, the need for increased translator 
productivity is urgent. The PANI3LOSS system addresses this dilemma 
by integrating MT with machine-aided translation (MAT). The main 
measure of progress in the development of the PANGLOSS system is 
a gradual increase in the level of automation. The current PANGLOSS 
MT system typically generates sub-sentence-length units of the target 
text. Any remaining gaps are treated by lexicon lookup. A mixture 
of these two kinds of components is presented to the user using the 
CMAT (Component Machine-Aided Translation) editor, which was 
designed to facilitate the transformation of this output into a high- 
quality text. An experiment evaluating the utility of the CMAT editor 
demonstrated its usefulness in this task, and provides useful guidance 
for further development. 
1. Introduction 
Fully automated machine translation of unconstrained texts is 
beyond the state of the art today. The need for mechanizing the 
translation process is, however, very urgent. It is desirable, 
therefore, to seek ways of both speeding up the process of 
translating texts and making it less expensive. In this paper 
we describe an environment that facilitates the integration 
of automatic machine translation (MT) and machine-aided 
translation (MAT). 
This environment, called the Translator's Workstation 
(TWS)\[5\], has been developed in the framework of the PAN- 
GLOSS machine translation project. 1 The main goal of this 
project is to develop a system that will, from the very begin- 
ning, produce high-quality output. This can only be attained 
currently by keeping the human being in the translation loop. 
The main measure of progress in the development of the Pan- 
gloss system is the gradual increase in the level of automation. 
PANGLOSS MARK I translates from Spanish into English, al- 
though additional source languages are planned. The analyzer 
used in this configuration is a version of the ULTRA Spanish 
analyzer from NMSU\[2\], while generation is carried out by 
the PENMAN generator from ISI\[4\]. The Translator's Work- 
1pANGLOSS is a joint project of the Center for Machine Translation at 
Carnegie Mellon University (CMU), the Computing Research Laboratory 
of New Mexico State University (NMSU), and the Information Sciences 
Institute of the University of Southern California (ISI). 
station provides the user interface and the integration plat- 
form. It is similar in spirit to systems such as the Translator's 
Workbench\[3\]. 
The processing in PANGLOSS goes as follows: 
1. an input passage is broken into sentences; 
2. a fully-automated translation of each full sentence is at- 
tempted; if it fails, then 
3. a fully-automated translation of smaller chunks of text is 
attempted (currently, these are noun phrases); 
. the material that does not get covered by noun phrases is 
treated in a "word-for-word" mode, whereby translation 
suggestions for each word (or phrase) are sought in the 
system's MT lexicons, an online bilingual dictionary, and 
a set of user-Supplied glossaries; 
. The resulting list of translated noun phrases and trans- 
lation suggestions for words and phrases is displayed in 
a special editor window, where the human user finalizes 
the translation. 
This entire process can be viewed as helping a human trans- 
lator, by doing parts of the job automatically and making the 
rest less time-consuming. 
We have designed and implemented an intelligent post-editing 
environment, the CMAT (Component Machine-Aided Transla- 
tion) editor. 
2. The User's View 
The CMAT editor allows the user to move, replace or delete 
output text elements, called components, with at most two 
mouse actions. The main user interface tool is a dynamically- 
changing popup menu available for each component. The 
ordering of alternate selections in the menus changes as the 
tool is used, to reflect the most recent user choices. 
Suppose the user selects a region of source text by highlight- 
ing it and submits it to be machine-translated. The result 
appears in a target window as a string of components, each 
196 
surrounded by "<<" and ">>" characters. 2 A mouse click 
anywhere within a single component brings up a CMAT menu 
for that component. In Figure 1, the user has clicked on the 
word "increase". A CMAT menu consists of three regions, 
each separated by a horizontal line. From top to bottom these 
are: 
• The LABEL region, which contains the word or phrase in 
the source text that produced this particular component) 
• The FUNCTION region, which contains the post-editing 
Move, Delete, Modify, and Finish functions. When the 
user selects Move, the component disappears, and the 
mouse pointer changes shape, indicating that a Move is 
in progress. The component is reinserted into the text at 
the nearest word break to the point where the user clicks 
the mouse again. Delete simply deletes the component. 
Modify pops up a window that allows the user to type 
in a new alternative (see next bullet). Finish removes 
the component markers, indicating that CMAT editing for 
this component is finished. 4 
• The ALTERNATIVE region contains alternative transla- 
tions of the source word or phrase. The source word or 
phrase is also present as an alternative, when available, 
as translators may wish to leave some source language 
words temporarily in the target text, and return to them 
later. Selecting one of the alternatives replaces the orig- 
inal selection for this component with the alternative, 
while the latter becomes an alternative in the alternative 
region. 
Figure 1: A typical CMAT menu 
An additional menu-base editing feature allows the user to 
change the morphology of a word with a single mouse ac- 
tion (Figure 2). This menu changes verb inflection or the 
determiner on a noun phrase, stripping any old morphological 
features before adding the new one. 
Using these popup menus, the user can move, replace, modify, 
or delete an output component with one or two mouse actions, 
rapidly turning the string of translated words and phrases into 
a coherent, high-quality target language text. Note that the 
user is not forced to use the CMAT editor at any particular time. 
Its use can be intermingled with other translation activities, 
according to the user's preferences. 
2If two components with different internal forms have the same string, it 
is followed by a colon and an integer. 
3Note that this information is not always available in noun phrase 
translation. 
4The user may also choose to wait and remove all the markers at once, 
for either a selected region or the whole buffer, using a selection in the TWS's 
main menu bar or a special keystroke. 
Figure 2: A typical CMAT morphology menu 
3. The CMAT Editor 
As part of the TW$, the CMAT editor is implemented in Com- 
mon LISP. It communicates through CLM (the Common 
LISP-Motif interface)J1\] to use Motif widgets inside of the 
X11 window system. 
The CMAT editor views text as a list of components. These 
197 
components are of three types: 
1. M\[T-generated strings. Phrases translated by the MT 
system are represented simply as the generated target 
language string, and are not further processed by CMAT. 
2. Glossary entries. Phrases not translated by the MT sys- 
tem, but found in the user glossaries, are each represented 
by a component list, a list containing the source string 
(source language phrase), the identifier : GLOSS, and a 
glossary entry list: a list of the possible target language 
phrases corresponding to the source language phrase. 
3. Dictionary entries. Words not covered by- either of the 
above are represented by a component list containing the 
source string, the identifier :M? and a target language 
string list: a list of the corresponding target language 
words as found in the MT system's lexicons; and finally 
the identifier : DICT and a dictionary entry list: a list 
of target language words found in the machine-readable 
dictionary. 
The CMAT editor uses a knowledge base and a working mem- 
ory. The knowledge base stores static information for a 
component's menu, while the working memory provides a 
mapping between the knowledge base and the components 
currently present in the target buffer. This separation is nec- 
essary because any given component generally occurs more 
than once in a given text, but there is only one menu associated 
with a particular component. 
Knowledge base structures are indexed by their component 
source strings. These structures contain four slots, one slot 
each for :GLOSS, :NT, and :DICT lists, plus a fourth slot 
containing the candidate list. This list is a union of the first 
three lists, with the elements' positions varying to reflect cur- 
rent estimates of their likelihood of being chosen by the user. 
Initially, the items from the target language string list appear 
first in the list and glossary entries appear second, since these 
items are more likely to be the correct translations of a source 
string in our domain. 
When a component list is passed to the CMAT editor to be 
displayed, the latter first checks to see if a structure for the 
component already exists in the knowledge base. If an entry 
does not exist, one is created. Then the first component is 
chosen from the candidate list and displayed with brackets 
in the editor window. In the working memory, a pointer to 
the knowledge base entry is stored, indexed by the displayed 
component. 
When the user clicks the mouse within a CMAT component, 
the system must use the actual character string as the index 
into the working memory, and from there get the index into 
the knowledge base. 5 The list of alternative translations for 
the component can then be obtained from the knowledge base 
structure. 
If a component is Moved in the editor window, nothing 
changes in the internal representation of the CMAT editor. 
When a component is Deleted, the pointer in the working 
memory is removed. If an alternative translation is chosen 
from the candidate list, the old component is replaced with a 
new component in the CMAT editor. The pointer in the work- 
ing memory is removed from its old location and stored under 
the new component. The new candidate is also moved to the 
front of the candidate list as the most likely candidate for fu- 
ture use. When a component is Modified, the new alternative 
entered by the user is stored in the knowledge base, and then 
treated as if it had just been chosen. 
When the component's markers are removed, either singly or 
en masse, the component's pointer in the working memory 
is removed, but the entry in the knowledge base remains. 
These are retained in order to provide a summary of the user's 
preferences, for the frequent case where future translations 
contain these components. This summary can be saved as a 
file, which can be loaded into the knowledge base in a later 
editing session, or analyzed by system developers. 
4. The Evaluation of the Tool 
In order to evaluate the effectiveness of this tool, we compared 
editing with the CMAT editor versus editing with just the basic 
Emacs-like text editor in which it is embedded. We conducted 
two experiments comparing CMAT and non-CMAT editing effi- 
ciency, one using monolinguals and one using translators. 
4.1. Experiment I 
Method. The monolingual task was to take the output of the 
MT system and, using as reference an English translation that 
was previously produced manually, produce the "same" text 
using either the CMAT editor or the regular editor. The time re- 
quired for each text-editing session was recorded. Keystrokes 
and mouse actions were automatically counted by the inter- 
face. 
As test texts, we used two of the texts from the 1992 DARPA 
MT evaluation. To shorten total experiment time and provide 
a reasonable number of sample texts, we broke each text 
into two halves of roughly equal size, at paragraph breaks, 
resulting in four text samples. 
Two subjects were presented with the samples in the same 
order. Their use of the CMAT or the plain Emacs editor on 
SThis is due to details of the CLM interface, and is the reason for mark- 
ing identical components that have different internal data structures with a 
colon and an integer: otherwise there would be no way to locate the correct 
associated data structure. 
198 
different samples was arranged to provide as much variation 
as possible in practice effects and acclimatization, so that 
these could be cancelled out during analysis. A few days 
later, subjects repeated the procedure, reversing the use or 
non-use of the CMAT editor. Since practice effects should be 
more uniform in a simple editing task than in translation (the 
task is much less intellectually challenging), we felt that texts 
could be reused if practice effects are taken into account in 
analysis. 
Subjects were instructed to produce a "close paraphrase" of 
the example translation, since any two translators will pro- 
duce slightly different correct translations of the same text. 
Subjects were also instructed not to use the CMAT Modify 
function, since it causes the editor to learn during use, making 
analysis even harder. 
Analysis. Given the above ordering of test runs, one can bal- 
ance practice effects, subject differences, and text differences 
simply by normalizing the total editing times for a subject on 
each run through the texts. That is, if we divide the editing 
time for each text by the total time for the entire set of texts in 
the given run, the variation between normalized editing times 
between subjects should reflect variations in the efficiency of 
editing. For example, in Figure 3, we see that for Session 1, 
Subject I spent a greater fraction of time using CMAT (0.2413) 
than Subject 2 spent editing it in a regular editor (0.2198), 
while for Session 2, the fraction of total time was the same 
with either editor. 
0.4 
0.3 
0.2 
quite helpful. It could be the case that the CMAT makes the 
job easier without making it faster, but we had the definite 
impression that it makes translating faster as well as easier. 
We therefore investigated further. 
Normalized keystroke and mouse action counts are shown in 
Figures 4 and 5. Here we see that while the CMAT editing 
sessions had 1/2 to 1/3 the number of keystrokes, they had 
between 2 and 9 times as many mouse operations. This is 
significant, since mouse actions are slower than keystrokes. 
J • CMAT 0.5 
0.4 ~~ 
0.3 
0.2 
0.1 
0.0 
1 2 3 4 5 6 7 8 
Editing session 
Figure 4: Normalized keystroke counts 
0.5 • CMAT 
0.4 
0.1 0.3 
0.0 
1 2 3 4 5 6 7 8 
Editing session 
Figure 3: Normalized editing times 
From comparing these normalized times, it appears that the 
CMAT actually slows subjects down. This contradicts the 
universal subjective impression of all CMAT users that it is 
0.2 
0.1 
0.0 
1 2 3 4 5 6 7 8 
Editing session 
Figure 5: Normalized mouse action counts 
199 

tional information available to translators, and that measure 
any trade-off between quality and speed of translation. 
In the second experiment, the normalized total-edit time ratios 
between the two texts for Subject 5 were essentially identical 
to the rough draft ratios, indicating that this ratio is indeed a 
good indicator of the relative difficulty of the two passages. 
It is interesting to note that Subject 4, whose data point had to 
be thrown out because his CMAT times were twice the length 
of his non-CMAT times, corresponds closely to the level of 
familiarity our translators had with the CMAT editor in the first 
MT evaluation in 1992. An important part of our preparation 
for the 1993 MT evaluation will be training the test subjects 
in the most efficient use of our tools. 
5. Conclusion and Future Work 
The CMAT editor, in conjunction with often fragmentary MT 
and word-for-word translation, allows the translator to pro- 
duce high-quality translations more quickly and easily than 
the simple combination of a text editor and an online dictio- 
nary. It will remain a crucial module in PANGLOSS until the 
MT system reaches the point of translating full sentences on 
a regular basis. 
These experiments provide initial evidence that the CMAT ed- 
itor is indeed effective, and have been very useful in pointing 
out areas for improvement: 
• The current CMAT design requires the use of the mouse. 
Since mouse actions are often slower than keystrokes, 
we will provide keybindings for all CMAT commands, 
including viewing and selecting alternative translations. 
This should not be technically difficult. 
• The users need to be taught the most effective strategies 
for using the CMAT, such as only using the mouse if they 
are fast with it, and generally not using the CMAT after 
their rough draft is finished. 
• Currently the CMAT menu often does not contain the 
correct answer, due to the low-quality of the online dic- 
tionary. This dictionary is currently being replaced, and 
we expect the coverage to be much improved for the next 
MT evaluation. 
3. Kugler, M., G. Heyer, R. Kese, B. yon Kleist-Retzow, G. 
Winkelmann, 1991. The Translator's Workbench: An Envi- 
ronment for Multi-Lingual Text Processing and Translation. In 
Proceedings of MT Summit III, Washington, DC. 
4. Mann, W., 1983. An Overview of the Penman Text Generation 
System. In Proceedings of the Third AAAI Conference (261- 
265). Also available as USC/information Sciences Institute 
Research Report RR-83-114. 
5. Nirenburg, S., P. Shell, A. Cohen, P. Cousseau, D. Grannes, 
C. McNeilly, 1992. Multi-purpose Development and Opera- 
tion Environments for Natural Language Applications, In Pro- 
ceedings of the 3rd Conference on Applied Natural Language 
Processing (ANLP-92), Trento, Italy. 
References 
1. B~icker, A., C. Beilken, T. Berlage, A. Genau, M. Spenke, 1992. 
CLM- A Language Binding for Common Lisp and OSF/Motif: 
User Guide and Reference Manual, Version 2.1, Technical re- 
port, German National Research Center for Computer Science. 
2. Farwell, D., ¥. Wilks, 1990. ULTRA: a Multi-lingual Machine 
Translator. Memoranda in Computing and Cognitive Science 
MCCS-90-202, Computing Research Laboratory, New Mexico 
State University, Las Cruces, NM, USA. 
201 
