The MATE Annotation Workbench: User Requirements 
Jean Carletta and Amy Isard 
tICR(~, University of Edinb,,,'gh 
J. Carletta@ed. ac. uk, hmy. Isard~ed. ac. uk 
Abstract 
The M A'I'E project (Telemati('s L1~4-8370) aims to 
facilitate the re-use of spoken dialogue resour,'es an,l 
to foster etnl)iricai invesl, igation o1" (lialoguc by pro- 
viding a workl)ench which ,:an l)e used to atlnotal.c 
and explore the. relationships among diflb.rent stru(-- 
tures within a (lialoguc corl),m. This I)aper des~:ribe.s 
the hltended functionality of the workl)¢~nch by ref- 
eren('e to the needs of several tyl)as of l)rO.sl)ectiw~ 
users. It should be considered a position i)aper about 
what kin(l of technological supl)ort the user (:olnnni- 
nity requires. The workl)en(:h itself is scheduled to 
be relea.se(l in I)et'emhe.r 19!)9, with further dew:lop- 
nlent likely t)eyon(I that. 
1 Introduction 
Many people wish to annotate spoken (li~dogu(~ (:or-- 
I)ora with co(led information. This information can 
come in many forms for many different i)url)oses. 
Sorne ol' it will be linguisti(:; for instance, the anno- 
tation /nay tel)resent i)art-of-sl)eech information or 
syntactic stru(:tur(. ~, for us(. in language modelling. 
Some of it will be non-linguistic, representing infor- 
mation about the co,nmunicative situation or about 
events such a.s coughing or gesturing. If the dia- 
logue being armotated is being conducted with or 
mediated by technology, some of it may be sl)ecilic 
to that technology .. for instance, showing the re- 
suits of speech recognition in line with a human tran- 
scription of the same material in order to highlight 
wt~ere the dialogue model, broke down. Although 
some kinds of annotation, such a.s dialogue act in- 
formation, come up again and again, it is impossible 
to prejudge what kinds of annotations people who 
work with corpora will find usefld even when con- 
sidering a quite restricted set of coding purposes. 
Currently, corpus annotation is a very costly exer- 
cise not just because of the time which it takes coders 
to make the coding distinctions, but also because 
there is little etfeetive technological support for an- 
notation. The MATE Workbench \[9\] is intended to 
address the need for technology by providing a sin- 
gle interface to all of the basic functionalities which 
corpus annotators need, but with enough flexibility 
that different I)rojects can to i~,',)vi(I,. ,lill'~r,.nt kinds 
of annotation and that inforlnation ('ktn I{¢ ~ i)()rt.¢.,I 
between tint Workl)en('h anti otlwr al)l)lical.i~,n,~. Tlu" 
workbel|ch is a standalonct t()¢)l wril.l.¢.n ill .lay;t, s,~ il, 
will I)e able to ru. with(,uL r,.,'tmq)ilathm ,m Illany 
different I,lal.l'ornm including I'( Is, Mats, ;t{ul Ill,ix 
lna(:hines. 
2 Why This is Itard 
'l'he'single llJOSt iml),Jrtanf, obstacb, t.o ann*Jtai,i~)n 
tool design is the fat:t l, hat ~:,,rpus anlu,tat.i~ms ar,. 
not n~:essarily hierarchically arrange'd, m;.king it. 
difficult to design data stru,%iires and aJlo,ril, hlim 
which can hart(lie th,~ui ,~tliciently bul. whirl, ar~- als,, 
llexiIJh ~. enough to jznak,~ it, ¢:;my I,o inq~b~tn-nt w-w 
annotation st'henies. I"(,r i,stanc,., diah,g,n~- I,iKht 
useflnJly be annotated fi,r int.ent, i,mal M,l'llCl, llr*-, ill 
tonational strll¢:|,ur¢~, {llzd sylll, a¢:ti,: strtl¢:l, llr,-, ;tlSd, 
in I'a(:t, for the inost del.ailcd empirh:al w¢,rk, ;tll t,f 
these annotations Inight be n,~,~(h~d shnnall.an~:,mHly 
so that relatiollshij)s anlonl.~ th{~ln ¢:an I,: ,.xpb,rt-d. 
I~a(:h type ofstrllctilr¢: (:all r(~|'~r to SOlll¢" shar,,I I,;,s,: 
lev(:l of transcripti(m, whether (}rthogral~hi{: ~Jr ph,~ 
i|etic, with thnhlg hlforlnatiol{ wllt:r~ I,he sp¢:r:,:h sip, 
ual is available. Each Lype of strlu:ture IlJight in 
volvo several kinds of tags (for h,stanc% dial,,g0,,~ 
IIIOVei'{ alid gallle8, or l)arf,-ofslJCi:cJJ lal,ell¢cd l.,,k,:{m) 
phra.ses, and senten,:es) but can I)e ,:,msid,:r-d 1,,, I,,: 
broadly hierarchic:el. I low(-.ver, th,.-hi~:rar,:hies them 
8eJvc:s l\[lay or lnay not IJear any r¢:la, tionsJlilJ f.r~ ,:;t*:J{ 
other, a.s in l"igure I. 
Although it is possible to design I,,Jols whi,:h ,::m 
handle such coxnplex arrangements t)l" t;tgs, il, i,', 
Hlljtlh easier to (\]¢) SO when ;I, ¢:on¢:r*:f~*~ sr:t of t;tEh 
is envisioned than to ilnpJP.n, Jr..nL a Itj,:neri,: ~,,lill.i,m 
which can be adapted 1,,, otlJ*:r tag s,~l.s. 'lh,:r,~l;,r% 
existing tools, (reviewed in MA'FE deliv,:ral,l,: 3.1 
\[2\]), such a~s I)AT \[8\] t(md to support parti,:.lar ,:,,,1 
ing schemes, at best allowing tags to be r,:nan,,:d ,,r 
extra categories to be added to a tag set wil.hh~ a 
strict family resemblan(-(- (for exampl% nl:l \[5\]), a,d, 
in fax:t, few irnplement sets of' Lag,.i whi~:l, ,:r,,,~s i, 
this way. In ax\]dition, e.xisting t,,~ls tend t,, haw 
fixed methods for displaying information an,l lix,:,l 
11 
o 
"4 ~ iL__\] 
\ 
o 
4 
;> 
I-4 
f~ 
Figure 1: Overlapping Hierarchies 
12 
j 
actions for coding. Since tools are often built for 
particular coding exercises, the user interface is also 
fixed, with no way of altering it to suit personal 
preferences. Although these limitations are under- 
standable, they are not insurmountable. The MATE 
Workbench aims to improve on the current state-of- 
the-art by exploiting recent developments in corpus 
representation which allow a more flexible solution. 
3 A Step Forward, but not Magic 
The technological solution to the flexible represen- 
tation of overlapping tag hierarchies is to use the 
XML markup language \[11\]. XML can be used to 
describe any sort of coding, and the coding struc- 
ture can be described in a Document Type Defini- 
tion (DTD) which describes what tags are possible, 
and where they can occur. It is not possible for 
XML tags to overlap within a document, but one 
can create overlapping hierarchies of markup in the 
same corpus by using "hyperlinks", which serve not 
just to point to elements in a different hierarchy of 
markup but also to include them for structural pur- 
poses \[10\]. For example, if dialogue moves are made 
up of words; and sentences are made up of the same 
words, as in Figure 1, then both can be linked to the 
same copy of the words, providing a way of relating 
the two structures. An example XML representation 
of a short dialogue fragment is given in Figure 2. 
Of course, XML is designed to be machine- 
readable, and should rarely (if ever) be inspected 
directly. The MATE project will use "stylesheets" 
based on the XSL language \[12\], which allow one to 
express operations on an XML corpus which can be 
used to typeset it for the human reader, choosing 
both what annotations to make visible and how 
they should be displayed. Using MATE stylesheets, 
it will also be possible to specify the link between 
user actions and modifications to the XML which 
is being displayed, and thus implement coding 
interfaces. Stylesheets are written as a sequence of 
rules which match the input and give instructions 
about what to do for the output. A simple example 
of such a rule is: 
<msl:template match="($a: dummy);"> (I) 
<result> (2) 
<msl:text>A d.mmy.</msl:text> (3) 
</result> 
</msl:template> 
Line (1) introduces one stylesheet rule. There 
will usually be several rules in a stylesheet. The 
'match' attribute tells us which elements this rule 
will apply to. In this case, the rule will match 
<dummy> elements in the input document. In the 
MATE stylesheet language, matching is done using 
the MATE query language; query construction is 
supported in the workbench by means of a graph- 
ical user interface. Line (2) and following lines gives 
the template of the rule. This is a description of the 
elements which will be created in the output doc- 
ument. In this case <result> is a literal element, 
and the result of this rule will be that all <d~mmy> 
elements will be converted into 
<result>A dummy. </result> 
elements. 
Rules in a stylesheet are applied in order of oc- 
currence starting with the top level element in the 
document. There is a mechanism for top-down left- 
to-right traversal of the input document hierarchy, 
and a default rule for unmatched elements. 
Using XML and XSL allows a flexible solution to 
the problem of technological support, but this solu- 
tion is not magical. For instance, XML and XSL 
do not remove the need to understand how tags re- 
late to each other; they simply make it easier to 
specify a good machine-readable representation of 
complex tag relationships and to display these rela- 
tionships for the human reader. To see what benefits 
this technology will bring, it is necessary to analyse 
the capabilities and requirements of different types 
of potential workbench users separately. 
4 User Types 
Type 1: The Coder 
At least for sites involved in large scale coding exer- 
cises, data coders are typically the cheapest labour 
source available. They do not wish to know any- 
thing about how the coding interface works or even 
how different sets of tags relate to each other. Their 
needs are fairly simple: an intuitive coding interface 
so that they can concentrate on the code distinc- 
tions, documentation of how to use the interface - al- 
though, human nature being what it is, we find that 
no amount of written material will replace good ver- 
bal instruction - and the coding instructions nearby, 
preferably on-line. The MATE Workbench will en- 
courage fulfilment of these needs by providing guide- 
lines and slots for documentation within the coding 
modules which define the coding task, through the 
example coding schemes which come with with the 
workbench, and by providing sufficient coding ac- 
tions for a wide range of interface designs. Thus the 
main benefit to coders is simply a side effect of hav- 
ing good support for the implementation of coding 
schemes via well-defined coding modules. Another 
potential benefit for longer-term coders is the ability 
to reconfigure the user interface which controls inter- 
action with the workbench components to suit per- 
sonal preferences, for instance, by rearranging the 
menus and buttons. 
13 
Y 
</game> 
Speaker A moves f'de 
<move id="Am I" type="instruct" hre f="A_word_file#id(A l)..id(A6)'7>\[ / 
Speaker A words file 
<word id= "A 1" start=-"!" end="2">go</word> 
<noi id="A2" start="2" end="3" type="cough"/> 
<word id="A3" start="3" end="4">aro--</word> 
<word id="A4" start="4" end="5">above</word> 
<word id="AS" start="5" end="6">the</word> 
Both speakers games file 
<game type="instruct"> 
<move_sequence href=-"A_move_file#id(Am t )"/> 
<move_sequence href="B_move_file#id(Bm 1 )"/> 
Speaker B moves file | 
<move id="Bm I" type="acknowledge" href="B_word_file#id(B 1 )"/~ 
I <word id="A6" start=-"6" end="7">swamp</word> -..... 
.... ~., ~ <ip href="A_word_file#id(A 1 )"/> apeager A syntax tile 
\ \[ <ip href="A word_file#id(A3)..id(A6)"/> 
<sentence> X x 
<pos type="v" hmf="A_word_file#id(A 1)"/> x x 
<phrase type="p"> 
<pos type="prep" href="A word file#id(A4)"/> \ 
<phrase type="n"> 
<pos type="det" href="A_word_file#id(A5)"/> 
<pos type="n" href="A_word_file#id(A6)"/> 
</phrase> 
</phrase> 
</sentence> 
I Speaker B words file \[ 
<word id="B 1" start="8" end="9">okay</word> I 
\] Speaker B intonation file \[ 
\[ <ip href="B word_file#id(B 1)/>\] 
Speaker A disfluency file 
<disfluency type="s"> <reparandum> 
<dw href="A_word_file#id(A3) "/> </reparandum> 
<repair> <dw href="A_word_fite#id(A4) "/> 
</repair> 
</disfluency> 
Figure 2: XML File Structure 
Type 2: The Coding Consumer 
There are several possible types of consumers of ex- 
isting coded data. Some users might wish to check 
the relationships in the data for things which are sta- 
tistically aberrant, mining the data pre-theoretically 
for whatever stands out. Others might wish to ex- 
port some part of the data in order to train on the 
relationships which are present in it, in the hopes 
that some theoretically-motivated relationship will 
improve performance in, for instance, a spoken di- 
alogue system. Straight theoreticians might wish 
to inspect particular relationships in order to test 
specific research hypotheses. Whatever the reason 
for interest in the corpus, consumers are united in 
their need to ask questions of the corpus, looking 
for places which match a specific form, and to dis- 
play the results. Using the coded data well requires 
the mathematical capacity to understand the kind 
of structural information represented graphically in 
Figure 1, since otherwise the questions which the 
user asks will be meaningless. The more theoreti- 
cally motivated the user, the more important this 
requirement is. Of course, this is true for all work 
with complex tag sets, not just those represented 
within MATE. In addition, where new kinds of dis- 
play are required to match specific explorations of 
the data, new stylesheets will be required. The 
main benefits of the workbench for coding consumers 
are (a) the possibility of combining many different 
kinds of annotation on one data source, (b) a well- 
specified query language for exploring the relation- 
ships among the tags, and (c) methods for exporting 
different cuts on the data to other packages for fur- 
ther theoretical or statistical analysis. 
14 
Type 3: The Coding Developer 
Many people wish to design their own coding 
schemes, either to improve on the reliability or suit- 
ability of an existing scheme or in order to test a 
particular research question. These coding develop- 
ers may hire type 1 coders, but they are quite likely 
to do their coding themselves. This group of users 
has the hardest job. Designing a complete corpus 
requires the mathematical capacity not just for un- 
derstanding structures such as that represented in 
Figure 1, but also for constructing new ones and 
mapping these into the sorts of file structures repre- 
sented in Figure 2. This is true whether or not the 
corpus is represented in XML but is sometimes hid- 
den away as something which only the software de- 
veloper truly understands. One need not understand 
the relationships among all the tags on a corpus in 
order to install a new coding level, but one must at 
least be able to hook a new tag set into some part 
of the existing structure. Although this may seem 
onerous for the user, in reality most of the require- 
ments are the same as they were before; users who 
wish to do something new have to understand what 
it is they are trying to do. The only additional re- 
quirement is that instead of developing their own ad 
hoc data representations and mappings between the 
data, the screen, and user actions, users need to un- 
derstand how data is represented in XML and how 
to write stylesheets expressing these mappings. The 
benefit of XML and XSL for this activity is that they 
are likely to be better-structured and more flexible 
than anything coding designers will come up with 
for themselves, especially if they are not experienced 
software developers, and that there are many exist- 
ing tools for support, with more on the way. 
5 Our Solution: Pre-Implemented 
Schemes Plus Development Tools 
It is difficult to address all the user types at once, 
and since so little support currently exists for cod- 
ing designers, it will be impossible to get the facil- 
ities for them right the first time. The fact that 
there are many users who do not necessarily want 
to implement their own coding schemes leads us to 
a staged solution. The MATE Workbench will be 
distributed with coding tools and basic display ca- 
pabilities for a range of coding schemes at various 
levels of annotation from prosody and morphosyn- 
tax to dialogue acts and communication problems. 
These schemes are being chosen (or, in some cases, 
developed), based on an extensive review \[4\]. In ad- 
dition to simply being practical and reliable schemes 
to represent their levels of coding, they are being 
chosen to represent as wide a spread of coding types 
as possible in order to test the workbench design. 
Current coding tools have also been reviewed \[2\] in 
order to inform us both about good design for tools 
for the chosen schemes and about the range of ca- 
pabilities needed in the workbench. These schemes 
and the tools implemented for them can be used to 
allow users to develop a sense of the workbench ca- 
pabilities, and for users who do not wish to imple- 
ment new coding modules, they may be all that is 
required. The introduction of new schemes will be 
supported by tools for authoring XML corpora and 
stylesheets, and by the use of the existing scheme 
implementations as examples to modify. 
6 Basic Functionalities 
Keeping in mind our description of the basic user 
types, these are the basic functions which the MATE 
workbench will provide. 
6.1 Display 
Given a coded corpus, the display function will show 
on screen a human-readable version of the data. 
This display will be produced from the data using an 
XSL stylesheet, although there is no reason for the 
user to know that. Display options will include the 
size ~nd placement of windows, text colour, font, and 
size, and text layout such as lists and tables. The 
display may include a speech waveform if one is as- 
sociated with the dialogue, and user actions (such as 
clicking on an area of text) may be associated with 
further display information. 
Some users believe that our flexible approach to 
document typesetting means that conceptualising 
complex relationships within a data set will become 
easier - that is, that using the workbench will help 
to clarify their thinking about what to look for. In 
a sense, it will, because the ability to write special- 
ist display stylesheets will allow the user to create 
views which abstract away to the right sections of 
the data. However, stylesheets do not enforce the 
creation of a usable data display. In particular, it is 
just as possible to overload a display with too much 
information using this technique as using any other 
(and perhaps more tempting, because the stylesheets 
make this easier to do). The basic limitation on con- 
ceptualising data relationships is human, and not a 
product of coding technologies. 
6.2 Query 
Given a coded corpus, the query function will allow 
the user to construct a query which will match some 
part of the data set, and then will either extract 
that part (which can then either be exported or sent 
via a stylesheet for display) or count the number 
of matches, for performing frequency analyses. The 
MATE query language \[6, 7, 3\] contains constructs 
which allow the expression of either hierarchical or 
temporal relationships among a set of tags. This 
means that structural constraints Can be given nat- 
urally (such as asking for all response moves within 
• 15 
dialogues of a particular type, or verbs within rel- 
ative clauses), but that cross-hierarchy constraints 
can also be expressed (such as asking for all disflu- 
encies which occur during a particular type of into- 
national phrase). The MATE workbench includes a 
point-and-click query formulation support tool; al- 
ternatively, queries can be typed at a command line. 
6.3 Coding 
Coding tools will allow the user to add an annota- 
tion corresponding to a particular coding scheme. 
Coding interfaces will be specified by means of 
stylesheets. Typical coding actions might include 
using the mouse to specify a location, to sweep ar- 
eas of text, or to bring up a text window or menu 
by which tagging details can be entered. 
6.4 Transcription 
The transcription process should be highly individ- 
ualised for a particular corpus depending on how 
recordings were obtained and for what purposes 
the corpus has been collected. Getting the pro- 
cess wrong can add months and great expense to 
a project. Good transcription requires software, 
such as spelling checkers, which would be difficult 
to provide within a Workbench. In addition, even 
if good transcription tools were supported by' the 
workbench, many projects would not be able to 
use them because they contract out transcription 
work to secretarial agencies which are only willing 
to quote for the work based on the model of audio- 
typing using standard word processing packages. As 
a result, we would expect most users to wish to do 
their transcription elsewhere, using other software, 
and to transfer their transcripts into the workbench 
when they are complete. On the other hand, many 
users experimenting the system will wish to input 
small amounts of data so that they can test out the 
coding schemes and the workbench on new mate- 
rims. In the first instance, we intend to provide a 
very simple transcription facility which will suffice 
for this purpose but which one would not wish to 
use for large-scale transcription, at least not with- 
out a great deal of thought about the alternatives. 
6.5 Import 
The more existing corpora the workbench will work 
with, the more useful it will be when it is introduced. 
Unfortunately, current corpora are in a wide range 
of formats, many of which bear little relationship 
to XML. We intend to supply two conversion tools 
with the workbench which will handle conversion 
from BAS-Partitur and Entropic xwaves xlabel files 
into XML. Corpora produced to EAGLES recom- 
mendations \[1\] require minimal conversions. Users 
of other formats must support their own conversion 
processes, the software for which can be installed in 
local copies of the workbench. 
6.6 Export 
Just as users of existing corpora may wish to import 
data, they may wish to export codings into another 
format, for instance, so that they can apply exist- 
ing automatic annotation techniques to it. As with 
importation, possible export formats are too numer- 
ous and varied for us to implement converters for 
them all. Users who supply their own converters 
will again be able to install them into local copies 
of the workbench. Printing and postscript output 
will be available as a function closely allied to dis- 
play. Stylesheets can be used to produce other out- 
put formats which give specific views of the data; 
for instance, it is possible to use them to construct 
HTML or tabular information for input into spread- 
sheets or statistical software. We are still consider- 
ing how best to export information for visualisation 
of complete data sets, as required for data mining 
techniques. 
7 Tools for Developers 
Ther~ are two basic functions which will be re- 
quired for corpus and coding scheme developers: 
adding a new coding level, and creating or editing a 
stylesheet. For adding a new coding module, we in- 
tend to support good practice by creating a template 
for storing information about each type of coding 
which leaves space for describing working practice, 
who coded each file, exactly what form of the coding 
manual was used, and so on. We are still considering 
what sorts of tools will best facilitate the DTD edit- 
ing and stylesheet creation essential for new coding 
schemes, but here the workbench may itself be of ser- 
vice. XSL is an XML language, and current develop- 
ments in XML suggest that DTDs will soon be writ- 
ten in XML itself using "XML schemata" \[13, 14\], 
so that DTD and stylesheet editors could be writ- 
ten quickly using the workbench. Editors written in 
this way could abstract away from the syntactic de- 
tails which users find so difficult to deal with, leaving 
them to concentrate on the structure of the corpus 
additions. 
8 Timescale 
Obviously, this is an ambitious project, and the 
tools for developers will take some time to settle, 
not least because they require developments to the 
underlying markup languages. The workbench is 
scheduled to be released in December 1999. By this 
time we would expect to provide reasonable support 
for using the coding schemes which we have imple- 
mented. Note that this still leaves coding scheme 
and tool developers in a better situation than they 
have been previously, since, given th e willingness to 
learn XML and XSL, software developers will al- 
ready be able to use the workbench to implement 
16 
new coding schemes, coding tools, and display func- 
tions. This process should be faster than starting 
from scratch, especially with the examples imple- 
mented in the workbench. 
9 Acknowledgements 
This work was funded by the European Union 
as part of the MATE project (Telematics LE4- 
8370). We are grateful to project participants at 
our partner sites (NIS, Odense; CSELT, Turin; DFE, 
Barcelona; DFKI, Saarbriicken; IMS, 
Stuttgart; ILC, Pisa; and TID, Madrid) and to the 
MATE Advisory Panel for informing our thinking 
on these issues. 

References 
The EAGLES Consortium. EAGLES. 
http://www.ilc.pi.cnr.it/EAGLES/home.html. 

Amy Isard et al. MATE Deliverable 3.1. 
ht tp: //mate.nis.sdu.dk/about/deliverables.ht ml. 

Andreas Mengel et al. MATE Query Lan- 
guage, http://www.ims.unl-stuttgart.de/ men- 
gel/.MATE/Specs/quer.html. 

Marion Klein et al. MATE Deliverable 1.1. 
http://mate.nis.sdu.dk/about/deliverables.html. 

Giovanni Flammia. The Nb annotation tool. 
ht tp://www.sls.lcs.mit .edu/flammia/Nb.html. 

Andreas Mengel and Uli Heid. Query language for ac- 
cess to speech corpora. In Forum Acustieum, March 
1999. (ASA,EAA,DEGA). 

Andreas Mengel and Uli Held. A query language for 
research in phonetics. In ICPhS 99 (International 
Congress of Phonetic Sciences), August 1999. 
University of Rochester. The DAT system. 
ht tp://www.cs.rochester.edu/research/trains/ 
annotation/. 

The MATE Project. Mate webpages. 
http://mate.mip.ou.dk/. 

Henry Thompson and David McKelvie. Hyperlink se- 
mantics for standoff markup of read-only documents. 
In SGML Europe'97, May 1997'. 

The W3C. Extensible Markup Language. 
http://www.w3.org/TR/REC-xml. 

The W3C. Extensible Stylesheet Language. 
http://www.w3.org/TR/WD-xsl. 

The W3C. Schema for object-orlented XML. 
http://www.w3.org/TR/NOTF_,-SOX. 

The W3C. XML Schema requirements. 
ht tp://www.w3.org/TR/NOTF-r xml-schema-req. 
