File Information
File: 05-lr/acl_arc_1_sum/cleansed_text/xml_by_section/metho/96/c96-2156_metho.xml
Size: 23,875 bytes
Last Modified: 2025-10-06 14:14:11
<?xml version="1.0" standalone="yes"?> <Paper uid="C96-2156"> <Title>Multi-Modal-Method: A Design Method for Building Multi-Modal Systems</Title> <Section position="3" start_page="0" end_page="925" type="metho"> <SectionTitle> 2 Multi-modal Processing vs Event-driven Programming </SectionTitle> <Paragraph position="0"> M uld-nmda,l interfa,ce is one oft,he a,d,,'a.nced ilHer Face beyond present 6', \[! Is. lJreseul, (;{!ls axe in tegra, l, ion of objecl orienled comImlin 9 and cvc~ldriv en prog.rav~ m in9.</Paragraph> <Paragraph position="1"> One of I,he nlost iH\]por(~a,v,t i.now~.t, io.s in computer i>rogi:aniiiiing during the past <leca.de ha.s been the developnle, ul, of &quot;Object oriented&quot; (:ouq)utiug. Viewing sofl\[,wa.re ('OlH\[)onellt, s as if they are _r>hysicaJ objects, cha.r~cterizaJJe via.</Paragraph> <Paragraph position="2"> class/sut>class rela, l, ions based on sinq)le lea Jutes and/el: how \[llnct, ions of t;he objecl.s differ, is a power\['ul inel, a.phor. Tile l)i:o~ra.iiililei&quot; (:ii.ii ilow hiia.gine coluplex sys/,eills a.s l>uill; up of these silii pler ol)jeci,s, liiiicii a.s a. c, hiht builds a. la.l'ge si.i'/IC. tlire oul, o\[&quot; sillil)le, l>uihihig 1)locks or a.li a.rchilecl ;t..i'l'{i.ligjes ;~.. Puncl, iona.1, yei, aesl.hel, ica.lly a.l)i)ea.iing edifice \['rolii COliipoiielll,s such a,s woodeii \]>+~illliS a.iid ineta.l gh'ders. 'l\]iinkhig o\[&quot; the coiilpiiter screeN, the whidows Oil l, ha.l, soi'eeli, itnd eveli the bits in those whidows a.s shnple objecls coiiil)osed togeLller int,o a, i)owerletil editor has l)eell a.I\] <;x l, reniely coinpeliing vision for iril,er\['~lx:e designers.</Paragraph> <Paragraph position="3"> In f~tcl,, ot:,jecl,-oi:ien/.ed progra, innlhig ha.s t)e.coine a, COl'llersl, Olle of inLei:fa,ce design, mid I,\]le doillililt.Ill, ineta.phor il\] inter\['a.ce \[,rogra.iiil ni lig systeiiis. Ilowever, SOlile r(\](;el\]l, systelllS lla.ve gOlie beyolid ot>jeci;s for (lea.ling with inierfa, ce develop luenl;. This is t)eca.use, especia.lly in willdow-based systelilS> SOllie tyi)e,s of inl, er\['a.ce COllipOileiils (\]o iiol; fit; well wheii viewed as &quot;objccls.&quot; 'l'hinldiig el&quot; the illOtiSe ,'l.s it physicaJ eiltity for the l)rogr;illl Iller to lise iliitkes perfect sense, btlt 'viewing a, &quot;iiiollse, (:lick;' as a, li object seeliiS less conipelling. ,qinlila.rly, other actions, such as sketchiug wiili a light I)ell, sca.iltlilig 0, (\]ocuillent, or sl)ea.king a.</Paragraph> <Paragraph position="4"> seliteilce Ca, llilOl, t)e 1;hough of&quot; a.s physica.l en/,itiesj but i:a.ther iiltlSt be viewed as &quot;'events&quot; which occur Oil a.li objeci;. Thus> \['or exa.ilil)le , tools on \'Viii dows like Visua.l Bask ha.re been lea.niug /.owa.rd a, progra.iiiinhig inelLhodo\]ogy i, ha.I, a,llows nol, olliy ol>je<:ts, but also event,-l>ased progra+nuning.</Paragraph> <Paragraph position="5"> It, is our contention t3m/{ while evenC-ba.sed pro gramnfing is a. step ill the right direction, it. does not go fa.r enough. In pa.rticular, we <:laitu that it is the order of events in a. sequettce that is critical.</Paragraph> <Paragraph position="6"> This is especially true in a nmld-moda.l iuterfaee where eveut, s may l>e coming from a set of different; conqmtationa+l device.s, each runifing separately.</Paragraph> <Paragraph position="7"> In such an interface, a. mouse click, a spoken uttera.nce, a drawing with a. light pets, and some typed comttta.nds mat have t.o be integra.ted into a single inl>ut. The ordering of the input events is clearly a critical fitctor in understanding the meaning of such inputs, aim &quot;parsing&quot; such astring requires a.</Paragraph> <Paragraph position="8"> more principled approach than simply expecting an application t,o handh~ the plethora of <tiverse inlJuts its all ++heir forths.</Paragraph> <Paragraph position="9"> The major purl>ose oF this paper is to define a. frameworl( and <\]esign methodology for a cosnpul.illg model which can inl.erl>reC a set; of events, particuhu'ly iu the area. of nmlti-mo(hd interfa.ce design, lit the next section we describe this idea.</Paragraph> <Paragraph position="10"> more fully and develol> a simple example.</Paragraph> </Section> <Section position="4" start_page="925" end_page="928" type="metho"> <SectionTitle> 3 Understanding Event Streams </SectionTitle> <Paragraph position="0"> N iga.y and Cou taz (1993) divided uutlCi-modal sysl.etllS iuto four categories. They are defined by two independent features; fllsior~ and 'use of r, odalily.</Paragraph> <Paragraph position="1"> &quot;l,'usion&quot; covers the possil)le combination of different types of data. the a.l>sence of fusion is ca.lled &quot;indel)eudent&quot; whereas the l)resence is referred to as &quot;coml)il\]ed&quot;. aUse of modaliCies&quot; expresses the tempora.l availability of multiple ntoda.lities. This dimension covers the a.bseuce or presence of Imam.1 lelism at the user int~erface. &quot;Parallel lisa;' allows the user to employ multiple modalities sintulta+ neously. &quot;~'Sequential&quot; forces the user I,o use the modalil, ies one after another. In this paper, we (lea\] with Cite :'synergist+it&quot; category, the most. difficult among t, he Corn' categories.</Paragraph> <Paragraph position="2"> A simple example shows how difficult it, is to understa.nd synergistic user expressions. Consider the example of a chiht who is using a nnlltinmdia encyclopedia system whicls provides a, mix of speech recognition (and language processing) and a. mouse. The chiht states &quot;Ca.n this, do this,&quot; pointing at a picture on the screen and clicking the mouse during the first %his&quot; and then choosing all itmn front a lllelltl during the second. The syslenl must realize that the first, point is, say, a. pict.ur<2 of a particular animal a.ud the second is the tttetm item &quot;fly.&quot; Somewhere, the system itlusl, creale a.Jl internal representation of this query that conforms to some data (or knowledge) base query la.nguage.</Paragraph> <Paragraph position="3"> In tile object-.orienCed metaphor, some sort of centraJ application object is in cha.rge, and must send messages to the screeu, the mouse, and the voice system asldng for input upon activation. This sys= tern then synthesizes that information and produces a query such as &quot;\[QU l,\]l{Y: Func-of <Object l)inosaur-bitmap-7><:ntenu item I,'I,V >\]&quot; which it is progra.mnmd to answer.</Paragraph> <Paragraph position="4"> Note, however, that as the central system ol>ject is in cha.rge, it, must send messages (or otll erwise cosltact) the wu:ious modalities of intera.ctiou to be aware of tlte. possibility of input. This can be arbitrarily hard, especially as we consider that the number of utodalities wi\]l keep growing as user interface technology design comin ues. Even R)r this simple example the same query can be a.sked many ways: the child could speak &quot;can a. ptera.smdon fly?&quot;; could choose from the menu aquery-I)utcCion,&quot; point at the dinosaur, and then mouse &quot;fly&quot;; could type t.o a. conmm.ml lilac &quot;query:flmction PT1) Fly&quot;; or any other COllll>iim.tion of these capal>ilities. The central ol<}ecl coordinating all these modalities IIItlSt sm,l ap propriate messages at approl)riate times to ea.cll <)r the drivers of the wu:ious devices, and theu iimsl syuthesize the answers that are received.</Paragraph> <Paragraph position="5"> Unfortunately, the situation is made even luore conlplex by the fact that the system ca.nnot extt'acC a\]\] inputs alsd colnbine them in sonle sin@e ltla.Slller. The sequence in which the inputs are. received can be critical tha.t is, the %vent stream&quot; must be aua.lyzed as a.n ordered set of events which determine tile interaction. If the chi\]<l says ~'ls this (points a.t elel>hant) bigger than this (points aC pteranodon)?&quot; then the system must recog nize in which order tile poiuCs and the anaphoric references occur. Simply recognizing /he query concerning the elephant and pteranodon is uot enough; we must understand (and process) theni in the correct order.</Paragraph> <Paragraph position="6"> The computatiollal met.aphor we prefer is nol Chat+ of objects, but rather that of l>rocessing the stream of events in a. gra.nuna.tical mamler. Thus, instead of having a central object initiating sollm sort of message passing, we view each of the indi vidual interaction techniques a.s producing reports concerning the events which occur and the t.imitlg of these events (e.g., the mouse in the aJ>ow', s<:e nario will simply report &quot;<Mouse-Click :Xpos 300 :Ypos 455 :start 2700 :end 273.5>.&quot;) Using the example, :'can this do this&quot;, we describe \]tow sophisticate synergistic iuputs should be processed more precisely. Figure 2 shows four which is ilq>ut lotlg a.t&quot;ter the, first mouse input (ff)r example, I tnimtt:e a.fter). I, this case, the seeottd inouse i,\[>ut is ig.ored l>eca.use of l imeo+lt I>y t.he syst, eln. OI\]ly I,he first mouse inlmt is iuterpreted.</Paragraph> <Paragraph position="7"> :l'hereff)re, ea.se d ix l>roc<'sse(\] tim sa, me a+s case 2. A tluml>el: of tnulti.-l,K)da.1 int, erfa.ces ha.ve bee,, de.velol)ed. There axe cevta.inly severa.l a.pplica.tion fieh\]s in which nmlti ||loda.I systems a.re a.l>l>liea. ble. The+' include: design and editiug, pt:esenta.t icm, infi:)rt,m.l:ion rett:ieva\], and educe.lion. Step 2: Mode. and media selection The tmHtber ;u,d type of Jl,o(les a,ttd media.</Paragraph> <Paragraph position="8"> sllottl(l be deternfined. Gettera\]\]y, niode arid lue,:lie. do not. ha.re a. otle-t.O-Olle eorresl>Olldell(:e. For exatnple, a.lt.hottgh speech inl>ttt a.H,:t keyl>oa.rd in put use+ different media+, they a.re t.vea.ted a.s the sa.nm mode beta.use they a.re used and interpreted idemica, lly.</Paragraph> <Paragraph position="9"> Step 3: Corpus collection The eorl)us of multi-i~mda.\] expressi<ms to tlm a,pplica, tio. is collected. This process is the su.,+ as that, for tm.latra,l la, ngua,ge processing.</Paragraph> <Paragraph position="10"> Step 4: Corpus analysis The collected corl>tls is mta.lyzed, l,:a.ch expres sion iu the COrl>US shouht I>e a, na.lyzed I>a.sed ou L\]le R)Howiug cl:il.eria..</Paragraph> <Paragraph position="11"> Economy: l)oes the exl>ression save a, tlse, r's la.bor? I,\]aeh expressiotl is exa.,t,iued as to whether it; ca.n sa.ve a. use, t:'s fa.bor v:hett t rmm ferring his/her iut, entio, to the a.pplica.tioa system. For example, in .t+ piet+tu'e <\]ra+',ving tool, if a. user is a.llowed t<) point a.1. a. si>e eific ol>jeet while sa,ying %Jelete&quot;, he/she ca.tl sa.ve ht.bor, be(:a.use he/she does ,of ha:.+'e Io cha, llge, t.hc IHouse positiotl frol~l the CIILIIVIhS to it. lllellll item a.t t.he lllelltPS |)ill' a.rea., a+lld lille.ill Fl:Olll the tDelltl })lt.l' it.Fee, to (:he (:3.+llV&'-;. Plausit:,ility: 1,:a.eh exl>ression is exandued as to whether: it. is likely to be used in a+ i'ea.1 appliea.tiot|. As desct'ibed t+etow, writing gra.tlmm.rs for tuulti--tttoda.1 interfaces requires mu,::h more effort tha.tt f<::,r single tt~<::,da.\] iJl teJq+'aees. O.ly frequently used ex\[>ressi<)us should be selected ca.refi|lly. The sp<,.ech mode is be/.l.er I&quot;or selecting a.n itetn anmllg n ta.rge mltu\]>er of ca, ndida, tes, such as choos i,g a. {'it.)&quot; ua.me a.lllong all cities in the I.!S:\. Ou the of.her ha+ml, a..w, uu iHterfa.(:e is bet t.er I&quot;or sefe,::l.il~g one a.luong a+ small tmtIll>,:'.r o f e a,u did a,t,es.</Paragraph> <Paragraph position="12"> The set. ,:)r the select;ed expressiorts t)ecOllles tim seed for the specifiea.tion of the desigm:d t,,ult.i |t|oda.1 sysl.etu.</Paragraph> <Paragraph position="13"> Step 5: Specification Design The diflieulty level oft.he interface (tesig,, should be (lel.er,l,hted />ased cm the a.ualysis of' sele(:le(l corpus e+xpressio,ls. Thet:e a.re five dil-liculty levels of multi-modal input exi>ressions (Ta.b\]e 1): Level 1: Single mode input: l,',veu it, a. tl,ulti ,node.1 syst, em, users oR,e ||wa,nt t,o express /heir i,Ientions with si,gle modal expressions. For example, I:>oiul, ing a,t a,u existing object, thee select hlg &quot;delete;' from the menu.</Paragraph> <Paragraph position="14"> Lewd 2: All mode inputs express identical contents: I.',a.eh tt,ode input, expresses a.n i(hm t.ical cc:,ntelfl.. I&quot;or exa.utl>le, poitH;ing a+/; a,n exisl.~ ing ot>ject, then selecting &quot;delete&quot; from the lllel\]l+l, while saying &quot;delete the reeta ugle&quot;.</Paragraph> <Paragraph position="15"> Level 3: A eoinbination of incomifle, te mode inputs eomph;lnent each other: Each t,lod<~ input does not. expresses tf|e <::otHelltS }>y itself.</Paragraph> <Paragraph position="16"> Each mode input complements other mode inputs; thus they express a. single content. For exampie, pointing a.t a.n existing object, while saying &quot;'delete&quot;.</Paragraph> <Paragraph position="17"> Level 4: Each mode input is contradictory: The contents generated from independent lnode inputs axe contra.dictory one ~nother. For exampie, sa.ying &quot;delete the circle&quot;, while pointing at. a. rectangle object which hides the specified circle object on the screen. Contra.dictions a.re often solved by context a.na.lysis.</Paragraph> <Paragraph position="18"> Level 5: A COlnbination of mode inputs still lacks something: The contents genera.ted from the combination of the interpretations genera.ted fl'om individua.l mode inputs a.re insufficient. For example, sa.ying &quot;move it. here&quot;, while pointing a.t a. specific point. The point should be unified with &quot;here&quot;, a.nd a.n object specified by&quot; &quot;it&quot; should be interpreted as the last referred object. This type of interpreta.tion requires of context a.na.lysis.</Paragraph> <Paragraph position="19"> It becomes more dimcult to interpret expressions as the level increases. Especia.lly, since level's 4 a.nd 5 require tight iutegra.tion with context a.na.lysis, interfa.ce designers should consider whether the applica.tion users really need these levels or not.</Paragraph> <Paragraph position="20"> Step 6: Architecture Design Any multi-moda.l system can ha.re a. multi agent a.rchitecture beta.use ea.ch mode processing is ea.sily ma.pped to a.n independent a.gent. There are two extreme types of architecture which ma.na.ge the agents. One is bh~ckboard a.rchitecture where a.gen ts excha.nge ilfforma.tion using a shared men,ory ca.lled a. bla.ckboa.rd. 'l'he a.rchitecture fits multi-moda.1 systems whose multi-modM expres sions a.re sophistica.ted a.nd integra.ted with context. a.na.lyses. The other is subsumption a.rchitecture where ea.ch a.gent a.cts ra.ther independently. ln forma.tion excha.nge pa.ths between a.gents a.re limited. The a.rchit.ecture fits multi-lnodM systems whose multi-moda.l expressions a.re simple a.nd slereotyped. Ma.ny a.ctuaJ multi-modaJ system a.rchitectures are combina.tions of these extrelne a.rchitectures.</Paragraph> <Paragraph position="21"> Step 7: Grammar rule writiug Each selected mu\]ti-moda\] expression is defined by the corresponding gra.mma.r rule to interpret it.</Paragraph> <Paragraph position="22"> The gra.mma.tica.l ffa,lnework for the mult.i-moda.l expressiou should ha.re the following functiona.lities: null (1) Modes should be interpreted equally and indei)(mdently. If ea.ch mode is trea.ted in the same ma.nner as tha.t of a na.tura.l la.ngua.ge mode, synta.x a.nd semantics of inputs of ea.ch mode are defined with gramlna.r fornmla.tion.</Paragraph> <Paragraph position="23"> Thus, complex multi-modM expressions can be defined declara.tively a.nd more easily.</Paragraph> <Paragraph position="24"> (2) Mode interpretations shouhl be referred to one another. Inputs of ea.ch mode shouhl be interpreted independently. However, the interpretation of such inputs should be referred to by other mode interpretations. There ~re a.mbiguities which a.re solved only by integrating pa.rtiM interpreta.tions of rehtted modes. For example, if a. user sta.tes &quot;this recta.ngle&quot;, pointing at a. different type of object overlapping the recta.ugle object, the a.m biguity of the object pointing nmst be solved by comparing the two mode interpreta.tions.</Paragraph> <Paragraph position="25"> (3) Mode interpretation should handle temporal inforlnation. Tempora.l ilfformat.ion of inputs, such as input a.rriva.1 time a.nd the interva.1 between two inputs, is importa.nt in interpretitlg multi-rood a.1 iuputs.</Paragraph> <Paragraph position="26"> Multi-Moda.l 1)CG (MM-I)CG) supports these functiona.lities. MM-DCG is a superset of 1)(7(\[; (Pereh'a. 1980); everything possible in 1)CG is a.lso possible in MM-I)CG. MM-I)CG has two ma~jor extensions: t. MMq)CG ca.n receive ,~rbitra.ry llUlllbers Of input strea,ms, while 1)CG ca,n receive only one. A single gnunm~r rule in MM I)C(; cain allow the coexistence of gra.nnna.tica.l ca.t egories, thus Mlowing for their iutegra.tiou.</Paragraph> <Paragraph position="27"> 2. hi MM-1)CG, ea.ch individual piece of input da.ta, is required to a.tta.ch the beginning time a.nd t\]Ie end time as its time sta.mp. Using the time sta.mp, MM-I)CG a.utom~tica\]ly ca.1 null file cych: iLera.tes. When the system rea.ches the lll;t.t;tJre sl;a..ge,, the syst:ctn is released to end users.</Paragraph> </Section> <Section position="5" start_page="928" end_page="929" type="metho"> <SectionTitle> 5 Case Study </SectionTitle> <Paragraph position="0"> This section describes the design process of a.</Paragraph> <Paragraph position="1"> mult.i-nmda.l drawing 1,ool along with tim ttmlt.imodal-nlethod. The following is tile trace of the (lesigtt process.</Paragraph> <Paragraph position="2"> Step l: Task soh;c+tioIl Since there has I)ee!l .,dgnifica.nt research oil develot>ing mult, i .,odal drawing tool (l\[iyoshi 199d; Niga,y 1993; \,% 1993; I~,ellik 1993), the application fiehl is l>rolnising. Step 2: Mode. and media seJection In tl,is exl>criu,ent , we R)<'+use<l on only input, t.odes. In put. modes include speech, keyboard a.d mouse inputs. These input nJodes a.re synergistic. Oub.</Paragraph> <Paragraph position="3"> l>ut modes include l>ictures and text, but outputs axe llOt synergistic.</Paragraph> <Paragraph position="4"> Step 3: Corpus collect'oil We co\]le<:l.ed about, two humlred nmlti-ttnoda.1 exln'essions front pol,ent, iaJ users a+s it,st.r,Jctiol~s for t.he i~lulti:moda.1 dra.wing tool. The users had exl>erience wit+h using cxisti.g dra:e,,ing t+ools.</Paragraph> <Paragraph position="5"> Step 4: Corpus analysis The following are some of tile result.s of l~he a.nalysis of the. collected corpus.</Paragraph> <Paragraph position="6"> * Users want. to use various ,nixed modes ac cording to the sil;ua.tions dmy are dealing with.</Paragraph> <Paragraph position="7"> +D Users wa.tlt. Lo use abridged expressions, whi<:h causes integration of multi-modal interpret.at.ion and cont.ext analysis.</Paragraph> <Paragraph position="8"> * Users wa.ttt I.o handle exisI;ing objects a.s a set.. I,'or example, &quot;Cha+,ge+ tile col<),' of all circles. ,., Users want. \[.o ha.ndle exist.trig objects whi<:h are not shown on t.he display. For example, asking &quot;'how many re<:tangles a.re hid<letl ()tit, of the canvasT'.</Paragraph> <Paragraph position="9"> + Users wa.nt t.o use+ l.he tuouse a.mbiguously. For exa.nq)le, saying &quot;l)e\]ete this circle&quot;, while I>oinl.ittg a.t a point, a~u:ay fl'o,n but near the circle. Such ambiguous pointing can be <:orre<-t\]y interprete.d only whett multi-,imdal expressio, is a\]lowe, d.</Paragraph> <Paragraph position="10"> Step 5: Specification Design The a.ua.lysis taught us tllaJ. ~,ulti-,,,odal drawhtg tools should support level% d and 5 (the most dill, cult levels) to meet ordinary users: rcquirenm.l.s. The sped\[,catkins were determiued based on these require ll\]ell\[;s.</Paragraph> <Paragraph position="11"> Step 6: Architectm'e Design Since tlw, re= quired specification is tim most, difl\]ctJt synergy lewJ, (.he a+rchit.ect.ure is blackl>oa.rd a.rc\],itecl;tJrc where, ea.ch agent can ex<'ha.t,ge infor,~m.t.ion in va.rying ways.</Paragraph> <Paragraph position="12"> Stell. 7': Gr3.llllllal' rule. writing After tlu~ a.nalysis, about, forty expressions were selected, a.ud va.ria.tions of ea.ch selected expression were a.lso genera.ted a.nd a.dded. (-~rammat rules were. de.fined corresponditlg to each mull:i, tnoda.1 expres sion. Figure 4 shows a part <)f the grammar rules written in MM-I)CG. The rules define how to in l.erl)reI, a.n hupera.i.ive sentence like &quot;l)elete this circle&quot; wil.h va.riet.ies of expressions. It allows the spokeu uttera.nce mode(speech sl.rea.ul), l.he tylm it\] ,node (keyl>oa.rd strem,), a.ud the mouse l>oilll ittg mode 0hOUSe stxeam), l{ules iu the level I sec 1.ion define single tnoda.\] e+xprcssio,m. In tim level 2 section, whethe, r di\[l'erent, mode hq>uts express identica.l cotg.ents is examined. The combina.tion of the verb_by_multimodal/1 clause a+ud the secolul object/1 clause+ is m\] exami>le of the level 3 exl>ressions, lit the le.vel 4 sect.iou, select_right_meaning/3 enclosed inside curly brackets { and } is a. Prolog predicate which detertnines the correct mea.lfit,g using cot,text analysis whet, (lilTere.,,t tuo(le iulmtS genera.re contradictory meanings. Such a. l>redica+i.e is defined it\] a task-specific ltlal,tleF. Ill the level 5 section, find_appropriate_termt/2 enclosed inside curly brackei.s { a,d } is a. l'rolog pred,.</Paragraph> <Paragraph position="13"> ca.re which finds a.u a.ppropria.te term ttshkg (:ou texl. analysis whe. the cond>inat.io,t of gcneraJed tuea.ui,g of all modes still lacks htrort\]m.tion. Su<:}l a predica.Le is also defined it, a t.a.sk-spe+cific lira, .er. A trivial heuristic rule exmnple is &quot;to use the ,|tost recently a.ppea.red t.erm&quot;.</Paragraph> <Paragraph position="14"> C, ra.J,n,ar writers should understand that. the re,tuber of grammar ,:tiles for muld-tttoda.l int<:vfaces becomes much larger than for any single moda.1 int+erfitces. If there are three triodes; :U\], M2, attd M3, a.nd the mJmbers of granum:u' rules imperative(meaning(Action, Object)\] -- > verb(Action\], object(Object\]. wrb(Aetion)-- > verbJ)y_menu(Action\].</Paragraph> <Paragraph position="15"> verb(Action)-- > verbJ~y_multimodal(Action).</Paragraph> <Paragraph position="16"> w.rbJ~y_menu(Action)-- > menu(Menu_it.era, Act:ion\]. verloJ)y_inult:im~dal(delete) -- > (speech or keyboard\]:\[delete\]. ulen u (in ellu_i t eln_2~l, delete\].</Paragraph> <Paragraph position="17"> object(Oh)) -- > ,ioun_phrase(Obj).</Paragraph> <Paragraph position="18"> obj~-ct(Ol<i\] -- > pointing(Oh)).</Paragraph> <Paragraph position="19"> for ea.ch mode a.re; (~1, G2, and Ca. Then, the totaJ number of the multi-nloda.l gra.mn|m' rules is the sun1 of the gramma.r rules of a.ny combination of these three modes. Thus, the tota.l number, (7~o,,/ is: M~ , ;V\[~ ,Ms D_ ,5' The a.bove steps took about two ma, n month for the first cycle. The most. time COl~suming steps were step 4 and step 7.</Paragraph> </Section> class="xml-element"></Paper>