<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-3004">
  <Title>Virtual Modality: a Framework for Testing and Building Multimodal Applications</Title>
  <Section position="6" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
5 Experiments
</SectionTitle>
    <Paragraph position="0"> The experimental setup is depicted in Figure 4. The core of the system is the Galaxy Communicator architecture extended with the Batchmode server (as explained in Section 3 and shown in more details in Figure 2). It must be noted that although the sentences are taken from dialogues, each sentence is processed independently so the focus of attention is the new aspect introduced by the Virtual Modality.</Paragraph>
    <Paragraph position="1"> There are two runs for each experiment. First, the original sentences are input to the Batchmode server and then passed to the Voyager application via the Galaxy architecture. The outcomes are the corresponding frames from the Language Understanding server (the Context Resolution server is not invoked due to the absence of context in this case). The second run takes the Virtual Modality data, namely the new sentences with the deictic references and the accompanying data for the Virtual Modality channel (semantic value, begin-end markers). The output frames are produced by the Language Understanding module and further processed by the Context Resolution server to resolve deictic references. null The last step of the execution is the comparison of the frame pairs: one frame for the original sentence and the other for the Virtual Modality data.</Paragraph>
    <Paragraph position="2"> The results presented below are from the very initial tests; clearly more work is needed to justify the concept of Virtual Modality, as well as to fully investigate the utilization of the generated data in testing.</Paragraph>
    <Paragraph position="3"> The initial experiments were run on 436 sentences, which represent a small portion of the entire database.</Paragraph>
    <Paragraph position="4"> The results indicate that if only one deictic reference per sentence is used with zero deviation, the generated output frames are identical to the original sentence output frames in 82.03% of the cases. The erroneous results occurred when the preposition and a chosen deictic form together formed an ungrammatical expression (e.g.</Paragraph>
    <Paragraph position="5"> &amp;quot;how about on over there?&amp;quot;). The data generation process requires further refinements to decide whether a preposition can be used with a randomly selected deictic expression.</Paragraph>
    <Paragraph position="6"> In sentences with two deictic references only 78.26% agreement was achieved. The major reason for this is the incorrect replacement of highways and highway numbers with deictic references by the data generation process. Also, awkward combinations of deictic references result in incorrect resolution. All these problems will be addressed in future work.</Paragraph>
    <Paragraph position="7"> Additionally, since the current version of the Context Resolution server has no built-in time limits for resolving deictic references, future work will aim to incorporate some kind of temporal considerations and adaptivity. The Virtual Modality data creation process supports the generation of a large amount of timeshifted versions of the original data, which can be used for further testing of the system's temporal robustness.</Paragraph>
  </Section>
class="xml-element"></Paper>