<?xml version="1.0" standalone="yes"?>
<Paper uid="W02-1713">
  <Title>XtraGen -- A Natural Language Generation System Using XMLand Java-Technologies</Title>
  <Section position="8" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
6 Evaluation
</SectionTitle>
    <Paragraph position="0"> At the end of a software development phase any newly created system must proof in an evaluation phase whether it reaches its predefined goals. (Mellish and Dale, 1998) This is especially true in an industrial context with commercial applications as in our case.</Paragraph>
    <Paragraph position="1"> The context for the evaluation of XtraGen was provided by X-Booster (Beauregard et al., 2002) which is another project at our site that was concurrently developed with our system. This system is an optimized implementation of Slipper (Cohen and Singer, 1999), a binary classifier that induces rules via boosting and combines a set of those classifiers to solve multi-class problems. It was the goal to successfully integrate XtraGen into this system.</Paragraph>
    <Paragraph position="2"> The motivation behind this is based on the fact that common classification systems are quite non-transparent in regard to their inner workings. Therefore it becomes rather difficult to understand how and why certain classification decisions are made by those systems.</Paragraph>
    <Paragraph position="3"> We departed at exactly this point: XtraGen was to be used to automatically generate texts that explain the learning phase of the X-Booster and hence make the classification more transparent. As an additional &amp;quot;gadget&amp;quot; we wanted to create the explanations in all the languages that are spoken at our company site: English, German, French, Italian, Russian, Bulgarian and Turkish.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
6.1 Integration Tasks
</SectionTitle>
      <Paragraph position="0"> As the very first task of the integration procedure we needed to answer the question what to actually output to the user and in which exact format this output should be. We decided on producing a description of the complete learning phase with two kinds of output texts: One targeted at experts and one for novice users. The format chosen for the final output was HTML.</Paragraph>
      <Paragraph position="1"> Now we needed to adapt the code of X-Booster slightly to make the meta data about the learning phase accessible from the outside. To do so we wrote some small methods that simply returned in XML format the meta data which were only stored in internal variables up to this point.</Paragraph>
      <Paragraph position="2"> The next step was to add to X-Booster's own code the code for calling the generation engine and for transforming the result into the final output-format. This was done as described in section 5.2.</Paragraph>
      <Paragraph position="3"> Finally (and most importantly) the generation grammars for the different languages were developed. This happened in the way that we first set-up a prototypical grammar for English which we tested extensively. Then in a second step the grammars for the other languages were modelled according to this exemplary one. For this we worked together with different native-speaking colleagues that translated the original English grammar into their language.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
6.2 Sample Template and Output
</SectionTitle>
      <Paragraph position="0"> returned after running X-Booster together with the integrated XtraGen on a given training set. This result was produced by using the English generation grammar and the parameters set for producing texts targeted at novice users.</Paragraph>
      <Paragraph position="1"> The number of documents is 37, divided into 2 different categories. The results have been produced using 3 fold-cross-validation which means that the data-set is divided into 1/3 test-set and 2/3 training-set.</Paragraph>
      <Paragraph position="2"> The learner is trained on the training-data only and evaluated on the test-set which has not been presented before. We repeat this process 3 times on non-intersecting testdata. null The overall result is then computed finally as the average of all performed tests. The average accuracy reaches 100.0% which is achieved by applying 5 rules.</Paragraph>
      <Paragraph position="3">  The template is complete in the sense that all different aspects of a template are exposed and it is only simplified in the respect that similar parts of the template are left out.</Paragraph>
    </Section>
  </Section>
  <Section position="9" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
7 Outlook and Future Work
</SectionTitle>
    <Paragraph position="0"> Because the above described evaluation proofed to be quite successful it was decided to further deploy XtraGen at our site and to integrate it into new products and projects. One of the first of these projects is Mietta-II (Multilingual Information Environment for Travel and Tourism Applications), an European Commission funded industrial project with the goal of developing a comprehensive web search and information portal that specializes on the tourism sector. (Additional application scenarios are already envisaged for a possible later stage of the project.) In this environment we will apply natural language generation to produce texts and messages for various types of media such as dynamically generated web pages, paper leaflets, hand-held devices and in particular cellular phones. For the last of those media we are exploring the possibility of producing voice-enabled output with a dedicated voice server that is based on VoiceXML (World Wide Web Consortium, 2002) or JSML (Sun Microsystems, 1999).</Paragraph>
    <Paragraph position="1"> We are experimenting at the moment with the different possible outputs and on how these outputs can be encoded in a generation grammar enriched with VoiceXML or JSML tags.</Paragraph>
  </Section>
class="xml-element"></Paper>