<?xml version="1.0" standalone="yes"?>
<Paper uid="W01-1301">
  <Title>Generating a 3D Simulation of a Car Accident from a Written Description in Natural Language: the CarSim System</Title>
  <Section position="6" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
5 Results and Discussion
</SectionTitle>
    <Paragraph position="0"> The CarSim system has been implemented and evaluated over the MAIF corpus. The assessment method does not consist, as usually done with IE systems, in calculating a precision and a recall.</Paragraph>
    <Paragraph position="1"> Our objective is to design a system that carries out the whole processing chain, that is from a written report up to a 3D animation. Therefore, we preferred to compare the simulation with the understanding and mental representation of the scene that could have a human reader. This implies that some aspects of the formal description are not taken into account when evaluating the system, e.g. we assume that the value of the INITIAL DIRECTION parameter is less important than the positions of the vehicles relatively to each other. Hence, we considered that the result is acceptable as far as the latter is correct.</Paragraph>
    <Paragraph position="2"> According to such criteria, we considered that the simulation provided by the system corresponds, in 17% of the texts, with what could have imagined a human being. Figure 3 &amp; 4 show the two collisions described in Text A4.</Paragraph>
    <Paragraph position="3"> Failure cases have many difierent grounds.</Paragraph>
    <Paragraph position="4"> They may be related either to the IE task, to the simulator, or to a lack of cooperation between the two subsystems. Evaluating separately each subsystem leads to a better understanding of the actual limits of the system.</Paragraph>
    <Paragraph position="5"> Feeding the simulator with manually written formal descriptions provides a good way to evaluate it for itself. According to such tests, the CarSim system generates an acceptable simulation of almost 60% of the reports. This implies that the results of the overall system will be lower. CarSim's simulator does not succeed in simulating manually written formal descriptions because  of three main causes: expressivity of the formalism that does not cover all possible accidents (e.g. synchronization between event chains of difierent objects), the restricted number of scenarios considered by the CarSim visualizer and the limited database of 3D graphical objects. Depending on the text, the failure is the result of either only one of these restrictions or a combination. Future work on the project will focus on these issues.</Paragraph>
    <Paragraph position="6"> The e-ciency of the IE task varies with the nature of extracted information. First, the results clearly depend on the accuracy with which the system can correctly extract impacts, that is flnd the verb representing the collision and also resolve the actor, the victim and possibly their participating parts5. This task is successfully accomplished in 69% of the texts6. In addition, the system correctly extracts EVENTS in 35% of the texts. This means that in 35% of the texts, all the events are properly extracted with a good ordering.</Paragraph>
    <Paragraph position="7">  are erroneous.</Paragraph>
    <Paragraph position="8"> Concerning time and space information, the system provides only simple mechanisms to obtain them. Our system is at an early stage and our objective when designing it was to see whether such an approach was feasible. It represents a sort of improved baseline with which we can compare further results. At this time, the temporal information known by the system is restricted to the events associated with dynamic objects. Our method assumes that they are given in the text in the same order they occur in reality. This is a simpliflcation that proves wrong in some reports. Further improvements could take into account tenses of verbs, temporal adverbs and prepositions, so that the system could determine the real chronological relationships between events.</Paragraph>
    <Paragraph position="9"> A similar comment can be given with regards to spatial information. In CarSim, the spatial conflguration (the background of the scene) is given mainly by the type of roads. The extraction of participating parts also provides additional information that in uence the relative positions of the vehicles when colliding. During preplanning stage, the system checks the consistency of the FD and tries to resolve con icts between the different information. At present, initial directions of the vehicles depend only on the background of the scene, that is the road conflguration. The co-ordinates are also chosen arbitrary from the beginning. See for example the tree referred as tree1 in Text A4: no information about its location is given in the text. The only facts relative to it that we can deduce from the original report are its existence and its involvement in a collision. Moreover, the problem of choosing a referential from which to calculate coordinates is quite unsolvable for texts that do not mention it explicitly. The IE task could involve deeper semantic analysis that provides means of constructing a more global spatial representation of the scene.</Paragraph>
  </Section>
class="xml-element"></Paper>