<?xml version="1.0" standalone="yes"?>
<Paper uid="W97-1503">
  <Title>Participatory Design for Linguistic Engineering: the Case of the GEPPETTO Development Environment</Title>
  <Section position="4" start_page="18" end_page="19" type="metho">
    <SectionTitle>
4 The Formalism for Linguistic Data
</SectionTitle>
    <Paragraph position="0"> CLs participating in the working group suggested a Typed Feature Logic oriented (Carpenter, 1992) formalism. The reasons were as follows: * TFL formalisms provide a way for breaking down the structure of linguistic data, allowing for a clear separation between the description of abstract linguistic types and that of grammatical rules and lexical entries. 4 This facilitates knowledge encapsulation as well as a modular architecture of linguistic data. Such a modularity can play an important role in the reuse of existing data; * typing secures to a high degree the consistency of the linguistic data. This speeds up the process of data editing and debugging; * the formalism is well known and many basic unification algorithms are available; * it meets the demands of many current linguistic theories, e.g. LFG, GPSG, HPSG, etc.</Paragraph>
    <Paragraph position="1"> TFL specifications are compiled into a graph format, where each node represents a Typed Feature Structure (TFS). Types and the type hierarchy have been implemented by adapting the encoding schema proposed by (Ait-Kaci et al., 1989) to the TFL format. This permits to efficiently handle very large type hierarchies as well as to account in a straight-forward way for type disjunction. The standard TFL formalism has been modified to accommodate:  phrase-structure-like skeleton should be provided. This seems to better address their expectations and ways of thinking than a system in which grammar rules are absent, as it is normally possible in type-oriented linguistic formalism (e.g. HPSG).</Paragraph>
    <Paragraph position="2">  possibly external, modules; such a fact can be specified by means of external constraints; * External constraints providing ,explicit links to external modules, e.g. morphological processors, independent KBs, etc.; * Directives for the unifier. For instance, it is possible to force the unifier to consider in the first place the paths that have been observed to cause more frequent failures (Uszkoreit, 1991).</Paragraph>
    <Paragraph position="3"> * Macros.</Paragraph>
    <Paragraph position="4"> Declaration statements and external constraints greatly enhance the modularity and portability of the LEAs developed by means of GEPPETTO, by allowing the reuse of existing processors and/or data.</Paragraph>
  </Section>
  <Section position="5" start_page="19" end_page="20" type="metho">
    <SectionTitle>
5 The GEPPETTO Environment
</SectionTitle>
    <Paragraph position="0"> In this section some of the characteristics of GEPPETTO are outlined, focusing on those aspects that specifically meet user needs. A more detailed description of GEPPETTO is contained in (Ciravegna et al., 1996).</Paragraph>
    <Paragraph position="1"> In GEPPETTO an application consists of two main parts: a (set of) processor(s) and a Linguistic System. The latter is the collection of all the sets of linguistic descriptions relevant for the characterization of a given corpus. Given the kind of formalism adopted, namely TFL, a Linguistic System consists of: a type hierarchy, a grammar, a lexicon, and a set of macros. The concept of linguistic system is not simply conceived as a set of the four components just mentioned but it is a complex object with a central role in GEPPETTO: much of the development of LEAs is centered around linguistic systems. CLs edit, browse, and update linguistic systems. They can reuse existing linguistic systems, or parts thereof, to produce new ones.</Paragraph>
    <Paragraph position="2"> GEPPETTO maintains a conceptual distinction between browsing/editing and testing/debugging. Actually, browsing/editing can be performed independently by different users, whereas testing/debugging can require a strict cooperation between different typology of users. This is so whenever an error is due to unexpected interactions between data and processors. These observations emphasize the advantage of a single environment for the whole development cycle: different users have dedicated facilities for development, but a common environment for integrating and testing.</Paragraph>
    <Paragraph position="3"> We now turn to a discussion of the facilities and tools provided to the different users.</Paragraph>
    <Section position="1" start_page="19" end_page="19" type="sub_section">
      <SectionTitle>
5.1 Supporting the Linguistic Engineer
</SectionTitle>
      <Paragraph position="0"> LER main task is the study of architectural requirements (together with PM). He/she also controls the compliance of the LEA with the initial requirements.</Paragraph>
      <Paragraph position="1"> To this end, GEPPETTO provides support for: (a) the rapid prototyping of architectures by assembling already existing processors and linguistic systems, and (b) tests against a test corpus. Both data and processors are seen by the LER as black boxes that can be combined by means of a graphical interface.</Paragraph>
      <Paragraph position="2"> When the architecture meets the requirements, a delivery system can be produced. It contains the selected linguistic system and processor(s), and excludes the GEPPETTO development environment.</Paragraph>
    </Section>
    <Section position="2" start_page="19" end_page="19" type="sub_section">
      <SectionTitle>
5.2 Supporting the Processor Manager
</SectionTitle>
      <Paragraph position="0"> PM task is to identify the processors that can satisfy the architectural requirements. She/he can choose among the processors made available by GEPPETTO 5 or link external ones to the environment. In the latter case, an API is provided to connect the external processor to the GEPPETTO world. Once a new processor has been properly linked, it is completely identical to the other default processors: it can be selected via the graphical interface, it can take advantage of the debugging/testing facilities, and so on.</Paragraph>
      <Paragraph position="1"> Via API, it is also possible to interface LEAs with other kinds of external modules, e.g. modules which make available functionalities not provided by the environment (e.g. Knowledge Bases or morphological analyzers).</Paragraph>
      <Paragraph position="2"> PM can also choose among different unification algorithms that have been designed to: * carefully control and minimize the amount of copying needed with non-deterministic parsing schemata (Wroblewski, 1987) (Kogure, 1990); * provide a better match between the characteristics of the unifiers and those of the linguistic processors. Indeed, different linguistic processors may profit of different unification algorithms. The availability of different unification algorithms allows the user to choose the one which best fits the needs of the particular linguistic processor at hand.</Paragraph>
    </Section>
    <Section position="3" start_page="19" end_page="20" type="sub_section">
      <SectionTitle>
5.3 Supporting the Computational Linguist
</SectionTitle>
      <Paragraph position="0"> A considerable amount of effort has been devoted to create suitable (specialized) graphical tools for CL. Recall that CL main task is to build a linguistic system satisfying the application requirements. The graphical environment must allow CL to ignore low-level details as much as possible, and concentrate on the linguistic aspects of data description.</Paragraph>
      <Paragraph position="1"> CLs can build a linguistic system both by pasting already existing components (and modifying them 5At present, GEPPETTO features two chart-based parsers (a bidirectional Head-Driven Bottom-Up (Satta and Stock, 1989) and a CYK-like) and a Head-Driven Bottom-Up non-deterministic generator (Pianesi, 1993).</Paragraph>
      <Paragraph position="2"> We plan to make available a wider array of processors in the near future.</Paragraph>
      <Paragraph position="3">  when necessary) and by building it from scratch. 6 As the data forming the parts of a linguistic system differ in attributes, global organization and functions, specialized graphical tools have been designed for browsing/editing the type hierarchy, the grammar, the lexicon and the macros.</Paragraph>
      <Paragraph position="4"> The main tools for the CL are: * a grapher for browsing and editing the type inheritance hierarchy. It displays mouse sensible nodes and allows to add/delete/modify nodes, as well as to modify the hierarchy itself; * browsers for data sets such as lexicon, grammar and macros. They allow to add/delete/modify/copy elements in the data sets, as well as to undertake actions on the data set as a whole (e.g. compiling it); * editors for editing and modifying properties of single lexical entries, grammar rules, macros and type hierarchy nodes. They include editors for TFL descriptions, feature appropriateness statements, etc. TFL-syntax error checking, TFL description compilation and TFS visualization are supported. Documentation and SCurrently GEPPETTO provides some standard resources for Italian: a type hierarchy, two lexica and two grammars.</Paragraph>
      <Paragraph position="5"> comment notes can be attached to each item; * interactive and post processing debugging tools (at now mainly a sophisticated chart browser).</Paragraph>
      <Paragraph position="6"> Facilities are also supplied for computing statistics about performances on test suites. In particular, it is possible to detect points where unification failures arise. Such results can be exploited either to handtune the linguistic systems to the corpus needs, or by feeding them into a module which forces unification algorithms to consider unification failure hypothesis first, this way speeding up the whole processing.</Paragraph>
    </Section>
  </Section>
  <Section position="6" start_page="20" end_page="21" type="metho">
    <SectionTitle>
6 PD at Work: the Debugging Tools
</SectionTitle>
    <Paragraph position="0"> The PD working group suggested to divide the tools for testing and debugging into interactive facilities (such as tracers and steppers to follow the application of grammar rules during processing), and &amp;quot;postprocessing&amp;quot; facilities. In the working group it was agreed that naive interactive tools can be quite difficult to be fully exploited given the great number of rule applications and unifications that happen during parsing. In order to reduce the number of rule applications and unifications traced, it is necessary to have a very powerful (and complex) language which makes the user able to program the tracer; but usually tracer's expressive power is quite difficult to  be fully taken advantage of. Moreover, it is important that the tools are (relatively) easy to use, so that they can be usefully exploit also by users not necessarily expert of that particular tool or by time-to-time users. Given these considerations and also the fact that all the processors currently available are chart-based (and hence all the results produced during processing are still available at the end of the processing itself), the discussion focused on the post-processing tools.</Paragraph>
    <Paragraph position="1"> Within such tools, the chart browser plays a central role. To better discuss its characteristics, paper mockups were jointly created and ,evaluated. Such an effort produced a highly detailed description of the tool functionalities and of its layout; in particular, the kind of information and actions (showing parse/generation trees, TFS descriptions associated with edges) to be made available to the user, the different viewpoints on edges and vertices, etc.</Paragraph>
    <Paragraph position="2"> As it turned out, the chart browser window is the starting point for the exploration of the structures produced during processing. The tool (cf. figure 2) allows the user * to see the edges either in a strictly sequential way or as organized around the objects connecting them (i.e. vertices for the parser and constraints for the generator); * to filter edges according to their type (active/inactive edges), to their categories, etc.; * to browse through the wide and complex data structures produced; * to activate auxiliary tools.</Paragraph>
    <Paragraph position="3"> The chart browser is a fairly standard debugging tool; in GEPPETTO the adoption of a User Centered approach permitted to design a flexible and extendible tool, which is the central structure for browsing through the elements built during processing. null Besides the chart browser facilities described above (and already implemented), the working group faced the problem of how to single out the failures happened during parsing and to understand their causes. Browsing edges in the chart it is possible to identify (guess) possible error points and to concentrate the focus on them: it was envisaged the possibility of selecting some edges in the chart and run the parser on them in a special mode. During this special running mode G~.PPETTO reports diagnostic messages on the causes of the failure: missing grammar rules/lexical items, failure during unification, etc. If the failure is due to unification, the involved paths are reported.</Paragraph>
  </Section>
  <Section position="7" start_page="21" end_page="22" type="metho">
    <SectionTitle>
7 Evaluations with Users
</SectionTitle>
    <Paragraph position="0"> The implemented system was assessed by means of a formative evaluation (Nielsen, 1993), to test its general usability and the quality of the proposed solutions. null Participants The testing group consisted of eight people from our department. Participants had different degrees of expertise in NLP, though none of them had ever used GEPPETTO before, nor had participated in the PD process. Participants were not required to have any previous knowledge of the TFS formalism.</Paragraph>
    <Paragraph position="1"> Procedure Users were given the manual in advance but they were not required to read it before the test, nor any training occurred before the testing phase. During the experiment, users were allowed to freely consult the manual. Each participant was asked to perform 4 tasks: 1. architecture definition and composition: the participant was required to create her/his personal LEA by composing existing linguistic resources and processors to form a new architecture; null 2. lexicon update: the participant had to insert lexical entries in the lexicon, and to run the parser over a sentence containing the new terms; 3. hierarchy manipulation and grammar update: the participant was asked to modify the type hierarchy by acting on its graph. Furthermore, she/he had to modify the grammar. Finally, by browsing the resulting parse tree, the sub-ject was asked to verify the correctness of the changes; 4. test suite run: lastly users had to load an existing test suite (a file), to add the sentence of task 2 and to run their architecture over it; results of the running had to be saved in a log file.</Paragraph>
    <Paragraph position="2"> During the experiment, participants were requested to verbalize their thoughts. This method, known as thinking-aloud, permits an easy detection of the problematic parts of the human-computer interaction as well as to understand how users perceive the system (Nielsen, 1993). An experimenter sat near the participant, taking notes on occurring problems and stimulating the subject to express her/his thoughts. After the test phase, the experimenter interviewed each participant, discussing the problems she/he run into, gathering suggestions on possible improvements, and investigating user satisfaction. All performances were videotaped to allow successive analysis.</Paragraph>
    <Paragraph position="3"> Results The choices done and implemented into GEPPETTO supported naive users in moving around and acting in a complex and unfamiliar environment. Even participants who had not read the manual and had only a little experience in NLP were able to complete the tasks in less then one hour. 7 Through 7This time is definitely low considering that users were required to comment their actions, were allowed  observations and interviews it could be verified that participants reached a good understanding of the system and judged it positively.</Paragraph>
    <Paragraph position="4"> Some weaknesses in the interface design were identified: problems mainly limited to common graphical user interfaces mistakes, e.g. lack of feedback in resource status, and to the understanding of the terminology developed during PD (naming problem) emerged. Identified problems may be solved with a limited revision of the graphical interface.</Paragraph>
    <Paragraph position="5"> Experiments demonstrated that the adoption of PD can bring intuitiveness also in the design of a complex LEADS: even users without any experience with GEPPETTO and limited knowledge in NLP were able to easily understand the system organization and to effectively use its tools to accomplish non trivial tasks.</Paragraph>
  </Section>
class="xml-element"></Paper>