<?xml version="1.0" standalone="yes"?>
<Paper uid="C96-2120">
  <Title>NP_Modification Diathesis Tense Aspect Modalit Sentence Types Coordination Negation</Title>
  <Section position="3" start_page="0" end_page="0" type="intro">
    <SectionTitle>
2 Test Suite Design and Methodology
</SectionTitle>
    <Paragraph position="0"> Based on a survey of existing test suites and an analysis of the diagnostic and evaluation requirements of both NL technology developers and users, TSNLP has developed the methodology for the construction of core test data, that is, test items reflecting central language phenomena and that are applicable to a wide range of applications, including parsers, grammar checkers, and controlled language checkers (Balkan et al. (1996)).</Paragraph>
    <Paragraph position="1"> The TSNLP methodology is designed to optimize (i) control over' test data, (ii) progressivity, and (iii) systematicity. These are necessary qualities for an adequate, reusable test suite, which are difficult to find in test corpora. The methodology also addresses the specific goals of TSNLP to produce multi-purpose, multi-user, and multilingual test suites.</Paragraph>
    <Paragraph position="2"> Control over test data What makes test suites valuable in comparison to corpora is that they can focus on specific linguistic phenomena and that each phenomenon can be presented both in isolation and controlled combinations in which as many linguistic parameters as possible are being kept under control. This is particularly the case when a phenomenon is illustrated by systematic variation over the parameters used to describe this phenomenon, while all other parts of the test items remain constant.</Paragraph>
    <Paragraph position="3"> Vocabulary is an aspect of the test data that needs to be controlled. TSNI,P achieves this by restricting the vocabulary in size as well as in domain. Categorially and semantically ambiguous words are avoided where possible and only included when ambiguity is explicitly tested for.</Paragraph>
    <Paragraph position="4"> Additionally, TSNLP attempts to control the interaction of phenomena by keeping the test items as small as possible. Therefore, a number of guidelines for this purpose (such as use declarative sentences and avoid modifiers and adjuncts) is provided. null Progressivity Progressivity is the principle of starting h'om simple test items and increasing their complexity. In TSNLP, this aspect is addressed by requiring that each test item focuses only on a single t)henomenon (or rather subphenomenon or even feature) which distinguishes it from all other test items. This principle not only ensures systernaticity during the test data (:onstruction but also allows test data users to apply the test data in a progressive order obtained from the special attribute presupposition in the phenomena classification. Thus, the precise identification of the coverage of a system and of its defteieneies is rendered easier.</Paragraph>
    <Paragraph position="5"> Systematicity Systematicity refers to the depth of coverage of a test suite, with respect to both well-formed and ill-formed items. Systematicity in TSNLP is achieved for well-formed items by the explicit classification of test items according to phenomena and sub-phenomena. Negative test data permits testing for overgeneration as well as for coverage, ill-formed items are derived from well-formed ones by systematic variation of the parameters through the application of one (or more) of four operations, namely:  * REPLACEMENT (e.g. change of person) (l)h'ench) L' ingdnieur vient.</Paragraph>
    <Paragraph position="6"> (lq'ench) *L' ingdnieur viens.</Paragraph>
    <Paragraph position="7"> * am)rrION (e.g. of an object NP) (German) Dcr Managcr arbeitet.</Paragraph>
    <Paragraph position="8"> (German) *Dcr Manager arbeitet den Vortrag.</Paragraph>
    <Paragraph position="9"> * DELETION (e.g. of an obligatory complement) (German) Der Mana.qer hiilt den Vortrag.</Paragraph>
    <Paragraph position="10"> (German) *Der Manager hiilt.</Paragraph>
    <Paragraph position="11"> * PERMUTATION (e.g. inverting word order ) (English) He saw the boy.</Paragraph>
    <Paragraph position="12"> (English) *He the boy saw.</Paragraph>
    <Paragraph position="13">  In general, tile systematicity of test data was greatly enhanced through the use of special-purpose tools in the data construction and validation process (see section 5 below).</Paragraph>
    <Paragraph position="14"> Multillnguallty Multilinguality is achieved in the TSNLP test suites by covering the same range of phenomena in English, French and German, and adopting the same classification for these phenomena in the three languages. Furthermore, the choice of related terminology for the categolial and structural description contributes to I;he comparability and consistency of the test items (see section 4 for details).</Paragraph>
    <Paragraph position="15"> Documentation To enhance the usability and extensibility of TSNI,P results, a three-vohnne user guide is under preparation providing clear instructions for the assessment of the methodology, test data, and tools developed.</Paragraph>
  </Section>
class="xml-element"></Paper>