<?xml version="1.0" standalone="yes"?>
<Paper uid="C88-2085">
  <Title>ERROR DIAGNOSING AND SELECTION IN A, TRAINING SYSTEM FOR SECOND LANGUAGE LEARNING Wolfgang Menzel Zentralinstitut fur Sprachwissensehaft</Title>
  <Section position="3" start_page="0" end_page="414" type="metho">
    <SectionTitle>
I. INTRODUCTION
</SectionTitle>
    <Paragraph position="0"> Error anticipation is a key issue so far, as systems for computeP assisted language learning are to be enhanced with diagnostic and explanatory capabilities. Traditional approaches do always rely on the central assumption that any mistake possibly done by a student can be foreseen by the author of a teaching exercise in order to supply the tutoring system with a collection of adequate responses to all the different errors and error combinations which might occur in real training sessions.</Paragraph>
    <Paragraph position="1"> Considering the enormous flexibility of natural language, however, this assumption is only justified for small classes of very simple .type exercises, and most efforts are spent into devising sensible teaching aids despite the absence of appropriate diagnostic techniques. The author is forced to concentrate on single teaching programs without almost any chance to generalise from his results and to reuse parts of it in other contexts later.</Paragraph>
    <Paragraph position="2"> More ambitious solutions certainly can be expected, if it becomes feasible to derive the diagnostic and explanatory abilities of the desired system directly from r u 1 e s for what is considered to be a correct construction in a sensible restricted subset of natural language, instead of using pre-selected lists of possibly occurring answers. In this respect, research in intelligent systems for second language learning sooner or later will follow similar lines as a parallel development in the field of expert systems, where a drastically improved diagnostic behaviour is attempted by modelling structure and function of the application domain (yielding so called &amp;quot;deep modelled&amp;quot; expert systems, which then are able to investigate faults by simulating them) rather than collecting huge amounts of symptom-oriented regularities to be used in a straightforward inference procedure on a surface level (c. f.</Paragraph>
    <Paragraph position="3"> for instance Davis 1984).</Paragraph>
    <Paragraph position="4"> By including semantic, syntactic and morpho-syntactie regularities into a single solution Weischedel et al. (1978) proposed a system fop intelligent language instruction that likewise turned out to be the most ambitious approach so far. Even today, almost exactly ten years.later, its main pPemises and challenging predictions obviously do not fully match the real pPospects in the field.</Paragraph>
    <Paragraph position="5"> ~ven worse, this as well as more recent attempts (Barehan et al. 1985, Schwind 1986) are strongly oriented on the notion of a &amp;quot;typical error&amp;quot;. Thus they clearly fall short of the deep-modelling ideal and ape connected with a number of critical shortcomings. These shortcomings can be avoided, if it becomes possible to identify limited application domains within the area of natural language instruction which allow a training system to be strictly based on the following basic prieiples:  (I) Supply the system with only knowledge about correctness.</Paragraph>
    <Paragraph position="6"> (2) Devise a diagnosis procedure which is  independent of the content of the knowledge base.</Paragraph>
    <Paragraph position="7"> Currently used semantic representation schemes in general seem much too rough to capture all the rather subtle distinctions  between an acceptable natural language expression and a deviant one in a sufficientl~ systematic way. Even if the necessary means would be available the problem of actually coding the desired information for a realistic domain remains to be solved.</Paragraph>
    <Paragraph position="8"> Better prospects for a successful implementation of these two principles can certainly be expected on the syntactic level. Nevertheless, in order to diagnose structural errors ~lll currently known solutions rely exclusiw~ly on the use of &amp;quot;marked&amp;quot; rules, representing typical mistakes and being intended to invoke appropriate error messages. This practice is clearly inconsistent with prineipl,~ (I) and comes out as nothing more than a simple shift of the necessity to anticipate student behaviour onto a new - not necessarily more perspicuous - level.</Paragraph>
    <Paragraph position="9"> Moreover, it leaves the system with all the consequences of a parsing grammar bloated with numbers Of fault-specific rules.</Paragraph>
    <Paragraph position="10"> The domain I believe is currently ripest for a strictly deep-modelled prototype is the area of morpho-syntactie correctness conditions which have to be satisfied in a given (if nec(!ssary artificially fixed) syntactic environm~nt. These consistency constraints fop well-formed syntactic trees comprise agreement relations as well as case government and, as Weischedel st al. (1978) acknowledge, constitute at least for languages like German a major error source. They can easily be expressed as predicates over structured sets of morpho-syntactic features and, if taken together, form simple networks. Here again, all current approaches fall back on either short-cut solutions (e.g. take the fir~t predicate evaluating to false) or simple h~uristic guesses when trying to diagnose (esp. to locate) consistency errors. Heuristics (or the use of rules-of-thumb) typically is the method of one's choice when being f~ced with a, by its nature, ill-defined C/,r extremely complex problem. Obviously the domain of. morpho--syntactie regularities belongs to neither the one nor the other category.</Paragraph>
    <Paragraph position="11"> Accordingly, an investigation into the problem of diagnosing consistency violations has shown that an efficient and extremely precise diagnostic procedure can be devised, which provides a sound basis for the generation of comprehensible explanations (including proposals for a proper solution), for a valuation of the learning progress and for an adaptive determination of the further traininq strategy. Moreover, it turned out that non-trivial training exercises based exclu-sively on simple morpho-syntaetic correctness conditions can well be used independently of an error sensitive * parser and additionally are connected with a number of pedagogical and performane~ gains (for a detailed discussion see Menzel, 1987 and 1988). In this ease the student can be guided by an appropriate sentence context in a smooth and unconstrained way to exactly produce those fragmentary utterances which have the desired fixed syntactic structure.</Paragraph>
  </Section>
  <Section position="4" start_page="414" end_page="418" type="metho">
    <SectionTitle>
II. CONSISTENCY CONSTRAINTS
</SectionTitle>
    <Paragraph position="0"> Consistency constraints to be used as a basis for diagnosing students' faults are expressed in a quite common way as binary predicates which hold over structured feature sets of syntactically related word forms (source and destination). Feature sets, as usual, are attached to word forms via the dictionary or a morphological analysis. To specify the part of a feature set relevant to a specific correctness condition correctness predicates are augmented by a third argument, a category: agree\[&lt;souree&gt;,&lt;destination&gt;,&lt;eategory&gt;\].</Paragraph>
    <Paragraph position="1"> Simple examples of correctness predicates define symmetrical relations between equivalent parts of the two involved feature sets, as does the number agreement between the determiner and the governing noun in a German noun phrase: agree(det, noun, number\].</Paragraph>
    <Paragraph position="2"> Other predicates are needed to account for agreement between different parts of two feature sets. One example are selectional demands of a source to be matched with the corresponding properties of a destination, another is given by predicates coping with the distinction between e.g. different gender values belonging to stem or ending of German possessive pronouns.</Paragraph>
    <Paragraph position="3"> A basically different type of predicates does not specify a source but instead gives a condition to be fulfilled by the destination: satisfy\[ &lt;destination&gt;, &lt;condition&gt;\].</Paragraph>
    <Paragraph position="4"> This type is always used in one of the following eases:  (I) Conditions of pure structural origin so that no particular source can be given, e.g. the nominative of the subject: satisfy\[ noun-of-subject, ease=nora).</Paragraph>
    <Paragraph position="5"> (2) Case government of verbs, e.g. : satisfy\[ noun-of-dative-object, ease=dat\]. (3) Ordinary' agreement relations where, because of the limits of a specific exercise, it is not necessary or not possible to specify the source of the condition explicitly (cut-off ares in the constraint network).</Paragraph>
    <Paragraph position="6"> (4) Word class restrictions, e.g. a noun  phrase determiner can be an article as well as a possessive or demonstrative pronoun:  satisfy\[det, oat=(art, poss-p, dem-p)\].</Paragraph>
    <Paragraph position="7"> To achieve a Uniform treatment, internally the satisfy-predicates are converted into ordinary agreement-predicates between an artificially created feature expression at a nowhere source (indicated by ***\] and the desired destination: satisfy\[ noun, case=ace\] ==&gt; agree\[ ***, noun, ease\] .</Paragraph>
    <Paragraph position="8"> A single training exercise usually is described by a number of correctness predicates which can be combined to form a simple constraint network. The arcs in this network represent correctness predicates and the connected nodes ape identified by the source and destination arguments. A rather simple type exercise is, for example, the correct insertion of a German possessive pronoun into a carefully generated context like: &amp;quot;lob habe Schal und Mutze verloren.</Paragraph>
    <Paragraph position="9"> Hat jemand ... Sachen gesehen?&amp;quot; The exercise asks the student to have regard to the agreement of the pronoun with both, the subsequent noun (&amp;quot;Sachen&amp;quot;) and the corresponding antecedent (&amp;quot;Ich&amp;quot;\]. It results in a constraint network consisting of three nodes and nine arcs. Two of the nodes (antecedent and noun\] are associated with the context whereas ~the third (possessive\] represents the student's response. The network is equivalent to the following list of correctness predicates: satisfy\[ possessive, cat=pose-p\] satisfy\[ noun, cat=noun\] satisfy\[ antecedent, cat=( noun, peps-p\] \] agree\[ antecedent, possessive, stem-number\] agree\[ antecedent, possessive, stem-gender\] agree\[ antecedent, possessive, stem-person\] agree\[ noun, possessive, case\] agree\[ noun, possessive, number\] agree\[ noun, possessive, gender\].</Paragraph>
    <Paragraph position="10"> As a slightly more ambitious example serve the formation of a German local according to the fixed structural pattern</Paragraph>
    <Paragraph position="12"> (preposition determiner adjective noun\] to be inserted into sentences like &amp;quot;Des Gold habe ich ... gelegt.&amp;quot; This time the network consists of five nodes and 16 edges. All the nodes correspond to word forms in the student's response, but depending on the sentence context only a subset of exaetly four nodes (e.g. (prep-4 dot adj noun\]\] is instantiated to the incoming word forms: satisfy\[ prep-3, cat=prep\] satisfy\[ prep-3, select=location\] satisfy\[ prep-4, cat=prep\] satisfy\[ prep-4, select=direction\]  satisfy\[ noun, cat=noun\] satisfy\[ det, cat=( art, pose-p, des-p\]\] satisfy\[ adj, oat=adj\] agree\[ prep-3, noun, ease\] agree\[ prep-4, noun, ease\] agree\[ noun, dot, case\] agree\[ noun, dot, number\] agree\[ noun, dot, gender\] agree\[ noun, adj, case\] agree\[ noun, adj, number\] agree\[ noun, adj, gender\]  agree\[ det, adj, inflectional-degree\]. III. DIAGNOSIS Diagnosis is mainly based on constraint propagation techniques. The nodes in the constraint network are treated as variables which receive their values (feature sets) by means of a pattern matching procedure on the student's input.</Paragraph>
    <Paragraph position="13"> The procedure consists of four parts being invoked sequentially. Two of them (the hard core of diagnosis\] are intended to detect and - if desired to locate errors in the student's response. The other two refine diagnostic results by applying transformational rules and preference criteria in order to provide a sound basis for th@ generation of appropriate explanations.</Paragraph>
    <Paragraph position="14"> The reason to draw a clear distinction between the error detection and localization components is one of mere technical nature. The separation has been introduced to considerably speed up the handling of error-free utterances, since it allows the time expensive localization procedure to be activated only if indeed an error ocurred and the student actually did ask for an explanation of his mistake.</Paragraph>
    <Paragraph position="15"> (A\] error detection Error detection, in fact, is a direct proof procedure for the correctness of the utterance. Trying to show the absence of errors, it evaluates the relevant predicates of the constraint network, taking into consideration all the morpho-syntactic readings of the word forms concerned. The values of the network variables ape constantly updated according to the results of an ordinary feature set unification until finally a state is reached which satifies all relevant predicates simultaneously.</Paragraph>
    <Paragraph position="16"> Given the ease this proof cannot be established, one or possibly several mistakes of: the student have to be assumed and upon request a detailed analysis of the error reasons may become necessary.</Paragraph>
    <Paragraph position="17"> ( B\] error localization Error localization is carried out by a simulation of constraint violations. For e a c h predicate in the network the procedure fol:Lows up the consequences of assuming the student did ignore the existence of exactly this particular regularity. This assumption is modelled by temporarily adding the negated predicate to the knowledge base and following up its consequences around the network. Theoretically such a modification of the knowledge base is equivalent to reasoning based on Re(tar' s famous &amp;quot;closed world assumption&amp;quot; (Re(tar, 1978): If you cannot proof P, add not(P) to the set of premises.</Paragraph>
    <Paragraph position="18"> By adding it as a premise the diagnosis procedur~ treats a negated predicate as nothing more than merely an error hypothesis. It can be raised to the level of a confirmed error description only after its consequences for othc~r predicates in the network have carefully been investigated and properly recorded. In contrast to a typical short-cut solution or a heuristics-based approach the diagnosi~ procedure thus carries out a thorough analysis of all possible error hypotheses. Hence, it is particularly qualified to diagnose multiple faults as well as errors with an ambiguous interpretation.</Paragraph>
    <Paragraph position="19"> Technically this kind of reasoning is achieved by simply resuming error detection with just the logical complement of the usual feature unification results* Error detection provides for this purpose and collects all necessary data on a separate stack.</Paragraph>
    <Paragraph position="20"> Despite the rather' tiny size of a typical constraint network, the simulation of constraint violations results in a procedure of high complexity and effective measures are required to keep the size of the search space limited* Hence, in a widely accepted way error localization is restricted to the search of minimal diagnoses (that is, results including only a minimum number of constraint violation hypotheses)* (C) error transformation According to the fundamental distinction between rules and facts in a knowledge base two different types of misconceptions have to be distinguished in the diagnostic results: (a) rule violations: the ignorance of correctness conditions ( expressed by predicates Jn the network) and (b) faetual faults: the lack of knowledge about specific morpho~syntactic properties of a word form (expressed by a feature set in the dictionary.</Paragraph>
    <Paragraph position="21"> An integration of both types of misconceptions into a single diagnosing procedure neither yields an efficient solution nor turns out to be really necessary. In print(. ple, rule violations and factual faults repr'esent different views on one and the same phenomenon and .a particular error can equally well he explained in terms of rules or in terms of facts. Both types should be convertible into each other without much difficulty. Accordingly, error localization has been designed to exclusively concentrate on the analysis of rule violations and to leave open the problem of factual faults. A total ignorance of factual faults, of course, would be an intolerable restriction on the diagnostic capabilities of the system, since then it interprets errors of whatever kind in terms of rule violations only.</Paragraph>
    <Paragraph position="22"> Hence, a transformational component has been added which is able to infer factual faults from the hitherto obtained diagnostic results. The necessary information is usually contained in typical structural error configurations involving the violation of adjoining predicates for one and the same morpho-syntactic category. One or several simultaneous constraint violations and\[ not\[ agree\[ Xi~ Y, C\] \] , not\[ agree\[ Xn, Y, C\] \] \] or and\[ not\[ agree\[ Y, Xi, C\] \], not\[ agree\[ Y, Xn, C\] \] \] can be transformed into a factual fault description, iff (i) the set of constraint violations is complete with respect to Y, i.e. there is no other X which appears as source or destination argument of a predicate agree\[ X~ Y, C\] or agree\[ Y,X, C\] within the knowledge base, and (2) consistency holds for all neigbouring nodes of Y: agree\[Xi,Xj,C\].</Paragraph>
    <Paragraph position="23"> Under these circumstances Y is said to be isolated with respect to C and without loss of generality a new description can be generated, stating the student's misconception to assume a wrong feature value concerning category C at the word form Y: not\[ agree(~** Y, C\] \] resp.</Paragraph>
    <Paragraph position="24"> not( agree( Y, ~*, C\] \] again using *** to indicate a non-specified source or destination. In particular three types of structural configurations have to be distinguished: (I) A single constraint violation concerning two isolated nodes X and Y can be transformed into two alternative factua\] error descriptions: not\[ agree( X, Y, C\] \] ==&gt; exor( not\[ agree\[ ***, Y, C\] \] not\[ agree( X, **~, C} \] \] .</Paragraph>
    <Paragraph position="25">  (2) A single constraint violation with only. , 417 one isolated node Y is equivalent to a single factual error description: not\[ agree\[ X, Y, C\] \] ==&gt; not\[ agree{ ***, Y, C\] \] . (3) Several constraint violations for an  isolated node Y can be summarized by a single factual error description (c. f. the general ease above).</Paragraph>
    <Paragraph position="26"> The decision whether to apply a transformation or not is influenced by two criteria, the tolerable complexity of diagnostic results {it is raised by (I), maintained by { 2) and reduced by (3)) and the intended teaehing strategy (e. f. section IV).</Paragraph>
    <Paragraph position="27"> (4) error generalization Error generalization aims at an as high as possible aggregation of elementary errors (constraint violations or factual faults) into a more concise description. Generalization schemata can be defined for all three argument positions of a violated correctness predieate~ e.g. for a category generalization: null and{ not\[ agree\[X~ Y, CI\]\]~</Paragraph>
    <Paragraph position="29"> which for the purpose of explanation later might be paraphrased like &amp;quot;Missing CI and C2 agreement between X and Y.&amp;quot; Other ~eneralization rules are intended to collapse alternative diagnostic readings which merely are artJ fi~et~ resulting from the technical layout of dictionary entries.</Paragraph>
    <Paragraph position="30"> Besides these stringent generalization schemes sometimes very weak ones turn out to be useful as well. They are partieularly suited to generate an at least partially comprehensible explanation which in an opaque error situation may give a rough indication of the error location instead of annoying the student with plenty of detail: and{ not{ agree\[XI,Y, CI\]\], \[ not{ agree{ X2, Y, C2\]\]\] ==&gt; not{ agreeE**x,Y,(C1~C2)\]\].</Paragraph>
    <Paragraph position="31"> The result of a weak generalization then may be explained to the student perhaps in a sentence like: &amp;quot;There is something wrong with CI and C2 at Y.&amp;quot; Since it is logically incomplete, however, a weak generalization can no longer serve as a reliable basis to derive repair suggestions from, neither for internal use nor for a presentation to the student.</Paragraph>
    <Paragraph position="32">  There is some reason to assume the general four step scheme of diagnosis not being restricted to violations of consistency constraints, but rather being adaptable to other types of errors, namely structural ones, as well. Again, the procedure has to be based on a detection and proper localization of elementary error types with respect to a simple model of correctness, e.g. a recursion-free PSG. In this context the elementary error types are substitution, insertion and omission of single word forms. Other types (e.g. extraposition or mutual interchange of forms) can later be inferred from the primary results by means of transformation rules.</Paragraph>
    <Paragraph position="33"> Generalization finally allows to condense error descriptions into more complex structural units (e.g. constituent based errors).</Paragraph>
    <Paragraph position="34"> IV. ERROR SELECTION Error selection is an integral part of several steps of the diagnosis procedure. It tries to keep the number of concurrent error descriptions as small as possible. Error localization selects on the number of primary errors per error description, but since transformation as well as generalization can influence the resulting complexity, their results in turn are subject to a decision with regard to a minimal number of errors. Although the side effect of further reducing the complexity of minimal diagnoses is normally welcomed, error transformations have to be sensitive to the intended teaching strategy. Diagnosing a constraint violation exactly pinpoints the conceptual mistake of the student but does not always give useful hints about where to correct the utterance. In these cases, despite being more complex, an indication of factual faults may sometimes be more helpful, since it better points out the existing possibilities for an improvement o~ the partially wrong solution.</Paragraph>
    <Paragraph position="35"> On the other hand, even less complex factual faults may be quite undesired if they result in doubtful correction proposals (e.g. in a given semantic context a gender-of-thenoun error can hardly be repaired by simply exchanging the noun in question).</Paragraph>
    <Paragraph position="36"> Despite the selection of minimal diagnoses on several levels, sometimes the diagnosing component provides a number of alternative minimal diagnoses. They finally are evaluated and weighted accordi&amp;quot;g to a number of preference criteria: (a) a structural measure which depending on whether a conceptual description or a repair suggestion is preferred may select errors higher up or deeper down in a hierarchical syntactic representation of the utterance, (b) the degree to which an error explanation can be assumed to be helpful for the student (if possible, certai, types of explanations are suppressed), (c) hints, resulting from an obstinate repetition of one and the same error type.</Paragraph>
  </Section>
  <Section position="5" start_page="418" end_page="418" type="metho">
    <SectionTitle>
V. CONCLUSIONS
</SectionTitle>
    <Paragraph position="0"> absolutely exact localization of an error either by pointing out the erroneous form (plus the categories and values involved) or by properly reflecting an ambiguous correction possibility: 73 percent.</Paragraph>
    <Paragraph position="1"> approximative localization by only indi~ caring the violated constraint predicate without being able to decide between source or destination: 23 percent.</Paragraph>
    <Paragraph position="2"> Although it is connected with a number of interesting advantages, deep modelled reasoning clearly should not be considered the sine qua non of future tools for language instruction. Based on the very idea of a closed world, it is by definition limited to the rather narrow domains of knowledge about natural language which allow a highly reliable and complete description of the necessary correctness conditions. Unfortunately, neither high precision nor completeness can well be attributed to major parts of currently available grammar models.</Paragraph>
    <Paragraph position="3"> Nevertheless, two possibilities for an advantageous application of the , presented diagnosis procedure can be identified even today: bad localization either by returning several rule violations or by ignoring an alternative correction possibility: 4 percent. Notice that not a single instance of wrong, that is~ misleading diagnostic results occurred.</Paragraph>
    <Paragraph position="4"> The system currently runs on 16-bit-micros (with drastic reductions on 8-bit-micros, as well) and has been tested on a number of non-trivial exercises for German. Experience has shown that the approach allows diversified and intensive training and supplies surprisingly precise explanations for most of the erroneous utterances.</Paragraph>
    <Paragraph position="5"> (I) &amp;quot;stand alone&amp;quot; use in simple exercises similar to the examples in section II.</Paragraph>
    <Paragraph position="6"> agreement presented</Paragraph>
  </Section>
class="xml-element"></Paper>