File Information
File: 05-lr/acl_arc_1_sum/cleansed_text/xml_by_section/metho/99/w99-0409_metho.xml
Size: 23,058 bytes
Last Modified: 2025-10-06 14:15:31
<?xml version="1.0" standalone="yes"?> <Paper uid="W99-0409"> <Title>Exploiting the Student Model to Emphasize Language Teaching Pedagogy in Natural Language Processing</Title> <Section position="2" start_page="55" end_page="57" type="metho"> <SectionTitle> 1 Design of the Grammar </SectionTitle> <Paragraph position="0"/> <Section position="1" start_page="55" end_page="56" type="sub_section"> <SectionTitle> 1.1. Grammatical Formalism and Implementation </SectionTitle> <Paragraph position="0"> The grammar for the German Tutor is written in ALE (The Attributed Logic Engine), an integrated phrase structure parsing and definite clause programming system in which grammatical information is expressed as typed feature structures (Carpenter & Penn 1994).</Paragraph> <Paragraph position="1"> The grammar formalism used is derived from l Schwind \[1990a\], p. 577.</Paragraph> <Paragraph position="2"> Head-driven Phrase Structure Grammar (Pollard & Sag 1994). This theory is one of a family which share several properties. Linguistic information is presented as feature/value matrices. Theories in this family are to varying degrees lexicalist, that is, a considerable amount of grammatical information is located in the lexicon rather than in the grammar rules. For example, Figure 1 illustrates a minimal lexical entry for geht. The subcategorization list of the verb, notated with the feature subj, specifies that geht takes a subject which is minimally specified as a singular noun. Rules of grammar specify how words and phrases are combined into larger units according to the subcategorization list. In addition, other principles govern how information such as the head features, which inter alia determine the grammatical category of a constituent, is inherited. 2 restriction on unification, namely that two categories A and B fail to unify if they contain mutually inconsistent information (Gazdar & Pullum 1985). However, this inconsistent information constitutes exactly the errors made by second language learners. For example, if the two categories A and B do not agree in number a parse will fail. To overcome this restriction, we relax the constraint on number agreement by changing its structure so that, rather than checking that the noun is singular, the system records whether or not the subject of geht is in the singular. To achieve this, the noun is no 2 Inheritance in feature-value matrices is indicated by multiple occurrences of a coindexing box labeling the single value.</Paragraph> <Paragraph position="3"> longer marked as \[num .sg\], but instead the path numlsg terminates with the values error or correct. For example, for a singular noun phrase, the value of the path numlsg is correct, while it is error for a plural noun phrase. The two pa~ial lexical entries are given in Figure 2(a) and The verb geht records the value of s g from its subject (Figure 3). If the value of the path numlsg is correct, the subject is in the singular. In case of a plural noun, geht records the value error for number agreement) phon < geht > head v \[cat \[head n\] \]Jl cat subj \[ ' \[ \[content lindex \[num \[sg \[~\]\] descriptor \[main_clause \[vp_num \[sg ~\]\] Figure 3 : Recording Number Features for geht</Paragraph> </Section> <Section position="2" start_page="56" end_page="57" type="sub_section"> <SectionTitle> 1.2 Phrase Descriptors </SectionTitle> <Paragraph position="0"> The goal of the parser and the grammar is the generation of phrase descriptors, each of which 3 For an analysis of errors in linear precedence, see \[Heift 98\]. describes a particular grammatical constraint, its presence or absence in the input sentence and the student's performance on this constraint.</Paragraph> <Paragraph position="1"> Phrase descriptors correspond to structures in the Student Model and are the interface medium between the Student Model and other modules in the system.</Paragraph> <Paragraph position="2"> A phrase descriptor is implemented as a frame structure that models a grammatical phenomenon. Each member of the frame consists of a name followed by a value. For example, subject-verb agreement in number is modeled by the frame \[number, value\] where value represents an as yet uninstantiated value for number. If the grammatical phenomenon is present in the student's input, the value is either correct or error depending on whether the grammatical constraint has been met or not, respectively. If the grammatical constraint is not present in the student's input, the feature value is absent. Consider examples (4a) and (b): (4a) *Er gehen.</Paragraph> <Paragraph position="3"> (4b) Ergeht.</Paragraph> <Paragraph position="4"> He is leaving.</Paragraph> <Paragraph position="5"> The phrase descriptor for subject-verb agreement in number in example (4a) is \[number,error\], while that for the sentence in (b) is \[number,correct\]. For either sentence, (4a) or (b), the information will be recorded in the Student Model. A system presented with (4a), however, will also instruct the learner on the nature of subject-verb agreement in number.</Paragraph> <Paragraph position="6"> In addition to the grammatical features defined in HPSG the grammar uses a type descriptor representing the description of the phrase that the parser builds up. This type is set-valued and is initially underspecified in each lexical entry. During parsing, the values of the features of descriptor are specified. For example, one of the members of descriptor, vp num in Figure 3, records the number agreement of subject-verb in a main-clause. Its value is inherited from the .~g feature specified in the verb geht. Ultimately, descriptor records whether the sentence is grammatical and what errors were made.</Paragraph> </Section> </Section> <Section position="3" start_page="57" end_page="58" type="metho"> <SectionTitle> 2 The Role of the Student Model in </SectionTitle> <Paragraph position="0"/> <Section position="1" start_page="57" end_page="57" type="sub_section"> <SectionTitle> Analysis and Feedback </SectionTitle> <Paragraph position="0"> The initial goals of the analysis of the results of parsing a student's sentence are selecting the appropriate parse and, from it, selecting the error (if there is one) that the system will focus on for instruction.</Paragraph> <Paragraph position="1"> A parse of a sentence is a collection of phrase descriptors. For example, the phrase descriptor given in (5) indicates that the learner has violated subject-verb agreement in number in a main clause.</Paragraph> <Paragraph position="2"> (5) \[main_clause \[vp_num \[sg error\]\]\] The constraint that generated this descriptor has a correlate in the Student Model, in this case a record labelled vp_nummaincl. For each grammar constraint, the Student Model keeps a counter which, at any given instance in the evaluation process, falls in the range of one of the three learner levels, given in (6a) - (c).</Paragraph> <Paragraph position="3"> (6a) novice: 20 _< X <_ 30 (b) intermediate: 10 _< X < 20 (c) expert: 0 _< X < 10 Initially, the learner is assessed with the value 15 for each grammar constraint, representing the mean score of the intermediate learner: Once a student uses the system, the Student Model adjusts the counter of each grammar constraint accordingly. If a grammatical constraint has been met, the counter is decremented. If the constraint has not been met, the counter is incremented and, ultimately, a feedback message is displayed to the learner.</Paragraph> <Paragraph position="4"> The result of the parsing process is a set of collections of phrase descriptors, each collection representing a separate parse. The step of winnowing this set down to a single collection is performed by a set of licensing conditions.</Paragraph> </Section> <Section position="2" start_page="57" end_page="58" type="sub_section"> <SectionTitle> 2.1 Selecting the Desired Parse </SectionTitle> <Paragraph position="0"> A sentence which is unambiguous in many other NLP applications can nonetheless result in multiple sentence readings in a system designed 4 The intermediate learner has been chosen as a reasonable default. While the messages might be initially too overspecified for the expert and too underspecified for the novice, they will quickly adjust to the actual learner level. tO parse ill-formed input. For example, consider a system which relaxes the constraint on grammatical case. Without case marking, the parser has no way of knowing if a constituent is a subject or a verb complement. As a result, more than one sentence analysis will be produced and the errors flagged in each sentence reading can vary. For instance, for the sentence Sie liebt er the parser can assign at least two legitimate syntactic structures. The two sentence readings are given in (7a) and (7b).</Paragraph> <Paragraph position="1"> (7a) *Sie liebt er. (7b) Sic liebt er.</Paragraph> <Paragraph position="2"> Sie liebt ihn. It is her he loves.</Paragraph> <Paragraph position="3"> She loves him.</Paragraph> <Paragraph position="4"> For the sentence Sie liebt er, er could be taken to be either the direct object or the subject of the sentence. Assuming that the choice between the two parses were arbitrary, sentence structure (7a) where er is the object of the sentence contains an error and would yield the feedback This is not the correct case for the direct object. In contrast, the alternative sentence reading given in (7b) where er is the subject of the sentence and the direct object sie is topicalized contains no errors.</Paragraph> <Paragraph position="5"> The example illustrates two important points.</Paragraph> <Paragraph position="6"> First, an algorithm that selects the appropriate parse by counting the numbers of errors flagged in each parse and selecting that which has the least number oferrors \[as in Weischedel (1978), Covington & Weinrich (1991)\] is inadequate. If the student is at an introductory level, the appropriate analysis is the sentence reading given in (7a), the parse that has more errors. 5 The algorithm promoted here uses instead a set of licensing conditions to mark sentences for the rules that were used to parse them and select the appropriate sentence reading on the basis of the likelihood of a grammatical construction. The task of the licensing conditions in this example is to distinguish multiple sentence readings conditioned by word order. During parsing, three of the syntactic rules, the Subject-Head, the Head-Subject-Complement, and the Head-Subject rule each assign a distinct licensing descriptor. Any of the three licensing conditions 5 Object topicalization is a rare construction and not explicitly taught at the introductory level where the focus is on the grammar of more commonly used rules of German.</Paragraph> <Paragraph position="7"> can license a sentence. After parsing, the Licensing Module prioritizes multiple sentence readings so that, in the event of a choice, the parses are selected in a particular order. The chosen parse is passed on to the next module of the system for further processing.</Paragraph> <Paragraph position="8"> The second point important to the current discussion is that it is not sufficient to exclude all other alternatives whenever they appear. An advanced student may be practicing object topicalization and in that case the system should preferably choose the alternative parse given in (7b).</Paragraph> <Paragraph position="9"> This consideration illustrates the importance of the Student Model to the problem of sorting out a set of parses to find the intended interpretation. To provide input to the licensing conditions, we generate a figure representing the student's overall mastery of the grammar by averaging expertise levels on each grammar constraint in the Student Model. A threshold of expertise is set, below which the analysis in (7a) is preferred and above which that in (7b) is chosen.</Paragraph> <Paragraph position="10"> After licensing, a single parsehas been selected and a learner model update on each of the grammatical constraints present in students' input has been extracted. A single parse, however, can contain a number of errors and the system has to decide on how to communicate these errors to the learner. The following section will discuss the task of filtering multiple errors to select the appropriate one.</Paragraph> </Section> <Section position="3" start_page="58" end_page="58" type="sub_section"> <SectionTitle> 2.2 Filtering Multiple Errors </SectionTitle> <Paragraph position="0"> A further challenge in analyzing student input is presented by multiple errors. The sentence given in (8a) illustrates an example.</Paragraph> <Paragraph position="1"> (8a) *Heute die Kindern haben gespeilt mit das Auto.</Paragraph> <Paragraph position="2"> (8b) Heute haben die Kinder mit dem Auto gespielt.</Paragraph> <Paragraph position="3"> Today the children were playing with the car.</Paragraph> <Paragraph position="4"> In example (8a) the student made the following five errors: 1. word order: the finite verb haben needs to be in second position 2. word order: the nonfinite verb gespielt needs to be in final position 3. spelling error with the past participle gespielt * 4. wrong plural inflection for the subject</Paragraph> </Section> <Section position="4" start_page="58" end_page="58" type="sub_section"> <SectionTitle> Kinder </SectionTitle> <Paragraph position="0"> 5. wrong case for the dative determiner dem From a pedagogical and also motivational point of view, a system should not overwhelm a student with instructional feedback referring to more than one error at a time. Little research has been done in Computer-Assisted Language Learning regarding the volume of feedback for different kinds of learners at different stages in their language development. However, van der Linden (1993) found that &quot;feedback, in order to be consulted, has to be concise and precise.</Paragraph> <Paragraph position="1"> Long feedback (exceeding three lines) is not read and for that reason not useful. ''6 She further states that displaying more than one feedback response at a time makes the correction process too complex for the student. The task for an Intelligent Language Tutor is to develop an error filtering mechanism that incorporates language teaching pedagogy. The sheer amount of feedback should not overwhelm the student. In addition, if feedback messages are displayed one at a time they need to be ordered in a pedagogically sound way.</Paragraph> <Paragraph position="2"> To filter the possible errors, an Error Priority Queue is implemented. This queue takes student errors and selects the most important error.</Paragraph> <Paragraph position="3"> Criteria for selection can be set by the language instructor based on her knowledge of the difficulty of a grammatical construction, the likelihood of an error and/or the focus of the exercise.</Paragraph> <Paragraph position="4"> However, the Student Model can also be invoked to rank errors. One criterion for ranking is the students' performance history as indicated by the Student Model: the grammar constraint most often violated will be reported first. The rationale for this criterion is that this grammatical property has been mastered the least and therefore needs the most attention.</Paragraph> <Paragraph position="5"> After student errors have been ranked and the most important one has been selected, the system needs to generate instructional feedback messages to be displayed to the learner. This is</Paragraph> </Section> </Section> <Section position="4" start_page="58" end_page="60" type="metho"> <SectionTitle> 6 Van der Linden \[1993\], p. 65. </SectionTitle> <Paragraph position="0"> achieved by an Analysis Module which will be discussed in the following section.</Paragraph> <Section position="1" start_page="59" end_page="60" type="sub_section"> <SectionTitle> 2.3 Generating Instructional Feedback </SectionTitle> <Paragraph position="0"> A further difficulty in ILTSs lies in framing instructional feedback to student input. In a typical student-teacher interaction, feedback depends on the students' previous performance history. Inexperienced students require detailed instruction while experienced students benefit best from higher level reminders and explanations (LaReau & Vockell 1989).</Paragraph> <Paragraph position="1"> For instance, in example (9a) the student made an error with the determiner einen of the prepositional phrase. Von is a dative preposition and Urlaub is a masculine noun. The correct article is einem.</Paragraph> <Paragraph position="2"> (9a) *Sie tr~iumt von einen Urlaub.</Paragraph> <Paragraph position="3"> (9b) Sie tr~iumt von einem Urlaub.</Paragraph> <Paragraph position="4"> She is dreaming of a vacation.</Paragraph> <Paragraph position="5"> In the German Tutor, the Analysis Module generates instructional feedback of different levels of specificity. The pedagogical principle underlying this design is guided discovery learning. According to Elsom-Cook \[1988\], guided discovery takes the student along a continuum from heavily structured, ~ tutordirected learning to where the tutor plays less and less of a role. Applied to feedback, the pedagogy scales messages on a continuum from least-to-most specific guiding the student towards the correct answer.</Paragraph> <Paragraph position="6"> For the error in example (9a), the sy, stem generates feedback of increasing abstraction that the instruction system can use when interacting with the student. The level of the learner, either expert, intermediate, or novice according to the current state of the Student Model, determines the particular feedback displayed. The responses, given in (10a) - (c) correspond to the three learner levels for the error in example (9a), respectively: (10a) There is a mistake with the article einen of the prepositional phrase.</Paragraph> <Paragraph position="7"> (10b) There is a mistake in case with the article einen of the prepositional phrase.</Paragraph> <Paragraph position="8"> (10c) This is not the correct case for the article einen of the prepositional phrase. Von assigns the dative case, For the expert, the feedback is most general, providing a hint to where in the sentence the error occurred (prepositional phrase). For the intermediate learner, the feedback is more detailed, providing additional information on the type of error (case). For the beginner, the feedback is the most precise. It not only pinpoints the location and type of the error but also refers to the exact source of the error (dative preposition).</Paragraph> <Paragraph position="9"> The Analysis Module is implemented in DATR \[Evans and Gazdar 1990\], a language designed for pattern-matching and representing multiple inheritance. For each grammar constraint, the Analysis Module creates three categories of instructional feedback corresponding to the three learning levels. Provided with three categories of feedback, the system selects an error response suited to students' expertise. The student level is determined by the numerical value for each grammatical constraint maintained in the Student Model. Each value is adjusted each time the learner interacts with the system.</Paragraph> <Paragraph position="10"> For example, the grammar constraint pp-dat records the student's performance on dative assigning prepositions. A learner who violates the constraint on dative prepositions will, at first, obtain the feedback message for the intermediate. If the student commits the same error in subsequent exercises, s/he will soon be assessed a novice. At this point, the system will display the more detailed feedback message suited to the beginner. However, each time the student applies the grammatical constraint correctly, the Student Model records the success. After demonstrating proficiency, the student will again be assessed as intermediate, or, even expert. Maintaining a large number of grammatical constraints allows for a very detailed portrait of an individual student's language competence over a wide-range of grammatical phenomena.</Paragraph> <Paragraph position="11"> After instructional feedback for student input has been generated; the feedback message is passed to the Teaching Module. The Teaching Module interacts with the learner. It displays instructional feedback and, at the end of an exercise set, shows the student's performance history on each grammatical constraint. Students performance history informs learners and instructors of the grammatical construction the student has mastered as well as the ones that require remedial work.</Paragraph> <Paragraph position="12"> Conclusion In this paper, we have described a Student Model that implements language teaching pedagogy in guiding the analysis of student input in an ILTS for German. The Student Model keeps a record of students' previous performance history which provides information essential to the analysis of multiple parses, multiple errors, and the level of interaction with the student.</Paragraph> <Paragraph position="13"> For multiple parses, the system implements licensing conditions which select one of the possible parses by taking into account the likelihood of the error. The likelihood of an error is determined by the performance level of the student as indicated by the Student Model. For multiple errors, the system implements an Error Priority Queue which takes student errors and selects the most important error. Criteria for selection can be set by the instructor or evoked by the Student Model. The Student Model ranks errors with respect to students' performance history. Finally, by consulting the Student Model, the Analysis Module selects instructional feedback of different levels of specificity.</Paragraph> <Paragraph position="14"> From a language teaching perspective, the system reflects a pedagogically informed, student-centered approach. System decisions are based on a dynamic Student Model rather than static computational factors. As a consequence, the learning process is individualized throughout the analysis of student input.</Paragraph> </Section> <Section position="2" start_page="60" end_page="60" type="sub_section"> <SectionTitle> Further Research </SectionTitle> <Paragraph position="0"> The ILTS described in this paper has been implemented on the World Wide Web. While the system encompasses all the necessary grammar rules, we are currently expanding the lexicon for an introductory course of German.</Paragraph> <Paragraph position="1"> Our immediate goal is to test the system with learners of German to assess accuracy. Long-term goals include expanding the Student Model. Student performance is only one criterion to individualize the language learning process. The native language of the student as well as different learning styles might also be key factors in the analysis of student input.</Paragraph> </Section> </Section> class="xml-element"></Paper>