File Information
File: 05-lr/acl_arc_1_sum/cleansed_text/xml_by_section/metho/04/w04-1709_metho.xml
Size: 23,425 bytes
Last Modified: 2025-10-06 14:09:16
<?xml version="1.0" standalone="yes"?> <Paper uid="W04-1709"> <Title>Sentence Completion Tests for Training and Assessment in a Computational Linguistics Curriculum</Title> <Section position="3" start_page="0" end_page="0" type="metho"> <SectionTitle> 2 The philosophy behind SETs </SectionTitle> <Paragraph position="0"> (R utter, 1973) creates an extensive topology for assessments. He distinguishes between open, semi-open and closed tasks. The distinction derives from the type of answer expected from the learner: There is no certain answer the author expects (open tasks), the author expects a certain answer the learner has to create themselves (semi-open tasks), the learner has to choose the right answer(s) from given possibilities (closed tasks). Multiple Choice tasks (MC) belong to the closed tasks.</Paragraph> <Paragraph position="1"> The topology presented by R utter is not restricted to the easy tasks. You will also nd so-called \Erweiterungswahlaufgaben&quot; in the class of closed assigns. This task consists of a piece of information the tested person has to extend so as to create a coherent piece of new information. The learner can choose suitable extensions from a given list. R utter's description includes the hint that these tasks are hard to design but present a very clear structure for the test person.</Paragraph> <Paragraph position="2"> Our Sentence Completion Tests can be seen as an instance of such Erweiterungswahlaufgaben. The learner has to answer a complex question in near-free-form on the basis of extensive choices of possible answer components supplied by the system. There will be answer components considered indispensable, some considered correct but optional, others categorised as outright wrong, and others still rated as irrelevant to the question at hand.</Paragraph> <Paragraph position="3"> The required components of the answer will all have to occur in the answer but not in a xed order.</Paragraph> <Paragraph position="4"> In concrete terms this means that a learner will author an answer in the form of a complete sentence, in a piecemeal fashion and under the guidance of the system, by picking answer fragments from dynamically adapted choice menus popping up as the answer is growing. At each step the user input will be checked by the system against the answer model that contains all the expected answer parts, essential relationships between them, and possible restrictions on the order of these parts. At each step as well as at the very end the system can generate feedback for the user which will make him understand why and in which aspects his answer is correct or incorrect.</Paragraph> <Paragraph position="5"> 3 How to use a SET All SETs are presented under a web interface.</Paragraph> <Paragraph position="6"> The student has to start a browser2 and choose a single SET3.</Paragraph> <Section position="1" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 3.1 Basic elements of a SET </SectionTitle> <Paragraph position="0"> The student sees four coloured elds4, each labeled with a number and a functional description. These elds are: 1. Text up to now 2. Comments/Feedback 3. List of elements to continue 4. Preview Text up to now contains the question and the answer in its present state. List of elements to continue consists of possible continuations of the answer. Clicking on one of the radio buttons activates the Preview showing all the options that will become available once the learner has committed himself to the given continuation. That way the user is always aware of the consequences his choice might have. The listing eld includes two submit buttons, one for submitting the choice, one for undoing the last choice. The element list will show the elements in di erent order each time the user reloads or restarts a SET.</Paragraph> <Paragraph position="1"> The crucial eld is the one for Comment/Feedback. The user does not merely get a \Right - Wrong&quot; feedback but rather If the answer contains the correct components but a wrong relationship the feedback will point this out and invite the user to try again and nd the correct combination.</Paragraph> <Paragraph position="2"> to give the student a familiar feeling for the assessment situation.</Paragraph> <Paragraph position="3"> If the answer consists of correct components as well as of wrong ones the feedback will say so and point out which components are wrong.</Paragraph> <Paragraph position="4"> If the answer is one of the correct ones, the feedback will approve this solution and mention the other possible correct answers. This way for every possible combination of answer components the user gets a di erent optimised feedback.</Paragraph> <Paragraph position="5"> The text inside the feedback eld is displayed as HTML so that it is possible to include links to related SETs, back to the lecture notes or associated material. A feedback text also can include a link to a new SET, as a followup. Sometimes it is useful to have the system generate a comment before a complete answer has been created by the learner. Once the learner has chosen a certain number of wrong answer components he will get suitable feedback before nishing. In this case the feedback is used to warn the user that he is on a completely wrong track and that he ought to undo some of the last choices, or to start again from scratch.</Paragraph> </Section> <Section position="2" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 3.2 A sample SET </SectionTitle> <Paragraph position="0"> See gure 1 for a sample session with SET.</Paragraph> <Paragraph position="1"> The initial question is: \Was ist ein Parser?&quot; (What is a parser?).</Paragraph> <Paragraph position="2"> Here the user chose \Ein Parser ist eine Prozedur&quot; (\A parser is a procedure&quot;) as next element in the third eld. This will be the beginning of his answer. Clicking on the corresponding radio button activated the preview in the fourth eld. Before submitting the choice, the user can think about the combinations his choice will allow. The preview shows 4 possibilities to continue with the description of the aim of this procedure.</Paragraph> <Paragraph position="3"> If the user is satis ed with his choice, he will click the submit button \Auswahl best atigen&quot; (Con rm choice). This will result in reloading the site with the new information.</Paragraph> <Paragraph position="4"> Text bisher (Text up to now) will contain the question, the beginning of the answer and the fragments added by the learner so far \Ein Parser ist eine Prozedur&quot;. The feedback eld will still be empty. Auswahl der Fortsetzungen (List of elements to continue) will show all possible continuations. Vorschau (Preview) will be empty until the user clicks on one of the radio buttons in the list of elements to continue. This sequence of actions will be repeated until the user has created a complete sentence.</Paragraph> <Paragraph position="5"> He then gets the feedback. If he is not satis ed with one of his choices before nishing, he can undo the last choice, or simply restart the SET.</Paragraph> <Paragraph position="6"> In case the user is on a probably wrong way he will get feedback before nishing the SET. See gure 2 for an example. The user created an answer start \Ein Parser ist eine Prozedur, um die syntaktische Korrektheit einer Sprache...&quot; (\A parser is a prozedure to ... the syntactical correctness of a language&quot;). The intervening feedback points to the principle of correctness concerning certain constructions of languages and prompts the user to undo the last decission(s). (\Was wohl soll die syntaktische Korrektheit einer Sprache sein?! Nur einzelne Konstruktionen einer Sprache k onnen korrekt oder inkorrekt sein. Einen Schritt zur uck!&quot; Figure 3 shows the nished SET. The user followed the hint in the intervening feedback shown in gure 2. He removed the part \einer Sprache&quot; (\of a language&quot;). The answer created by the user is \Ein Parser ist eine Prozedur um die syntaktische Korrektheit eines Satzes zu ermitteln&quot; (\A parser is a procedure to detect the syntactical correctnes of a sentence\). Clearly, this answer is not correct. It describes rather an acceptor than a parser. The comment says so and then o ers a correct definition with a hint to the latin origins of Parser.</Paragraph> </Section> <Section position="3" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 3.3 Training mode and assessment </SectionTitle> <Paragraph position="0"> mode SETs can be used in E-Learning for training as well as for assessments. Self-assessment can be seen as an instrument for training { users get elaborate feedback for their answers and are invited to try again.</Paragraph> <Paragraph position="1"> In the training/self-assessment mode users get feedback after completing the answer or while composing it. The feedback always takes into account all components collected up to that point as well as the user's undo or redo actions. The user is allowed to undo a decision as often as he likes. This way nding the right answer is a question of either knowing it or following the hints in the feedback.</Paragraph> <Paragraph position="2"> In the assessment mode the user gets a number of points credited. The points total is compiled the same way the feedback is created. Depending on the answer fragments chosen by the learner, and on their order, the points total will be computed. It is also possible to chain several SETs one after another5, collect the credits collected in each of them, and present the grand total at the very end.</Paragraph> <Paragraph position="3"> The user can be allowed to use the undo button in di erent manners. Three settings are possible: The undo button can be used as often as the learner wants but each use is logged in the background.</Paragraph> <Paragraph position="4"> 5SETs can be linked in linear or network like fashion via HTML links or followups in the comments. Each use of the undo button results in a deduction of a certain number of points, and its use is logged.</Paragraph> <Paragraph position="5"> The use of the button is allowed only a pre-set number of times { if the user tries to undo more often, the button is disabled.</Paragraph> <Paragraph position="6"> That way tutors can track whether the student arrived at the answer by merely trying out all possible continuations.</Paragraph> <Paragraph position="7"> 4 How to create a SET What does an author have to consider when creating a SET? First, he has to decide which answer elements the user can choose from at any given step. Second, he must make sure that any of the answer components o ered as a choice at a given step will contribute to a well-formed sentence only. Finally, helpful and syntactically well-formed comments have to be de ned for any of the possible answers.</Paragraph> <Paragraph position="8"> What the presentation of a SET ultimately boils down to is running a Finite State Automaton (FSA), with answer components as states and user choices as input. This is done by a Prolog program as the back-end for a single SET. As input it takes the SET speci c Prolog automaton, the path up to now, and the current choice of the user. As output it creates the new current answer, the new list of elements to continue, the preview, comments, paths and points. The author of a SET has thus to write a (potentially large) FSA. This is a tedious and error-prone task. How can this be done e ciently and reliably?</Paragraph> </Section> <Section position="4" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 4.1 The machinery behind a SET </SectionTitle> <Paragraph position="0"> Developing the automaton normally starts with the author writing a number of possible correct and incorrect answers, in a way similar to the development of an ordinary MC. The author then marks where these sentences could be split into fragments. Splitting must allow the combination of various sentence fragments from di erent sentences in a way that only well-formed passages result. To limit the number of such combinations the author can de ne constraints that explicitly include or exclude certain combinations.</Paragraph> <Paragraph position="1"> To increase readability, answer fragments that are of the same syntactic type can be collected in boxes. It is, however, advisable to create distinct boxes for correct fragments, wrong fragments, and indi erent fragments of the same syntactic type; this makes the design of complex automata considerably easier. Each box has an ID, in-going and outgoing boxes6, information concerning speci c constraints on allowed combinations, and (positive or negative) credits the user will collect when choosing this element. Boxes are linked by vectored edges to create a number of paths through the answer fragments, each one of which will de ne a complete and syntactically well-formed sentence.</Paragraph> <Paragraph position="2"> Splitting answer sentences into fragments that can be combined freely creates, of course, a large number of potential answers (in fact, a potentially in nite number). It would be clearly impossible to write individual comments for each of these answers. We overcome this obstacle by generating comments, semi-automatically in some cases, and fully automatically in others. The semi-automatical creation relies on the fact that each answer fragment can be rated according to its correctness and relevance for a given question. It is relatively easy to attach, to a limited number of \strategically important&quot; answer fragments, comment fragments specifying in what way they are (in)correct and (ir)relevant. We then have SET collect the comment fragments of all answer fragments chosen by the learner, and combine them into complete and syntactically well-formed comments that refer to the individual parts of an answer and point out super uous, missing, or wrong bits, in any degree of detail desired by the author. We can even generate comments on the basis of arbitrarily complex logical conditions over answer fragments, thus identifying, among others, contradictions in answers. That way we can generate a potentially in nite number of comments on the basis of relatively few comment fragments. This is the semi-automatic creation of comments, taking into account the local properties of an answer path. We also allow the fully automatic creation of comments that take into consideration the global properties of answer paths. Thus the fact that a learner used the undo button very often in various places, or took a very circuitous way to arrive at his answer, may be detected by measuring global values of the answer path and can then be commented upon automatically7. For a detailed documentation see (Brodersen and Lee, 2003).</Paragraph> </Section> <Section position="5" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 4.2 Developing a sample SET </SectionTitle> <Paragraph position="0"> Clearly the author of a SET must be supported in the design of even moderately complex FSAs.</Paragraph> <Paragraph position="1"> To this end we developed an authoring tool called Satzerg anzungstest-Ersteller-Interface (SETEI), a Java application with a GUI. It uses 7Resulting in comments like \You used the undo button way too often.&quot; or \Correct but your answer could have been much shorter&quot;, etc.</Paragraph> <Paragraph position="2"> a text-based format for saving data and has an export function to create the FSA. Figure 4 shows the nal stages in the development of the SET \Was ist ein Parser?&quot; (\What is a parser?&quot;) used as example in section 3.2.</Paragraph> <Paragraph position="3"> The box in the left upper corner is the start box, containing the question. Boxes 1, 2, 3, 4, 6, 7, 8, 9, 13 are answer boxes containing answer fragments. Boxes 10, 11, 12, 14 are comment boxes containing comments for complete answers or certain combinations of answer parts (box 14).</Paragraph> <Paragraph position="4"> One of the boxes, box 14, is selected, and inside this box the text element 72 is selected. As the boxes o er limited space the full text of a selected element is shown at the very bottom of the window. Here we can also see the box number, fragment number, and the credits attached to the selected answer fragment.</Paragraph> <Paragraph position="5"> These credits can be used, in assessment mode, to grade the answer. Creating, lling, and modifying boxes is a matter of a few clicks.</Paragraph> <Paragraph position="6"> The possible answer paths are represented, obviously, as vectored edges between boxes.</Paragraph> <Paragraph position="7"> Each path must end in a comment box.</Paragraph> <Paragraph position="8"> Two paths contain three boxes { 1!8!9 and 1!2!7 Two paths contain four boxes { 1!2!3!7 and 1!2!6!13 One path contains ve boxes { Possible answers in the above example may thus consist of three, four or ve parts. Since each answer box contains at least two text elements this automaton de nes many more answers than there are paths. On path 1!2, for instance, the user can combine each element in box 1 with each element in box 2. Connections between boxes are created or deleted by simple dragging or clicking operations. Whenever a circular connection is created, even an indirect one, the user is asked whether this is what he really wanted to do.</Paragraph> <Paragraph position="9"> The top menu in the window contains the various tools for the manipulation of boxes.</Paragraph> <Paragraph position="10"> Thus, to see all text elements in one box plus all the in-going and out-going boxes and the constraints for elements, the author may use the box browser Ansicht (view). The browser presents a magni ed view on the given box with additional functionalities to edit the box content. The user can also zoom out and see the bare structure of the entire FSA, without box contents, can select sub-parts of the automaton and try them out in isolation, etc.</Paragraph> <Paragraph position="11"> To allow intermediate feedback, comment boxes may be placed in the middle of the FSA (such as, in this SET, comment box 14). All answer paths end with a comment box to give feedback after creating a complete sentence.</Paragraph> </Section> </Section> <Section position="4" start_page="0" end_page="0" type="metho"> <SectionTitle> 5 Where to use SET </SectionTitle> <Paragraph position="0"/> <Section position="1" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 5.1 Where we use SETs </SectionTitle> <Paragraph position="0"> Since winter term 2003/2004 we use SETs at our institute as a training and self-assessment tool in introductory courses on CL. They are often used as nal element in learning units intended for self-study by students. These learning units each cover one particular aspect of Computational Linguistics that may be unfamiliar to part of the audience (such as regular expression, tokenising, tagging or parsing).</Paragraph> <Paragraph position="1"> They are organised around Problem-based Interactive Learning Applications.8 While simple skills can be tested with standard MC methods, for more general and more abstract types of knowledge SETs turned out to be a much better solution. Any type of question that would, ideally, require a free form answer can be turned into a SET. These are de nitional questions (\What is a parser?&quot;) as well as a questions requiring comparisons between concepts (\How does a parser di er from an acceptor?&quot;) and the description of procedures (\What are the processing steps of a transfer Machine Translation system?&quot;). It is important that SETs can determine, and comment upon, non-local properities of answers. Thus a SET can detect contradictions between di erent parts of an answer, or a wrong sequencing in the description of processing steps (say, putting tokenising after parsing), or repetitions, all of which may occur in parts of an answer that are arbitrarily far removed from</Paragraph> </Section> <Section position="2" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 5.2 Real Examples of SETs </SectionTitle> <Paragraph position="0"> SETs have been developed mainly for the introductory classes in Computational Linguistics at Zurich but new tests for more advanced courses are under development. Since classes are taught in German, all SETs are in German, too.</Paragraph> <Paragraph position="1"> Students can access SETs in two ways: As most SETs are used in Learning Units students will encounter SETs for the rst time when they are working their way through the Learning Units.</Paragraph> <Paragraph position="2"> When preparing for exams students want to have random access to SETs. For this reason all SETs ever developed are accessible via one big collection, our Setcenter. The Setcenter www.cl.unizh.ch/ict-open/satztest/setcenter.html o ers a check-box list to create a customised web page containing a short introduction to SETs, help for using them, and a list of links to the chosen SETs. For a rst look at SETs the page www.ifi.unizh.ch/cl/ict-open/satztest/, with pre-de ned examples from outside the eld of Computational Linguistics, may also be useful.</Paragraph> <Paragraph position="3"> Most of the SETs we developed ask questions about the basic concepts and terms of the eld. Some examples are listed in table 1.</Paragraph> <Paragraph position="4"> In some case we also \abuse&quot; SETs to function itself as authoring tool with feedback facilities. In one case students are asked to Intro to CL 1 Intro to CL 2</Paragraph> </Section> </Section> <Section position="5" start_page="0" end_page="0" type="metho"> <SectionTitle> CL </SectionTitle> <Paragraph position="0"> write speci c rules for a chunking grammar.</Paragraph> <Paragraph position="1"> In a SET, they get a set of rule elements to choose from (pre-terminal and terminal categories, parentheses, Kleene star, etc.) and have to combine them, step by step, creating a grammar rule in the process. If their choice of a symbol is completely o track (such as a grammar rule beginning with a closing parenthesis) they are warned right away. Otherwise the structure of the completed rule is commented upon. If the rule is not correct, users are sent back to the beginning. Otherwise they are sent to a subsequent SET, with a more demanding task. That way, by chaining SETs, we teach them to write increasingly complex chunking rules, under close guidance of the system. This turned out to be a very promising use of SETs.</Paragraph> <Section position="1" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 5.3 Use of SET in other topics </SectionTitle> <Paragraph position="0"> The question arises whether it would be possible to use SETs in elds other than CL.</Paragraph> <Paragraph position="1"> In general, in all elds where short textual descriptions are the best way to answer questions, SETs are a good way to automatise training and testing. SETs are of particular interest to the Arts and Humanities, but the Medical Sciences might also be a eld that could bene t form SETs (for instance, a picture is presented and the user is asked to describe what seems important or abnormal).</Paragraph> </Section> </Section> class="xml-element"></Paper>