File Information
File: 05-lr/acl_arc_1_sum/cleansed_text/xml_by_section/metho/05/i05-6009_metho.xml
Size: 18,133 bytes
Last Modified: 2025-10-06 14:09:41
<?xml version="1.0" standalone="yes"?> <Paper uid="I05-6009"> <Title>Error Annotation for Corpus of Japanese Learner English</Title> <Section position="5" start_page="73" end_page="75" type="metho"> <SectionTitle> 4 Error Tags in the NICT JLE Corpus </SectionTitle> <Paragraph position="0"> In this section, we introduce the error annotation scheme we used for the NICT JLE Corpus.</Paragraph> <Paragraph position="1"> We are aware that it is quite difficult to design a consistent error tagset as the learner errors extend across various linguistic areas, including grammar, lexis, and phoneme, and so on. We designed the original error tagset only for morphological, grammatical, and lexical errors, which are relatively easy to categorize compared with other error types, such as discourse errors and other types of errors related to more communicative aspects of learners' language. As shown in Figure 3, our error tags contain three pieces of information: POS, morphological/grammatical/lexical rules, and a corrected form. For errors that cannot be categorized as they do not belong to any word class, such as the misordering of words, we prepared special tags. The error tagset currently The tags are based on XML (extensible markup language) syntax. One advantage of using XML is that it can clearly identify the structure of the text and it is also very beneficial when corpus data is utilized for web-based pedagogical tools or databases as a hypertext. The error tagset was designed based on the concept of the ICLE's error tagging, that is, to deal with as many morphological, grammatical, and lexical errors as possible to have a generic error tagset. However, there are several differences between these two tagsets. For example, in the ICLE, only replacement-type errors are linguistically categorized, and redundant- and omission-type errors are not categorized any more and just called as &quot;word redundant&quot; or &quot;word missing&quot;, while in our error tagset, all these three types of errors are linguistically categorized.</Paragraph> <Paragraph position="2"> Although our error tagset covers major grammatical and lexical errors, annotators often have difficulties to select the most appropriate one for each error in actual tagging process. For example, one erroneous part can often been interpreted as more than one error type, or sometimes multiple errors are overlapping in the same position.</Paragraph> <Paragraph position="3"> To solve these problems, tagging was done under a few basic principles as follows.</Paragraph> <Paragraph position="4"> 1) Because of the limitation of XML syntax (i.e. Crossing of different tags is not allowed.), each sentence should be corrected in a small unit (word or phrase) and avoid to change a sentence structure unnecessarily.</Paragraph> <Paragraph position="5"> 2) If one phenomenon can be interpreted as more than one error type, select an error type with which an erroneous sentence can be reconstructed into a correct one without changing the sentence structure drastically. In this manner, errors should be annotated as locally as possible, but there is only one exception for prefabricated phrases. For example, if a sentence &quot;There are lot of books.&quot; should be corrected into &quot;There are a lot of books.&quot;, two ways of tagging are possible as shown in a) and b).</Paragraph> <Paragraph position="6"> a) There are <at crr= &quot;a&quot;></at> lot of books.</Paragraph> <Paragraph position="7"> b) There are <o_lxc crr= &quot;a lot of&quot;>lot of</o_lxc> books.</Paragraph> <Paragraph position="8"> In a), just an article &quot;a&quot; is added before &quot;lot of&quot;, while in b), &quot;lot of&quot; is corrected into &quot;a lot of&quot; as a prefabricated phrase. In this case, b) is preferred.</Paragraph> <Paragraph position="9"> 3) If multiple errors overlap in the same or partly-same position, choose error tags with which an erroneous sentence can be reconstructed into a correct one step by step in order to figure out as many errors as possible. For example, in the case that a sentence &quot;They are looking monkeys.&quot; should be corrected into a sentence &quot;They are watching monkeys.&quot;, two ways of tagging are possible as shown in c) and d).</Paragraph> <Paragraph position="10"> c) They are <v_lxc crr= &quot;watching&quot;> looking</v_lxc> monkeys.</Paragraph> <Paragraph position="11"> d) They are <v_lxc crr= &quot;watching&quot;> looking<prp_lxc2 crr= &quot;at&quot;> </prp_lxc2></v_lxc> monkeys.</Paragraph> <Paragraph position="12"> In c), &quot;looking&quot; is replaced with &quot;watching&quot; in one step, while in d), missing of a preposition &quot;at&quot; is pointed out first, then, &quot;looking at&quot; is replaced with &quot;watching&quot;. In our error tagging scheme, d) is more preferred.</Paragraph> <Section position="1" start_page="75" end_page="75" type="sub_section"> <SectionTitle> 4.1 Advantages of Current Error Tagset </SectionTitle> <Paragraph position="0"> Error tagging for learner corpora including the NICT JLE Corpus and the other corpora listed in Section 3 is carried out mainly by categorizing &quot;likely&quot; errors implied from the existing canonical grammar rules or POS system in advance. In this sub-section, we examine the advantages of this type of error tagging through research and development done by using these corpora.</Paragraph> <Paragraph position="1"> Tono (2002) tried to determine the order in which Japanese learners acquire the major English grammatical morphemes using the error tag information in the JEFFL Corpus. Izumi and Isahara (2004) did the same investigation based on the NICT JLE Corpus and found that there was a significant correlation between their sequence and Tono's except for a few differences that we assume arose from the difference in the languga e production medium (written or spoken). Granger (1999) found that French learners of English tended to make verb errors in the simple present and past tenses based on the French component of the ICLE.</Paragraph> <Paragraph position="2"> Izumi et al. (2004) also developed a framework for automated error detection based on machine learning in which the error-tagged data of the NICT JLE Corpus was used as training data. In the experiment, they obtained 50% recall and 76% precision.</Paragraph> <Paragraph position="3"> Error tagging based on the existing canonical grammar rules or POS system can help to successfully assess to what extent learners can command the basic language system, especially grammar. This can assist people such as teachers who want to improve their grammar teaching method, researchers who want to construct a model of learners' grammatical competence, and learners who are studying for exams with particular emphasis on grammatical accuracy.</Paragraph> </Section> </Section> <Section position="6" start_page="75" end_page="75" type="metho"> <SectionTitle> 5 Future Improvement </SectionTitle> <Paragraph position="0"> Finally, let us explain our plans for future improving and extending error tagging for the</Paragraph> </Section> <Section position="7" start_page="75" end_page="78" type="metho"> <SectionTitle> NICT JLE Corpus. </SectionTitle> <Paragraph position="0"/> <Section position="1" start_page="75" end_page="76" type="sub_section"> <SectionTitle> 5.1 Problems of Current Error Tagset </SectionTitle> <Paragraph position="0"> Although the current error tagging scheme is beneficial in the ways mentioned in 4.1, it cannot be denied that much could be improved to make it useful for teachers and researchers who want to know learners' communicative skills rather than grammatical competence. The same can be said for learners themselves. In the past, English education in Japan mainly focused on developing grammatical competence in the past. However, in recent years, because of the recognition of English as an important communication tool among peoples with different languages or cultures, acquiring communicative competence, especially production skills, has become the main goal for learners. One of the most important things for acquiring communicative skills might be producing outputs that can be understood properly by others. In other words, for many learners, conveying their messages clearly is often more important than just producing grammatically-correct sentences.</Paragraph> <Paragraph position="1"> It is necessary to make the current error tagset more useful for measuring learners' communicative competence. To do this, firstly we need to know what kind of learners' outputs can be understood by native speakers and in what cases they fail to convey their messages properly. By doing this, it should become possible to differentiate fatal errors that prevent the entire output from being understood from small errors that do not interfere with understanding.</Paragraph> <Paragraph position="2"> Another goal of studying English for learners, especially at the advanced level, is to speak like a native speaker. Some learners mind whether their English sounds natural or not to native speakers. In the current error tagging, both obvious errors and expressions that are not errors but are unnatural are treated at the same level. It would be better to differentiate them in the new error annotation scheme.</Paragraph> </Section> <Section position="2" start_page="76" end_page="76" type="sub_section"> <SectionTitle> 5.2 Survey for Extending Current Error Tagset </SectionTitle> <Paragraph position="0"> To solve the problems of our current error tagging system discussed in 5.1, we decided to do a survey to: 1) Identify fatal errors and small ones by examining &quot;learners' outputs that can be understood properly by native speakers&quot; and &quot;those that do not make sense to native speakers&quot;.</Paragraph> <Paragraph position="1"> 2) Identify unnatural and non-nativelike expressions and examine why they sound unnatural.</Paragraph> <Paragraph position="2"> We will do this mainly by examining the learner data corrected by a native speaker. Correction by NS We asked a native speaker of English to correct raw learner data (15 interviews, 17,068 words, 1,657 sentences) from the NICT JLE Corpus and add one of the following three comments (Table 2) to each part.</Paragraph> <Paragraph position="3"> Comment 1 It is obviously an error, but does not interfere with understanding.</Paragraph> <Paragraph position="4"> Comment 2 The meaning of the utterance does not make sense at all.</Paragraph> <Paragraph position="5"> Comment 3 It is not an error, and the utterance makes sense, but it The person who did the corrections is a middle-aged British man who has lived in Japan for 14 years. He does not have experience as an English teacher, but used to teach Japanese Linguistics at a British University. Although he is familiar with English spoken by Japanese people because of his long residence in Japan and the knowledge of the Japanese language, we asked him to apply the corrections objectively with considering whether or not each utterance was generally intelligible to native speakers.</Paragraph> </Section> <Section position="3" start_page="76" end_page="77" type="sub_section"> <SectionTitle> Corrected Parts </SectionTitle> <Paragraph position="0"> A total of 959 errors were corrected and 724 of these were labeled with Comment 1, 57 with Comment.</Paragraph> <Paragraph position="1"> In order to examine what kind of differences can be found among errors labeled with these comments, we categorized them into four types (morpheme, grammar, lexis, and discourse) depending on which linguistic level each of them belongs to based on corrected forms and additional comments made by the labeler (Table As a whole, the most common type was grammar (481), but most of the grammatical errors (or cases of unnaturalness) were labeled with Comment 1, which implies that in most cases, the grammatical errors do not have a fatal influence making the entire output unintelligible. The second-most common type was lexical errors (or cases of unnaturalness) (407). Half of them were labeled with Comment 1, but 23 errors got Comment 2. This means that some errors can interfere with understanding. Discourse errors accounted for a fraction of a percent of all errors (65). However, compared with other types of errors, the percentage of Comment 2 was the highest (14 out of 65), which means that discourse errors can greatly interfere with the intelligibility of the entire output. The main difference between the discourse errors labeled with Comment 2 and those labeled with Comment 3 was that most of the latter related to collocational expressions, while the former involved non-collocational phrases where learners need to construct a phrase or sentence by combining single words. In the following sections, we examine the characteristics of each type of error (or cases of unnaturalness) in detail.</Paragraph> <Paragraph position="2"> Comment 1 Half of the Comment 1 errors were grammatical ones. Most of them were local errors such as subject-verb disagreement or article errors. There were 286 lexical errors, but in most cases, they were not very serious, for example lexical confusions among semantically similar vocabulary items.</Paragraph> <Paragraph position="3"> Comment 2 Most of the Comment 2 errors had something to do with lexis or discourse.</Paragraph> <Paragraph position="4"> 1) Too abrupt literary style (discourse error) ex) I've been to the restaurant is first. I took lunch. The curry the restaurant serves is very much, so I was surprised and I'm now a little sleepy.</Paragraph> <Paragraph position="5"> ex) T: How are you? L: I'm very fine.</Paragraph> <Paragraph position="6"> better barb2right I'm fine.</Paragraph> <Paragraph position="7"> 5) There are more appropriate words or expressions. (discourse/pragmatic-level of unnaturalness) ex) To go to high school in the mainland, I went out of the island.</Paragraph> <Paragraph position="8"> better barb2right ... I left the island.</Paragraph> </Section> <Section position="4" start_page="77" end_page="78" type="sub_section"> <SectionTitle> 5.3 Limitation of Current Error Annotation Scheme </SectionTitle> <Paragraph position="0"> It is obvious that discourse and some types of lexical errors can often impede the understanding of the entire utterance.</Paragraph> <Paragraph position="1"> Although our current error tagset does not cover discourse errors, it is still possible to &quot;just&quot; assign any one of error tags to the erroneous parts shown in 5.2. There are two reasons for this. One is that, in the current error tagging principle, it is possible to replace, add or delete all POS in order to make it possible to &quot;reconstruct&quot; an erroneous sentence into a correct one. The other reason is that since discourse structure is liked to grammatical and lexical selections, it is possible to translate terms for describing discourse into terms for describing grammar or lexis.</Paragraph> <Paragraph position="2"> However, annotating discourse errors with tags named with grammatical or lexical terms cannot represents the nature of discourse errors. Since discourse errors are often related to intelligibility of learners' outputs, describing those errors with appropriate terms is quite important for making the current error tagset something helpful for measuring learners' communicative competence. We will need to know what kind of discourse errors are made by learners, and classify them to build in the error tagset. Some parts labeled with Comment 3 were also related to discourse-level problems. It would be beneficial to provide learners with feedback such as &quot;Your English sounds unnatural because it's socio-linguistically inappropriate&quot;. Therefore, it is also necessary to classify discourse-level unnaturalness in learners language.</Paragraph> </Section> <Section position="5" start_page="78" end_page="78" type="sub_section"> <SectionTitle> 5.4 Works for Expansion to New Error Tagset </SectionTitle> <Paragraph position="0"> We decided the basic principles for revising the current error tagset as following.</Paragraph> <Paragraph position="1"> 1) Classify second language discourse errors and building them into a new error tagset.</Paragraph> <Paragraph position="2"> 2) Differentiate unnatural expressions from errors. Information on why it sounds unnatural will also be added.</Paragraph> <Paragraph position="3"> 3) Add information on linguistic level (morpheme, grammar, lexis, and discourse) to each tag.</Paragraph> <Paragraph position="4"> 4) Do a further survey on how we can differentiate errors that interfere with understanding and those that do not, and add information on error gravity to each tag.</Paragraph> <Paragraph position="5"> Classifying discourse errors will be the most important task in the tagset revision. In several studies, second language discourse has already been discussed (James, 1998), but there is no commonly recognized discourse error typology. Although grammatical and lexical errors can be classified based on the existing canonical grammar rules or POS system, in order to construct the discourse error typology, we will need to do more investigation into &quot;real&quot; samples of learners' discourse errors.</Paragraph> <Paragraph position="6"> Adding the information on linguistic level (morpheme, grammar, lexis and discourse) to each tag is also important. From the survey, we found that the linguistic level of errors is strongly related to the intelligibility of the entire output. If linguistic level information is added to each error tag, this might help to measure the intelligibility of learners' utterances, that is, learners' communicative competence.</Paragraph> </Section> </Section> class="xml-element"></Paper>