<?xml version="1.0" standalone="yes"?>
<Paper uid="W05-0201">
  <Title>Applications of Lexical Information for Algorithmically Composing Multiple-Choice Cloze Items</Title>
  <Section position="2" start_page="0" end_page="1" type="abstr">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> Computer-assisted item generation (CAIG) allows the creation of large-scale item banks, and has attracted active study in the past decade (Deane and Sheehan, 2003; Irvine and Kyllonen, 2002). Applying techniques for natural language processing (NLP), CAIG offers the possibility of creating a large number of items of different challenging levels, thereby paving a way to make computers more adaptive to students of different competence. Moreover, with the proliferation of Web contents, one may search and sift online text files for candidate sentences, and come up with a list of candidate cloze 1A portion of results reported in this paper will be expanded in (Liu et al., 2005; Huang et al., 2005).</Paragraph>
    <Paragraph position="1"> items economically. This unleashes the topics of the test items from being confined by item creators' personal interests.</Paragraph>
    <Paragraph position="2"> NLP techniques serve to generate multiple-choice cloze items in different ways. (For brevity, we use cloze items or items for multiple-choice cloze items henceforth.) One may create sentences from scratch by applying template-based methods (Dennis et al., 2002) or more complex methods based on some pre-determined principles (Deane and Sheehan, 2003).</Paragraph>
    <Paragraph position="3"> Others may take existing sentences from a corpus, and select those that meet the criteria for becoming test items. The former approach provides specific and potentially well-controlled test items at the costs of more complex systems than the latter, e.g., (Sheehan et al., 2003). Nevertheless, as the Web provides ample text files at our disposal, we may filter the text sources stringently for obtaining candidate test items of higher quality. Administrators can then select really usable items from these candidates at a relatively lower cost.</Paragraph>
    <Paragraph position="4"> Some researchers have already applied NLP techniques to the generation of sentences for multiple-choice cloze items. Stevens (1991) employs the concepts of concordance and collocation for generating items with general corpora. Coniam (1997) relies on factors such as word frequencies in a tagged corpus for creating test items of particular types.</Paragraph>
    <Paragraph position="5"> There are other advanced NLP techniques that may help to create test items of higher quality. For instance, many words in English may carry multiple senses, and test administrators usually want to test a particular usage of the word in an item. In this case, blindly applying a keyword matching method, such as a concordancer, may lead us to a list of irrelevant sentences that would demand a lot of postprocess- null ing workload. In addition, composing a cloze item requires not just a useful sentence.</Paragraph>
    <Paragraph position="6"> Figure 1 shows a multiple-choice item, where we call the sentence with a gap the stem, the answer to the gap the key, and the other choices the distractors. Given a sentence, we still need distractors for a multiple-choice item. The selection of distractors affects the item facility and item discrimination of the cloze items (Poel and Weatherly, 1997). Therefore, the selection of distractors calls for deliberate strategies, and simple considerations alone, such as word frequencies, may not satisfy the demands.</Paragraph>
    <Paragraph position="7"> To remedy these shortcomings, we employ the techniques for word sense disambiguation (WSD) for choosing sentences in which the keys carries specific senses, and utilize the techniques for computing collocations (Manning and Sch&amp;quot;utze, 1999) for selecting distractors. Results of empirical evaluation show that our methods could create items of satisfactory quality, and we have actually used the generated cloze items in freshmen-level English classes.</Paragraph>
    <Paragraph position="8"> For broadening the formats of cloze items, we also design software that assists teachers to create listening cloze items. After we defining a metric for measuring similarity between pronunciations of words, our system could choose distractors for listening cloze items. This addition opens a door to offering different challenging levels of cloze items.</Paragraph>
    <Paragraph position="9"> We sketch the flow of the item generation process in Section 2, and explain the preparation of the source corpus in Section 3. In Section 4, we elaborate on the application of WSD to selecting sentences for cloze items, and, in Section 5, we delve into the application of collocations to distractor generation. Results of evaluating the created reading cloze items are presented in Section 6. We then outline methods for creating listening cloze items in Section 7 before making some concluding remarks.</Paragraph>
  </Section>
class="xml-element"></Paper>