<?xml version="1.0" standalone="yes"?>
<Paper uid="W03-0208">
  <Title>Automatic Evaluation of Students' Answers using Syntactically Enhanced LSA</Title>
  <Section position="2" start_page="0" end_page="0" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> Computer based education systems are useful in distance learning as well as for class-room learning environment. These systems are based on intelligent tutoring systems(ITS's) which provide an interactive learning environment to students. These systems first familiarize a student with a topic and then ask questions to assess her knowledge. Automatic evaluation of students' answers is thus central to design of an ITS that can function without the need of continuous monitoring by a human. Examples of ITS's that use natural language processing to understand students' contribution are CIRC-SIM (Glass, 2001), Atlas (Freedman et al., 2000), PACT (Aleven et al., 2001) etc. These systems use a parser to derive various levels of syntactic and semantic information and rules to determine the next dialog move. They perform quite well with short answers in a limited domain, but are limited to take arbitrarily long free-text input and are difficult to port across domains. These limitations can be alleviated by using latent semantic analysis(LSA), a recently developed technique for information retrieval (Deerwester et al., 1990), knowledge representation (Landauer et al., 1998), natural language understanding and cognitive modeling (Graesser et al., 1999; Graesser et al., 2000) etc. LSA has been used in various ITS's like AutoTutor (Wiemer-Hastings et al., 1998), Intelligent Essay Assessor (Foltz et al., 1999), Summary Street (Kintsch et al., 2000), Apex (Dessus et al., 2000) etc.</Paragraph>
    <Paragraph position="1"> LSA is a statistical corpus-based natural language understanding technique that supports semantic similarity measurement between texts. Given a set of documents in the tutoring domain, LSA uses the frequency of occurrence of each word in each document to construct a word-document co-occurrence matrix. After preprocessing, singular value decomposition is performed to represent the domain knowledge into a 200 to 400 dimensional space. This space is then used for evaluating the semantic similarity between any two text units.</Paragraph>
    <Paragraph position="2"> In an ITS, LSA is used to evaluate students' answers with respect to the ideal answers to questions in the domain (Graesser et al., 2000). This is done by finding the match between a student's answer and the ideal answer by calculating the cosine similarity measure between their projections in LSA space. This information is used to provide interactive response to the student in terms of hint, prompt,question etc.</Paragraph>
    <Paragraph position="3"> It has been found that LSA performs as good as an intermediate expert human evaluator but not so well as an accomplished expert of the domain. This may be because LSA is a 'bag-of-words' approach and so lacks the word-order or syntactic information in a text document. But for correct automatic evaluation of students' answers, a model should consider both syntax and semantics in the answer. So, one obvious way to improve the performance of LSA is to incorporate some syntactic information in it.</Paragraph>
    <Paragraph position="4"> In order to add syntactic information to LSA, recently there has been an effort in (Wiemer-Hastings and Zipitria, 2001), where a word along with its part-of-speech (POS) tag was used to construct the LSA matrix, thus capturing multiple syntactic senses of a word. But this approach, called tagged LSA, deteriorated the performance. In another attempt (Wiemer-Hastings and Zipitria, 2001), similarity between two sentences was calculated by averaging the LSA based similarity of sub-sentence structures like noun phrase, verb phrase, object phrase etc. This approach, called as structured LSA (SLSA), could improve the performance in terms of sentence-pair similarity judgment. But its performance in terms of evaluating students' answers was poorer than that of LSA(Wiemer-Hastings, 2000).</Paragraph>
    <Paragraph position="5"> We propose here a model called Syntactically Enhanced LSA (SELSA), where we augment each word with the part-of-speech (POS) tag of the preceding word. Thus instead of word-document co-occurence matrix, we generate a matrix in which rows correspond to all possible word - POS tag combinations and columns correspond to documents. A preceding tag indicates some kind of syntactic neighbourhood around the focus word. Depending on the preceding tag, the syntactic-semantic sense of a word can vary. Thus SELSA captures finer resolution of syntactic-semantic information compared to mere semantics of LSA. This finer information can therefore be used to evaluate a student's answer more accurately than LSA.</Paragraph>
    <Paragraph position="6"> We compare the performance of SELSA with LSA for the AutoTutor cognitive modeling task (Graesser et al., 1999). This involves evaluating students' answers to questions in three areas of computer science viz. hardware, operating system and networking. The performance is measured in terms of various criteria like correlation, mean absolute difference and number of correct /emphvs false evaluations by humans and by computer.</Paragraph>
    <Paragraph position="7"> SELSA is found better than LSA in terms of robustness across thresholds as well as in terms of evaluating more answers correctly, but it is having less correlation measure with human than LSA.</Paragraph>
    <Paragraph position="8"> The organization of this paper is as follows. The next section describes LSA and its applications in ITS's. In section 3, we describe the proposed SELSA model. The experimental details are given in section 4 followed by discussion on results in section 5.</Paragraph>
  </Section>
class="xml-element"></Paper>