<?xml version="1.0" standalone="yes"?>
<Paper uid="N04-1009">
  <Title>A Probabilistic Rasch Analysis of Question Answering Evaluations Rense Lange Integrated Knowledge Systems</Title>
  <Section position="2" start_page="0" end_page="0" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> For a number of years, objective evaluation of state-of-the-art computational systems on realistic language processing tasks has been a driving force in the advance of Human Language Technology (HLT). Often, such evaluations are based on the use of simple sum-scores (i.e., the number of correct answers) and derivatives thereof (e.g., percentages), or on ad-hoc ways to rank or order system responses according to their correctness.</Paragraph>
    <Paragraph position="1"> Unfortunately, research in other areas indicates that such approaches rarely yield a cumulative body of knowledge, thereby complicating theory formation and practical decision making alike. In fact, although it is often taken for granted that sums or percentages adequately reflect systems' performance, this assumption does not agree with many models currently used in educational testing (cf., Hambleton and Swaminathan, 1985; Stout, 2002). To address this situation, we present the use of Rasch (1960/1980) measurement to the HLT research community, in general, and to the Question Answering (QA) research community, in particular.</Paragraph>
    <Paragraph position="2"> Rasch measurement has evolved over the last forty years to rigorously quantify performance aspects in such diverse areas as educational testing, cognitive development, moral judgment, eating disorders (see e.g., Bond and Fox, 2001), as well as olfactory screening for Alzheimer's disease (Lange et al., 2002) and model glider competitions (Lange, 2003). In each case, the major contribution of Rasch measurement is to decompose performance into two additive sources: the difficulty of the task and the ability of the person or system performing this task. While Rasch measurement is new to the evaluation of the performance of HLT systems, we intend to demonstrate that this approach applies here as well, and that it potentially provides significant advantages over traditional evaluation approaches.</Paragraph>
    <Paragraph position="3"> Our principal theoretical argument in favor of Rasch modeling is that the decomposition of performance into task difficulty and system ability creates the potential for formulating detailed and testable hypotheses in other areas of language technology. For QA, the existence of a well-defined, precise, mathematical formulation of question difficulty and system ability can provide the basis for the study of the dimensions inherent in the answering task, the formal characterization of questions, and the methodical analysis of the strengths and weaknesses of competing algorithmic approaches.</Paragraph>
    <Paragraph position="4"> As Bond and Fox (2001, p. 3) explain: &amp;quot;The goal is to create abstractions that transcend the raw data, just as in the physical sciences, so that inferences can be made about constructs rather than mere descriptions about raw data.&amp;quot; Researchers are then in a position to formulate initial theories, validate the consequences of theories on real data, refine theories in light of empirical data, and follow up with revised experimentation in a dialectic process that forms the essence of scientific discovery.</Paragraph>
    <Paragraph position="5"> Rasch modeling offers a number of direct practical advantages as well. Among these are:  * Quantification of question difficulty and system ability on a single scale with a common metric.</Paragraph>
    <Paragraph position="6"> * Support for the creation of tailor-made questions and the compilation of questions that suit well-defined evaluation objectives.</Paragraph>
    <Paragraph position="7"> * Equating (calibration) of distinct question corpora so that systems participating in distinct evaluation cycles can be directly compared.</Paragraph>
    <Paragraph position="8"> * Assessment of the degree to which independent evaluations assess the same system abilities.</Paragraph>
    <Paragraph position="9"> * Availability of rigorous statistical techniques for the following: - analysis of fit of the data produced from systems' performance to the Rasch modeling assumptions; - identification of individual systems whose performance behavior does not conform to the performance patterns of the population as a whole; - identification of individual test questions that appear to be testing facets distinct from those evaluated by the test as a whole; - assessment of the reliability of the test - that is,  the degree to which we can expect estimates of systems' abilities to be replicated if these systems are given another test of equivalent questions; - identification of unmodeled sources of variation in the data through a variety of methods, including bias tests and analysis of residual terms.</Paragraph>
    <Paragraph position="10"> The remainder of the paper is organized as follows.</Paragraph>
    <Paragraph position="11"> First, we present in section 2 the basic concepts of Rasch modeling. We continue in section 3 with an application of Rasch modeling to the data resulting from the QA track of the 2002 Text REtrieval Conference (TREC) competition. We fit the model to the data, analyze the resulting fit, and demonstrate some of the benefits that can be derived from this approach. In section 4 we present simulation results on test equating. Finally, we conclude with a summary of our findings and present ideas for continuing research into the application of Rasch models to technology development and scientific theory formation in the various fields of human language processing.</Paragraph>
  </Section>
class="xml-element"></Paper>