<?xml version="1.0" standalone="yes"?>
<Paper uid="N03-2033">
  <Title>Library and Information Studies, Queens</Title>
  <Section position="3" start_page="0" end_page="0" type="intro">
    <SectionTitle>
3 Methodology and Results
</SectionTitle>
    <Paragraph position="0"> Multiple research methods were used. Firstly, we conducted focus-group sessions to elicit key quality aspects from news analysts. Secondly, we performed experts and students quality judgment experimental sessions.</Paragraph>
    <Paragraph position="1"> Thirdly, we identified a set of textual features, ran programs to generate counts of the features, and performed statistical analysis to establish the correlation between features and users' quality ratings.</Paragraph>
    <Paragraph position="2"> Two focus group sessions were conducted during March and April of 2002. Participants included journalism faculty members, professional editors, and a number of journalists from a local newspaper Albany Times Union. Nine information quality criteria were considered to be salient to the context of news analysis: Accuracy, Source reliability, Objectivity, Depth, Author credibility, Readability, Conciseness, Grammatically Correctness, and Multiple Viewpoints.</Paragraph>
    <Paragraph position="3"> A computerized quality judgment system that incorporated the nine quality aspects was developed. One thousand medium-sized (100 to 2500 words) news articles were selected from the TREC collection (Voorhees, 2001) with 25 relevant documents each from five TREC Q&amp;A topics.</Paragraph>
    <Paragraph position="4"> We recruited expert and student participants for judgment experiments. Expert sessions were performed first and ten documents judged by experts were selected and used as the training and testing material for the student participants. The entire judgment experiment period ran from May to August of 2002. As a result, each of the 1,000 documents was rated twice, by two different judges, one at Albany, and one at Rutgers.</Paragraph>
    <Paragraph position="5"> There were high inter-judge agreements between Albany and Rutgers. Figure 1 is the normality plot of the difference between scores assigned by Rutgers' judges and Albany's judges on the variable of &amp;quot;accuracy,&amp;quot; with a mean almost equals to zero (with range from - 9 to + 9). The curves of the other eight quality variables are similar to the one below, indicating a very insignificant disagreement in judgments.</Paragraph>
    <Paragraph position="6">  judgments on the aspect of &amp;quot;Accuracy&amp;quot; Principle component analysis (PCA) revealed the same two components from Albany data as from Rutgers data. As shown in Figure 2, one component (the lower one) consists of &amp;quot;credibility&amp;quot;, &amp;quot;source reliability&amp;quot;, &amp;quot;accuracy&amp;quot;, &amp;quot;multi-view&amp;quot;, &amp;quot;depth&amp;quot;, and &amp;quot;objectivity.&amp;quot; The second component (the upper one) consists of &amp;quot;grammar&amp;quot;, &amp;quot;readability&amp;quot;, and &amp;quot;verbose and conciseness&amp;quot;. Together they explain 58% of the variance.  We recoded users' scores 1 to 5 as low and scores 6 to 10 as high. We split the 1,000 documents into two halves by random selection. In our training round the first half was used to estimate the parameters that would give best discriminant and logistic regression functions. In our testing round, we applied the functions to the other half to predict the quality criteria of the documents. null  half training and testing) by two methods We then employed stepwise discriminant analysis to select the dominant predictive variables from a range of 104 textual features. These features included elements of punctuations, special symbols, length of document segments, upper case, quotations, key terms, POS, and entities. Our further analysis suggested that certain text features are highly correlated with each of the nine aspects. null  At this point, we are able to produce good prediction of several aspects of information quality, including Depth, Objectivity, Multi-view, and Readability. The prediction testing and training for the remaining quality aspects are currently in progress. Tables 4 and 5 illustrate the results of training versus testing classification for the criteria of &amp;quot;objectivity&amp;quot; and &amp;quot;depth,&amp;quot; with ratings grouped into high and low categories.</Paragraph>
  </Section>
class="xml-element"></Paper>