<?xml version="1.0" standalone="yes"?>
<Paper uid="W97-0705">
  <Title>I I I I I I I I I I I I I I I I I</Title>
  <Section position="4" start_page="0" end_page="26" type="metho">
    <SectionTitle>
2 The FAN Protocol
</SectionTitle>
    <Paragraph position="0"> This protocol anus to assess the quahty of an abstract independently from the source text and the reformation It contains Assessment was therefore camed out by two jurors, who were not specialL~tS m the fields concerned, who read the 27 abstracts without hawng seen the source texts It takes approximately 10 minutes to assess an abstract The assessment grid contams 4 criteria, which are described below Crzterwn 1 Number of Anaphora Deprtved of Re.l~rents Given that an abstract Is created from sentences from the source text, It Is pusslble that a sentence contains an anaphora whose referent does not belong to the extracted sentence We must not forget that SERAPHIN does not detect referents, R detects what it considers to be indications of potentml anaphora wRhm sentence Pl, out of a closed hst Ot, thts, that, etc ), and then apphes a simple heunsm which revolves selecting the preceding sentence P0 which contains the potential anaphora On the one hand, thts heurnsm may prove to be msufficsent (no exphctt referent, the referent is located* m a sentence further up m the text), and on the other hand this heunsm is not apphed to sentence P0 (m order to avotd selec~ng sentences based on cnterta that are i this ~lll be the objectof another report  not &amp;quot;semantic&amp;quot;) We believe this crRenon to be a determining factor with regard to the legtbthty of the abstract, nevertheless the jurors raised several problems We will illustrate these via a number of examples Thus, m the following sentence, According to thls researcher, It ts a movement Which zs characterlsed by a deswe to go backwards, a deswe for a state m which the obstruction between subject and object no longer exists (Text N~) the anaphonc term is deprived of its referent but the legibility and coherency of the abstract are not really altered, because, m this textual context, the name of the researcher ~s not considered .to be an important piece of information Nevertheless, m order to restrict * the effects of interpretation, this case was considered to be an anaphora deprwed of a referent In the following sentence, . ..</Paragraph>
    <Paragraph position="1"> One could say that these first contacts between EDF and the resident created a precedent whtch was notvery favourable wath regard to estabhshmg peaceful relationships between EDF and the local populatton ln, mg near the hne (Text N*9) the potentially anaphonc term these may be interpreted as referring either to contacts described earher m the text, or to ' a chronological account which takes on meaning as one reads the entire text, and, m part, at the end of the sentence in question In the text under consideration, it Is the second interpretation which ts correct, but because the juror &amp;d not have access to the source text, we treated this case as an anaphora deprived of a referent Crlterron 2 Rupture of textual segments organtsed by Linear lntegratton Markers Various studies on textual lmgmsttcs (CharoUes 89, Adam, 90) have underlined the utdtty of locating lmgmstac markers m order to identify the dtscurswe organisataons which go beyond the sentence itself SERAPHIN identifies linear integration markers (MIL) from within a closed list (on the one hand, on the other ham~ firs@, secondly, etc ) m order to rebmld textual segments m the abstracts produced Thus, ff sentence P is selected, sentences Pi, P2, Pn which are linked to P via MIL's wdl also be selected This selection may fall, . either because the MILts not recogmsed (absent from the list,, elhpsts, spelling mistake, etc ), or because the maximum size of the abstract has been reached We shouldstress both the fact that the jurors can only detect ruptures in the textual segments, and not thetr completeness, and that argumentation connectors (such as zndeea~ furthermore, etc ) are not considered to be MIL's This decision was taken after a preliminary validation by ten or so readers, and was confirmed by the jurors SERAPHIN uses a special symbol \[ \] to show that two sentences are not adjacent m the source text, such as m the following example It ts easy to imagine the huge number of texts and wrztten documents that ts produced by a company hke EDF \[Y Of course, all &amp;quot;language productwn&amp;quot; may appear to come fi'om an abstract source, a umque source .(Text Ndeg6) This symbol avoids the problem of the reader mistakenly reconstructing argumentation chains Criterion 3 Presence of &amp;quot;tautologtcal&amp;quot; sentences :. A sentence is considered to be tautological, if the information It provides is completely independent of the source text, as m the following example Predicting the future is a difficult and uncertain exercise (Text Ndeg4) We were trying to detect abstracts whlch, although optimal from the point of view of the two criteria above, had merely been created from very general sentences, as ts often the case with certmn abstractsby authors In fact, this criterion is far too dependent on the knowledge that the reader has of the subject of the text, and the results of the protocol show that It Is not pertinent Crtterton 4 Legzbzhty of the abstract This crRermn, whose values are Very Back Mediocre, Gooa~ Very Good, Is an overall appreciation of the abstract Although they are highly subjectave, the &amp;quot;scores&amp;quot; given by the jurors varied very little, with just the two exceptions set out m table 1 below Juror Jl Juror J2 Text N deg 20 Mechocre Very Good Text Ndeg22 Good Very Bad .</Paragraph>
    <Paragraph position="2"> Table 1 &amp;quot;Dwergence of assessment between jurors Text Ndeg20 (TF1, Le Grand Bluff) Is a chronological description of the pnvatlsatmn of the television channel TF1 The successmn of events, the number of players revolved, metaphors such as &amp;quot;It ts at the foot of the wall that one judges the br:cldayer'; make the readmg of the abstract (and of the source text) difficult unless the reader has a good understanding of the subject concerned (exceptionally, juror 2 happened to know this subject well) Text N022 (Credlbdzty of command control systems concepts and tools) ~ a highly technical text outside the experience of the two jurors For the presentation of the results (Tables 2 and 3) we systematically chose the lowest &amp;quot;score&amp;quot; Interpreting the results of the FAN Protocol We cross-referenced the legibihty criterion with criteria 1 and 2 Contrary to our anginal hypothesis, there is no correlation between the two Indeed, the comments made by the two jurors show that overall readmg of the abstract * allows them to overcome any locahsed lack of understanding caused by the absence of anaphonc referents This conclusion must nevertheless be nuanced by the fact that there is a hmited number of mistakes m the abstracts that were analysed Assessmg abstracts simply on the basis of surface hngmstic mdlcators, and without calling upon the knowledge that the jurors may have of the subject concerned, remains a difficult problem \] Nommtakes \] Onemlstake \] Two mistakes \] Three mistakes \]</Paragraph>
    <Paragraph position="4"/>
  </Section>
  <Section position="5" start_page="26" end_page="27" type="metho">
    <SectionTitle>
3 The MLUCE Protocol
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="26" end_page="26" type="sub_section">
      <SectionTitle>
3.1 ObjecUves
</SectionTitle>
      <Paragraph position="0"> The arm of this protocol ts to enable potentml users to assess the quahty of automatic abstracts In this case, an abstract, and therefore its quahty, wdl not be defined m any absolute way, but rather m terms of the ways m which it can be used For example, tf a person is looking for the &amp;quot;proven&amp;quot; zdea wlthm a text, he wall need to understand the different stages of the argument, on the other hand, if he only wishes to observe any stmultaneous occurrences of two themes wRhm the same text, he no longer needs to have the arguments The assessment of quahty therefore depends on what one wishes to use the abstract for (Path et al, 1961) It ts therefore important to pre-def'me one or more uses of the abstract, and for each def'mRton to accurately measure the &amp;quot;dlstance&amp;quot; between the source text and its abstract We selected twoapphcatmns for automatlc abstracts winch were of partlcular interest to EDF * Apphcatton 1 the abstract is a tool which allows one to decide whether or not to read the source text * Apphcatwn 2 the abstract is a support for writing a synthesis of a written document The MLUCE protocol therefore alms at &amp;quot;measuring&amp;quot; how a given abstract meets these two oblectwes In sectmn 3 4, we set out the results on SERAPHIN, but this protocol was apphed, without much tmportance, to the RAFI system (Lehmam 95)</Paragraph>
    </Section>
    <Section position="2" start_page="26" end_page="26" type="sub_section">
      <SectionTitle>
3.2 Experiment procedure
</SectionTitle>
      <Paragraph position="0"> The procedure selected for assessmg a &amp;quot;summar\]smg&amp;quot; system has to be precise, complete and unambiguous, m order to - restrict the &amp;quot;reader effect&amp;quot; as much as possible - m other words, to hm\]t the vartatlon m assessmentbetween readers, due to then&amp;quot; different fields of experhse, different cultures or varying archetypes of abstracts,  - hmR the influence of the order m whtch the texts are read, - be.adapted to all types of text, and to take their differences rote account A pdot was requn.ed m order to adjust the procedure to the above requ~ments For the SERAPHIN assessment, we set up a jury of four readers - a qualified French language teacher, specmlLsed m teaching abstract and synthesis techmques, - a documentary researcher, working in a documentary umt, - two users, wRh different training backgrounds  For the first stage of the assessment, we gave each member of the jury seven texts, then. abstracts, and a hst of mstructlons explaining the approach to be used (the &amp;quot;jurors&amp;quot; each had different texts Each reader-assessor then had to  - read the documents m a pre-defined order (firstly, all the abstracts, then all the source texts), - fill m the reader's sheet (attached to each document) as he went along, - give Ins overall opmmn on the &amp;quot;comparison sheets&amp;quot;  provided for thts purpose The second stage of the assessment revolved analysing the sheets returned by the readers The-whole experiment (defmamn of procedure, and the actual assessment) lasted a total of eight months</Paragraph>
    </Section>
    <Section position="3" start_page="26" end_page="27" type="sub_section">
      <SectionTitle>
3.3 Criteria retained
</SectionTitle>
      <Paragraph position="0"> On the basis of the experiments camed out by Borko et al (1975), Edmunson (1969), Mathls et al (1973), Payne (1964) on the assessment of the quahty of &amp;quot;abstracts&amp;quot;, and m terms of the apphcat~ons defined (section 3 1), we set four criteria, and for each criterion we estabhshed the means of assessing it Apphcatlon I For the first apphcatton, the criteria defined in MLUCE are designed to assess the utdlty of the abstract as a statable dec~ston-makmg tool for the reader These criteria must allow one to judge whether the abstract contains the mformauon requwed to be able to decide whether or not to read the source text In order to do so, we wall say that the abstract must allow  abstract) which show the fields or natures of the texts sctenufic or techmcal, pohtlcal, sociological, polemical, general, prospective, retrospective, situational or state-of-the-art null * check the presence of the essentml Ideas Each reader underlines the ideas m text Tt which he feels to be essential, and checks that they are present m abstract R~ * avmd parasmc ideas Each reader highhghts sentences In R~ which should not be m RI, and the sentences m abstract Rt which are cut off from the context (essential ideas that have been cut short) Apphcauon 2 For the second application, the cntena defined m MLUCE * are designed to assess the utthty of the abstract as a support for writing a synthesis of a written document In order to do so, we will say that the abstract must allow us to * identify the fie!d or nature of the source text (criterion ldenUcal to application I) * check the presence of the essentml ideas (cntermn identical to apphcatmn 1) * hlghhght the logical lmkmg of ideas Each reader fills in two grids (one for the Source text and one for the abstract) m which the following argumentation links appear cause implying consequence, consequence ~mphes cause, proposmon of a solution, from particular to general, from general to pamcular, motivated juxtaposmon of facts, hstmg of facts, confrontatmn He then states whether the idea ~s &amp;quot;proven&amp;quot; in each of the documents he has read Finally, he assesses whether the abstract is clear, fmrly clear, not very clear or incomprehensible 3.4. Results on SERAPHIN and interpretation Identification of the subject (table 4) The texts submitted to the reader-assessors may cover several fields and be of several chfferent natures, which ts why the total number of texts shown m table 4 is greater than the number of texts studied (number stadled = 27) The categories hsted in table 4 were not explicitly defined to the jurors, ttts therefore possible that there is a certain amount of &amp;quot;subjectivity&amp;quot; m the categonsatlon of the texts Nevertheless, we supposed that each reader could lmphcitly and continuously m time divide the texts rote the categories proposed number of source texts for which  fmrly close to. 11 the text relatively 10 different from the text well away from 4 the text Table 5 Presence of the essential ideas Presence of essent*al ideas (table 5) The result of hlghhghtmg the &amp;quot;essential xdeas&amp;quot; m the source text, and of the reader marking the &amp;quot;parasmc ideas&amp;quot; that appear m the abstract, are grouped together m order to define a &amp;quot;proximity&amp;quot; mdlcator This mdieator ~ defined m the following way - we will define an abstract as being close to the text ff more than 75% of the sentences which make it up are among the essential ideas (highlighted) and less than 10% are parasmc ideas, -we will define an abstract as being fairly close to the text If between 50% and 75% of the sentences which make it up are among the essential ideas (highhghted) and less than 10% are parasmc Ideas, - we will define an abstract as being rela#vely different from the text If between 25% and 50% of the sentences which</Paragraph>
      <Paragraph position="2"> make it up are among the essential Ideas (hlghhghted) and less than i 0% are parasttlc ideas, * - we wzll define an abstract as being Well away from the text m all other cases Highhghtmg the logical sequence of arguments (table 6) We have supposed that a text was written, by hm author, m a precise mm (the &amp;quot;proven&amp;quot; idea) We *have ldenUfied 8 types of argumentaUon lmks which allowed the authors to construct their demonstratmn (rows m table 6) Like when identifying a field, the texts submRted to the jurors may link several types of argument, which is why the total number of texts shown m table 6 is greater than the number of texts studied (readers were asked, where necessary, to gwe detads of the order m which the different types appeared over the whole of the text)  Table 6 htghhghtmg the logical sequence of arguments Table 6 should be compared wzth table 4 Indeed, we have nouced, wtth regard to the texts studied, the absence m the abstract of the source of the argument that ,sm the original document Thus, a text whtch uses a gtven theory, leads to an abstract m whtch no theoretical base for the argument is given Thin might explain the bad performance of the &amp;quot;sctenttfic&amp;quot; or &amp;quot;prospective&amp;quot; texts and the sequences of &amp;quot;cause ~mphes consequence&amp;quot; or &amp;quot;consequence Imphes cause&amp;quot; types Quahty of the abstract (tables 7 and 8) After having determmed the field, the jurors noted the logical argumentatton sequencing, stated the &amp;quot;proven&amp;quot; tdea of the abstract, and filled m a grid m order to give their overall lmpress~on of the quahty of the abstract</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>