<?xml version="1.0" standalone="yes"?>
<Paper uid="C94-2105">
  <Title>Semantics WORD SENSE ACQUISITION FOR MULTILINGUAL TEXT INTERPRETATION *</Title>
  <Section position="6" start_page="668" end_page="669" type="evalu">
    <SectionTitle>
5. R,ESULTS
</SectionTitle>
    <Paragraph position="0"> I:igu.,c 5 shows I,he overall recall, l/recision a ud I&amp;quot;-IIIC~)SlII'(+ SCOI'(!S \['(:'1' SII()(~UN on t, lm E&gt;ur c,.)ltIigtiration o1' tile \[iual TIPS'I'I';II. (M tO(', 5) Iw.uchniark. I';:IV a.nd  JJV are the F, nglish and Japanese joint venture tests, and EME and JME are the two microelectronics test sets. R,ecall is the percentage of possible information that is correctly identified by the system. Precision is the percentage of information produced by the system that is correct. The F-measure is the geometric mean of recall and precision.</Paragraph>
    <Paragraph position="1">  Scores as low as 50 recall may appear low, and certainly leave room h)r improvement. A 50 recall measure means that the system only correctly recovered half of the possible information, on average, from each text. llowever, by t~ number of relative comparisons, these nmnbers are good. They are a significant improvement over previous benchmarks, and are close to (;he recall and precision scores of the GE system on nmch easier tests. The TIPSTER (,ask is quite difficult, wif, h trained human iiitelligence analysts often producing recall scores in the 70s.</Paragraph>
    <Paragraph position="2"> As we have pointed out, SHOGUN's recall was, on average, 37% higher than any other system in each configuration, although the precision was 13% lower than the system with the best precision in each toni(aural, ion. For example, the next best systeIrt in Fmglish joint ventm'es (l~aV) had 38 recall and 58 precis(old, and the next best system in Japanese .joint ventures (a different system) had 42 recall and 67 precision.</Paragraph>
    <Paragraph position="3"> Much of the difh.'rence in perff)rmance between SIIOGUN and other systems can be attributed to difticult portions of the task, where SHOGUN somethnes ln*d recall scores as rllllch as 3 or 4 tidies 3.8 high as other systems. The portions of the joint ventm:e template shown in Figure 3 are examples of' such components, l~ecause these were. the most knowledgeintensiw.' components of the task, we believe that the results validate SIIOG UN's approach to knowledge acquisition. Certainly the system had much better cover: age than other systems, and we attribute this result to the representation gild automation used in word sense interpretation.</Paragraph>
    <Paragraph position="4"> Figm:e 6 gives an inh)rmal analysis of the level and type ofeflbrt used ill each configuration. Although the Japanese scores were generally higher thtm English, the Japanese contigurations largely relied on the English knowledge development. The level of ell'or( for Japanese joint ventures was higher than English because the English system started out with nmch more than the aal&gt;anese system (for example, we ah:eady had a fairly well developed English nan\](? recognition eompoIlent). By contrast, the Japanese microelectronics configuration derived ahnost entirely fl:om the English, with ahnost no eflbrt required t}om ,lapanese speakers.</Paragraph>
    <Paragraph position="5"> Many other sites participated in the TIPSTF, I{, project and the MUC evahmtions, including two others \[Cowie and Pustqjovsky, 1993; Weischedel el al., 1993\] that covered both domains and both languages, and one ol, her \[Lehnert el al., 1993\] that ff)cused on lexical acquisition, although only in English. In addition,  l, here \]ms I)c(:n oliher signific,aut relidx~d work hi rolmst processing of I;exts, notM)ly \[llobbs cl al.&gt; 1992\]; how ever, this rc,semmh has gener~dly emphasize(l synt,;u%ic coverage rather th;m lcxical (:overage. Finally, rc,latcd researc,h in lexical acquisition \[Zernik, ;1991\] focuses on corm lexicnl resources r~tller I,|l;.tli on c,ustomizing the lexicon through the use of a rel)resentative corlms.</Paragraph>
    <Paragraph position="6"> I\]enc,e, the rese.;u:ch i;}iat we have t)reseni, ed has advanced the state of the a, rt \])()tit in t, he use of I;\]ie corpus I,o ideni, ify word sl!iise in|'ornw.l, ion ~il(I the denionstrn: Lion of lrutll,ilingua.\] CalmbililAos.</Paragraph>
  </Section>
class="xml-element"></Paper>