<?xml version="1.0" standalone="yes"?>
<Paper uid="C94-1042">
  <Title>Comlex Syntax: Building a Computational Lexicon</Title>
  <Section position="5" start_page="269" end_page="271" type="evalu">
    <SectionTitle>
4 Types and Sources of Error
</SectionTitle>
    <Paragraph position="0"> As \])art of the process of refining the dictionary and assuring its quality, we have spent considerable resources on reviewing dictionary entries and on occasion have had sections coded by two or even four of the elves.</Paragraph>
    <Paragraph position="1"> This process has allowed us to make some analysis of the sources and types of error in the lexicon, and how these errors might be reduced. We. can divide the sources of error and inconsistency into four classes: 1. errors of classification: where an instance of a word is improperly analyzed, and in particular where the words following a verb are not properly identified with regard to complement type. SI)eeific types of problems include misclassifying adjuncts as arguments (or vice versa) and identifying the wrong control features. Our primary defenses against such errors have been a steady refinement of tile feature deseril)tions in ollr nlanlla\] and rel';ular grou I) review sessions with all the elves. Ill particular, we have developed detailed criteria for making adjunct/argument distinctiolis \[O\].</Paragraph>
    <Paragraph position="2"> A 1)reliminary study, conducted on examples (drawn at random from a corpus not used for our concordance) of verbs beginning with &amp;quot;j&amp;quot;, indicated that elves were consistent 93% to 94% of the time in labeling argument/adjunct distinctions following our criteria and, in these eases, rarely disagreed on the subcategorization. In more than half of the cases where there was disagreethen(, the elves separately flagged these as drillcult, ambiguous, or figurative uses of the verbs (and therefore would probably not use them its the basis for assigning lexical features). The agreement rate for examl)les whicti were not flagged was 96% to 98%.</Paragraph>
    <Paragraph position="3"> 2. oniitted features: where an ell' omits a Dature because it is not suggested by an example in the concordance, a citation ill the dictionary, or the elf's introspection. In order to get an est.ilnate of the niag,itude of this problem we decided to establish a measure of coverage or &amp;quot;recall&amp;quot; for the subcategorization Dal.ures assigned by our elves.</Paragraph>
    <Paragraph position="4"> &amp;quot;lb do this, we tagged the first 150 &amp;quot;j&amp;quot; verbs from a randomly selected corpus from a part of the San Diego Mercury which was not inchlded in our concordance and then compared the dictionary entries created by our lexicographers against the tagged eorptis. The restllts of this colnparison are sliown in Figure 3.</Paragraph>
    <Paragraph position="5"> ~Phe &amp;quot;(~omplements only&amp;quot; is the percentage of instances in the corpus covered by the subcategorization tags assigned by the elves and does not include the identification of i~rly l)rel)ositions or adverbs. 'l'lie &amp;quot;(~oinl~lements only&amp;quot; would correspond rougllly to the type of inforinal, ion provided by OALI) and l,I.)()(\]Jl'\] 4. The &amp;quot;COlllpielnc:nl,s ql&gt;relmsitions/l)articles&amp;quot; colliirin inehides eli the fl,al.ures&gt; tllal, is it, eonsidel'S the correct idenl,illcation of the conip\]einent l)\]ilS the sp,~cilie preposil.ions aiid adverbs i't!(lllil'e(\] by eert~thi compleillonl.s. Ttie two COlllliiliS of (igiii'es iUlder &amp;quot;Ci)nil)lenients-t-I&gt;rel&gt;ositions/l'ari.icles ', show tim resuits with and without the enumeration of dh'oetional l)reposltlons.</Paragraph>
    <Paragraph position="6"> We have recently changed ollr approach to i, he classification of verbs (like &amp;quot;riin '&gt;' &amp;quot;send &gt;&gt;, &amp;quot;jog '&gt;, &amp;quot;wall:', &amp;quot;juml;') wliieh take a long list of directional l)rel)ositlons, by l)roviding our entering prograin with a P-D/I{ option on the preposition llst.</Paragraph>
    <Paragraph position="7"> 'l'his option will automatically assign a list of directional prepositions to the verb and thus will saw.' tirne and eliminate errors of rriissing prepositions. In some eases this apl)roaeli will provide 4 I~I)OCI~ does provide some preposltloiis and particles.</Paragraph>
    <Paragraph position="8">  a prel)osition list that is a little rich for a given verb I)ut we have decided to err on the side of a slight overgeneration rather thall risk missing ally prel)ositions which actually occur. As you can see, the removal of the ILl)IlLs from consideration improves the in(lividual elf scores.</Paragraph>
    <Paragraph position="9"> The elf union score is the union, of the lexical entries for all fcmr elves. These are certainly nulnbets to be proud of, but realistically, having the verbs clone four sel)arate times is not I)ractical.</Paragraph>
    <Paragraph position="10"> llowew~r, in our original proposal we stated that because of the complexity of the verb entries we wouhl like to have them done twice. As can be seeil in l:igure 5, with two passes we su('ce,,d hi raising individual percentages in all cases.</Paragraph>
    <Paragraph position="11"> We would like to make clear that evell in tim two cases where our individuM lexicographers miss 18% and 13% of the complements, there was only one instance in which this might have resulted in the inability to parse a sentence. This was a missing intransitNe. Otherwise, the missed cOnll)lernents wouhl have been analyzed as adjuncts since they were a combination of prepositional phrases and adverbials with one case of a suhordinal.e ccmj line(ion ~as&amp;quot;.</Paragraph>
    <Paragraph position="12"> We endeavored to make a comparison with LDOCE on the measurement. This was a bit difficult since LDOCE lacks some con,plements we have and combines others, not always consistently.</Paragraph>
    <Paragraph position="13"> For instance, our PP roughly corresponds to either  1,9 (our Pl'/al)Vl') or l)rep/adv + T1 (e.g. &amp;quot;on&amp;quot; 3.</Paragraph>
    <Paragraph position="14"> + TI) (our I'I'/I'AI{;F-NP) but in some cases a  I)relmsition is mentioned but the verb is classified as iIltr~tllSii, iVe. 'Hie stra, ightl(Fw;u'd colnparisor~ has I,I)O(~E (illdhlg 7;~(~t) of f,he Lagged COml)lemenl.s hut a softer measure eliminating complements that I,I)OC, E seems to 1)e lacking (I'AILT-NP-I'P, ILPOSSIN(I, PP-Pl ~) aM Mlowing for a 1)P coral)lenient for &amp;quot;joke&amp;quot;, although it is no(. specitied, results ill a l'ml'celfl.ag(; of 79.</Paragraph>
    <Paragraph position="15"> We haw~ adOld.ed tw. lines of defense against the prohh!m of omitted features, l&amp;quot;irsg, critical entries (particularly high fre(luency wM)s) have been done independently by two or more elves. Second, we are dewq.pinp; a IIIO1'(~ balanced C(H'plIH for t\]lo dv,,s to ,,,~,ilt. i~)~c,.,,,, st,,di~,s (e4';., \[1\]) co,lh.,,, our {)\])serv;ttions I,h:d, I'(':d, ures SllCh as sub('ategorizati()n patterlls luay di\[i'l!r sui)stantiatly betweell corpora. We began with a corpus f,'ol,~ a single newspaper (S:m .lose Mercury News), but lutve since added the Ih'own corpus, several literary works from the l,ibrary of America, scientific abstracts fl'oul the U.S. I)epartment of Energy, aml ;ill additional newspal~er (the Wall ~treet Jnur-Iiztl). In ,~xl, ending the corpus, we h&amp;ve limited mlrselves to texts which would lie readily awdlable I,o nlenlliers of the l,inguistic I)ata Consortium.</Paragraph>
    <Paragraph position="16"> exl:e.ss ft~atul'es: when ;m elf assigns a spurious feature throu.gh incorrect extrapolation or analo.gy from available examples or introspection. Because of our desire Io obtain relatively complete foatllre sets, even figr infrequent verl)s, we have pernfit- null ted elves to extrapolate from the citations fotmd.</Paragraph>
    <Paragraph position="17"> Such a process is bound to be less certain than the assignment of features from extant examples.</Paragraph>
    <Paragraph position="18"> Ilowever, this problem does not appear to be very severe. A review of tile &amp;quot;j&amp;quot; verb entries produced by all four elves indicates that the fraction of spurious entries ranges from 2% to 6%.</Paragraph>
    <Paragraph position="19"> d. fl~zzy features: feature assignment is defined in terms of the acceptability of words in particular syntactic frames. Acceptability, however, is often not absolute but a matter of degree. A verb may occur primarily with particular complements, but will be &amp;quot;acceptable&amp;quot; with others.</Paragraph>
    <Paragraph position="20"> This problem is eompmmded by words which take on particular features only in special contexts.</Paragraph>
    <Paragraph position="21"> Thus, we don't ordinarily think of &amp;quot;dead&amp;quot; as being gradable (*&amp;quot;Fred is more dead than Mary.&amp;quot;), but we do say &amp;quot;deader than a door nail&amp;quot;. It is also compounded by our decision not to make sense distinctions initially. For examl)le, many words which are countable (require a determiner before the singular form) also have a generic sense in which the determiner is not required (*&amp;quot;Fred bought apple.&amp;quot; but &amp;quot;Apple is a wonderflfl flavor.&amp;quot;). For each such problematic feature we have prepared gnidelines for the elves, but these still require considerable discretion on their part.</Paragraph>
    <Paragraph position="22"> 'Fhese problems have emphasized for ns tbe impof tanee of developing a tagged corpus in conjunction with the dictionary, so that frequency of occurrence of a feature (and frequency by text type) will be available. We have done stone preliminary tagging in paraim with the completion of our initial dictionary. We expect to start tagging in earnest in early summmer.</Paragraph>
    <Paragraph position="23"> Our plan is to begin by tagging verbs in the Brown corpus, in order to be able to correlate our tagging with the word sense tagging being done by tim Word-Net group on the same corpus \[7\]. We expect to tag at least 25 instances of each verb. If there are not enough occurrences in tim Brown Corlms , we will use examples from the same sources as our extended corpus (s~e above).</Paragraph>
  </Section>
class="xml-element"></Paper>