<?xml version="1.0" standalone="yes"?>
<Paper uid="C96-1011">
  <Title>Unsupervised Learning of a Rule-based Spanish Part of Speech Tagger</Title>
  <Section position="4" start_page="54" end_page="56" type="intro">
    <SectionTitle>
3 Experiments and Results
</SectionTitle>
    <Paragraph position="0"> I&amp;quot;or training and testing of the tagger, we have randomly l)icked articles from a large (274MB) &amp;quot;H Norte&amp;quot; Mexican newspaper corl)uS, and sel)arated tlwm into the training and test s(+ts. The test set; (17,639 words) was t, ngged matmally for comparison agaittst the system-tagged texts. For training, wc partitioned the de, velopment set into sev(:ral dilt'erent-sized sets in order to st(: the elfeels of training corpus sizes. The 1)reakdown can  (+t+ each word in the test set, the accuracy is 78+0% (78.0% with the simple verh tag set). The awwage I'()S amhiguity per word is 1.52 (1.49) including t)unctuation tags arr(I 1.58 (1.56) excluding l)Unctuat, ion tags. For co,nparison, the accuracy of lh'ill's unsupervised English tagger was 95.1% using 120,000-word Penn Treel)ank texts. Ills initial state tagging accuracy was 90,7%, whictl is considerably higher than our Sl)a, ish case (78.6%:).</Paragraph>
    <Section position="1" start_page="54" end_page="55" type="sub_section">
      <SectionTitle>
3.1 Eth;ct of Tag Set
</SectionTitle>
      <Paragraph position="0"> Our tirst set of experiments tests the etDct of the I'()S tag eomt)lexity. We used both the Siml)le verl) tag set (5 tags) and the c, otnplex verb tag set (42 tags), which is shown in &amp;quot;l'~l)le 3, where * can be either IS(l, 2S(~, 3S(;, IPL, 2PI+, or 3PI+. In tim case of siml)le verb tag set, tense, person and numl)er information is discarded, leaving only a &amp;quot;V&amp;quot; tag and the lower four tags in the table.</Paragraph>
      <Paragraph position="1"> The scores witlr the siml)le verb tag set fur different sizes of training sets are found in Tabh~ 4, and those with the complex verb tag set in 'l'a ble 5. For these two experiments, (,he Learner was set to have a tight restriction on using context for learning (i.c, the freedom parameter was set to 1) and a loose restriction on context tbr applying the learned rules (i.e., l-lagfrecdom 10). q'he l{,ule Tagger was given a moderately-tight restrictiotb on using context for reduction rule application (i.e., r-lagJ'rccdom 2).</Paragraph>
      <Paragraph position="2"> In goner'M, the scores are slightly higher using the siml)le verb t~g set over the complex verb</Paragraph>
      <Paragraph position="4"> This rule was learned h~te in tile learning process when most I'/SU1KJONJ pairs had already been reduced, llowever, as olle Call see frolll t\]le COiltext of the rule, it will apply in a large number of eases in a text. The Rule Tagger notes this and applies the rule early, thus incorrectly changing many P/SUI~C()NJ pairs to SUBC()NJ and reducing the accuracy of t, he tagging. Since this phenomenon never occurred in any of the other learning rims, one can see that the learning pro eess can be heavily influenced by the choice of it, put texts.</Paragraph>
    </Section>
    <Section position="2" start_page="55" end_page="56" type="sub_section">
      <SectionTitle>
3.2 Effeet of Rule Application
Parameters
</SectionTitle>
      <Paragraph position="0"> The next tests performed involved using rules generated above and changing 1)arameters to the Rule 'l'agger to see how the scores wouhl be influenced.</Paragraph>
      <Paragraph position="1"> In the following test, we used tile simi)le verb tag set rules but varied the r-tagfrccdom parameter and the scq parameter. The results can be found in Table 6.</Paragraph>
      <Paragraph position="2"> tag set (91.8% vs. 90.3% for the &amp;quot;Medimn&amp;quot; corpus). This behavior is most likely due to the fact that, some verb tense/person/number combinations e~mnot easily be distinguished from context, so the Learner was unable to find a rtfle that would disambiguate them.</Paragraph>
      <Paragraph position="3"> As can be seen from the tables, performance increased as the size of the learning set incre, ased up to the &amp;quot;Medium&amp;quot; set, where the score levelled otf. With very small learning sets, the system was unable to tlnd sulticient examples of phenomena to produce reduction rules with good coverage.</Paragraph>
      <Paragraph position="4"> One surprising data t)oint in the simple verb tag set experiments was the &amp;quot;Full&amp;quot; score, which dropped Mmost 9% fi'om the &amp;quot;Medium&amp;quot; score. After analyzing the results more closely, it was found that the l,earner had learned a very spec, i\[ie rule regarding tile reduction of prel)osition/subordinate~-conjunction eombinations late in the learning process. The learned rule was:</Paragraph>
      <Paragraph position="6"> Although the wu'iations are slight, the best value for the r'-lagfl'c, edom l)arameter seems to be at an ambiguity level of 2. It seellls that the strategy of reducing the ambiguity as quickly as possible (best-rule-first) is better than following the ordering of the rules by the l,earner. This \[nay well be due to the fact that the ordering of the rules as produced by the Learner is dependent on the training texts. Since the test set was a differeat set of texts, the ordering of the rules was not as applicable to them as to the training texts, and so the tagging performance suffered.</Paragraph>
    </Section>
    <Section position="3" start_page="56" end_page="56" type="sub_section">
      <SectionTitle>
3.3 Etfe, rt (ff Hand+tagged Tex(;s
</SectionTitle>
      <Paragraph position="0"> Afl+-er ex+ttnining l+-h(; result;s fi'om l+-he aJ)(~v(~ expcr imcnl+-s, wc rea,lized l+-hal, sonm of (,he (:h~scd-cl;uss words in Spanish ;~re a, ltnosl, always amhiguous (e.g., preposil+-ions are usually ~unl~ig;uous bel,we(m  1 )I{EP a, nd S U B(:()NJ, a, nd del+-errnine, rs bel;we('n I) 1'3'1' a, ud 1' R()). This m('aus (;hal, l;h(~ l,ea, rncr will ?~,ever \[ea, rn a rule I+-o dismul)igu;tl,e l, hcs(~ clos(:dclass (:~+ses I)e(:+mse l, here will r;u'ely he ulmml)i,gttotis C()ll(;c:xl+-s ill I, he l+-raining I,ex(,s (,agge(\[ hy 1;11(' ini(;iaJ Si;al,e A lttlO(,&amp;(;or. 'l'ha(, is, un\]i\],m ()\[)('II-(:\[&amp;SS  w(,'ds, wc will no(, littd .cw ltJta,ntl)iguotts ch,s('d class words i, l+-exl+-s prccis(~ly I)(:(:;mse there is oil\[y a closed set; of t;hcm+ 'l'hus, wc decided I+-o illl, ro(bite a, st~la\[\] tltll\]t\])(:r of' \]la, lt(\[-(,~Lgg(x\] l;exl;s illl,o (,lie 1;l:a, ining sel; given (;o the l,earner. Since t, he }l;m(\[ t~tgg(;(\[ 1;exI;s \[l&amp;Ve ~C/corI'(~C(;&amp;quot; (~X&amp;III\[)I0S ()\[+ V,~LI'IOIlS l)h(:notn(',ua,, I, he l.eartmr s\]toul(l \])e a\])\[e (,C/) lin(I good exa, nq)les in t;h(,~+ I+-(, learn l'ro~+.</Paragraph>
      <Paragraph position="1"> For our t,esl+-s, wc (h~litmd four set,s o\[&amp;quot; hat.lt, agged texts t,h;U, wc a, dded t+()the &amp;quot;Sttmll&amp;quot; (306(~ wor(\[s) set, o\[' at~tbigu()us\[y l,aggcd l,exl,s. The  Again, (,he l,e~rner w~m ,'set l,o have a. l+-i,ghl, rcsl+-rici;ion on using cont, exl+- for h+arn\]ttg (,fr'ccdom l) ;m(I a loose restric(,ion on col|l+-ex(, \['or ;t.ppJyill,g l+-hc \],:,a, rn(;(\[ rul(;s (la.qJ}'ecdom, 10). 'l'h(&gt; I{,ulc Tagger wa, s giw~n a itlodera,l,ely-t, ighl+- rest, riot,ion on using (:OIl\[,(;Xl+- \['or rt':(lll('l,iotl rule a,i)l+lit:al, i()t~ (J:r'rcdom 2). The bcst-rul(',-Jir,sl mode of I, he l{mle Tagg(:r was Ilscd, The resull,~, ~s shown iu Table 8, a+rc sligi~l, ly belA,('r l, han wh(;n using only ;m~l)igttously Lagged l+-eXl,S, It is inl;eresl, it~g I+-o note tl\];d, l,\]m higher ~-tc(:tu'a, cy w:-ts achieved wit, h fewer ruh'.s. Itl fact, ;d\[ expe, rimcnl,s resull;ed iu \[ea, rnhtg a lil,l+-h' (~ver  wa, l|l+-(xI l,o knuw i\[&amp;quot; (,he itfl;r()(lucl, ion o\[' ha, rid-ULgg;cd texts into tim &amp;quot;Full&amp;quot; aud~iguo,.tsly 1,a~g,.'d set would improw~ il,s r,M;h.er low score (or. 'l'ahie 4). Wc performed an experilJtcnl, using sitnplc w~rb tags, the &amp;quot;l,'ull&amp;quot; ambiguously tagged text;s, ~md the &amp;quot;Full&amp;quot; ha, nd-t;agged l+-exts. Tim resu\[l+-s were d22 rules learned with :-~ score of 92.1%, which tied with (,he &amp;quot;Sm'MI&amp;quot; ambiguously l,agged set, for achieving l,he highest, .,tccura.('y of all o\[&amp;quot; the lem',i,g/ta,ggine; runs, a~ full 13.5% higher than using ,o I,.~;-u'nittg.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>