<?xml version="1.0" standalone="yes"?>
<Paper uid="C02-1071">
  <Title>Integrating Shallow Linguistic Processing into a Unication{based</Title>
  <Section position="6" start_page="56" end_page="56" type="evalu">
    <SectionTitle>
6 Experiments and Results
</SectionTitle>
    <Paragraph position="0"> The twoexperiments described in this section were used to evaluate the performance of the integratedsystembothw.r.t. ecientprocessingandrobustness. null In therstexperiment, ourgoalwastoperformacomparativestudyoftheprocessingtime null  ofourALEPgrammarbeforeandaftertheintegration of the PoS tagger and chunker. For this experiment, therefore, we required testing cases whichwere already fully covered byour grammar before the integration of the tagger and chunker. In this experiment, we used a subset of the test suites wehaveusedinthe LS{GRAMandtheMELISSAprojects.</Paragraph>
    <Paragraph position="1"> In the second experiment, our goal was to investigatetowhatextenttheALEPgrammar beneted from the default lexical entries in terms of robustness. In this experiment, we tested our system on test corpus whichwas  This information was not manually encoded, but it was extracted from the lexical resources developed in the project PAROLE (Melero and Villegas, 1998).</Paragraph>
    <Paragraph position="2"> selectedrandomly.</Paragraph>
    <Paragraph position="3">  a){ ExperimentA Toevaluate the eciency of the system, we dened twotestsuites and run them with our  ALEPgrammarbothbeforeandaftertheintegrationoftheshallowprocessingtools. null  The rst test suite included short instructivesentences or queries from the corpus of the MELISSA project  and sentences we selected from the dierent test suites wehave used for diagnosis and evaluation purposes in the LS{GRAM and the MELISSA projects.</Paragraph>
    <Paragraph position="4">  Test cases were selected according to: (i) the syntactic function of the chunk e.g. subject, complement and adjunct, for nominal chunks, complementandadjunct,foradjectivalchunks, etc.;; (ii) the position of the chunk in the sentence, and (iii) the category and the number ofnon{headelements. Thistestsuiteincluded  type ofsentential structures, weevaluated our system with much more complex sentences, showinga high interaction ofphenomena. For this, we used an article |from the newspaper  Test suites and corpora are the two tools traditionally used for evaluating and testing NLP systems. The main properties of test suites are: systematicity,control over data, exhaustivity, and inclusion of negative data. Test corpora, by contrast, reect naturally occurring date (cf. (Lehmann et al., 1996)).  Experiments have been run in a 128 Mb Ultra Sparc{ 10. Mean CPU time values were calculated for 50 samples. null  NL utterances which users made in interacting with ICAD, an administrative purchase and acquirement handling system, employed at ONCE (Organizacion Nacional de Ciegos de Espa~na), dealing with budget proposals and providing information to help decision makers.  These test suites are organized on the basis of a hierarchical classication of linguistic phenomena. Test suites including cases with interaction of phenomena and negative cases are also included.</Paragraph>
    <Paragraph position="5">  The reduction of the sentence length is due to the fact that elements that are wrapped together in a chunk by the preprocessing module are lifted to the parsing component of the grammar as a unique element. \ElDiarioVasco&amp;quot;|of250wordsfromtheLS{ GRAMcorpus.</Paragraph>
    <Paragraph position="6"> Two experiments have been carried on, rst byintegrating the PoS tags into ALEP and then the chunk mark{ups. For the rst experiment, the reduction of morphosyntactic ambiguity an average of 0.40 reduces the processingtimeoftheoverallprocessby45.9% (35.9% on average per sentence). For the secondexperiment,thesystemprocessingtime is reduced by52.6%(anaverageof 42.7% per sentence). Here, parsing speed{up is due to the fact that byintegrating chunk mark{ups, we do not only avoid generating irrelevant constituentsnotcontributingtothenalparse tree but wealsoprovide part of the structure thattheanalysiscomponenthastocompute.</Paragraph>
    <Paragraph position="7">  b){ ExperimentB Theevaluationoftheeectofdefaultlexical entries on the ALEP grammar was done with freeinputtext. Hereweuseda300wordarticle from\ElPais&amp;quot;(September2001).</Paragraph>
    <Paragraph position="8"> In running the second experimentwe observed that our rst approach ensured that the accuracy of the grammar |percentage of input sentences that received the correct analysis |remained the same, even though 67.7% of major words which appeared in the article was not encoded in the ALEP lexicon. The precision of the grammar|percentage of inputsentencesthatreceivednosuperuous(or wrong) analysis|, however, was be very low, wegotanaverageof8analysispersentence. By adding framinginformation tothe PoStagsof ourexternallexiconwereducedovergeneration uptoanaverageof2.5analysispersentence.</Paragraph>
    <Paragraph position="9"> Besides, our system provides structural robustness to the high{level processing. We observed that a number of linguistic structures which could not be handled by the grammar  A detailed analysis of the results showed us that, while in processing simple sentences, as the ones we included in the rst test suite, the most relevant factor for improving processing time was the reduction of the numberoftokens of the sentences, in processing complex sentential constructions, e.g. sentences included embedded clauses, eciency gains were mainly due to the reduction of the morphosyntactic ambiguity, since this drastically reduced the structural ambiguity.</Paragraph>
  </Section>
class="xml-element"></Paper>