<?xml version="1.0" standalone="yes"?>
<Paper uid="W05-1520">
  <Title>Statistical Shallow Semantic Parsing despite Little Training Data</Title>
  <Section position="3" start_page="186" end_page="186" type="intro">
    <SectionTitle>
3 Experiments and Results
</SectionTitle>
    <Paragraph position="0"> We train all our systems on a training set of 477 sentence-frame pairs. The systems are then tested on an unseen test set of 50 sentences. For the test sentences, the system generated frames are compared against the manually built gold standard frames, and Precision, Recall and F-scores are calculated for each frame.</Paragraph>
    <Paragraph position="1"> Table 1 shows the average Precision, Recall and F-scores of the different systems for the 50 test sentences: Voting based (Voting), Maximum Entropy based (ME), Support Vector Machine based (SVM), Language Model based with unigrams (LM1) and Language Model based with trigrams (LM2). The F-scores show that the LM2 system performs the best though the system scores in general for all the systems are very close. To test the statistical significance of these scores, we conduct a two-tailed paired Student's t test (Manning and Schtze, 1999) on the F-scores of these systems for the 50 test cases. The test shows that there is no statistically significant difference in their performances.</Paragraph>
  </Section>
class="xml-element"></Paper>