<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-0832">
  <Title>SENSEVAL Automatic Labeling of Semantic Roles using Maximum Entropy Models</Title>
  <Section position="4" start_page="5" end_page="5" type="metho">
    <SectionTitle>
3 Results
</SectionTitle>
    <Paragraph position="0"> SensEval-3 provides the following data set: training set (24,558 sentences/ 51,323 frame elements/ 40 frames), and test set (8,002 sentences/ 16,279 frame elements/ 40 frames). We submit two sets to SensEval-3, one (test A) is the output of all above processes (identifying frame elements and tagging them given a sentence), and the other (test B) is to tag semantic roles given frame elements.</Paragraph>
    <Paragraph position="1"> For test B, we attempt the role classification for all frame elements including frame elements not matching the parse tree constituents. Although there are frame elements that have two different semantic roles, we ignore those cases and assign one semantic role per frame element. This explains why test B shows 99% attempted frame elements. The attempted number for test A is the number of frame elements identified by our system.</Paragraph>
    <Paragraph position="2"> Table 4 shows the official scores for these tests.</Paragraph>
    <Paragraph position="3">  In the official evaluation, the precision and recall are calculated by counting correct roles that overlap even in only one word with the reference set. Overlap score shows how much of an actual FE is identified as an FE not penalizing wrongly identified part. Since this evaluation is so lenient, we perform another evaluation to check exact matches.</Paragraph>
  </Section>
class="xml-element"></Paper>