<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-2802">
  <Title>Towards Measuring Scalability in Natural Language Understanding Tasks</Title>
  <Section position="7" start_page="0" end_page="0" type="concl">
    <SectionTitle>
6 Conclusion and Future Work
</SectionTitle>
    <Paragraph position="0"> We have discussed various measures for evaluating performance of individual components and systems and for estimating the corresponding task complexities. Additionally, we demonstrated the feasibility to employ an entropy-based metric for tasks that are heterogeneously structured in terms of their markable/attribute setup as well as attribute/value distribution. That means it can be applied to any corpora even if they feature disjunct attributes with different values of their set sizes. In a first study, on such a heterogeneous task, we have shown that the results of this generally applicable entropy-based metric line up correspondingly to increases and decreases in task difficulty.</Paragraph>
    <Paragraph position="1"> In our minds this metric for measuring task difficulty can now be employed to approach the question of measuring scalability. Since it is now feasible to manipulate task sizes and difficulties in a controlled and measurable fashion, future experiments and studies can be performed that do almost exactly the opposite from current evaluations of systems and components. That is, instead of keeping the task - test corpus - identical and measuring the performance of different methods, we can now keep the method identical and measure its performance on tasks differing in their difficulty. Hereby, some open question still have to be solved, such as evaluating and determining suitable performance measures and formalizing the specific dimensions of scalability that can be measured using this approach, e.g. scalability in terms of performance on problems that are equally difficult but vary in sizeversus problems that vary in size and difficulty to name a few.</Paragraph>
  </Section>
class="xml-element"></Paper>