<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-2920">
  <Title>CoNLL-X shared task on Multilingual Dependency Parsing</Title>
  <Section position="4" start_page="0" end_page="149" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> Previous CoNLL shared tasks focused on NP chunking (1999), general chunking (2000), clause identification (2001), named entity recognition (2002, 2003), and semantic role labeling (2004, 2005). This shared task on full (dependency) parsing is the logical next step. Parsing is an important preprocessing step for many NLP applications and therefore of considerable practical interest. It is a complex task and as it is not straightforwardly mappable to a &amp;quot;classical&amp;quot; segmentation, classification or sequence prediction problem, it also poses theoretical challenges to machine learning researchers.</Paragraph>
    <Paragraph position="1"> During the last decade, much research has been done on data-driven parsing and performance has increased steadily. For training these parsers, syntactically annotated corpora (treebanks) of thousands to tens of thousands of sentences are necessary; so initially, research has focused on English. During the last few years, however, treebanks for other languages have become available and some parsers have been applied to several different languages.</Paragraph>
    <Paragraph position="2"> See Section 2 for a more detailed overview of related previous research.</Paragraph>
    <Paragraph position="3"> So far, there has not been much comparison between different dependency parsers on exactly the same data sets (other than for English). One of the reasons is the lack of a de-facto standard for an evaluation metric (labeled or unlabeled, separate root accuracy?), for splitting the data into training and testing portions and, in the case of constituency tree-banks converted to dependency format, for this conversion. Another reason are the various annotation  schemes and logical data formats used by different treebanks, which make it tedious to apply a parser to many treebanks. We hope that this shared task will improve the situation by introducing a uniform approach to dependency parsing. See Section 3 for the detailed task definition and Section 4 for information about the conversion of all 13 treebanks.</Paragraph>
    <Paragraph position="4"> In this shared task, participants had two to three months3 to implement a parsing system that could be trained for all these languages and four days to parse unseen test data for each. 19 participant groups submitted parsed test data. Of these, all but one parsed all 12 required languages and 13 also parsed the optional Bulgarian data. A wide variety of parsing approaches were used: some are extensions of previously published approaches, others are new. See Section 5 for an overview.</Paragraph>
    <Paragraph position="5"> Systems were scored by computing the labeled attachment score (LAS), i.e. the percentage of &amp;quot;scoring&amp;quot; tokens for which the system had predicted the correct head and dependency label. Punctuation tokens were excluded from scoring. Results across languages and systems varied widely from 37.8% (worst score on Turkish) to 91.7% (best score on Japanese). See Section 6 for detailed results.</Paragraph>
    <Paragraph position="6"> However, variations are consistent enough to allow us to draw some general conclusions. Section 7 discusses the implications of the results and analyzes the remaining problems. Finally, Section 8 describes possible directions for future research.</Paragraph>
  </Section>
class="xml-element"></Paper>