<?xml version="1.0" standalone="yes"?>
<Paper uid="W03-1719">
  <Title>The First International Chinese Word Segmentation Bakeoff</Title>
  <Section position="6" start_page="0" end_page="0" type="concl">
    <SectionTitle>
6 Summary and recommendations
</SectionTitle>
    <Paragraph position="0"> We feel that this First International Chinese Word Segmentation Bakeoff has been useful in that it has provided us with a good sense of the range of performance of various systems, both from academic and industrial institutions. There is clearly no single best system, insofar as there is no system that con- null sistently outperformed all the others on all tracks. Even if there were, the most one could say is that for the four different segmentation standards and associated corpora, this particular system outperformed the others: But there could be no implication that said system would be the most appropriate for all applications.</Paragraph>
    <Paragraph position="1"> One thing that we have not explicitly discussed in this paper is which type of approach shows the most promise, given the different submissions. While we are familiar with the approaches taken in several of the tested systems, we leave it up to the individual participants to describe their approaches and hopefully elucidate which aspects of their approaches are most responsible for their successes and failures; the participants' papers all appear in this volume. We leave it up to the research community as a whole to decide whether one approach or another shows most promise.</Paragraph>
    <Paragraph position="2"> We believe that there should be future competitions of this kind, possibly not every year, but certainly every couple of years and we have some specific recommendations on how things might be improved in such future competitions: 1. It may be a good idea to insist that all participants participate in all tracks, subject of course to the restriction that participants may not be evaluated on data from their own institution.</Paragraph>
    <Paragraph position="3"> The decision this time to let people pick and choose was motivated in part by the concern that if we insisted that people participate in all tracks, some participants might be less inclined to participate. It was also motivated in part by the different Chinese coding schemes used by the various corpora, and the possibility that someone's system might work on one coding scheme, but not the other.</Paragraph>
    <Paragraph position="4"> However with sufficient planning, perhaps giving people a longer period of time for training their systems than was possible with this contest, it should be possible to impose this restriction without scaring away potential participants. null 2. We would like to see more testing data developed for the next bakeoff. While the test sets turned out to be large enough to measure significant differences between systems in most cases, a larger test set would allow even better statistics. In some cases, more training data will also be needed.</Paragraph>
    <Paragraph position="5"> Given the problems noted by some of the participants with some of the data, we would also  like to see more consistently annotated training and test data, and test data that is more representative of what was seen in the training data. 3. We would like to expand the testing data to include texts of various lengths, particularly short strings, in order to emulate query strings seen in commercial search engines.</Paragraph>
    <Paragraph position="6"> 4. Finally, one question that we did not ask that should have been asked was whether the tested system is used as part of a commercial product or not. It is often believed of natural language and speech applications that deployed commercial systems are about a generation behind the systems being developed in research laboratories. It would be interesting to know if this is true in the domain of Chinese word segmentation, which should be possible to find out if we get a good balance of both.</Paragraph>
    <Paragraph position="7"> For the present, we will make the training and test data for the bakeoff available via http://www.</Paragraph>
    <Paragraph position="8"> sighan.org/bakeoff2003 (subject to the restrictions of the content providers), so that others can better study the results of this contest.</Paragraph>
  </Section>
class="xml-element"></Paper>