<?xml version="1.0" standalone="yes"?>
<Paper uid="W02-1201">
  <Title>A Study in Urdu Corpus Construction</Title>
  <Section position="3" start_page="0" end_page="0" type="intro">
    <SectionTitle>
2 XML
</SectionTitle>
    <Paragraph position="0"> The natural choice these days for storing a corpus is in an XML format. An XML format provides needed standardization so that a user who is unfamiliar with the corpus data, but familiar with a given XML DTD, can interface with the corpus fairly efficiently. At its best, software that has been previously designed to handle a corpus marked up in a given XML structure can handle a new corpus marked up in the same structure. This is advantageous because someone does not have to comb through the new corpus trying to understand its design in order to redesign the software that interfaces with the corpus. The designer of a corpus is always familiar with his/her own design, so one advantage of using an XML language to mark up a corpus is to make the corpus readily available to other researchers.</Paragraph>
    <Paragraph position="1"> We chose the Corpus Encoding Standard (CES) XML DTD to mark up our corpus [2]. The main enclosing tag in this DTD is &lt;cesCorpus&gt; which is broken into main parts, &lt;cesHeader&gt; and &lt;cesDoc&gt;.</Paragraph>
    <Paragraph position="2"> The header &lt;cesHeader&gt; contains meta information about the corpus data such as, date created, creator's name and contact information, description of the source, categories of the content, the writing system of the language being stored, how hyphenation in the source text is handled, and much more information (Figure 1).</Paragraph>
    <Paragraph position="3"> The document tag &lt;cesDoc&gt; is where the actual text of the language of interest is stored. Each document is itself marked up with metadata specific to each document, like topic and source information for every separate document in the corpus. null The language data inside the &lt;cesDoc&gt; tags can be marked up simply with a paragraph tag &lt;p&gt; (Figure 2) or they can be more elaborately marked up with tags of semantic value (e.g., date, number, measure, name, term, time, foreign word) and formatting value (e.g., figure, table, p, sp, div, caption) (Figure 3). Tags that indicate formatting features such as 'caption' are important because they can be used, for example, to automatically determine the topic of a story.</Paragraph>
    <Paragraph position="4"> The actual implementation of tagging Urdu script at a detailed level presents a display problem for our XML editor of choice, XML Spy. Upon looking at Figure 3, which is an excerpt from XML Spy, one may think that the word order of the paragraph is out of order. At the display level, the word order is out of order--it is barely humanreadable, but at the storage level, the text is perfectly tagged and will process correctly. In Figure 4, we show, in a human-readable format, the order in which the Urdu text and English tags are stored. If an XML editor were optimized to display a right-to-left language with left-to-right tags, this is how we imagine the text would look. More importantly though, this is the order in which XML Spy currently stores the Urdu corpus.</Paragraph>
    <Paragraph position="5"> We began the corpus building process by storing Urdu documents at the paragraph level with no other tags peppering the data. However, we intend to hand tag the data for parts of speech so the data can be used to train and test natural language processing algorithms.</Paragraph>
    <Paragraph position="6">  optimized to display a right-to-left language with left-to-right tags</Paragraph>
  </Section>
class="xml-element"></Paper>