<?xml version="1.0" standalone="yes"?>
<Paper uid="P02-1026">
  <Title>Entropy Rate Constancy in Text</Title>
  <Section position="5" start_page="4" end_page="4" type="concl">
    <SectionTitle>
5 Conclusion and Future Work
</SectionTitle>
    <Paragraph position="0"> We have proposed a fundamental principle of language generation, namely the entropy rate constancy principle. We have shown that entropy of the sentences taken without context increases with the sentence number, which is in agreement with the above principle. We have also examined the causes of this increase and shown that they are both lexical (primarily for open-class parts of speech) and non-lexical.</Paragraph>
    <Paragraph position="1"> These results are interesting in their own right, and may have practical implications as well. In particular, they suggest that language modeling may be a fruitful way to approach issues of contextual influence in text.</Paragraph>
    <Paragraph position="2"> Of course, to some degree language-modeling caching work has always recognized this, but this is rather a crude use of context and does not address the issues which one normally thinks of when talking about context. We have seen, however, that entropy measurements can pick up much more subtle influences, as evidenced by the results for determiners and prepositions where we see no caching influence at all, but nevertheless observe increasing entropy as a function of sentence number. This suggests that such measurements may be able to pick up more obviously semantic contextual influences than simply the repeating words captured by caching models. For example, sentences will di er in how much useful contextual information they carry. Are there useful generalizations to be made? E.g., might the previous sentence always be the most useful, or, perhaps, for newspaper articles, the rst sentence? Can these measurements detect such already established contextual relations as the given-new distinction? What about other pragmatic relations? All of these deserve further study.</Paragraph>
  </Section>
class="xml-element"></Paper>