<?xml version="1.0" standalone="yes"?>
<Paper uid="P98-1075">
  <Title>Growing Semantic Grammars</Title>
  <Section position="4" start_page="0" end_page="0" type="metho">
    <SectionTitle>
2 Grammar development
</SectionTitle>
    <Paragraph position="0"> If we analyze the traditional method of developing a semantic grammar for a new domain, we find that the following stages are involved.</Paragraph>
    <Paragraph position="1">  1. Data collection. Naturally-occurring data from the domain at hand are collected.</Paragraph>
    <Paragraph position="2"> 2. Design of the domain model. A hierarchical structuring of the relevant concepts in the domain is built in the form of an ontology or domain model.</Paragraph>
    <Paragraph position="3"> 3. Development of a kernel grammar. A grammar that covers a small subset of the collected data is constructed.</Paragraph>
    <Paragraph position="4"> 4. Expansion of grammar coverage. Lengthy, ar null duous task of developing the grammar to extend its coverage over the collected data and beyond. 5. Deployment. Release of the final grammar for the application at hand.</Paragraph>
    <Paragraph position="5"> The GsG system described in this paper aids all but the first of these stages: For the second stage, we have built a simple editor to design and analize the Domain Model; for the third, a semi-automated way of constructing the Kernel Grammar; for the fourth, an interactive environment in which new semantic mappings are dynamically acquired. As for the fifth (deployment), it advances one place: after the short initial authoring phase (stages 2 and 3 above) the final application can already be launched, since the semantic grammar will be extended, at run-time, by the non-expert end-user.</Paragraph>
  </Section>
  <Section position="5" start_page="0" end_page="454" type="metho">
    <SectionTitle>
3 System architecture
</SectionTitle>
    <Paragraph position="0"> As depicted in Fig. 1, GsG is composed of the following modules: the Domain Model Editor and the  Kernel Grammar Editor, for the authoring stage, and the SouP parser and the IDIGA environment, for the run-time stage.</Paragraph>
    <Section position="1" start_page="451" end_page="452" type="sub_section">
      <SectionTitle>
3.1 Authoring stage
</SectionTitle>
      <Paragraph position="0"> In the authoring stage, a developer s creates the Domain Model (DM) with the aid of the DM Editor.</Paragraph>
      <Paragraph position="1"> In our present formalism, the DM is simply a directed acyclic graph in which the vertices correspond to concept-labels and the edges indicate concept-subconcept relations (see Fig. 2 for an example). Once the DM is defined, the Kernel Grammar Editor drives the development of the Kernel Grammar by querying the developer to instantiate into grammar rules the rule templates derived from the DM.</Paragraph>
      <Paragraph position="2"> For instance, in the DM in Fig. 2, given that concept {suggest_time} requires subconcept \[time\], the rule template \[suggest_time\] &lt; \[time\] is generated, which the developer can instantiate into, say, rule (2) in Fig. 3.</Paragraph>
      <Paragraph position="3"> The Kernel Grammar Editor follows a concreteto-abstract ordering of the concepts obtained via a topological sort of the DM to query the developer, after which the Kernel Grammar is complete 3 and  main model DM if and only if for each arc from concept i to concept j in DM there is at least one grammar rule headed by concept i that contains concept j. This ensures that any idea expressible in DM has a surface form, or, seen it from another angle, that any in-domain utterance has a paraphrase  ing task. A dashed edge indicates optional subconcept (default is required), a dashed angle indicates inclusive subconcepts (default is exclusive).</Paragraph>
      <Paragraph position="4">  (1) \[suggestion\] ~-- {suggest_time} (2) {suggest_time} ~-- how about \[time\] (3) \[time\] ~ \[point\] (4) \[point\] 4---- *on {day_of_week} *{time_of_day} (5) {day_of_week} ~--- Tuesday (6) {time_of_day} 6--- afternoon  A '*' indicates optionality.</Paragraph>
      <Paragraph position="5"> the NLU front-end is ready to be deployed.</Paragraph>
      <Paragraph position="6"> It is assumed that: (i) after the authoring stage the DM is fixed, and (ii) the communicative goal of the end-user is expressible in the domain.</Paragraph>
    </Section>
    <Section position="2" start_page="452" end_page="454" type="sub_section">
      <SectionTitle>
3.2 Run-time stage
</SectionTitle>
      <Paragraph position="0"> Instead of attempting &amp;quot;universal coverage&amp;quot; we rather accept the fact that one can never know all the surface forms by which the concepts in the domain can be expressed. What GsG provides in the run-time stage are mechanisms that allow a non-expert end-user to &amp;quot;teach&amp;quot; the meaning of new expressions. The tight coupling between the SouP parser 4 and the IDIGA s environment allows for a rapid and multi-faceted analysis of the input string. If the parse, or rather, the paraphrase automatically generated by GSG 6, is deemed incorrect by the end-user, a learning episode ensues.</Paragraph>
      <Paragraph position="1"> that is covered by G.</Paragraph>
      <Paragraph position="2">  performed in natural language only, a generation grammar is needed to transform semantic representations into surface forms. To that effect GSG is able to cleverly use the analysis grammar in &amp;quot;reverse.&amp;quot; By bringing to bear contextual constraints, Gso can make predictions as to what a sequence of unparsed words might mean, thereby exhibiting an &amp;quot;empathic&amp;quot; behavior toward the end-user. To this aim, three different learning methods are employed: parser predictions, hidden understanding model, and end-user paraphrases.</Paragraph>
      <Paragraph position="3">  Similar to Lehman (1989), learning in GsQ takes place by the dynamic creation of grammar rules that capture the meaning of unseen expressions, and by the subsequent update of the stochastic models. Acquiring a new mapping from an unparsed sequence of words onto its desired semantic representation involves the following steps.</Paragraph>
      <Paragraph position="4"> 1. Hypothesis formation and filtering. Given the context of the sentence at hand, Gsc constructs hypotheses in the form of parse trees that cover the unparsed sequence, discards those hypotheses that are not approved by the DM r and ranks the remaining by likelihood.</Paragraph>
      <Paragraph position="5"> 2. Interaction with the end-user. The ranked hypotheses are presented to the end-user in the form of questions about, or rephrases of, the original utterance.</Paragraph>
      <Paragraph position="6"> 3. Dynamic rule creation. If the end-user is satisfied with one of the options, a new grammar rule is dynamically created and becomes part of the end-user's grammar until further notice.</Paragraph>
      <Paragraph position="7"> Each new rule is annotated with the learning episode that gave rise to it, including end-user ID, time stamp, and a counter that will keep track of how many times the new rule fires in successful parses, s  As suggested by Kiyono and Tsujii (1993), one can make use of parse failures to acquire new knowledge, both about the nature of the unparsed words and about the inadequacy of the existing grammar rules. GsG uses incomplete parses to predict what can come next (i.e. after the partially-parsed sequence 7I.e., parse trees containing concept-subconcept relations that are inconsistent with the stipulations of the DM. SThe degree of generalization or level o.f abstraction that a new rule should exhibit is an open question but currently a  Principle of Maximal Abstraction is followed: (a) Parse the lexical items of the new rule's right-hand-side with all concepts granted top-level status, i.e., able to stand at the root of a parse tree.</Paragraph>
      <Paragraph position="8"> (b) If a word is not covered by any tree, take it as is into  the final right-hand side. Else, take the root of the parse tree with largest span; if tie, prefer the root that ranks higher in the DM.</Paragraph>
      <Paragraph position="9"> For example, with the DM in Fig. 2 and the grammar in Fig. 3, What about Tuesdayf is abstracted to the maximally general what about \[time\] (as opposed to what about \[day_of_week\] or what about \[point\]).</Paragraph>
      <Paragraph position="10">  predictions. Initially only the temporal expression is understood... null in left-to-right parsing, or before the partially-parsed sequence in right-to-left parsing). This allows two kinds of grammar acquisition:  1. Discovery of expression equivalence. E.g., with the grammar in Fig. 3 and input sentence What about Tuesday afternoon? GsQ is able to ask the end-user whether the utterance means the  same as How about Tuesday afternoon? (See Figs. 4, 5 and 6). That is because in the process of parsing What about Tuesday afternoon? right-to-left, the parser has been able to match rule (2) in Fig. 2 up to about, and thus it hypothesizes the equivalence of what and how since that would allow the parse to complete. 9 2. Discovery of an ISA relation. Similarly, from input sentence How about noon? GsG is able to predict, in left-to-right parsing, that noon is a \[time\].</Paragraph>
      <Paragraph position="11"> 3.2.3 Hidden understanding model As another way of bringing contextual information to bear in the process of predicting the meaning 9For real-world grammars, of, say, over 1000 rules, it is necessary to bound the number of partial parses by enforcing a maximum beam size at the left-hand side level, i.e., placing a limit on the number of subparses under each nonterminal to curb the exponential explosion.</Paragraph>
      <Paragraph position="12">  of unparsed words, the following stochastic models, inspired in Miller et al. (1994) and Seneff (1992), and collectively referred to as hidden understanding model (HUM), are employed.</Paragraph>
      <Paragraph position="13"> * Speech-act n-gram. Top-level concepts can be seen as speech acts of the domain. For instance, in the DM in Fig. 2 top-level concepts such as \[greeting\], Cfarewell\] or \[suggestion\], correspond to discourse speech acts, and in normally-occurring conversation, they follow a distribution that is clearly non-uniform. 1deg * Concept-subconcept HMM. Discrete hidden Markov model in which the states correspond ldegNeedless to say, speech-act transition distributions are empirically estimated, but, intuitively, the sequence &lt;\[greeting\], \[suggestion\]&gt; is more likely than the sequence &lt; \[greeting\], \[farewell\]&gt;.</Paragraph>
      <Paragraph position="14"> to the concepts in the DM (i.e., equivalent to grammar non-terminals) and the observations to the embedded concepts appearing as immediate daughters of the state in a parse tree. For example, the parse tree in Fig. 4 contains the following set of &lt;state, observation&gt; pairs:  model in which the states correspond to the concepts in the DM and the observations to the embedded lexical items (i.e., grammar terminals) appearing as immediate daughters of the state in a parse tree. For example, the parse tree in Fig. 4 contains the pairs: {&lt;\[day_of_week\], tuesday&gt;, &lt; \[time_of_day\], afternoon&gt;}.</Paragraph>
      <Paragraph position="15"> The HUM thus attempts to capture the recurring patterns of the language used in the domain in an asynchronous mode, i.e., independent of word order (as opposed to parser predictions that heavily depend on word order). Its aim is, again, to provide predictive power at run-time: upon encountering an unparsable expression, the HUM hypothesizes possible intended meanings in the form of a ranked list of the most likely parse trees, given the current state in the discourse, the subparses for the expression and the lexical items present in the expression.</Paragraph>
      <Paragraph position="16"> Its parameters can be best estimated through training over a given corpus of correct parses, but in order not to compromise our established goal of rapid deployment, we employ the following techniques. null  1. In the absence of a training corpus, the HUM parameters are seeded from the Kernel Grammar itself.</Paragraph>
      <Paragraph position="17"> 2. Training is maintained at run-time through dynamic updates of all model parameters after each utterance and learning episode.</Paragraph>
      <Paragraph position="18">  If the end-user is not satisfied with the hypotheses presented by the parser predictions or the HUM, a third learning method is triggered: learning from a paraphrase of the original utterance, given also by the end-user. Assuming the paraphrase is understood, 11 GsG updates the grammar in such a fashion so that the semantics of the first sentence are equivalent to those of the paraphrase. 12 11 Precisely, the requirement that the grammar be complete (see note 3} ensures the existence of a suitable paraphrase for any utterance expressible in the domain. In practice, however, it may take too many attempts to find an appropriate paraphrase. Currently, if the first paraphrase is not understood, no further requests are made.</Paragraph>
      <Paragraph position="19"> 12Presently, the root of the paraphrase's parse tree directly becomes the left-hand-side of the new rule.</Paragraph>
    </Section>
  </Section>
  <Section position="6" start_page="454" end_page="454" type="metho">
    <SectionTitle>
4 Preliminary results
</SectionTitle>
    <Paragraph position="0"> We have conducted a series of preliminary experiments in different languages (English, German and Chinese) and domains (scheduling, travel reservations). We present here the results for an experiment involving the comparison of expert vs. non-expert grammar development on a spontaneous travel reservation task in English. The grammar had been developed over the course of three months by a full-time expert grammar writer and the experiment consisted in having this expert develop on an unseen set of 72 sentences using the traditional environment and asking two non-expert users is to &amp;quot;teach&amp;quot; Gs6 the meaning of the same 72 sentences through interactions with the system. Table 1 compares the correct parses before and after development.</Paragraph>
    <Paragraph position="1"> It took the expert 15 minutes to add 8 rules and reduce bad coverage from 27.01% to 13.51%. As for the non-experts, end-user1, starting with a similar grammar, reduced bad parses from 22.97% to 12.17% through a 30-minute session 14 with GsG that gave rise to 8 new rules; end-user2, starting with the smallest possible complete grammar, reduced bad parses from 41.89% to 22.98% through a 35-minute session 14 that triggered the creation of 17 new rules.</Paragraph>
    <Paragraph position="2"> 60% of the learning episodes were successful, with an average number of questions of 2.91. The unsuccessful learning episodes had an average number of questions of 6.19 and their failure is mostly due to unsuccessful paraphrases.</Paragraph>
    <Paragraph position="3"> As for the nature of the acquired rules, they differ in that the expert makes use of optional and repeatable tokens, an expressive power not currently available to GSG. On the other hand this lack of generality can be compensated by the Principle of Maximal Abstraction (see note 8). As an example, to cover the new construction And your last name?, the expert chose to create the rule: \[requestmame\] ~ *and your last name tSUndergraduate students not majoring in computer science or linguistics.</Paragraph>
    <Paragraph position="4"> 14 Including a 5-minute introduction.</Paragraph>
    <Paragraph position="5"> whereas both end-user1 and end-users induced the automatic acquisition of the rule: \[requostmame\] ~ CONJ POSS \[last\] name. 15</Paragraph>
  </Section>
class="xml-element"></Paper>