<?xml version="1.0" standalone="yes"?>
<Paper uid="W99-0402">
  <Title>ELICITING NATURAL SPEECH FROM NON-NATIVE USERS: COLLECTING SPEECH DATA FOR LVCSR</Title>
  <Section position="7" start_page="8" end_page="10" type="evalu">
    <SectionTitle>
5 Analysis and Examples
</SectionTitle>
    <Paragraph position="0"> Although transcription and analysis of the data we have collected so far is in the beginning stages, we have observed patterns that lead us to believe that our protocol is meeting our goals of eliciting speech from non-native speakers that is representative of what they would use in a real system and that begins to uncover patterns that are different from those native speakers use and will be useful in acoustic and language modeling. null The analysis in this section is based on transcribed data from 12 speakers. For comparison, we recorded three native speakers doing the same task the non-native speakers did (with English prompts). This is not a large sample, but gives us some evidence to support our intuitions about what native speakers would be likely to say.</Paragraph>
    <Section position="1" start_page="8" end_page="9" type="sub_section">
      <SectionTitle>
5.1 Qualitative Analysis
</SectionTitle>
      <Paragraph position="0"> Examples 1-3 show some sample utterances produced by the non-native speakers. In each example, the first sentence represents the prompt that would have been used for elicitation (speakers were actually given short bullets). Example 1 was selected t o exemplify how speakers were influenced in their use of phrasal and colloquial verbs when given an Englishprompt.</Paragraph>
      <Paragraph position="1"> We observed that when prompted to ask for directions or travel time, native speakers almost always used the expression &amp;quot;get to.&amp;quot; Non-native speakers often used this form when given an English prompt containing it, but almost never when given an L1 prompt.</Paragraph>
      <Paragraph position="2"> 1. Ask how to get to the aquarium.</Paragraph>
      <Paragraph position="3"> How do I get the aquarium? Please let me know how do you go the aquarium? I'd like to go to Aquarium.</Paragraph>
      <Paragraph position="4"> I want to go to the aquarium so please let me know how to go to there In the data we have transcribed so far, 25 of 55 uses of get to were by non-native speakers, while 45 of 56 uses of go to were by non-native speakers.</Paragraph>
      <Paragraph position="5"> Example 2 illustrates how number agreement can be influenced by the Enghsh prompt. Although nativespeakers often misspeak and disobey agreement rules in conversational speech, there are situations in which we observed that they are consistently careful, and the pattern any + Npl, when appropriate, was one. The non-native speakers, on the other hand, consistently produced any + Nsing when not primed by an English prompt. &amp;quot;Any&amp;quot; was also often used where a native speaker would use &amp;quot;a.&amp;quot; 2. Ask if there are any \[restaurants nearby / tickets available... \].</Paragraph>
      <Paragraph position="6"> Is there any restaurant around here? is there any good place to visit is there any available ticket do you have any special exhibition now is there any subway around Of the 105 instances of use of the word &amp;quot;any,&amp;quot; 52 were followed inappropriately by a singular noun. When the pattern &amp;quot;any place&amp;quot; is removed from the list, 52 out of 81 instances were grammatically incorrect in this way. To compare, 1 of 21 instances in the native sample were grammatically incorrect. Prescriptively incorrect grammar is expected in spontaneous speech even by native speakers. However, when non-native speech consistently strays from patterns observed in native speech, the bigram and tri-gram contexts used to model language at the sentence level can no longer be relied upon.</Paragraph>
      <Paragraph position="7"> Of course, by using an L1 prompt we are influencing the speakers in the opposite direction, priming them to produce a translation of an L1 word and form an awkward English sentence around it when they might not do so in spontaneous system use. It is difficult to know whether this is the case with example 3. On the one hand, the speaker is clearly translating  the Japanese term nyuujouryou (entrance fee).</Paragraph>
      <Paragraph position="8"> On the other hand, speakers consistently built a sentence around the word &amp;quot;fee&amp;quot; where a native speaker would use the pattern &amp;quot;how much does X cost&amp;quot; regardless of what Japanese term was used.</Paragraph>
      <Paragraph position="9"> 3. Ask how much admission costs How much is the fee for entrance? How much is fee for entering? How much is the fee for admission? Although it was the element of the task that the speakers liked the least, the handling of unfamiliar expressions showed us how important it was to prompt users with specific queries that they might not know how to express. In real-world use, an application would have to handle such utterances, but in a more free-form data collection scenario speakers might avoid asking such questions altogether. We included among the Japanese prompts expressions which have no obvious English equivalent in order to observe how speakers expressed themselves when they did not know what the right English expression would be. Speakers were very inventive and almost always cameup with an understandable English utterance, as shown in Figure 1 (displayed on the following page).</Paragraph>
      <Paragraph position="10">  non-native speakers in the tourist information task. Corpus size is displayed on the x axis and vocabulary size is displayed on the y axis.</Paragraph>
    </Section>
    <Section position="2" start_page="9" end_page="10" type="sub_section">
      <SectionTitle>
5.2 Quantitative Analysis
</SectionTitle>
      <Paragraph position="0"> Figure 2 shows the vocabulary growth rate for native and non-native speakers in the tourist information task that was our domain for these experiments. Interestingly, the vocabulary growth seems to be faster for non-native  speakers than for native speakers. The curve for native speakers in another similar domain (travel arrangement) for which we have much more data was similar to the curve for native speakers shown in Fig. 2; in fact, the vocabulary size for this bigger corpus did not reach the size of the non-native corpus at 5600 words until 10,000 word tokens had been seen.</Paragraph>
      <Paragraph position="1"> We also looked at trigram perplexity of the data collected in the different pilot experiments measured with respect to a model built on the large travel arrangement data set. Although the test corpora were very small, we found that the corpus collected from non-native speakers using English prompts was very similar in terms of perplexity to the corpus collected from native speakers in the tourist information task. Conversely, the corpus collected from non-native speakers using Japanese prompts showed over 1.5 times the perplexity of the native corpus.</Paragraph>
      <Paragraph position="2"> This indicates that the character of the two non-native corpora are quite different, and that incorporating the Ll-prompted data in training a statistical language model will increase the predictive power of the model with respect to non-native speakers.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>