<?xml version="1.0" standalone="yes"?>
<Paper uid="H93-1075">
  <Title>A SIMULATION-BASED RESEARCH STRATEGY FOR DESIGNING COMPLEX NL SYSTEMS*</Title>
  <Section position="6" start_page="373" end_page="374" type="evalu">
    <SectionTitle>
4. SAMPLE RESULTS
</SectionTitle>
    <Paragraph position="0"> The variability inherent in people's language, whether spoken or written, poses a substantial challenge to the successful design of future NL systems. One aspect of this research has been a comprehensive assessment of the linguistic variability evident in people's speech and writing at various levels of processing, including acoustic, lexical, syntactic, and semantic. Full reports of these results are forthcoming \[11, 14\]. Special emphasis has been placed on identifying problematic sources of variability for system processing, as well as an explanation of the circumstances and apparent reasons for their occurrence. In connection with these analyses, one goal of this researclh program has been to identify specific interface techniques that may naturally channel users' language in ways that reduce or eliminate difficult sources of variability, so that more robust system processing can be achieved. In particular, the impact of selecting a particular input modality or presentation format is being examined, so that future system designers will have the option of choosing a particular modality or format because doing so will minimize expected performance failures of their planned NL systems.</Paragraph>
    <Paragraph position="1"> To briefly illustrate the research theme of reducing linguistic variability through selection of modality and format, the results of an analysis related to syntactic ambiguity are summarized. Two indices of relative ambiguity were measured for all phrasal and sentential utterances that people spoke to an unconstrained format (SNF), wrote in an unconstrained.format (WNF), spoke to a form (SF), or wrote in a form (WF). Two different estimates of parse ambiguity were computed to check for convergence of results. First, utterances produced under the different simulation conditions were parsed using DIALOGIC \[9\], a robust text processing system developed at SRI that employs a broad coverage grammar. Second, a summary was computed of the number of canonical parses produced by DIALOGIC, through a mapping of each DIALOGIC parse to an emerging national standard parse tree representation called PARSEVAL form 2 \[2\]. The average number of DIALOGIC and PARSEVAL parses generated per utterance for the different simulation conditions is summarized in Table 1, along with the percentage of all utterances in each condition that were phrases or sentences and therefore appropriate for parsing. null None of the subjects produced phrases or sentences when writing to a form, so none of the simple utterances from PAFtSEVAL form is designed to reflect agreement among computational linguists simply on the major constituent bracketlngs, so PARSEVAL identification of syntactic structures should tend to represent the commonalities among many different systems.</Paragraph>
    <Paragraph position="2">  VAL parses per utterance as a function of modality and format.</Paragraph>
    <Paragraph position="3"> this condition were appropriate for parsing. The percentage of phrase and sentential utterances available for parsing was greater for unconstrained than form-based input, and greater for spoken than written input. Comparison of both parse metrics for unconstrained and form-based speech revealed that using a form significantly reduced the average number of parses per utterance, t (paired)</Paragraph>
    <Paragraph position="5"> VAL). When comparisons were made of the same subjects accomplishing the same tasks, the parse ambiguity of utterances in the unconstrained format averaged 232% higher for DIALOGIC and 157% higher for PARSEVAL than when communicating to a form. However, comparison of both parse metrics for speech and writing in an unconstrained format did not confirm that use of the written modality reduced the average number of parses per utterance, t (paired) = 1.18 (df = 14), p &gt; .10, one-tailed (DIALOGIC), and t &lt; 1 (PARSEVAL). That is, reliable reduction of parse ambiguity was obtained only through manipulation of the presentation format.</Paragraph>
    <Paragraph position="6"> This pattern of results suggests that selection of presentation format can have a substantial impact on the ease of natural language processing, with direct implications for improved system robustness. In addition, postexperimental interviews indicated that participants preferred form-based interactions over unconstrained ones by a factor of 2-to-1 in the present tasks. In particular, both the guidance and assurance of completeness associated with a form were considered desirable. This indicates that the a priori assumption that any type of constraint will be viewed by people as unacceptable or unnatural clearly is not always valid. Furthermore, such a presumption may simply bias system development away from good prospects for shorter-term gain. The application of this kind of interface knowledge will be important to the successful performance and commercialization of future natural language technology.</Paragraph>
  </Section>
class="xml-element"></Paper>