File Information
File: 05-lr/acl_arc_1_sum/cleansed_text/xml_by_section/metho/04/w04-2801_metho.xml
Size: 15,303 bytes
Last Modified: 2025-10-06 14:09:24
<?xml version="1.0" standalone="yes"?> <Paper uid="W04-2801"> <Title>Robustness versus Fidelity in Natural Language Understanding</Title> <Section position="4" start_page="0" end_page="0" type="metho"> <SectionTitle> 3 Spelling Correction </SectionTitle> <Paragraph position="0"> NUBEE's spelling corrector (Elmi and Evens, 1998) and robust parser are both part of the Carmel workbench and the interface between the two is predefined. The spelling corrector uses the parser's lexicon as its dictionary and attempts to fix spelling and typing errors in known words.</Paragraph> <Paragraph position="1"> Since the parser's lexicon is typically much smaller than a lexical database such as WordNet (Miller, 1990), there is a reduction in token ambiguity (i.e., the number of possible replacements to consider) but the spelling of unknown words will not be corrected. The simplification is also made that known words are never misspelled versions of other known words (e.g., typing &quot;their&quot; instead of &quot;there&quot;).</Paragraph> <Paragraph position="2"> The spelling corrector uses string transformations to attempt to repair spelling/typing errors. Because repeated transformations will map any input string to a word in the dictionary, transformations are given penalty scores and a threshold defines an allowable spelling correction.</Paragraph> <Paragraph position="3"> However, the spelling corrector's decisions are final; replacements whose penalty scores are below the threshold are entered directly into the parser's chart but the penalty scores are not passed on.</Paragraph> <Paragraph position="4"> To produce a record of spelling corrector transformations, we create a word transformation table for every new utterance. Each transformation is recorded in a table entry consisting of the original word, the transformed word (after spelling correction), and an associated confidence score. We have modified the spelling corrector to return a confidence score based on the number of alternatives that it proposes. We show below the transformation table recording the spelling corrector's output for the misspelled word &quot;socet&quot;: socet -> socket, 0.5 -> set, 0.5 In future work, we plan to modify the spelling corrector so that it outputs its penalty score.</Paragraph> </Section> <Section position="5" start_page="0" end_page="0" type="metho"> <SectionTitle> 4 Unknown Word Handling </SectionTitle> <Paragraph position="0"> Carmel's robust parser can skip words while attempting to find an analysis for the user input. True unknown words (not spelling errors) will be skipped because Carmel will have no information about their syntactic or semantic features. For some unknown words, this is the best solution because they represent concepts not having an obvious link to knowledge modeled by the system (users should be alerted to the system's limitation). However, we are aware of no work on attempting to recognize novel ways of referring to entities modeled by the system.</Paragraph> <Paragraph position="1"> Our approach is to find known synonyms (i.e., in NUBEE's lexicon) of unknown words using the Word-Net lexical database (Miller, 1990). In WordNet, words are connected to one or more synsets each corresponding to a distinct concept. Each synset will have a set of hyponymy (all the subtypes of the synset) and hypernymy (the supertype of the synset) links.</Paragraph> <Paragraph position="2"> Currently, we use a very simple search process to look for a known word whose meaning approximates the meaning of the unknown word. We assign a part-of-speech (POS) tag to the unknown word, and search the appropriate WordNet taxonomy. We retrieve the synsets associated with the word and run the search procedure SEARCH-WORD stopping when a known word is found.</Paragraph> <Paragraph position="3"> procedure SEARCH-WORD (SYNSETS) 1. SEARCH-DOWN (SYNSETS) 2. if height threshold not reached then SEARCH-WORD (hypernyms for SYNSETS) procedure SEARCH-DOWN (SYNSETS) 1. search all words having a synset in SYNSETS 2. SEARCH-DOWN (all hyponyms of SYNSETS) Nodes in the WordNet taxonomy close to its root have relatively general meanings (e.g., social-relation, physical-object) so we define a limit (height threshold) to how far the search can progress up the taxonomy. To make a record of unknown-word-handling transformations, we add additional entries to the word transformation table output by spelling correction. We use the size of the search space to calculate confidence scores, treating the set of replacement words retrieved in each step of the search process as equally likely. Consider the example of the unknown word &quot;cable&quot;. Step one of SEARCH-DOWN returns: &quot;telegraph&quot;, &quot;wire&quot;, and &quot;fasten-with-a-cable&quot;. &quot;wire&quot; is a known word, and &quot;cable&quot; is replaced by &quot;wire&quot; with a confidence of 0.33: cable -> wire, 0.33</Paragraph> </Section> <Section position="6" start_page="0" end_page="0" type="metho"> <SectionTitle> 5 Carmel Workbench </SectionTitle> <Paragraph position="0"> We use the Carmel workbench (Ros'e, 2000; Ros'e et al., 2003) for parsing and post-processing. In Carmel's AU-TOSEM framework: &quot;semantic interpretation [operates] in parallel with syntactic interpretation at parse time in a lexicon driven fashion. ... [Semantic] knowledge is encoded declaratively within a meaning representation specification. Semantic constructor functions are compiled automatically from this specification and then linked into lexical entries&quot; (Ros'e, 2000, p. 311).</Paragraph> <Paragraph position="1"> Carmel comes with a wide-coverage English grammar that is compatible with the wide-coverage COMLEX lexicon (Grishman et al., 1994). For each COMLEX entry that we wanted to add into NUBEE's lexicon, we specified its meaning as shown below for the words &quot;connect&quot;, &quot;battery&quot;, and &quot;wire&quot;.</Paragraph> <Paragraph position="2"> connect: connect', subject->agent, object->theme, modifier->destination battery: battery' wire: wire' This simplified example of the meaning specification assigns a predicate to each word, and in the case of a verb such as &quot;connect&quot; assigns a mapping from the syntactic roles of subject, object, and modifier to the semantic roles of agent, theme, and destination. This representation is domain-independent and reusable; it will always be the case that the subject of &quot;connect&quot; realizes the</Paragraph> <Paragraph position="4"> agent, the object realizes the theme, and that the destination (if present) will be realized as a modifier.</Paragraph> <Paragraph position="5"> Figure 4 shows a simplified version of the parser's output given the utterance, &quot;connect the battery to the cable&quot;. Recall from section 4 that the unknown word handler will replace &quot;cable&quot; with &quot;wire&quot; leading to the wirea0 predicates in figure 4.</Paragraph> <Paragraph position="6"> The two occurrences of the definite article &quot;the&quot; trigger the feature values (def defNP), marking that batterya0 and wirea0 occurred in definite NPs. The nouns, &quot;battery&quot; and &quot;wire&quot;, have the associated syntactic feature of being singular, and the features theme and destination mark the semantic roles of batterya0 and wirea0 .</Paragraph> <Paragraph position="7"> Dialogue systems use domain-specific reasoners to process the output of NLU sub-systems (e.g., to answer a user question or execute a user command, or to judge the correctness of a student answer). Such domain reasoners generally expect input in a predefined, domain-specific format necessitating a second stage of processing to convert the parser's output into the correct format.</Paragraph> <Paragraph position="8"> Our domain reasoner's representation for the connect action is a predicate, connecta0 a0 , taking as arguments, the two objects to be connected. Carmel provides support for building such predicates from the parser's output based on a declarative specification. Based on our specification for connecta0 a0 , Carmel's predicate mapper will produce the logical form shown in figure 5 (a copy of figure 2).</Paragraph> <Paragraph position="9"> During the parsing and post-processing stages, the string returned from pre-processing (spelling correction and unknown word handling) is transformed into a series of predicates. We currently do not keep track of all the connections between the predicates and the words that formed them. The predicate mapping stage is difficult to</Paragraph> <Paragraph position="11"> unravel; the mapping rules operate on semantic feature values (such as those shown in figure 4). There is no direct link between pieces of semantic feature values and the words that triggered them so it is difficult to associate the output of predicate mapping with words. See section 7 for more details and our interim solution.</Paragraph> </Section> <Section position="7" start_page="0" end_page="0" type="metho"> <SectionTitle> 6 Reference Resolution </SectionTitle> <Paragraph position="0"> For each predicate corresponding to a physical entity, the reference resolution module must decide whether the predicate refers to: a concept (a generic reading), a novel object (indefinite reference), or an existing object (definite reference). If the predicate refers to an existing object, the predicate (e.g., wirea0 a0 ) may match several objects in the domain reasoner but the speaker may only be referring to a subset of these objects.</Paragraph> <Paragraph position="1"> Figure 6 shows the example from figure 5 after reference resolution. The predicate batterya0 a0 is replaced by the name of the one battery present in figure 1, but wirea0 a0 could refer to any of the five wires in the circuit leading to the ambiguity depicted.</Paragraph> <Paragraph position="2"> NUBEE can query the dialogue system's history list to assign salience and calculate confidence scores for the transformations it makes. These transformations are stored in a table such as the one below (assume that &quot;|I|WIRE-1461&quot; had been mentioned previously but the other wires had not):</Paragraph> <Paragraph position="4"> In the next section, we will see how these table entries are matched with words from the input.</Paragraph> </Section> <Section position="8" start_page="0" end_page="0" type="metho"> <SectionTitle> 7 Improving Fidelity </SectionTitle> <Paragraph position="0"> We are currently focusing on improving fidelity for referring expressions: assigning confidence scores to the objects retrieved during reference resolution and linking referenced objects to the words used to refer to them. We make the assumption that all referring expressions are NPs and build a table of predicates (formed from NPs) and the words associated with those NPs. A sample entry is shown below.</Paragraph> <Paragraph position="1"> (wire'' [id2] def singular) <= ''the wire'' We run the following procedure on each NP in the one parse tree node covering the input (in the parser's packed representation there will only be one such node with the category utterance).</Paragraph> <Paragraph position="2"> procedure PROCESS-NP (NP) 1. run the predicate mapper just on NP to get its associated predicate 2. follow the children of NP to find the words associated with it The next step is consulting the spelling corrector and unknown word handler. In section 3, we introduced a table of word substitutions with associated confidence scores. This table can be used to replace the words found in the chart with the words actually typed by the user and to compute a global confidence score by multiplying the confidence scores of the individual words with the confidence score assigned during reference resolution. In section 4, we discussed unknown word handling for &quot;the cable&quot;; combining this result with the reference table computed above gives us: (wire'' [id3] def singular), 0.033, ''the cable'' One complication is that NPs can be associated with an ambiguous set of words. Consider a nonsense word in our domain, &quot;waters&quot;. The spelling corrector will propose &quot;watts&quot; and &quot;meters&quot; as possible replacements. In the parser's packed representation, the nouns &quot;watts&quot; and &quot;meters&quot; share the same node. A disjunctive set of features represents the ambiguity.</Paragraph> <Paragraph position="3"> We modified PROCESS-NP to deal with such cases by adding an additional step: 3. for each meaning, M of the NP 3.1 try to match M with one or more of the words associated with the NP (i.e., run the predicate mapper just on the word and see if it matches one of the meanings of the NP)</Paragraph> </Section> <Section position="9" start_page="0" end_page="0" type="metho"> <SectionTitle> 8 Discussion </SectionTitle> <Paragraph position="0"> Although there has been work on controlling the fidelity of individual components of the pipeline shown in figure 3, there has been little work considering the NLU sub-system as a whole. Gabsdil and Bos (2003) incorporate (speech recognizer) confidence scores into their logical form for elements that correspond directly to words in the input (rather than larger structures built through composition). Consider the example of the word &quot;manager&quot; and assume it has a speech recognition confidence score of 0.65. Gabsdil and Bos' parser will assign &quot;manager&quot; the semantic value of a0a2a1 :MANAGER(a3a5a4 ) where a0a6a1 is a handle and a3a7a4 a variable. This semantic value is given a confidence score of 0.65 the same as &quot;manager&quot;. To compute confidence scores for larger constituents they suggest to &quot;combine confidence scores for sub-formulas recursively&quot; (Gabsdil and Bos, 2003, p. 149).</Paragraph> <Paragraph position="1"> We have taken this idea further and explored the issues involved in computing confidence scores for larger constituents. Some of these issues are linked to our two-stage semantic analysis. However, Carmel's two-stage interpretation process (i.e., a domain-independent parsing stage and a domain-dependent predicate mapping stage) is not idiosyncratic to the Carmel workbench. Dzikovska et al. (2002) adopt such a two stage approach because their NLU sub-system is used in multiple domains (e.g., transportation planning, medication advice) necessitating reuse of resources wherever possible. Milward (2000) uses a two stage approach because it increases robustness. When the parser is not able to build a parse tree covering the entire input, there will still be a semantic chart composed of partial parses and their associated semantic feature values. For the domain of airline flight information, Milward defines post-processing rules that scan this semantic chart looking for information such as departure times. Our goal in this paper was to highlight the architectural trade-offs of such features on controlling fidelity.</Paragraph> </Section> class="xml-element"></Paper>