 OBLEM LOCALIZATION STRATEGIES 
FOR PRAGMATI~S ~ IN NATURAL-LANGUAGE FRONT ENDS• 
Lance A. Remshaw & Ralph ~L Welschedel 
Department of Ccaputer and Information Sciences 
University of Delaware 
Newark, Delaware 19716 USA 
ABSTRACT 
Problem localization Is the identification of 
the most slgnlflcant failures in the AND-OR tree 
resulting from an unsuocass/ul attempt to achieve a 
goal, for instance, In planning, backward-chnining 
inference, or top-down parnin~ We examine beurls- 
tics and strategies for problem localization in the 
context of using a planner to check for pragmatic 
failures in natural language input to computer sys- 
tems, such as a cooperative natural language 
interface to Unix •• . Our heuristics call for 
selecting the most hopeful branch at ORs, but the 
most problematic one at ANDs. Surprise scores and 
speclal-purpose rules are the maln strategies sug- 
gested to determine this. 
I PRAGMATIC OVERSHOOT AND PRCBLEM LOCALIZATION 
Even if the syntactic and semantic content of 
a request is correct, so that a natural language 
front end can derive s coherent representation of 
its meaning, its praamatlc content or the structure 
of the underlying system may make aSy direct 
response to the request impossible or mlsleadln~ 
According to Sondbelmer end Welschedel (Sondhelmer, 
1980), an input exhibits ~ ~ If the 
representation of its meaning is beyond the capa- 
bilities of the underlying system. Kaplan (1979), 
Mays (1980e), and Carberry (1984) have each worked 
on strategies for dealing wltb particular classes 
of such praamatlc failures. This paper addresses 
the problem of identifying the most si~ctflcant 
reason that a plan to achieve a user goal cannot be 
carried out. 
The approach to pragmatic fnilure taken In 
thls paper is to use a planner to verify the 
presumptions in a request. The presumptions behind 
a request become the subEoals of e plan to fulfill 
the request. Oslng Mays' (1980a) example, the 
query "Which faculty members take coursas?" Is here 
handled as an instance of an IDENTIFY-SET-~EHS 
• This material Is based upon work supported by 
the National Sclence Foundation under grants 
LST-8009673 and IST-8311~00. 
• • Unix is a trademark of Bell Laboratories. 
goal, and the pragmatlcs of the query are checked 
by looklng for a plan to achieve that goal. Deter- 
mining both that faculty members and courses do 
exist and that faculty members can take courses are 
subEoals within that plan. A presuppositlonal 
failure is noted if the planner is unable to com- 
plete a plan for the goal. 
Furthermore, £r~formation for recovery process- 
ing or expleaatory responses can be derived 
directly from the fniled plan by identifying what- 
ever blocked goal in the planning tree of subgoals 
Is most nignif~cant. Thus, in the example above, 
if the planner failed because It was unable to show 
that faculty can take courses, the helpful response 
would be to explain this presumption failure. We 
concentrate here on identifying the signifleant 
blocks rather than on generating natural language 
responses. 
The examples in this paper will be drawn from 
a pleaning System intended to function as the prag- 
matic overshoot component of a cooperative natural 
language interface to the Unix operating system. 
We chose Unix, much as Wilensky (1982) did for his 
Unix Consultant, as a fomiliar domain that was 
still complex enough to require interesting plan- 
ning~ In this system, the praRmatics of a user 
request are tested by building a tree of plan 
structures whose leaves are elementary facts avail- 
able to the operating system. For instance, the 
following planning tree Is built in response to the 
request to print a file: 
(PRINT-FILE ?user ?file ?device) 
& (IS-TEXT-FILE ?file) 
& (UP-AND-RUNNING ?device) 
& (READ-PERM ?user ?file) 
I (WORLD-READ-PERM-BIT-SET ?file) 
I (READ-PERM-USER ?user ?file) 
& (IS-O~NER ?user ?file) 
& (USER-READ-PERM-BIT-SET ?file)" 
\[ (READ-PERM-GROUP ?user ?file) 
& (SA~-GROUP ?user ?file) 
& (OROUP-REAI>-PERM-BIT-SET ?file) 
I (READ-PERM-SUPER-USER ?user) 
& (AUTHORIZED-SUPER-USER ?user) 
& (SUPEH-USER-PASSWORD-GIV~ ?user) 
(The children of AND nodes are preceded by amper- 
sands, and OR children by vertical bars. Initial 
question marks precede plea variables.) If a singie 
node In thls planning tree fails, say (IS-TEXT-FILE 
?file), that Information can be used In explnining 
the failure to the user. 
139 
The failure of certain nodes could also 
trigger recovery processing, as in the following 
example, where the failure of (UP-AND-RUNNING 
?device) triggers the suggestion of an alternative 
device: 
User: Please send the file to the laser printer. 
System: The laser printer is dowm 
Is the line printer satisfactory? 
This planning scheme offers a way of recognizing 
and responding to such temporarily unfulfillable 
requests as well as to other pragmatic failures 
from requests unfulfillable in context, which is an 
important, though largely untouched, problem. 
A difficulty arises, however, when more than 
one of the planning tree precondition nodes fail. 
Even in a tree that was entirely made up of AND 
nodes, multiple failures would require either a 
llst of responses, or else scme way of choosing 
which of the failures is most meaningful to report. 
In a plan tree containing OR nodes, where there are 
often many alternative ways that have all failed of 
achieving particular goals, it becomes even more 
important that the system be able to identify which 
of the failures is most significant. This process 
of identifying the significant failures is called 
"problem localization", and this paper describes 
heuristics and strategies that can be used for 
problem localization in failed planning trees. 
II HEURISTICS FOR PROBLEM LOCALIZATION 
The basic heuristics for problem localization 
can be derived by considering how a human expert 
would respond to someone who was pursuing an impos- 
aible goal. Hot finding any suosessful plan, the 
expert tries to explain the block by showing that 
every plan must fail. Thus, if more than one 
branch of an AND node in a plan fails, the most 
significant one to be reported is the one that the 
user is least likely to be able to change, since it 
makes the strongest case. (The planner must check 
all the branches of an AND node, even after one 
fails, to know which is most significant to 
report.) For instance, if all three of the children 
of PRINT-FILE in our example fail, (I~-TEXT-FILE 
?file) is the one that should be reported, since it 
is least llkely that the user can affect that node. 
If the READ-PERM failure were reported first, the 
user would waste time changing the read permission 
of a non-text file. Unix's actual behavior, which 
reports the first problem that it happens to dis- 
cover in trying to execute the co@mend, is often 
frustrating for exactly that reason. This heuris- 
tic of reporting the most serious failure at an AND 
node is closely related to ABSTRIP's use of "crltl- 
callty" numbers to divide a planner into levels of 
abstraction, so that the most critical features are 
dealt with first (Sacerdoti, 1974). 
The situation is different at OR nodes, where 
only a single child has to sueseed. Here the most 
serious failure can safely be ignored, as long as 
some other branch can be repaire~ Thus the most 
si~iflcant branch at an OR node should be the one 
the user is most likely to be able to affect. In 
• our example, READ-PERM-USER should usually be 
reported rather than READ-PERM-SUPER-USER, if both 
have failed, since most users have more hope of 
changing the former than the letter. There is a 
duality here between the AND and OR node heuristics 
that is llke the duality in the minimax evaluation 
of a move in a game tree, where one picks the best 
score at nodes where the choice is one's own, and 
the worst score at nodes where the opponent gets to 
choose. 
III STRATEGIES FOR PR~LEM LOCALIZATION 
Identification of the most significant failure 
requires the addition to the planner of knowledge 
about significance to be used in problea loealiza- 
tio~ Many mechanisms are possible, ranging from 
fixed, pre-set ordering of the children of nodes up 
through complex knowledge-based mechanlqms that 
include knowledge about the user,s probable goals. 
In this paper, we suggest a combination of statist- 
Ical "surprise scores" and speclal-purpose rules. 
Statistical ~UslnISurorise Scores 
This strategy relies on statistics that the 
system keeps dynamically onthe number of times 
that each branch of each plan has succeeded or 
failed. These are used to define a success ratio 
for each branch. For example, the PRINT-FILE plan 
might be annotated as follows: 
SUCCESSES RATIO 
(PRINT-FILE ?user ?file ?device) 
& (IS-TEXT-FILE ?file) 235 3 0.99 
& (UP-AND-RUNNING ?device) 185 53 0.78 
& (READ-PERM ?user ?file) 228 10 0.96 
FAILURES 
From these ratios, we derive surprise scores 
to provide some measure of how usual or unusual it 
is for a particular node to have succeeded or 
failed in the context of the goal giving rise to 
the node. The surprise score of a successful node 
is defined as 1.0 minus the success ratio, so that 
the success of a node llke I~-TEXT-FILE, that 
almost always succeeds, is less surprising than the 
success of UP-AND-RUNNING. Failed nodes get nega- 
tive surprise scores, with the absolute value of 
the score again reflecting the amount of surprise. 
The surprise score of a failed node is set to the 
negative of the success ratio, so that the failure 
of IE-TEXT-FILE would be more surprising than that 
of UP-AND-RUNNING, and that would be reflected by a 
more strongly negative score. 
Here is an example of our PRINT-FILE plan 
instantiated for an unlucky user who has failed on 
all but two preconditions, with surprise scores 
added: 
140 
SURPRISE 
SUCCESS/FAILURE SCORE 
(PR~T-FILE Ann Filel laser) 
& (IS-TEXT-FIIE Filel) -.99 
& (UP-AND-RUNNING laser) -.78 
& (READ-PERM Ann Filel) -.96 
I (WORLD-READ-PERF,-BIT-SET Filel) -.02 
\] (READ-PERM-USER Ann Filel) -.87 
& (IS-0WNER Ann Fllel) -.87 
& (USER-READ-PERM-BIT-SET Fllel) +.01 
J (READ-PERF,-GROUP Ann Filel) -.55 
& (SA~-GROUP Ann Filel) +.05 
-.58 
I -.02 
-.03 
-.02 
F 
F 
F 
F 
F 
F 
S 
F 
S 
& (GROUP-READ-PERM-BIT-SET Filel) F 
(BEAD-PERF~SUPER-USER Ann) F 
& (AUTHORIZED-SUPER-USER Ann) F 
& (SUPER-USER-PASSWORD-GIVEN Ann) F 
Note tbat the success of USER-READ-PERM-BIT-SET is 
not very surprising, since that node almost always 
succeeds; the failure of a node llke READ-PERM- 
SUPER-USER, which seldom succeeds, is much less 
surprising than the failure of UP-AND-RUNNING. 
We suggest keeping statistics and deriving 
surprise scores because we believe that they pro- 
vide a useful if imperfect handle on judging the 
signlflcence cf failed nodes. Regarding OR nodes, 
strongly negative surprise scores identify branches 
that in the past experience of the system have usu- 
ally succeeded, and these are the best guesses to 
be likely to succeed again. Thus READ-PERM-USER, 
the child of READ-PERM with the most strongly nega- 
tive score, turns out to be the most likely to be 
tractable. The negatlve surprise scores at a 
failed OR node give a profile of the typical suc- 
cess ratios; to select the nodes that are generally 
most likely to succeed, we pick the most surprising 
failures, those with the most strongly negatlve 
surprise scores. 
At AND nodes, on the other hand, the goal is 
to identify the branch that is most critical, that 
is, least likely to succeed. Surprisingly, we find 
that the most critical branch tends in thls case 
also to be the most surprlalng failure. In our 
example, IS-TEXT-FILE, which the user can do noth- 
ing about, is the most surprising failure under 
PRINT-FILE, READ-PERM is next most surprising, and 
UP-AND-RUNNING, for which simply waiting often 
works, comes last. Therefore at AND nodes, llke at 
OR nodes, we will report the child wlth the most 
negative surprise score; at AND nodes, this tends 
to identify the most critical failures, while at OR 
nodes, it tends to select the most hopeful. Note 
that the combined effect of the AND and OR stra- 
tegies is to choose from among all the failed nodes 
those that were statistically most likely to 
succeed. 
The main advantage of the statistical surprise 
score strategy is its low cost, both to design and 
execute. Another nice feature is the self- 
adjusting character of the surprise scores, based 
as they are on success statistics that the system 
updates on an onEolng basis. For example, the 
likelihood of GROUP-READ-PERM being reported would 
depend on how often that feature was used at a par- 
tlcular site. The main difficulty is that surprise 
scores are only a rough guide to the actual siEnl- 
ficance of a failed node. The true significance of 
a failure in the context of a particular command 
may depend on world knowledge that is beyond the 
grasp of the planning system (e.~, the laser 
printer is down for days this time rather than 
hours), or even on a part of the planning context 
itself that is not reflected in the statistical 
averages (e.~, READ-PERM-SUPER-USER is much more 
likely to succeed when READ-PERM is called as part 
of a system d,-,p ceamand than when it is called as 
part of PRINT-FILE). To get a more accurate grasp 
on the significance of particular failures, more 
knowledge-intenslve strategies must be employed. 
~. Svecial-Purnose Problem Localization Rules 
As a mechanism for adding extra knowledge, we 
propose supplementing the surprise scores with 
conditlon-action rules attached to particular nodes 
in the planning tree. The cendltlons in these 
rules can test the success or failure of other 
nodes in the tree or determine the hi~er-level 
planning context, while the actions alter the prob- 
lem localization result by changing the surprise 
scores attached to the nodes. 
The speclal-purpose rules which we have found 
useful so far add information about the criticality 
of particular nodes. Consider the following plan- 
aing tree, which is somewhat more successful than 
the previous one: 
SURPRISE 
SUCCESS/FAILURE SCORE 
(PRINT-FILE Ann File2 laser) 
& (IS-TEXT-FILE Flle2) S 
& (UP-AND-RUNNING laser) S 
& (READ-PERM Ann Flle2) F 
I (WORLD-READ-PERM-BIT-SET Flle2) F 
\] (READ-PERM-USER Ann File2) F 
& (IS-OWNER Ann File2) F 
& (USER-REAI~PERM-BIT-SET File2) 3 
I (READ-PERM-GROUP Ann Flle2) F 
& (SA~.-GROUP Ann Flle2) S 
& (GRODP-READ-PERM-BIT-SET Flle2) F 
I (READ-PERM-~PER-USER Ann) F 
& (AUTHORIZED-S~PER-USER Ann) S 
& (SUPER-USER-PASSWORD-GIVEN Ann) F 
+.01 
÷.22 
-. 96 
-.02 
-. 87 
-. 87 
÷.01 
-.55 
÷.05 
-.58 
-.02 
+. 97 
-.02 
Relying on surprise scores alone, the most signifi- 
cant child of READ-PERM would be READ-PERM-USER, 
since its score is most strongly negative. How- 
ever, since IS-OWNER has failed, a node which most 
users are powerless to change, it is clearly not 
helpful to choose READ-PERM-USER as the path to 
report. This is an example of the general rule that 
if we know that one child of an AND node is critl- 
cal, we should include a rule to suppress that AND 
node whenever that child fails. Thus we attach the 
followln8 rule to READ-PENM-USER: 
IF (FAILED-CHILD (IS-OWNER ?user ?file)) 
TH~ (SUPPRESS-SCORE 0.8) 
In our current formulation, the numeric argument to 
SUPPRESS-SCORE gives the factor (i.e., percentage) 
141 
by which the score should be reduced. The-rule's 
affect is to change READ-PERM-USER's score to-.17, 
which prevents it from being selected. 
With READ-PERM-USER suppressed, the surprise 
scores would then select READ-PERM-GROUP, which is 
a reasonable choice, but probably not the best one. 
While the failure of IS-~NER makes us less 
interested in READ-PERM-USER, the very surprising 
success of AUTHORIZED-SUPER-USER should draw the 
system's attention to the READ-PERM-SUPER-USER 
branch. We can arrange for this by attaching to 
READ-PERM-SUPER-USER a rule that states: 
IF ( ~CCESSFUL-CHILD 
( AUTH 0RIZ ED--qU PER-USER ?user)) 
THEN (ENHANCE-SCORE 0.8) 
This rule would change READ-PERM-SUPER-USER's score 
from -.02 to -.79, and thus cause it to be the 
branch of READ-PEBM selected for reportln~ 
While our current rules are ell in these two 
forms, either suppressing or enhancing a parent's 
score on the basis of a critical child's failure or 
success, the mechanlam of special-purpose rules 
could be expanded to handle more complex forms of 
deduction. For example, it mlght be useful to add 
rules that calculate a criticality score for each 
node, working upward frem preassigned scores 
assigned to the leaves. If the rules could access 
information about the state of the system, they 
could also use that in Judging criticality, so that 
an UP-AND-RUNNING failure would be more critical If 
the device was expected to be down for a long time. 
OtheF Problem Localization 
While our System depends on surprise scores 
and rules, an entire range of strategies is possi- 
ble. The simplest strategy would be to hand-code 
the problam localization into the plans themselves 
by the ordering of the branches. At AND nodes, the 
children that are more critical would be listed 
first, while at OR nodes, the lees critical, more 
hopeful, children would come first. In such a 
blocked tree, the first failed child could be 
selected below each node. A form of this hand- 
coded strategy is in force in a~y planner that 
stops exploring an AND node when a single child 
blocks; that effectively selects the first child 
tested as the significant failure in every case, 
since the others are not even explored. Hand- 
coding is an alternative to surprise scores for 
providing an initial comparative ranking of the 
children at each node, but it also would need sup- 
plementingwlth a strategy that can take account of 
unusual situations, such as our specisi-purpose 
rules. 
It might be possible to improve the parfor~- 
mance of a surprise score System without adding the 
complexity of special-purpose rules by using a for- 
mula that allows the surprising success or failure 
of a child to Inarease or decrease the chances o£ 
its parent being reported. While such a formula 
could perhaps do much of the work now done by 
special-purpose rules, it seams a harder approach 
to control, and one more likely to be sensitive to 
inaccuracies in the surprise scores themselves. 
Proper Level p..~Deta.4.1 
One final question concerns identifying the 
proper level of detail for helpful responses. The 
strategies discussed so far have all focused on 
choosing which of multiple blocked children to 
report, so that they identify a path frem the root 
to a leaf. Yet the leaves of the planning tree may 
well be too detailed to represent helpful 
responses. A selection strategy could report the 
node containing the appropriate level of detail for 
a given user. Modeling the expertise o£ a user and 
using that to select an appropriate description of 
the problem are significant problems in natural 
• language generation which we have not addressed. 
IV RELATED APPLICATION ARE~ 
While developed here in the context of a prag- 
matice planner, strategies for problem localization 
could have wide applicability. For instance, the 
MYCIN-llke "How?" and "why?" questions (Shortllffe, 
1976) used in the explanation components of many 
expert systems already use either the already-built 
successful proof tree or the portion currently 
being explored as a source of explanation~ Swat- 
tout (1983) adds extra knowledge that allows the 
system to Justify its answers in the user's terms, 
but the user must still direct the exploration. An 
effective problem localization facility would allow 
the System to answer the question "Why not?e; that 
is, the user could ask why a certain goal was not 
substantiated, and the System would reply by iden- 
tifying the surprising nodes that are likely to be 
the slgnlflcant causes of the failure. Such "Why 
not? n questions could be useful not only in expla- 
nation but also in debugEin~ 
/ 
In the same way, since the execution of a PRO- 
LCQ progr-m can be seen as the exploration of and 
AND-OR tree, effective problem localization tech- 
niques could be useful in debugging the failed 
trees that result frem incorrect logic programs. 
Another example is recovery processing in 
top-down paralng, such as using au~nented transi- 
tion networks (Woods, 1970). When an ATN fails to 
parse a sentence, the blocked parse tree is quite 
similar to a blocked planning tree. Weischedel 
(1983) suEaests an approach to understanding ill- 
formed input that makes use of meta-rules to relax 
some of' the constraints on ATN arcs that blocked 
the original parse. Recovery processing in that 
model requires searching the blocked parse tree for 
nodes to which meta-rules can be applied. A prob- 
lem localization strategy could be used to sort the 
142 
llst of blocked nodes, so that the most llkely can- 
didatea would be tested first. The statistics of 
success ratios here would describe likely paths 
through the grammar. Nodes that exhibit surprising 
failure would be prime candidates for mets-rule 
processiag~ 
Before problem lor~alization can be applied in 
these related areas, further work needs to be done 
to see how many of the heuristics and strategies 
that apply to problem localization in the planning 
context can be carried over. The larger and more 
complex trees of an ATN or PROLO~. program may well 
require development of further strategies. Ho~- 
ever, the nature of the problem is such that even 
an imperfect result is likely to be useful. 
V IMPLEMENTATION DE~CRIPTION 
The examples in this paper are taken frem an 
Interlisp implementation of a planner which does 
prs~atics checking for a limited set of Unix- 
do, sin requests. The problem localization c~- 
ponent uses a combination of surprise scores and 
special purpose rules, as desoA'ibed. The statis- 
tics were derived by running the planner on a test 
set of commands in a simulated Unix environment. 
VI CONCLUSIONS 
In planning-based pra~matlcs processing, prob- 
lem localization addresses the largely untouched 
problem of providing helpful responses to requests 
unfulfillable in context. Problem localization in 
the planning context requires identifying the most 
hopeful and tractable choice at OR nodes, but the 
most critical and problematic one at AND nodes. 
Statistical surprise scores provide a cheap but 
effective base strategy for problem localization, 
and condition-action rules are an appropriate 
mechanism for adding further sophistlcatio~ 
Further work should address (1) applying 
recovery strategies to the localized problem, if 
any recovery is appropriate; (2) investigating 
other applications, such as expert systems, 
back~ard-chnining inference, and top-down parsing; 
and (3) exploring natural language generation to 
report a block at an appropriate level of detail. 
VII REFER~ CE -~ 
Carberry, E $andra. "Understanding Pragmatically 
Ill-Formed Input. • ~ of ~he Intern~ 
1984. 
Kaplan, Samuel J. ~ ~ From a 
Portable Natural ~ Data ~ase Ouerv System. 
PbD. Dissertation, Computer and Information Sci- 
ence Dept., University of Pennsylvania, 1979. 
Mays, Eric. "Correcting Misconceptions About Data 
Base Structure. " ~ of the ~ of 
the Canadian Society for ~ Studies of 
~. Victoria, British Col,~bla, 
Canada, May 1980, 123-128. 
Maya, Eric. WFailtmes in Natural Language Systems: 
Applications to Data Base Query Systems. • 
~ of t~e Ltnn~ Ammal aa~Aonal Conre~ 
ence on ~ ~ (AAA~-~0~. Stan- 
ford, California, August 1980, 3~-330. 
Sacerdoti, F~ D. =Planning in a Hierarchy of 
Abstraction Spaces." ~ ~ 
(197~l), 115-135. 
$hortllffe, F. ~ Comvuter Based Medical Cons~t~- 
~ons: ~ (North-Holland, 1976). 
Sondheimer, N. and R. ~t Weischedel. "A Rule-Based 
Approach to Ill-Formed Input. • ~ of the 
8th ~ ~ on ~ 
~, 1 980. 
Swartout, Willlam R. "IPLA~: A System for Creat- 
ing and Explaining Expert Consultlng Programs. • 
~ 21 (1983), 285-325. 
Weischedel, Ralph ~ and Norman K. Sondheimer. 
• Meta-Rules as a Basis for ProcessinE Ill-Formed 
Input. = AmeriQan Journal of .~.Ji~JI.~4ZJ~ 
~ (1983) , to appear. 
Wilensk~, Robert. "Talking to UNIX in English: An 
Overview of UC." ~ of the 1982 National 
Co~e~nae of ~ ~ (AA~-~), 
103-106. 
Woods, Willi am A. "Transition Network Grammars 
for Natural Language Analysis." ~.dm~g£.i,Q/,,~l~ of 
the ~ 1.~ (Oct. 1970), 591-606. 
143 
