Understanding Human Action 
Charles F. Schmidt 
Rutgers University 
New Brunswick NJ 
I. INTRODUCTION 
Wittgenstein has said, "If a lion could 
talk, we could not understand him" (1958). 
The point of this rather cryptic comment is 
undoubtedly Wittgenstein's contention that 
language or "language games" are embedded in 
what he termed "forms of life." That is, we 
are able to understand each other not just 
because we share common knowledge about the 
syntactic and semantic conventions for the 
use of words, but also because we share 
common knowledge about the forms of life or 
social reality within which we live and act. 
Wittgenstein's remarkable lion would 
presumably not share our social reality nor 
we have knowledge of the lion's social 
reality. Consequently, Wittgenstein would 
contend that this lion's exhibition of 
speech would not result in our being able to 
communicate with him nor he with us. 
If Wittgenstein were here today, he 
might well contend that a computer that 
doesn't understand the forms of life of man 
will not really be able to understand the 
speech of man. 
Such an assertion would probably 
generate quite a bit of controversy amongst 
those of us interested in natural language 
understanding. Clearly, how one reacts to 
this assertion will depend upon how one 
wishes to explicate the notion of 
understanding. As a psychologist primarily 
interested in how persons understand and 
communicate with each other, I am inclined 
tO take Wittgenstein's contention quite 
seriously. 
The work on belief systems, which will 
be discussed here, has been aimed at 
exploring and explicating this relationship 
between language use or language games and 
the larger social reality or forms of life 
within which these language games are 
played. Consequently, the focus of this 
work has not been upon the parsing of 
sentences nor even upon the understanding of 
paragraphs, but rather upon the 
understanding of human action. 
In pursuing this line of research, it 
has been assumed that language use is not 
understood in this stronger sense of 
understand unless what someone says can be 
related to and understood in terms of the 
speaker's beliefs, intentions, and goals. 
The central notions in reasoning about 
human action are the concepts of a plan and 
of a reason or motive for performing a plan. 
A person's action is explained or understood 
when this action can be identified as part 
of the actor's plan and the reason why the 
actor chose to perform that plan can be 
given. 
A model of how persons reason about 
human action must include then: (I) 
196 
assumptions about how actors reason about 
human action, that is, how a person arrives 
at a plan; (2) assumptions about what can 
count as a reason for a person choosing to 
perform or carry out a plan of action; and, 
finally, in order to use this theory as a 
recognizer of plans one must include (3) 
procedures for moving from the observation 
or linguistic report of actions to the 
identification of the plans and motivations 
that have generated these reported or 
observed actions of other persons. 
Natural language is used to describe, 
explain and evaluate human action and 
speaking is itself a human action. Thus, 
natural language provides both the 
meta-language that is used to discuss human 
action and the vehicle for performing the 
various human actions which Austin, Searle, 
and others (cf. Austin, 1962; Searle, 1969; 
Strawson, 1964; Gandhi, 1974) have referred 
to as speech acts. Consequently, a goal of 
this research is to develop a system which 
will accept a linguistically encoded 
description of a sequence of actions and 
then be capable of answering questions about 
the beliefs and motivations which the system 
believes explains these actions. 
Since natural language represents the 
meta-language within which human action is 
described and explained, natural language 
has been examined in order to identify the 
concepts which are peculiar to the 
discussion of human action. Once these 
concepts are identified, then the problem is 
one of determining: (I) how to represent 
these concepts; (2) how these concepts are 
organized into a "theoretical" system; and 
(3) the nature of the processes which use 
this system of concepts to reason about 
reported or observed actions. In the 
development of BELIEVER some fairly concrete 
ideas about the representation and 
organization of these concepts have been 
developed and various procedures for using 
these concepts to identify the plans and 
purposes of the actors have been tried, but 
this latter problem remains an area where a 
great deal of work must yet be done. 
In order to try to communicate our 
thinking as succinctly as possible the 
introduction of technical notation will be 
avoided. The papers by Schmidt (1972; 
Schmidt and D'Addamio, 1973) and by Bruce 
(Bruce and Schmidt, 1974; Bruce, 1974) all 
focus upon representation and organization. 
The papers by Schmidt(1973) and Brown(1974) 
discuss the implemetation of earlier 
algorithms for plan recognition and the 
interested reader is referred to these 
papers for additional detail and technical 
discussion. 
The assumption that action is 
understood by attributing a plan and motive 
to the actor has highlighted the importance 
of the following types of concepts which are 
found in natural language. Those concepts 
which have figured prominently in the 
theoretical development are: (I) Acts such 
as: give, buy, say, ask, promise, help, ... 
(2) Act Relations such as: make possible, 
motivate, prevent, allow, ... (3) Cognitive 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
States such as: believe, know, expect, ... 
(4) Dispositional Properties such as: can, 
ought, ... (5) Motivational States such as: 
want, desire, ... (6) Role Relations such 
as: father of, friend of, ... and Role 
Terms such as: father, friend, customer, ... 
(7) Evaluative States such as: values, 
likes, hates, .... 
Conceptual types I through 3 above and 
the concept of "can" figure most prominently 
in reasoning about plans, whereas conceptual 
types 5 through 7 and the concept of "ough%" 
are importantly involved in reasoning about 
motive. 
II. PLANS AND THE REPRESENTATION OF ACTS 
Considered from the point of view 
developed here, a plan is not simply an 
appropriately structured sequence of 
actions. 
Rather, a plan is the internal 
representation or set of'beliefs about how a 
particular goal may be achieved that is 
attributed to the actor in order to explain 
the observed or reported actions of that 
particular actor. This distinction is 
important to recognize in thinking about 
human action. Failure to maintain this 
distinction can lead one to consider the 
following two statements to be equivalent: 
(A) P performed act(i) and state(k) was a 
consequence of act(i) and state(k) enabled P 
to perform act(j). (B) P performed act(i) 
and state(k) was a consequence of act(i) and 
state(k) enabled P to perform act(j) and P 
performed act(i) in order to be able to 
perform act(j). 
It is obvious from the way in which A 
and B have been stated that B implies A but 
not the converse. The second statement 
attributes to the actor the belief that 
doing act(i) would result in his being able 
to do act(j). Thus, the logic of explaining 
action in terms of plans involves more than 
a recognition of an intersection between a 
consequence of one action and a precondition 
on some subsequent action. If it did not 
our jails would be much fuller. 
If plans do involve beliefs of this 
type and if we agree that statement A does 
not imply statement B then precisely how are 
we able, as observers of actions, to move at 
times from an observation or report which 
satisfies A to a statement equivalent to B? 
One possibility is that we might directly 
observe beliefs. Except for a few staunch 
believers in ESP, this position seems to 
have very few adherents. A second 
possibility is that we are able to make 
observations of one kind or another which 
are independent of the observed actions 
which allow us to regularly and reliably 
infer the belief states of others. There 
are a few psychologists who seem willing to 
argue for this position in principle 
although I know of no psychological evidence 
to support this position. In any case, it 
seems unlikely that our commonsense theory 
of human action contains rules of inference 
of this type which have somehow escaped the 
notice of the scientists in psychology. 
197 
Having argued that beliefs are 
attributed to another neither through a 
process of deduction nor a process of 
induction, the problem of trying to 
characterize exactly what kind of "logic" is 
involved remains. The logic of this type of 
reasoning may be roughly characterized in 
the following way. 
First, if the observer has made some 
observation which is of the form of 
statement A, then the observer recognizes 
that this observation counts as one reason 
for believing that a statement of the form 
of B is the case. The observer may know or 
believe other propositions which may also 
count as reasons for believing a proposition 
of the form of B and the observer may know 
or believe still other propositions which 
may count as reasons for not believing that 
a statement of the form of B correctly 
characterizes the beliefs of the actor. 
Next, the consistency of these various 
reasons must be assessed. The action is 
understood when the observer has arrived at 
some consistent set of reasons for 
attributing to the actor certain beliefs and 
motives which explain the actor's 
performance of the action in question. 
Now if this accurately characterizes 
the reasoning process, then acts must, for 
purposes of reasoning about plans, be 
represented in a fashion that will yield 
access to the kinds of beliefs that must be 
attributed to the actor if his action is to 
be described using a particular act concept. 
For example, the action "P handing a quarter 
to R" where P and R are two different 
persons, may under various circumstances be 
described as an act of "buying," "repaying," 
"helping," and so on. Which particular 
description is most appropriate will depend 
upon whether or not there is a basis for 
attributing to the actor, P, the additional 
beliefs and motives that are implied by the 
concepts "buy," "repay," and "help. The 
class of actions termed speech acts has been 
extensively discussed and the nature of our 
approach to representing acts may be briefly 
exemplified by considering the speech act 
"request" or "ask." Table I presents in a 
very informal way the kinds of information 
which must be represented and associated 
with "request" in some way in order to 
capture the structure of the plan or subplan 
that "request" stands for. Those terms in 
Table I which have been capitalized form the 
basis for the formal representation of 
speech acts. Except for those statements 
which appear under the subheading of Outcome 
Possibilities, each statement refers to 
psychological states of the actors. Thus, 
an act name is simply a way of organizing a 
set of beliefs about how a "move" of this 
type might be related to other moves or 
actions and to the cognitive and 
motivational states of the actors. 
The statements referred to as the Can 
Conditions express those conditions which 
must be true in order for this action to 
have been planned. CI expresses the fact 
that the agent must have believed that he 
was able to transmit his message to the 
recipient. This is expressed at a very 
Act Name: Request 
Argument List: agent: A; recipient: R; message: M; requested 
response: X 
Can Conditions: CI: A EXPECTS that A CAN CAUSE some action 
such that that action results in R KNOWING A's message. C2~ A 
EXPECTS that R will CHOOSE to UNDERSTAND A's message C3: A 
BELIEVES that R BELIEVES certain propositions; AND A EXPECTS that 
\[R's KNOWING A's message AND R BELIEVING certain propositions\] 
will result in R BELIEVING: (I) A WANTS X (2) A WANTS R to CAUSE X 
(3) A BELIEVES that R CAN CAUSE X (4) A EXPECTS that A's 
REQUESTING may MOTIVATE R to CAUSE X (5) A BELIEVES tha,t R was NOT 
MOTIVATED to 
CAUSE X prior to A's REQUEST. 
Goal Hypotheses: GI: R BELIEVES that A WANTS R to CAUSE X 
G2: A's REQUEST may MOTIVATE R to CAUSE X 
Outcome Possibilities: O1: R will UNDERSTAND A's 
COMMUNICATIONACT 02: If Someone PERCEIVES A's message, then that 
Someone CAN UNDERSTAND A's COMMUNICATIONACT 
Motivational Hypotheses: MI: A WANTS R 
WANTS R to CAUSE X M2: A WANTS X 
M3: A WANTS R to CAUSE X 
to BELIEVE that A 
Normative Obligations: NI: If someone BELIEVES that A is 
COMMUNICATING then that someone BELIEVES that A OUGHT to 
UNDERSTAND A's message N2: If R BELIEVES that A is COMMUNICATING 
to R then R BELIEVES that R OUGHT to UNDERSTAND A's message N3: If 
R BELIEVES that A is REQUESTING that R CAUSE X AND R EXPECTS to 
NOT CAUSE X then R BELIEVES that R OUGHT to EXPLAIN to A why R 
EXPECTS to NOT CAUSE X 
TABLE I. Representation of the Action REQUEST 
198 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
general level by statingthat any causing 
which the agent performs which brings about 
the recipient's knowledge of the agent's 
message can count as an action done in order 
to partially make possible the performance 
of the request. For example, uttering a 
sentence in the recipient's presence, 
sending the recipient a telegram or letter, 
requesting another person to give the 
message to the recipient, etc. are all 
actions which could under the appropriate 
circumstances be viewed as part of the 
agent's requesting X of the reclpien~. 
Thus, the information in CI provides a basis 
for generating hypotheses about how other 
observed actions might be related to this 
plan of requesting. 
Statements C2 and C3 differ from CI in 
that whereas CI states that some action of a 
particular type must be taken by the agent, 
statements C2 and C3 simply state 
expectations that the agent must have. 
However, by stating what conditions the 
agent must expect, these statements may also 
provide a basis for recognizing that 
previous actions are related to the plan of 
requesting. For example, if the agent first 
addresses the recipient by name, this action 
may have been done in order to provide a 
basis for expecting that the recipient will 
listen to his request. Similarly, the agent 
may precede a request for a ride home with a 
request for information about whether or not 
the recipient drove to work today. This 
information would provide the agent the 
basis for forming a belief about condition 
(3) of C3. 
Whereas the Can Conditions provide a 
basis for generating hypotheses about how 
previous actions might fit into the plan of 
requesting, the Goal Hypotheses provide the 
information needed to generate hypotheses 
about how the action of requesting might 
itself fit into some larger plan of the 
agent. 
For example, G2 provides the basis for 
the hypothesis that the purpose of the 
agent's request was to create in the 
recipient some set of beliefs that will 
partially provide the reason or motivation 
for the recipient to cause X. 
The Outcome Possibilities express the 
information needed to recognize cases where 
A's plan goes awry or generates side-effects 
which enable latter actions which were not 
the focus of the agent's plan. For example, 
someone might overhear the agent's request 
and offer to fulfill the agent's request. 
This would then be a very different 
situation from the one that might result 
from a similar offer from the intended 
recipient of the request. 
In 01, 02 and in the various normative 
statements the concept UNDERSTAND is 
introduced. UNDERSTAND is being used here 
in a technical sense that deserves comment. 
From the point of view of BELIEVER, part of 
the actor's knowledge of his social world is 
that other person's are capable of 
interpreting and coming to an understanding 
of any observed action. That is, other 
persons also possess belief systems and 
therefore in communicating with another one 
must recognize that the overt action, what 
has been termed the message, will be 
interpreted by others within the context of 
their beliefs about the actor's plan and 
motive. Thus, the theory of human action is 
essentially recursive. This is recognized 
more explicitly in the statement of C3 which 
states that the agent must select a message 
which when taken together with the agent's 
beliefs about the recipient's current 
beliefs can be expected to result in the 
recipient arriving at an understanding of 
the agent's action as a request. This 
characteristic of communication helps to 
explain why most everyday communication 
occurs in such an abbreviated form. To use 
a non-abbreviated form can be insulting 
since it could be interpreted by the 
recipient as a presupposition by the agent 
that the recipient is unable to recognize 
and understand the other's action. The term 
COMMUNICATIONACT is used in 01 and 02 in 
recognition of the fact that the recipient 
or observer of an action may understand the 
agent's action in a way other than that 
intended by the agent. 
The Motivational Hypotheses and 
Normative Obligations represent statements 
which are most importantly involved in 
reasoning about motive. The Motivational 
Hypotheses provide the basis for generating 
hypotheses about the agent's wants whereas 
the Normative Obligations provide 
information about the motivational side 
effects of performing actions to which 
various social norms apply. For example, NI 
is essentially a statement of the sincerity 
norm which applies to all communication acts 
and asserts that the communicator ought to 
believe what he is communicating. N2 states 
that it is improper to ignore someone whom 
you believe is speaking to you. 
N2 is related to C2 since N2 provides 
the norm which can be used to provide the 
basis for expecting another person to try to 
understand the agent's message. Finally, N3 
states that if the recipient has understood 
the agent's action as a request and if the 
recipient is not planning to fulfill that 
request, then the recipient ought to explain 
to the agent why he will not honor the 
request. Thus, this norm provides a basis 
for generating hypotheses about the 
recipient's response to the agent's request 
or the agent's response to the lack of a 
response from the recipient. This latter 
case is especially important since one of 
the interesting characteristics of human 
action is that under certain circumstances, 
the lack of a response counts as an action 
which itself must be explained. 
Clearly, if an act of omission is to be 
recognized, the observer must have a basis 
for expecting the act that was omitted. N3 
also provides the basis for various 
strategic actions. For example, the 
panhandler who asks a bypasser for a dollar 
and then demands with great moral 
indignation an explanation for the • 
bypasser's lack of response is playing upon 
this particular norm. 
199 
Having focussed in some detail upon the 
representation of this one particular act, 
it is now possible to briefly and roughly 
state the position that this leads to 
concerning memory and inference. First, it 
is assumed that human conceptual memory is 
in some sense organized around various 
points-of-view. The representation sketched 
in Table I suggests the meaning" of 
"request" from our Belief System point of 
view. However, "request" also has a 
representation from a linguistic point of 
view and this would require that different 
kinds of information and relations be 
specified that would reflect this point of 
view. Thus, the general assumption is that 
concepts can play various roles in various 
"theories" that the Understander posseses. 
Which particular point of view is "active" 
at a particular time should depend upon the 
goals of the understanding system at that 
point in time. Thus, the position here is 
not that the kind of representation for the 
concept "request" that is presented in Table 
I replaces or is logically prior to a 
representation of "request" as a linguistic 
entity. 
Secondly, and more specifically with 
respect to the conceptual memory for 
concepts within the Belief Systems domain, 
the assumption has been made that the 
organization of knowledge reflects the role 
that that knowledge plays within the 
"theory" that the Understander uses to 
reason about the domain. The various 
subheadings provided in Table I and the 
discussion of this information was intended 
to reflect the view that the conceptual 
structures of human memory contain 
information about how they fit into or can 
be used to answer or reason about the 
various types of questions that must be 
generated and answered by a reasoning 
process that is attempting to understand an 
action in a particular way. Thus, these 
concepts are ,'hypotheses" laden structures 
which serve as the basis for generating the 
goals and subgoals of the reasoning process. 
From this point of view, reasoning about 
action is itself a goal driven process where 
the ',top-level" goals are to find a plan and 
a motive for the actor which explains his 
action. An attempt to view the actor's 
action as a request, provides the basis for 
generating new goals for the understanding 
system. That is, the request structure 
specifies the kind of additional information 
to be assembled which would count as reasons 
for viewing the observed act as a request. 
Finally, since it is assumed that the 
understanding of human action is a 
theory-driven process, it is also assumed 
that the representation of particular events 
in memory reflects the operation of this 
theory. Understanding is viewed as a 
constructive process and what will be 
remembered is not simply a representation of 
"what happened", but rather an organization 
and elaboration of what happened that 
reflects the way in which the observer has 
understood what happened. 
200 
Acknowledgements: G. Brown, B. Bruce, J. 
Goodson, A. Sedlak and N. Sridharan have 
all been involved at one time or antoher in 
this work on belief systems and they along 
with S. Amarel and W. Fabens have 
contributed to much of the work described 
here. Most of what makes sense in this 
paper is due to their various contributions. 
The nonsense is the author's own. 
REFERENCES 
Austin, J. L. How to Do Things with Words. 
Clarendon Press, Oxford, 1962. 
Brown, G. The BELIEVER System. Computer 
Science Department, 
Rutgers, 1974, NIH Report CBM-TR-33. 
Bruce, B. C. Belief Systems and Language 
Understanding. Computer Science 
Department, Rutgers, 1975, NIH Report 
CBM-TR-41. 
Bruce, B. C. and Schmidt, C. F. Episode 
Understanding and Belief Guided Parsing. 
Computer Science Department, 
Rutgers, 1974, NIH Report CBM-TR-32. 
Gandhi, R. Presuppositions of Human 
Communication. Oxford 
University Press, Delhi, 1974. 
Schmidt, C. F. Modeling of Belief Systems. 
Section 3. Second Annual Report of the 
Rutgers Special Research Resource on 
Computers in Biomedicine. 1973. 
Computer Science 
Department, Rutgers. 
Schmidt, C. F. A Computer Simulation Model 
of the Social Perception of 
Interpersonal Episodes. Computer 
Science 
Department, 1972, NIH Report CBM-TR-11A. 
Schmidt, C. F. and D'Addamio, J. A Model 
of the Common-Sense Theory of Intention 
and Personal Causation. Proc. Third 
International Joint Conference on 
Artificial 
Intelligence, Stanford, 1973. 
Searle, J. R. Speech Acts. Cambridge 
University Press, 
London, 1969. 
Strawson, P. F. Intention and Convention 
in Speech Acts. Ph~losophical Review, 
1964, LXXIII, 439-460. 
Wittgenstein, L. PhilosQphical 
Investigations. Macmillan, 
New York, 1958. (Tr. by G. E. M. 
Anscombe). 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
