"Expertness" from Structured Text.? 
R~CONSIDEI~ A Diagnostic Prompting Pragram 
Mark S. Turtle a'b, David D. Sherertz c, Marsden S. Btois a, Stuart Nelson d 
baS ection on Medical Information Science, A-16, Univ. Cal. San Francisco, San Francisco, CA 94143 
Computer Science Division -EECS, Univ. Cal. Berkeley, Berkeley, CA 947~0 
C(formerly) Section on Medical Information Science, UCSF 
C(currently) Tandem Computers, Inc., 19333 Vallco Park~-ay, Cupertino, CA 95014 
"Dept. of Medicine, State Univ. of New York at Stony Brook, Stony Brook. NY 11794 
All~tract: 
RECONSIDER is an interactive diag- 
nostic prompting program which uses 
simple information retrieval tech- 
niques to prompt a physician regard- 
ing possible diagnoses, given a list of 
positive patient findings. Its 
knowledge base consists of "struc- 
Lured text" definitions of 3262 
diseases and a synonym dictionary 
Patient findings, and their synonyms, 
are matched against inverted files of 
terms from the disease descriptions, 
the number and selectivity of the pa- 
tient findings matching terms in a 
given disease description determine 
that disease's "score", and the 
matched diseases are sorted on this 
score to form a preliminary 
differential diagnosis. Defimtions of 
diseases can be referenced for view- 
ing by name, or by their position in a 
differential While its first formal 
evaluation is not yet complete, the 
performance of RECONSIDER contin- 
ues to exceed the expectations of 
user and designer alike. 
1. \]ioU~tl~ ~ B~cl~ro~d 
A review of the various means by which 
medical knowledge is represented in symbolic 
form \[6,7\] led us to formulate the \[o\[Iowing 
speetrun~: 
Human. Machine 
Processible Processible 
Spectrum of 
Medical Knowledge Representation 
Schemes 
The two endpoints of the spectrum represent 
the limiting cases wherein 
knowtedge is difficult, or impossible, to 
process algorithmically, but transparent 
to medical personnel, e.g. free text; 
or easily processibte algorithmicatty, and 
dil~cuit to process by humans untrained 
in applied mathematics or computer sci- 
ence. e.g. a matrix of Bayesian probabili- 
ties, or a semantic network. 
Those attending this conference will be familiar 
with work at both ends of the spectrum, if not in 
medicine, then in other knowledge domains. 
Most will concede that the greatest "successes" 
in the field of expert systems has been achieved 
by those working at or near the right-hand end 
of the spectrum; and that progress has been 
most difficult to achieve at the Left-hand end of 
the spectrum. We concluded that. for the short 
run at least, those successes at the right-hand 
end would prove to be self-limiting - knowledge 
that was not readily accessible to and modifiable 
by the medical community at large could not 
remain in the mainstream of medical practice. 
Similarly, we saw no immediate prospects for a 
breakthrough in the algorithmic understanding 
of free text. though ~npressecL with accomplish- 
ments in the area of natural language access to 
databases \[9, I0\]. 
The dilemma these observations implied led 
us to formulate the following question: 
Co~r~ knowledge ubo=t diseases 6e 
1"e:presen.f.ed in. n fo'mw, f~ha/, is ea.s'i2y 
comprehended O~j ph.ysici~r~s not 
t~¢/rted iTt co~rt~tL~e~ science or 
aT~i.f~ci.aL i.n.te/Ii.gew.ce, (zv'td ~le~ s~il.l 
6e algori.l.hrn'i.cally lm'ocess~,ble to- 
~u~.rd sonic rne dic olAg u.s e f'u2 e~2~ 
Our initial attempts to answer this question led 
us to formulate yet another knowledge 
representation scheme, one which operated 
somewhat to the right of the human-processible 
end of the spectrum. Conceding the important 
role of zvo~-ds (rather than text) as conveyers of 
meaning in medicine, we focused on a hierarchi- 
ca/ nominal-attribute model, wherein nominals 
(electrons, cells, lungs, eta) were "defined" in 
terms of attributes (spin, neoplastic, congested, 
etc.). Obviously, nominals could be attributes of 
other higher level nominals, and attributes 
could be nominals at a lower level. The principle 
result of this model was the observation that 
some words had meaning only at certain levels - 
electrons could not be congested or neoplastic, 
nor could Lungs or cells have spin. While the 
idea of "levels of description" is not new, such 
levels were observed to be both well separated 
and powerful determiners of context in medi- 
cineJ In turn. well defined contexts implied, not 
lTh~ is not a tautology. La '=he world of ar*.ffac'.s (:na.n- n~ade 
nora.reals), levels ere not so well separated or orderly Until recent~ 7 one would not ordL, m~/y ~i.'tk of 'spark p:'.~' 
and 'computer' as hav~.~ c\]ose!y corrected .-neanmgs. but new elec~omc ~m~on =F~tem~ in caI~ combine ~oCu 
~n a mn~le sFstem. Bioiogzcai s~terr~ are not so :.~ee~y re- 
arrar~ed. 
surprisingly, well determined meanings for 
words, diminishing the need for syntax to clarify 
or disambiguate meanin 8. 
Our search for an body of knowledge on 
which to explore certain hypotheses regarding 
such a nominal-attribute model in medicine led 
us to regard a familiar but little used resource 
ill medicine in a new light. A corpus of com- 
puter readable disease deflmtions was seen to 
be a crude instantiation of the model. In this 
corpus each disease was given a name (a nomi- 
nal). and defined by its (usually clinical) attri- 
butes - the original motivation for the corpus 
being the standardization of disease nomencla- 
lure. The attributes were written in a tale- 
graphic, but otherwise easily readable style, and 
organized, for each disease, in a relatively stable 
format - a form we have chosen to call st~c- 
t~d ttzt. 
Superficially. the corpus had but one level 
of description, attributes of diseases. But each 
disease definition was divided, explicitly, into 
"contexts" (etiology. symptoms ..... lab. x-ray. 
...). and each disease was place in one (or. at 
most two) "body systems" (whole body. skin ..... 
urogenital .... ). These contexts and systems 
were obviously strong, if imprecise, determiners 
of context. 
Early experiments \[3.4\] with this corpus. 
the computer readable version of Cto-rtn2 Msd/- 
c,., I, tfo,',nu.~n ~ T~n~o~ (C~IT), 4¢h 
Edition \[II\]. explored the selective and associa- 
tive power of the words it employed, and 
confirmed our hypothesis that word use in it was 
both relatively consistent and systematic. We 
soon realized that the sharpest test of the abil- 
ity of words to convey meaning in this context 
was to evaluate the corpus as a knowledge base 
for a "diagnoses program" which would accept a 
description of the patient in the form of a list of 
words, such as 'pain. fever, jaundice .... '. The 
specific diagnostic problem we addressed was 
that of formulating a "differential diagnosis ''z 
\[12. 15\]. which included, as alluded to by Scad- 
ding \[21, 5\]. diseases that a physician might not 
othaT~ise think of. but. perhaps, should think of. 
Important to our attempt to formulate a 
diagnostic prompting program was not only that 
the knowledge base should be readily 
comprehensible, but, if the disease "prompts" 
were to be credible, the "reasoning" by which 
diseases were retrieved and ranked had to be 
equally accessible - a consultative criterion 
noted by Shortlfffe and co-workers \[P~. 23\]. 
In addition, the availability of a knowledge 
base contmning in excess of 3000 disease 
descriptions has allowed us to study phenomena 
that would be hard to reproduce in the context 
of most "expert systems". ~ For example. 
zA "dlfferentla/ diagnose" ts u.sua\].ly a List of dJxcases 
which represents the current L~'~:ing of a phymcian resard- 
tr~ poamble ~ha~n~em for • 81ven pauent, at a Iliven point m the diagnor..lc process. 
best know Cll~q~noms program, an expert sy~ern fm'merly named .~NTERbr~T - ,now eai~cd CADUCEUS, current- 
appended to this paper is a transcript of an 
interaction with RECONSIDER regarding a case 
of methanol poisoning supplied by one of the 
authors (SIN). None of the patient findings are 
particularly specific, but RECONSIDER places 
the correct diagnosis in Oth place, and deter- 
mines that most of the diseases near the top of 
the differential are "whole body" diseases, a 
group containing most toxicity diseases. If this 
differential were selected from among a few hun- 
dred diseases, or even from a knowledge base of 
toxicity diseases, the result would he more open 
to a variety of less favorable interpretations. 
Put differently, when one is retrieving from such 
a large knowledile base. one is more tolerant 
about the appearance of "false positives" 
(diseases that shouldn't be there) in the 
interests of minimizing the number of "false 
negatives" (diseases that should be there, but 
are not). 
Finally. RECONSIDER provides a test bed 
for the evaluation of some hypotheses regarding 
the kind of problems encountered representing 
and utilizing knowledge about the 'natural', as 
opposed to 'artificial'. world. Briefly. RECON- 
SIDER benefits from the high degree of struc- 
ture observable in diagnostic medicine, in spite 
of our ignorance in many areas, and the other- 
wise generally unappreciated stability and 
specificity of medical language regarding this 
structure. 
2. Zxtmtetattons? 
Non-medical audiences should be reminded 
of differing expectations regarding such mean- 
ing representation experiments. As computer 
scientists, two of us (MST & DDS) "knew" that 
meaning could not be represented satisfactorily 
by words alone: words were ambiguous, in gen- 
eral. and, besides, syntax was a partner with 
semantics, and to separate the two was to 
grossly distort the meaning of either. 4 We 
regarded early efforts as potentially interesting 
from the point of view of statistical linquistics - 
how did words and contexts associate? However. 
the medically trained member of the initial 
team (MSB) predicted the successful perfor- 
mance of RECONSIDER once he saw the results 
of some early word-counting experiments. 
Later. SN, an internist with a background in 
mathematics, anticipated the performance lim- 
iting aspect of RECONSIDER without ever using 
the program! (He predicted that inadequacies 
in the knowledge base would be more important 
than any shortcomings in the algorithms by 
which descriptions of patients were "matched" 
w~th the descriptions of the diseases.) 
\]y "understands" a few hundred dineaJee in the field of '--'Iter- aal medicine \[19, 18. 20, 18, 14\]. 
4A local example of failure m "~-text lear~hl~" WaS 
recently bro~l~tt to our attention \[13\]. \[n a search of docu- 
nnen~ in a daymblum collected for a m~ut regarding a large const~ction project, p~c/mo~ (the probability of • do.u- 
meat beu~ relevant) wu no better than 80~ which hush: have been acceptable except for ~.he fact that the reea~ (the 
probability that ,.ha relevant docunten~ wli\] be reuneved) was no better than 20~! 
125 
-3. An Example of 'Structured Text' 
CMIT was designed first for human users, as 
a reference of standard disease names (in book 
form it is about the size of the World Almanac), 
and second for computer applications. (The 
RECONSIDER-formatted CKIT definition of 
.tet/~yl 0~co/wL, to~c~t~J appears in the appendix 
of this paper.) The "structure" imposed on CMIT 
definitions is Largely external to the language of 
those definitions. 
First. the entire text of C~T iS organized in the 
aforementioned noTm~t~-c~tr~b~e form. the 
disease names being the nominals and the 
descriptions consisting of the attributes of the 
disease, s 
Second. each disease is assigned to one. or pos- 
sibly two, bod~j s'tjsts~s: 
,~Aots bco~ 
skin 
wtusc~oskeletal 
c m-d~ascu/m" 
h.mn~c & lymphatic 
g ~s ~s'o ~ e s ~nu~ 
zu~g~i~a~ 
e'n~ocr~ae 
n~ous 
¢pec~aZ sense ~rg~ts 
Third, each disease is described in po~ts: 
O~ terms (synonyms x, eponyrns) 
e~oto~ 
r~m~to,ns 
c or~pt ic at~ons 
p~Aofo~ 
z~ 
• i.~fe.Fff~.ce~ 6 
Fourth. within each :DaFt, the descending hierar- 
chy of se~.ences, ct~zuaes, and phrases (all 
inferrable from punctuation) are used relatively 
consistently to denote appropriate "chunks" of 
meaning. 
Thus, in this instance, structured tezt is 
tightly edited prose written in nominal-attribute 
form, employing external markers, and rela- 
tively consistent punctuation, style, and vocabu- 
Lary. Put differently. CMIT can be "structurally" 
parsed without the need to /ztfer any of the 
semantics from the text. (Again. a portion of 
this "parse" is what produces the "display" of 
the deflnLtaon of methyl -lcotwl, toz~c-iZy shown 
in the appendix.) 
~/t~ we are le~n~ .~m ore" eva~ua,Aon, the rm~n~s of dmeeses, even when they are descnp~ve r~aw.es (as CM\[T zs 
deslff~ed to encotu-age), are not aJways sufficzent ,.o deter- re.he which d~ease ~ beLn 8 spoken o!. Without the descr~p- 
t~orm (attributes) phymcmr~ world he u.nabLe ~.0 resolve the 
problems created by different ~\]'stems of disease nomencLa- tl~'e. 
SAn u'tcpca-tan~ feat'~u'e of the compu~.er readable vernon of CMIT is that it contm.ns references, men, on of which is aot 
m~de in the printed vermont 
4. The Current I~CONSIDKR Implementation 
4-. L The Inverted File 
Using abstract syntax to represent the 
structure in the text. C?41T was scanned and 
"parsed ''~ to produce a sequence of ts~w.~, each 
with the following attributes: 
ordinal position of teem in phrase 
ordinal position of phrase in clause 
ordinal position of clause in sentence 
ordinal position of sentence in part 
name of part 
disease 
body system(s) of disease 
Thus. a dictionary (containing in excess of 
20.000 such terms) was formed and CM.IT 
"inverted", so that each dictionary entry was fol- 
Lowed by pointers to every occurrence of that 
entry in CMIT. Included with every pointer were 
the seven attributes associated with each 
occurrence of that term. There are 333.211 
terra occurrences in CMIT. for an average of 
about 102 terms per disease, or 79 unique terms 
per disease, the difference being terms that are 
used more than once in a given deflnitior~ In 
principle, this "dictionary" could be used to 
reconstruct CMIT, as it r~preswrtts, in aZter,ta- 
t~ve for.7.at, e=actty tlw saw~ ~nformat~rn! 
This Large inverted file allows et~cient 
searching for terms in the text. The searches 
can be (I) constrained to a context (diseases of 
the skin). (2) constrained to textual proximity 
(adjacency. or membership within a clause), or 
(3) constrained to a definition part (symptoms 
only). 
4.2~ Synonym Dictionary 
A 15.388 term "synonym" dictionary a. 
includes words not in CMIT which are synonyms 
of words used in the CMIT definitions and words 
already in CMIT that are synonyms of each other 
(e.g. prur~tu,s and itcA g) These are parti- 
tioned a_mongst 4.165 "synonym classes" (the 
two or more words within each class are 
synonyms of each other). Search options allow 
searches with or without equiv~dencmg the 
synonyms, and with or without invoking 
hierarchical synonyms. The term "synonym" is 
used generously, as the dictionary is actually 
functioning as a kind of semantic net - connect- 
ing words with strong conceptual Links. It should 
also be noted that RECONSIDER does not 
employ "stemming". All variants of a term (and 
some phrases, e.g. abdominal penn ). including. 
in some cases, mis-speLlings, appear wittun a sin- 
gle • 'synonym class". Though we have not proven 
this, it is our opinion that this synonym diction- 
ary is what converts an interesting tool for 
research into medical term-use, into something 
vOnce age.u~ this par~e L~ .nOt identifylr~ "pa."~ of 
speech" Ln L~e conventional serme. Rat.her ".he ab~rac% t~ (a BICF grammar akan to those deflnLng program,'vzJ~ 
languages) encodes the meaning of :he ex'.erna~ mcrke..-s and 
ptmctuatic~W conventions employed in C.MIT. 
a(:ort.~J'ucted "0y Rod~ey Ludw~, U.D. and HTo ~ \]LD.. 
L2b 
-that functions not unlike an expert system. 
4.3. Searches 
Searches for a set of terms can require a 
match on every term. or a match on one or 
more of the terms in the set. In the latter case. 
matches are scored in a manner reminiscent of 
techniques used for literature and infm-mation 
retrieve/ by Salton. ~parck-Jones and others. 
and in particular Doszkocs \[8\]. The scoring a/go- 
rithm is illustrated in the next section. 
4. 4. The User-Interface 
RECONSIDER is an interactive user inter- 
face running on top of the inverted file and the 
search algorithms. It accepts terms, search 
modifiers, and requests for one of the two 
matching algorithms, formulates the appropri- 
ate query, searches the inverted files, computes 
the score of the diseases retrieved (if 
requested), constructs a body-system histogram 
(if requested), ranks the diseases if appropriate, 
and displays any disease definitions selected for 
viewing or browsing by the user. 
5. Performance 
S.L & Compartsen with two lY~ncet/e Expert 
When applied Lo the published cases diag- 
nosed by INTERNIST and PIP \[R0.17,16\], 
RECONSIDER produced the correct diagnosis 
(or diagnoses) at. or near, the top of the disease 
List produced by enterin~ the positive findings 
given Lo these programs \[5\]. (Again. CADUCEUS 
considers 300 diseases from internal medicine, 
and PIP considers 20 diseases featuring edema.) 
While these cases were often complex, a large 
amount of clinical information was available for 
each patient. 
5.2. Diagnostic Pr~nptin~: An Example 
We believe that RECONSIDER performs 
better, and much more usefully, at an earlier 
point in the diagnostic process, at a time prior 
to any extensive patient work-up, when the 
physician's "cognitive span" is widest \[2\]. 
For example, a patient presents with 
findings as noted at the beginning of the appen- 
dix. RECONSIDER begins by prompting for 
terms. The prefix ~s/is used by the physician- 
user to indicate that the succeeding terms are 
to be searched ~or in either the s'y~npton~s, or 
s~\]rts portions of Lhe disease descriptions. This 
grouping, a union of the two vocabularies, was 
necessitated by the non-consistent usage of 
terms in these contexts. ~ The phrase oObdo~r~,,/ 
pmL't will match (given the RECONSIDER options 
setected to run this case) any co-occurrence of 
these two words (or its synonyms) within a si~gie 
ctause. RECONSIDER responds with the 
synonyms it knows for the terms entered, and 
sThe ~ of terms withe CMIT did not follow t~e ~dioal do~rrm as to what was • ~m;~om, and what w,,, a ssgn. 
the number of diseases containing one of mor'e 
occurrences of each of the terms within the ss/ 
context. The response =bdo.tinaL pain\[ 191+80\] 
indicates that the pah" abdom~n~ pen occurs in 
191 diseases and that 80 additional diseases 
have been retrieved by the synonyms for ~bdom- 
paL-t, namely ccL/c\[3~\], csL~ck~/~16\], end 
pWut /n abdom4~48\]. The fact that 3.5÷15+48 
exceeds 80. and 191+35+18+48 exceeds 
191+180, indicates that some diseue definitions 
contain more than one term from this synonym 
class. 
The score (a measure of selectivity) for 
abdom~n~ p~n is 
0.917 = 1 - (271/3262) 
where 271 is the number of "disease 
occurrences" of abdomma~ p~, and 3252 is the 
total nurnbar of diseases in CMIT. A disease's 
score is the sum of the scores of the terms its 
description matched. 
Most physicians would probably conclude 
that the observation that the patienL smoked 
was not relevant to the patient's illness, but the 
term smo~ was entered here to show its obvi- 
ous effect on the disease List (it brings n~.ott~ur, 
to=/¢/~ and ~g ~pende.ce, ma=-/h~m~ 
nearer to the top, partly because it is so "seLec- 
tive"). It is not clear which 'part' of the disease 
descriptions the term ~wlo~g will be found in. 
so its search context is all/. and the same deci- 
sion is made with respect to =e/dos.iv. An/on gap 
~/dos/8 is not used in C~\]T, so we enter the 
more genera/ form. I° Entering swto~lk~g in the 
a~/context has the disadvantage that it brings 
in a reference to smoky, which is used as an 
adjective. 
The histogram displays the body system 
frequencies for the diseases near the top of the 
disease list (the top 4~, was selected by the user 
to include about the first "screen's worth" of the 
disease List - 8?9 diseases containing one or 
more of the terms entered, or their synonyms). 
A physician-user viewing the first screen- 
full of this ~st (the portion shown in the appen- 
dix) would next formulate a strategy for resolv- 
ing it, assuming the diagnosis was still noL 
immediately apparent. A methodical approach 
would note first that no disease matched all five 
entries (as no disease has a score of 4.738). 
Similarly, diseases #I, #2. and #3 would be ruled 
out by asking the patient appropriate questions. 
(If the patient were from Matin County, here in 
the Bay Area. we might focus our initial aLLen- 
Lion on #2, rn~.sh~'oont, toe'S.city, in response to 
recent news reports of cases of tt there - 
1°An a%ternpt on the par~ of the ,~er so enter witch g~ 
~na, whJJe !audable (it wou~d be very Selectee). wouid be greeted 5y a rr~essage 01at the :er.'n was not found m CMIT or 
its synonym, dictionary - Ln QI~s case because CM,'T predates 
wide v~e o+ '+ this ~es~. At t~e point the phy~e~a.~user must use hi~ ~" 
her own knowled~\[e of med~cme, to know ~hat ~he term ~'ldom Ls the bern. ~bst~tu~e under ~.hese c~r- 
cum~anc~. Looked at differently, our eva~uaUon ~ee.'u= to con.~Lrm ~h~t, in genera\], alor~ medical ~alowledge makes one 
a more effee~ve ~ECOH~ID~ user. ~f t.~e, we regard "~h~s 
as a po~Uve featm-~ ~' RRCON.elDER. 
127 
"~owledge that is not available tu RECON- 
SIDER.) Disease #4, ecto~p~, raises a more 
interesting issue. RECONSIDER does not have a 
model of gender (or of anything else), so a 
disease that occurs during pregnancy is not 
automatically ruled out when the patient is 
male. WhAle understandably distracting at first. 
users are soon comfortable ignoring such inclu- 
sions, especially since it's easy to understand 
RECONSIDER put the disease there. View- 
ing the C~\]T definition of disease #5. nej#u-Lbi~, s"It £o~ 
reveals that it is usually accompanied 
by a rich complex of symptoms, so while it can 
not be ruled out at this point, it becomes 
extremely unlikely. Since the patient is not an 
alcoholic, the definition of disease #6, rn.ethTjt 
~l.cohoL, Lozic%tll. suggests the possibility of 
occupational exposure (perhaps percutaneous 
or respkatory). Once considered, an appropri- 
ate test would confirm the existence of the toxic 
substance in the body. 
8. /k~l-Umr Experience 
We have not permitted RECONSIDER to be 
used '~iva" in a clinical context. In addition to 
the fact that evaluation of the program is not 
complete, the knowledge base is known to be out 
of date. Nonetheless since we have been able to 
move RECONSIDER to the MIS-UCSF VAX 11/750 
running UNIX~ (Berkeley 4.1) students, post- 
doctoral fellows and some faculty have been able 
to use the program. The initial reaction usually 
consists of the following three observations: (I) 
"Why is that disease there?" (sometimes it's 
there Legitimately, and sometimes not), (2) "How 
does such a dumb program do so well?" (refer- 
ring to RECONSIDER's lack of evident reasoning 
power), and (3) "What I need to be able to do now 
is ..." (1111 in your favorite interactive- 
knowledge-base user-feature). 
We tolerate the probiem alluded to by ques- 
tion (I) because it is more important, at this 
stage of development, not to miss important 
diseases, and because it is easier for a 
physician-user to reject totally inappropriate 
diseases than it is for the program to do so. 
Question (2) alludes to the point raised by the 
title of this paper. RECONSIDER can only be 
considered an "expert" (if at all) because its 
knowledge base is so Large (relative to what a 
physician can keep readily available in his or her 
head), and because of its performance. It is 
obviously not like a human "expert" un the way it 
a~'~ves at the disease list. And question (3) we 
take to be a comphment that reveals, among 
other things, that occasionally the utility of 
RECONSIDER is iu~uted not by the knowledge it 
eonteuns, but by the means we currently have of 
accessing it through the narrow window of a 23- 
line CRT terminal. 
Question (1) deserves further comment. 
The author (MST) has observed considerable 
user-discomfort caused by CMIT hexing diseases 
from several body systems near the top of a 
eUN1X is a produc~ of Ben Telephone Laboratories, ~nc. 
sorted disease list. Apparently, the cognitive 
dissonance is usually avoided by thinking about 
diseases by system, an the discomfort can be 
relieved by restricting the search (and thus the 
sorted list) to a single body system. The prob- 
lem with the latter practice is that the prelim- 
inary results of our evaluation reveals that con- 
textless (0~/searches) are the most e~Lcacious. 
on average. AS this is also the opposite of the 
behavior predicted by our model of context in a 
norruna/-attribute knowled4\[e base. further study 
is suggested. In any case, it may prove neces- 
sary to re-design the user-intorface to accomo- 
date some users' need to view deseases by sys- 
tem, within a contextless search. 
7. Evaluation 
A formal evaluation of RECONSIDER on i00 
serial admissions to a tertmry care medical 
ward. is in progress (and will be reported else- 
where), but the prelim/nary results are both encouraging 
and interesting. They are 
encouraging because the correct diagnoses is 
included so often in the first frame or two (and 
usually higher), and interesting because the 
difference between diagnostic programs, and 
diagnostic p~rn4~g programs is made quite 
clear, The former have a very specific goal. and 
it is easy to tell whether it is reached or not. A 
prompting program is evaluated against a 
different standard; not whether it is correct but 
whether it is halpfui And judging whether some- 
thing is helpful or not may be a subtle matter. 
If the correct diagnosis is included h~h on the 
List, the performance can be given a hiKh score. 
But if, instead, a listed disease closely related to 
the correct one has the result of directing the 
physician's attention to the correct body sys- 
tem, and finally the correct diagnosis, how is 
this to be scored? 
8. Suspected \[~mitat/ons: 
8.1. The \]Qaowtedge 
As has been the experience with similar 
projects, computer processing subjects 
"knowledge" to a harsh and unyielding l~ht. We 
anticipate that a half a man-year of "tuning" 
would significantly improve RECONSIDEEs per- 
formance, but that the next and much more 
serious Limitation will be the quality, uniformity, 
completeness, and timeliness of CMIT and the 
synonym dictionary. Given the opportunity to 
rewrite CMIT (and continue to do so on an on- 
going basis), or introducing A\] techniques to 
RECONSIDER (we have received many sugges- 
tions), we would choose the former. 
8.2, Other lJrnit~Uol~ 
Our experience to date has taught us that, 
in this context, negatives are ~nportant. Terms 
such as fe~r u~b.se~tt are teated as if/e-vet were 
a positive finding: while not fatal, such retrievals 
increase the number of false positives. Also 
users often wish to search using "rule-out", e.g. 
elirmnate all diseases from consideration 
128 
containing a certain term. or terms. Especially 
tricky would be interactions between these two 
uses of negation. 
On a more global level. CMITs homogeniza- 
tion of diseases contributes to confusion and 
loss of information. Congestive heart failure is 
listed as a disease under ~r(. fa~,ui'e, cov~ss- 
tt~e. as a symptom under ~art. AMoe~tm,~mye. 
~e--e. as a sign under Hart, AV/m,-tT~pA V, 
/mm-t. f~tv 0kge~rt and ~ sta~a'~. 
,fu.bv,dvuZar, and as a complication in, for exam- 
pie. tr~pznaso~a~s, ~awm~c~m. And to illus- 
trate the stress on the process of attempting to 
form a closed set of synonyms, the symptoms 
and signs of c0n~es~e ~m'~ ~s are 
described at various points as in cm-dio~.,~o- 
pa£A!@, but the phrase conges~bue heart f~ 
does not occur in that description. 
9. Futuure Imp{ementaU~um 
Given an opportunity to re-brnplement CMIT. 
we would retreat h-ore our original notion that it 
should not be modified (so as to prove that 
structured text could be used, intact, as a 
knowledge base). Rather we would maintaIn the 
inverted files dynamically, in a relational data- 
base. so as to facilitate modifications, and 
experiments with alternative knowledge 
representations and retrieval techniques. 
Specifically. we would investigate the difficulty 
of re-writIng CM1T to improve the quality and 
timeliness of the information it contained, to 
use a more standard model of disease nomencla- 
ture \[ 1\]. to evaluate alternative ways of handling 
negation (such as 2m~ug~e -bsent). and the 
allow users to specify necess/~ (a term m.usf 
occur, or not occur, In a disease description for 
it to be retrieved). 
RECONSIDER currently requires some 20 
MB o{ disk space. A dynamically revisable ver- 
sion would require at least twice that. making 
RECONSIDER a little Like an orphan elephant in 
already pressed medical computing environ- 
ments. A "production" version of RECONSIDER 
might fit in 15 MB, leaving two alternatives for 
the future: running RECONSIDER on the large 
address-space micro-based systems now avail.- 
able with large hard disks, or making it available 
on a network. We are toot~n~ into both possibili- 
ties. 
10. Conclumons 
In the context of medical diagnoses, and 
perhaps in other apptication areas. "structured 
text", as defined here, has been neglected as a 
means of representin 8 information in a form 
accessible to both humans and algorithms. If as 
Minsky has put it. "For a program, being smart 
is knowing a tot.", then carefulty edited and con- 
structed natural language text, available in 
computer-readable form, may facilitate the pro- 
cess by which programs come to "know a lot" 
and continue to "know a tot" as the knowledge 
evolves over time. 
We conclude by noting that ultimately tile 
usefulness of diagnostic aids such as RECON- 
SIDER. must await the verdict of users. If the 
cost and bother of their use is less than the 
benefit they are found to provide, we can expect 
them to make their way Into clinical practice. 
Up until the present thne. no diagnostic support 
program seems to have accomplished t~s. 
11. t~.knmeledgementa 
Future reports will include the performance 
of the case "enterere" who have labored to com- 
plete the task of formulating differentials for 
100 cases. As some of their reactions are 
Included here they are acknowledged below. 
Those case-enterers who are not co-authors are 
Mark P, ribaum, M.D.. Peter Harrison. M.D.. Hyo 
Kim. M.D.. Pauline Yelez. medical student. Ber- 
nard Winklmann, M.D.. Dale Yamashita. M.D. 
Append/x: 
A Cane of MethAnol Poisoning 
A 26 year old male was admitted to 
the medical ward of the SUNY Stony 
Brook hospital complaining of ab- 
dorninal pain, confusion, and vomit- 
ing It was noted that Lhe patient was 
a smoker A lab test had revealed an- 
ion gap acidosis. 
Enter ten'm: ss/abdo~rml pain, confusion, 
v~ni t in~ 
Signs or Syrrptcrm: .~x~Ti. nat pain\[191+O0\] (colic\[35 i, cotickytZ6\], pain in abdu',=.\[48\]); 
ccnfusiontBS+7\] (confused\[7\]) ; ~m-~tirv, E4e~l\] (erms~s\[2\], byperm~sis\[2\], 
r~r~m~sis\[1\], vm~tus\[9\]). 
Fnter tents: all/snaking,acidosis 
Signs or Synlptcrn~: _abd@~~l pa in\[191+80\] (cotic\[35J. colicky\[m\], pain in a~n\[4S\]): 
con fus ion\[SS+7\] (confused\[ 7\]) ; ~ting\[~+1\] (~sis\[2\], ~o,p=-,,=s~\[2\]. 
hypereTisis\[ I J. vani tusL9\]). all: ,r~kir~\[Z~S\] (,m£c, ta\]. s-mky\[1\]) . 
acidos i s\[37+1\] (acids'l~a\[ I \] ). 
Cu = m'zi: s 
Cazputing scores for Signs or SMTpLcrs Leto's 
Finished abdo-n\[r~ll pain, selectivity = 0.917 
Finished confusion, selectivity = 0.972 
Finished vcmtting, selectivity = 0.570 
C_xrrputing scores for all ternm 
Finished s'mking, selectivity = 0.99\] 
Finished acidosis, selectivity = 0.~39 
Sorting totaled d, sease scores... 
4.738 - rmxuTu'n total score 
679 diseases in this list 
}-\[isto~r:rn for the top 4~o of the \[tsL 
(27 diseases) 
• • e, • e,+) e+, • ,I + • + o+ Wno~e 5ody • *" " * " "" " " "" ""'"" "*'" " 
lhsc'a oF ¢e\[ e=a~ ''''" 
~sp ; .-a'+ o.-y "" 
Ca_"d : ovascu, a.- "* "=Wr= c--.wrp, hat : c ° 
\[~Og~; ta. ''°°°''''°°° 
F-~Id oc .": ne " 
Ne ."~'otJs "''" 
Signs or Synlptcrns: ,~m::krrnnal pain\[191+SO\]: 
confuston(8,5+7\] ; va.,m t ing{.'l.~"5+-I \]. 
all: ~--mkirg\[Z3+a3\]; acidosis\[37+l\] 
4.T38 - n'~xmn.m total score 679 dzseases m ~nzs 
!:st 
l 3,750 nlcor.lne, :oxlc:ty O0 
2 3.748 n%lsaro~% toxlc:ty O0 3 2.833 dr.~ dependence, n'e.-i~ama O0 
4 2.830 ec ia'rps ~ a 07 5 2,830 neIi"-':t:s, salt los:hE 07 
130 
6 2.7"~ rmthyl alcohol, to~:czty O0 
7 R.7~ food ;x)is~n~0 st m~hylococcal 8 2,776 ca=a, diabetic 08 
9 2.7~9 tlmiliutt toxicity O0 
10 2.7"J~ ars~nc, t~icity O0 
II ~.~ rngraizm syr~ram 00-09 
12 2.7~ p~yria, acute int~mtt(mt 
13 I,MI l~Fe.h~is withrretabolic, .'~tr i ti~ dis~den. O0 
14 1.961 e.~ban diomde, narcceis O~ 
15 l.g61 ~ti~ ¢~4~lopat~y (~ 16 I.~I cam, hepatic 06-09 
17 I.~ f~ syr~'a~'m, adult, 
18 I.~08 disrrhea, d~r~nc ~8 ~@ ~.~ kidr, sy, calcul~ 07 
00-08 
8 2.7?5 nethyt alcotx)t, toxicity 00 
(ss) abd~nai pain\[0.917\], (as) ~t~ng\[0.sv0\], 
{al) acictosis\[O.g{~J. 
O0 (8) methyl alc~oL, toa:icity 
Alterrmte tem'inology \[at\] 
toxicity, r~thyl alcohol 
wood aLcohol, toxicity; 
methanoL, toxicity. 
Etiology let\] 
- Inhalation of vapor, ingestion, 
percu~ absorption of 
flmm'ebte liquid widely used 
in inc~z try'; 
- effect of rmtabolization by body to 
fonmldehyde and fon'ric acid, with 
depressant action on ens; 
- tie, 200 pprnof air; 
- internal lethaL close, 60-P.50 ml or 2-5 oz. 
- OccxR~tiorml exposure: dry cleaning, 
organic synthesis; 
-rfanufacture of antifreeze, dyes, 
explosives, fuel, leather, plastics. 
- Acute poisoning from ir~estion, 
ir~;alation, or percutaneous absorption: 
fatigue: 
- headache; 
- r~u~ea; 
~> vcrra t ing; 
- vision zquaired; 
- phot~ia; 
- dizziness; 
~> in exposure to high concentration or 
ingestion of high dose. 
rr~ni festat ions n~re rmrked as sex'(~re 
upper abdtrr~nal co/icky patr~. 
sweating, possibly biir~lness. 
- O'~ronie poisoning from i,~-~lation, 
percutaneous absorpt i on: vi s ion 
~'zpaired initially, progressive: 
- fatigue: 
- Y~usea. 
Sigm \[sgJ 
- Acute poisoning: with ingestion. 
onset within 8-48 hours: 
- cyanosis: 
- cold, clmm/ skin; 
o eu\[Nnoria; 
- respiration sb-.llovc, 
- blood pressure low;, 
~> features of acidosis; 
- crm depression; 
- convulsions; 
- ca'1"a. 
- Onronic poisoning: ~zerratoid 
demmtitis; 
- conjunctivitis: 
- tracbei t is; 
- brocchi t is: 
- ~.tteady gait. 
- Course: in severe acute poisonir.~, 
rmrtality rate 25-50 percent; 
- inrrilder fon'm, recovery wiLhin 
~ek~ to rnmths; 
- vision, recal function possibly 
irrpai red pen~k,~mt ly~ 
- Treatn~nt: adrinistration of sodium 
bicartxl'mte orally or sodiu-n i~c:,~'e 
intravencuely tot acidosis: 
- irrigation of eyes with ~ater: 
- washing conternirated areas of body 
with soap, water; 
- c(nbating shock with oxygen, 
stkmlants; 
- oral ~tninistration of whiskey or 
intravenous administration of ~'0 
percent ethanol possibly 
inhibiting oxidation of rreth~r,o' 
to its toxic intemmdiates. 
Laboratory \[ lb\] 
- Methyl alcohol in expired ai:', t'_"l~e, 
blood; 
- fore'de acid in urine. 
- 0phthalrrDscopy: in acute ~.i~on!~g. 
di iatation of pupils, ccn, r~':~ ! ~:. ,,f 
vimml fields, hyper~'~a o" o1,'. i," d!~l< 
retinal ede'm.; 
- blind while discs, attenc:,'e,: ',~, -::- 
of optic atrophy. 
Pathot c3:,. \[pc\] 
- .Meningeal petechi~/; 
- cerebra\[ ederm; 
- necrosis of reti~li neurc.:-.~; 
- suh'rucose.l, subepicardia' s .... " 
h~T~r rhage; 
- ~arer~h}rmtous de~en~rat .. ~" . :. 
kich~ey. 
References \[rf\] 
Drei sbach 13 t,/:12 
bIJnter 561 ff 
Jot~nstone-mi 1 let t56./59 
PLtmkett 2~30,,"5 .t 
Thienes-hatey 68 

References

I. . ICD. £ CM- TM In£~ Clu.~cou~m~ 
ol ~seases, ~h RmtL~r~ Com~ssion on 
Professional and Hospital Activities. Green 
Road. Ann Arbor, Michigan (1970). 

2. Blois. M. 5. "Clh~cal judgment and comput- 
ers," New ~,~. ,Tour. Mad. 340B pp. 192-197 
(1980). 

3. Blois. M. S.. D. D. Sherertz. and M. 5. Turtle, 
"Word and Object In Disease Descriptions." 
Proc. of the 18th Annual Meeeting of Association for 
Computational Linguistics. Assoc. for Computational 
Linguistics, (1980). 

4. BloL% M. 5, D. D. Sherertz. and Nu 5. Tuttle. 
"The Algorithmic Processing of Medical Text 
Utilizing Context," ~c. of the 4th .~. 
Med~c,~ C~ (W¢~lvLv~to~. D. C., No~err~er. 2~ao). 
\]zzz. (~gso). 

5. Blois, M. S.. M. 5. Turtle. and D, g. Sherertz. 
"RECONSIDER: A Program for Generating 
Dit~erential Diagnoses." in Prec. of t~ 5th 
Ann. S~npo~.~ o~ Co,npu~cr Ainpt.~c~ 
Me~cu/ C~ (W~Amgt~n. O, C. , 
No'ue.%bsv', 1981), IEEE, New York (1981). 

6. Btois. M. S.. "Conceptual Issues in 
Computer-Aided Diagnosis and the Hierar- 
chial Nature of Medical Knowledge." Jou~'. 
Me~. P~U, 8pp. 29-48 (1983). 

7. Blois. M. 5. /~fo~mu,~ ~ ~ea~c~; An 
HY~r~ch, ia/ W~u;, Univ. of Cal~forma Press, 
Berkeley (1983 (in press)). 

8. Doszkocs, T.E., "'An Associative Interactive 
Dictionary (A/D) for 0n-tAns Bibiiograph,c 
Searching," Proc Alnm"i.cm~ Soc~t~t /or 
Info~tio,~ Sc/e~e 10 pp. 105-109 
(November, 1978). 

9. Epstein. Martin. "Natural Language Access 
to a Clinical Database," TR-51, Section on 
Medical Information Science, Univ. Cal. San 
Francisco, San Francisco. CA (1980). 

-10. Epstein, Martin N. and William Le~zis, 
"Methodology for Creation of and Access to 
a Clinical Database," ProceedL-tgs of t/m 
~rst In~e~,m.,~onal Cm'~J'erence on Medical 
Co.tputer Science, IEEE Computer Society, 
(September, 1982). 

11. Gordon. B. L.(Ed.), Our-rent /Jed~ca/ Info~- 
matiart and Te~,'mAnotogy, 4th Edil #.on, 
American Medical Association, Chicago 
(197t). 

12. Lindberg, D. A. B., I. R. Rowland, C. R. Buch, 
jr., W. F. Morse, and S. S. Morse, "CONSIDE~ 
A Computer Program for Medical Instruc- 
tion," in 8th IBM Me~Lc=L Svm.po~'mm., 
(1968). 

13. Maron, M.E., "Problems with Full-Text 
Searching," Office A1zto~n~t~on ConleTence 
Digest. AFIPS, (April, 1982). 

14. Miller, R. A., H. E. Pople, and J. D. Myers, 
"INTERNIST-l, An Experimental Computer- 
Based Diagnostic Consultant For General 
Internal Medicine," Ne~v E'n.g/.am.~ Jmarnal of 
Mediz/~'u~ 307. No. 8 pp. 468-476 (Aug. 19, 1982). 

15. Morse, S. S., L. R. Rowland, and D. A. Lind- 
berg, "CONSIDER Implementation." Techni- 
cal Report MOU-IS-TR-6, Univ. of Missouri, 
Columbia, MO (1971). 

l6. Myers, J. D.. H. E. Popte, and R. A. Mil/er, 
"INTERNIST: Can Artificial Intelligence 
Help?," pp. 251-269 in ~.%cal Dec/stons 
a~ l.~boTatowJ Use, ed. D. Fenderson,Univ. 
Minnesota Press, Minneapolis (1982). 

17. Pauker, S. G., G. A. CorD,, J. P. Kassirer, 
and W. B. Schwartz, "'Towards the simula- 
tion of clinical cognition: Taking a present 
illness by computer," A~er Jour Meal 
60 pp. 981-996 (t976). 

18. Pople, H. E., "The formation of composite 
hypotheses in diagnostic problem solving: 
an exercise in synthetic reasoning," in 
P~c. of the 5th Int Jo%nl Conf on ~rt~f. 
Intett ,. Cambridge, Mass. (1977). 

19. Pop\[e. H. J., J. Myers, and R. Miller, "DIA- 
LOG: A Model of Diagnostic Logic for Internal 
Medicme," F~roc \[nt Joint Ccnf. A/ , 
77ailsi. USSR. 1975, 0- 

20. Pople, H. J.. Heuristic Methods/o7" lmpo~tg 
Structur~ on Ill Structured Problemz: The 
Struc tur~rtg of ,;ted~c al Piagnostics (1980), 
American Association for the Advancement 
of Science. Washington, D.C. (in press). 

21. Scadding. D., "Diagnosis: The clinician emd 
the computer." Lancet, p. B76 (1987). 

22. Shortliffe, E. H., Computer-Based Medical 
Corts'ultat~ans: MYCfN, Elsevier, New York 
(1976). 

23. Shortiif\[e, E. H.. B. G. Buchanan. and E. A. 
Feigenbaum, "Knowledge engineering for 
medical decision making: a review of 
computer-based clinical decision aids." 
Proc \]EEE 67 pp. 1207-1224 (1979). 
