STPJkTEGX?. SELECTION FOR AN ATN STNTACT~C PARSER 
Giacomo Ferrari and Oliviero Stock 
Istituto dl tingulstica Computazionale - C~TR, Plsa 
Performance evaluation in the field of natural language processing is 
generally recognised as being extremely complex. There are, so far, no 
pre-established criteria to deal x~th this problem. 
I. It is impossible to measure the merits of a grammar, seen as the 
component of an analyser, in absolute terms. An "ad hoc" grammar, constructed 
for a limited set of sentences is, without doubt, more efficient in dealing with 
those particular sentences than a zrammer constructed for a larger set. 
Therefore, the first rudimentary criterion, when evaluating the relation~hlp 
between a grammar and a set of sentences, should be to establish whether this 
grammar is capable of analysing these sentences. This is the determination of 
linguistic coverage, and necessitates the definition of the linguistic 
phenomena, independently of the linguistic theory which has been adopted to 
recognise these phenomena. 
2. In addition to its ability to recognise and coherently describe 
linguistic phenomena, a grammar should be Judged by its capacity to resolve 
ambiguity, to bypass irrelevant errors in the text being analysed, and so on. 
This aspect of a grammar could be regarded as its "robustness" \[P.Nayes, R.Reddy 
1979\]. 
3. Examining other aspects of the problem, in the analysis chat we propose we 
will assume a grammar which is capable of dealing with the texts which we will 
submit to it. 
Let an ATN grammar tl, vlth n nodes, be of this type. N will be maintained 
constant for the following discussion. 
BY text we intend a series of sentences, or of utterances by one of the 
speakers in a dialogue. When analysing such a text, once a constant N has been 
assumed, it is likely that, in addition to the content (the arglm~ent of the 
discourse) indications will appear on the grammatical choices made by the author 
of the text (or the speaker) when expressing himself on that argument (how the 
argument is expressed). 
When these indications have been adequately quantified, they can be used to 
correctly select the perceptive strategies (as defined in \[Kaplan 72\]) to be 
adopted in order to achieve greater efficiency in the analysis of the following 
part of the text. 
4. For our experiments we have used ATNSYS \[Stock 76\], and an Italian 
grammar with n - 50 (127 arcs) \[Cappelli st at.77\]. In this system, search is 
depth-first and the parser Interacts with a heuristic mechanism which 
orders the arcs according to a probability evaluation. This probability 
evaluation is dependent on the path which led to the current node and is also a 
function of the statistical data accumulated during previous analyses of a 
"coherent" text. 
The mechanism can be divided into two stages. The first stage consists of the 
acquisition of statistical data; i.e, the frequency, for each arc exiting from a 
node, of the passages across that arc, in relation to the arc of arrival: for 
each arriving arc there are as many counters as there are exiting arcs. 
f {e)-*x. f (b}:y 
~f{,):w. f(b),* 
Fig. 1 
In this way, in Fig. I arc 1 has been crossed x times coming from a and y times 
coming from b. In the second stage, during parsing, in state S, if coming from 
a and w > x, arc 2 is cried first. 
113 
4.1 Thus, a first evaluation of the linguistic choices made is provided by the 
set of probability values assocla~ed to each arc. These figures can to some 
extent describe the "style" of any "coherent" text analysed. (For this one 
should also take into account the different lin~uistlc significance of each arc. 
In fact a CAT or PUSH arc directly corresponds to a certain linguistic 
component, while a JUMP or VIRT arc occurs in relation to the technique by which 
the network has been built, the linguistic theory adopted, and other variables.) 
4.2 The second part of the mechanism, ~he dynamic reordering of the arcs, 
coincides with a reordering of the co~prehension strategies. In this way, a 
matrix can be associated to each node, giving the order of the strategies for 
each arc in arrival. 
For each text T, there is a set of strategies ~ ordered as describod above. 
While the analysis of the probability values for distinct texts T and T" can 
give global indications of their lln~ulstlc characteristics, if we focus on ~he 
comprehension of the sentence, it is more meaningful to give nvaluatlons in 
relation to the sets of strategies, ~T and ~ , which are selected. 
Fig. 2 shows , for some nodes, a comparison between the orders of the arcs for 
the first Ii sentences from two texts, a science fiction nnvel (SFN, upper 
boxes) and a handbook of food chemistry (FC, lower boxes). The arc numbers are 
referred ~o the order in the original network. The figures which appear after 
the - in the heading indicate the number of parses for each sentence. An ec~cy 
box indicates the same order as that shown in the previous box. 
S b/,S2 1~ 1~ 
312 
~,~RT 312 
312 
;P/Qn 312 
312 
~/~36 ~ 1/R~'L1 51324 51342 
52134 52134 
S/~3~"l 51234 
512.14 
;V/R~'I 41 235 45123 41235 43125 
G~/Y 41 23 
4213 2413 
123 
Sl 213 213 
123 
GV/~I~ I 123 123 
G~N42 1 123 1123 
• t\[ ~11 .~4Sl 51342 
4123 41 Y~ 
4~I,1 
52143 
5.1 It is to be expected that thls mechanism, in an far as it Intrnduces a 
heuristics, will increase the efficiency of the system used for the linguistic 
analysis. The results of our experiments so far confirm this. This ir~roved 
efficiency can be measured in three ways: 
a) locally, in terms of the computational load, due to non-determinism, ~ich is 
saved in each node. In fact, by some experiments, it is possible to 
quantify the computational load of each type of arc. The computational load 
of a node is then a linear combination of these values and one can comgare it 
with the actual load determined by the sequence of arcs attempted in that 
point after the reordering. 
b) in terms of an overall reduction in computing time; 
C) in terms of penetrance, i.e. the ratio between the number of choices which 
actually lead to a solution and the total number of choices wade. 
5.2 If T is a text containing r sentences, the average penetrance will be: 
o .=..', 
where ~ stands for each of the sentences in T. 
If T is analysed using the set of strategies chosen for a different text, T °, 
then the penetrance is, on average, no greeter than with~ T • 
114 
In our experiments, for instance, the avera~_e oenetrance for the first text 
(SFN) parsed with its own strategies (~s##) is ~ed,SFN) = 0.52, while parsed 
with the strategies of the second text (Sty) is ~(5~,SFN) = 0.39. 
We have attempted to evaluate experlmentallv the relationship between the 
difference of the average penetrances, which we call dlscrepanc7 
and the distance between two sets of strategies. However we think we need more 
experimentation before formalizing this relationship. 
Returning to our science fiction novel, the discrenanc- using its set of 
strategies and the one inferred by the food chemistry text is 
6. In addition to the definition of a heuristic mechanism which is capable of 
in~rovinE the efficiency of natural language processing, and which can be 
evaluated as described above, our research aims at providing a means to 
chsracterise a text by evaluating the ~ramr~atical choices made by the author 
while expressing his argument. 
We are also attemptin~ to tako into account the expectations of the listener. 
In our opinion, the listener's expectations are not limited to the argument of 
the discourse but are also related to the way in which the argument is 
expressed; this is the equivalent of the choice of a sdb-grammar \[Kittredge 7~\] 
We intend to verify the existence of such expectations not only in literature 
or x~hen listening to long speeches, but also in dialogue. 
References 
I. Cappelll A., Ferrsri G., Horetti L., Prodanof I., S~ock 0.= "An 
Experimental ATN Parser for Italian Texts" Technical Report. LLC-CNR. Pisa 
1977. 
2. Kaplsn R.- "Augmented Transition Networks as Psychological t*~dels of 
Sentence Comprehension" Artificial Intelligence 3 1972. Amsterdam - flew York 
- Oxford. 
3. 8ayes P., Reddy R. - "An anatomy of Graceful Interaction in Spoken and 
written ~n~chine Communication', C~-CS-79-144, Pittsburgh PA, 1979. 
4. Kittredge g.- *Textual Cohesion Within Sublanguage.s: Implications for 
Automatic Analysis and Synthesis*, COLIN~ 78, ~ergen, 1978. 
5. Stevens A., Rumelhart D.- "Errors in ReadlnR:An Analysis Using an Augmented 
Transition Network Hodel of Grammar" in Horman D., Rumelhart D. eds., 
Explorations in Cognition, Freeman. S.Francisco, 1975, pp. 136-155. 
6. Stock o. - "ATN~YS: Un sisteme per l*analisi grammaticale automatics delle 
lingue naturali', NI-R76-29, IEI, Pisa, 1976. 
115 

