APPENDIX B:
MUC-5 TEST SCORES
This appendix contains the summary score reports for each system on the MUC-5 evaluation . The reports
are ordered by language/domain (EJV, JJV, EME, JME) and secondarily by site (in alphabetical order).
Deeper investigation into system performance may be done on the basis of template-by-template scor e
reports, which detail the performance of the system on each article in the test set . The template-by-template score
reports are not included in this appendix .
A brief introduction to reading the score reports is presented here . Some information on the performance
metrics may also be found in the paper, "Tipster/MUC-5 Information Extraction System Evaluation" by B .
Sundheim in this volume . Further description of the score report and further definition and explanation of the
scoring categories and performance metrics may be found in this volume in the paper titled "MUC-5 Evaluatio n
Metrics" by N. Chinchor and B. Sundheim .
The first column in the report, SLOT, contains one of the following three types of label:
A slot name, e.g., content
An object name, e.g., <template>
The word "subtotals"
Of the three types of rows, the one of primary interest is the one labeled with a slot name . The rows labeled with
an object name may be disregarded. The rows containing subtotals may be of interest to those interested in
performance on the various types of objects in the template .
The following tallies for the test set are presented in columns in the score reports :
POS
	
Number possible (= COR+PAR+INC+MIS), i .e., the number of fills a system should
generate. This value is somewhat variable among systems because of alternative
cardinalities of slot fills allowed for in the answer key .
ACT
	
Number "actual" (= COR+PAR+INC+SPU), i .e., the number of fills generated by the
system under evaluation
C OR
	
Number correct
PAR
	
Number partially correct
IN C
	
Number incorrect
S P U
	
Number spuriou s
MIS
	
Number missing
NON
	
Number "noncommittal ," i.e., number of null slot fills generated by system, where the fill
is also null in the answer ke y
The rightmost columns in the report contain the scores for the following extraction metrics:
ERR Error per response fill = (.SPAR+INC+SPU+MIS)/(COR+PAR+INC+SPU+MIS )
UND Undergeneration = MIS/P O S
OVG Overgeneration = SPU/AC T
SUB Substitution = (.SPAR+INC)/(COR+PAR+INC )
REC Recall = (COR+.5PAR)/PO S
PRE Precision = (COR+.SPAR)/ACT
None of the above metrics includes the NON scoring category in its computation . Readers who wish to
consider noncommittal fills to be correct may revise the formulas by including NON in all denominators and i n
those numerators where COR appears . Note, however, that the high undergeneration scores obtained by mos t
systems on many slots in the evaluation places in question the value of considering noncommittal fills to b e
correct.
36 2
The first two rows at the bottom of the report are the ALL-OBJECTS and MATCHED-ONLY rows.
These offer two options in the way the ERR, UND, OVG, SUB, REC, and PRE scores are computed.
ALL-OBJECTS gives the official overall tallies and overall ERR, UND, OVG, SUB, REC, and
PRE scores. The tallies in POS, ACT, COR, PAR, INC, SPU, MIS, and NON columns add up to
the ALL-OBJECTS values, but not all rows are considered. The rows that are not considered arc th e
object rows (i.e., those where the entry in the SLOT column is an object name), the subtotal rows (sinc e
they simply sum up the tallies for the scored slots in each object), and the following slot rows in th e
<template> object: doc-nr, doc-date, and doc-source (these appear only in the EJV and JJV full-
template reports).
MATCHED-ONLY contains tallies and scores given a method of scoring that eliminates slot-fill
penalties in the case where the system under evaluation failed to generate an expected object and in the cas e
where it generated a spurious object. The tallies in the slot rows do not sum up to the totals contained in
the MATCHED-ONLY row.
The scores for the metrics in the ALL-OBJECTS and MATCHED-ONLY rows are not computed by
averaging the scores in the individual slot rows, but instead are computed on the basis of the tally totals containe d
in the ALL-OBJECTS or MATCHED-ONLY row.
The remaining rows at the bottom of the report contain scores for the following metrics :
TEXT-FILTERING
		
Text filtering (document detection )
RICHNESS-NORMALIZED-ERROR Richness-normalized error (normalized version o f
error per response fill)
ERROR-RATE-PER-WORD
	
Error rate per word
F-MEASURES
	
F-measures (weighted combination of recall an d
precision)
The formulation of each of the above metrics is described below.
Text-filtering scores are computed on the basis of relevant/nonrelevant decisions made by th e
system for the text as a whole, not on the basis of tallies in the slot rows . Some of the score category
columns are used to contain the tallies for these decisions across the test set : COR (decides relevant
and relevant is correct), SPU (decides relevant and nonrelevant is correct), MIS (decides nonrelevan t
and relevant is correct), and NON (decides nonrelevant and nonrelevant is correct) . Text-filtering
formulas are therefore as follows : REC = COR/(COR+MIS); PRE = COR/(COR+SPU) ; UND
= MIS/(COR+MIS); OVG = SPU/(COR+SPU) .
Two values of richness-normalized error are computed . The formulas are as follows : Min-err =
Wrong/All-fills, Max-err = Wrong/Req-fills . There will be a range of fill counts from the
minimum number of fills required (Req-fills) to the maximum number of fills allowed (All-fills),
according to the answer key . These values are almost, but not quite, system-independent .
Error rate per word is computed as Wrong/Word-count . The scores for Japanese are not
accurate .
F-measure scores corresponding to three different weightings of precision relative to recall arc
computed. The P&R value weights precision the same as recall ; the 2P&R value gives twice th e
weight to precision ; the P&2R value weights precision half as much as recall . The general formul a
is F = ((32+1 .0)xPxR/(6 2xP)+R, where is the weighting factor.
363
EJV Summary Scores
BBN PLUM Syste m
1. Full-Template Test
SLOT PIS ACT COR PAR INC SPU MIS NON ERR UND OVG SUB REC PRE
template> 282 282 282 0 0 0 0 0 0 0 0 0 100 100
doc-nr 282 282 282 0 0 0 0 0 0 0 0 0 100 100
doc-date 275 282 273 0 2 7 0 0 3 0 2 1 99 97
doc-source 276 213 207 0 0 6 69 0 26 25 3 0 75 97
content 348 331 253 0 10 68 85 6 39 24 20 4 73 76
subtotals 348 331 253 0 10 68 85 6 39 24 20 4 73 76
<tie-ul;-relati 348 330 253 0 10 67 85 0 39 24 20 4 73 77status
348 330 210 0 53 67 85 0 49 24 20 20 60 64
entity 791 861 469 0 55 337 267 0 58 34 39 10 59 54
joint-venture 179 71 35 0 10 26 134 96 83 75 37 22 20 49ownership 103
99 46 0 2 51 55 150 70 53 52 4 45 46activity 387
226 115 0 17 94 255 17 76 66 42 13 30 5 1subtotals
1808 1587 875 0 137 575 796 263 63 44 36 14 48 55<entity> 977 831 646 0 51 134 280 0 42
29 16 7 66 78
name 873 737 459 30 111 137 273 17 53 31 18 21 54 64
aliases 359 347 231 5 8 103 115 321 49 32 30 4 65 67location
323 95 47 10 10 28 256 444 85 79 29 22 16 55
nationality 265 158 82 0 3 73 180 437 76 68 46 4 31 52
type 977 831 605 0 92 134 280 0 46 29 16 13 62 73
entity-rel 1156 940 579 0 133 228 444 0 58 38 24 19 50 62
person 118 16 10 0 1 5 107 567 92 91 31 9 8 62
facility 83 20 3 0 2 15 78 598 97 94 75 40 4 15subtotals
4154 3144 2016 45 360 723 1733 2384 58 42 23 16 49 65
<entity-relati 436 352 256 0 30 66 150 0 49 34 19 10 59 73
entityl 883 891 464 0 81 346 338 0 62 38 39 15 52 52
entity2 266 85 43 0 13 29 210 97 85 79 34 23 16 50
rel-ent2-to-e 436 352 178 0 108 66 150 0 64 34 19 38 41 50
status 431 352 195 0 90 67 146 0 61 34 19 32 45 55
subtotals 2016 1680 880 0 292 508 844 97 65 42 30 25 44 52
<activity> 386 213 135 0 17 61 234 0 70 61 29 11 35 63
industry 418 176 120 0 8 48 290 0 74 69 27 6 29 68site
552 67 24 1 15 27 512 29 96 93 40 39 4 36
start 103 0 0 0 0 0 103 98 100 100 • • 0
end 16 0 0 0 0 0 16 128 100 100 • * 0 *
revenue 23 0 0 0 0 0 23 131 100 100 * * 0
subtotals 1112 243 144 1 23 75 944 386 88 85 31 14 13 59
<industry> 412 176 127 0 11 38 274 0 72 66 22 8 31 72
type 381 174 106 0 26 42 249 0 75 65 24 20 28 61
prod-sery 809 176 67 14 57 38 671 27 91 83 22 46 9 42subtotals
1190 350 173 14 83 80 920 27 86 77 23 33 15 51<facility> 164 36 25 0 6 5 133 0 85 81 14 19 15 69
name 36 1 0 0 1 0 35 24 100 97 0 100 0 0
location 142 11 5 1 4 1 132 3 96 93 9 45 4 50
type 163 36 25 0 6 5 132 0 85 81 14 19 15 69
subtotals 341 48 30 1 11 6 299 27 91 88 12 27 9 64<person> 118
15 11 0 1 3 106 0 91 90 20 8 9 73name 118 15 8 1 3 3 106 0 93 90 20
29 7 57entity 118
16 10 0 2 4 106 0 92 90 25 17 8 62position 136
15 5 0 7 3 124 0 96 91 20 58 4 33subtotals 372 46 23 1 12 10 336 0 94
90 22 35 6 5 1
<ownership> 103 78 51 0 2 25 50 0 60 48 32 4 50 65
time 22 0 0 0 0 0 22 45 100 100 * • 0 *
owned 103 42 11 0 14 17 78 0 91 76 40 56 11 26
capital 28 24 20 1 0 3 7 28 34 25 12 2 73 85
owner% 376 171 78 4 30 59 264 3 82 70 34 28 21 47
subtotals 529 237 109 5 44 79 371 76 82 70 33 29 ' 21 47
<revenue> 23 0 0 0 0 0 23 0 100 100 * * 0 *
t 23 0 0 0 0 0 23 0 100 100 * * 0 *totaltal
15 0 0 0 0 0 15 0 100 100 * * 0 *
total-eta 11 0 0 0 0 0 11 0 100 100 * * 0 *
rate 8 0 0 0 0 0 8 0 100 100 * * 0 *
rate-eta 0 0 0 0 0 0 0 0 * * * * * *
subtotals 57 0 0 0 0 0 57 0 100 100 * • 0 *
<time> 149 0 0 0 0 0 149 0 100 100 * * 0 •after
16 0 0 0 0 0 16 0 100 100 • • 0 •
before 27 0 0 0 0 0 27 0 100 100 * * 0
(luring 91 0 0 0 0 0 91 0 100 100 • * 0 *
on 19 0 0 0 0 0 19 0 100 100 * * 0 *
subtotals 153 0 0 0 0 0 153 0 100 100 * • 0 *
364
•\w
	
• t 1 :T o
MATCHED-ONLY 6940
	
6037
	
4503 67 99 803
	
1706 3266 41 24 13 13 65 75TEXT-FILTERING 251
	
262
	
237 25
	
14 6 - 6 10 - 94 90
Wrong Req-fills All- i Min-err Max-errRICHNESS-NORMALIZED-ERROR
9667 .5 11803 12128 0.7971 0.819 1Wrong Word-count
Error-rateERROR-RATE PER-WORD 9667 .5 92862 0 .104 1
P&R 2P&R P&2RF-MEASURES
45 .95 53 .07 40.5 1
2.
	
Core-Template Test
SLOT POS ACT
	
COR PAR INC SPU MIS NON ERR UND OVG SUS REC PR E
<template> 282 282
	
282 0 0 0 0 0 0 0 0 0 100 100content
348 331
	
253 0 10 68 85 6 39 24 20 4 73 76subtotals 348
331
	
253 0 10 68 85 6 39 24 20 4 73 76<tie-up-relati
348 330
	
253 0 10 67 85 0 39 24 20 4 73 77status
348 330
	
210 0 53 67 85 0 49 24 20 20 60 6 4entity 791
861
	
468 0 56 337 267 0 58 34 39 11 59 54joint-venture
179 71
	
35 0 10 26 134 96 83 75 37 22 20 49
subtotals 1318 1262
	
713 0 119 430 486 96 59 37 34 14 54 56<entity>
977 831
	
645 0 52 134 280 0 42 29 16 7 66 78name
873 737
	
459 30 111 137 273 17 53 31 18 21 54 6 4aliases
359 347
	
230 4 9 104 116 319 50 32 30 4 65 67location 323 95
	
47 10 10 28 256 445 85 79 29 22 16 55
nationality 265 158
	
82 0 4 72 179 436 76 68 46 5 31 52type 977 831
	
604 0 93 134 280 0 46 29 16 13 62 73entity-rel 1156 940
	
578 0 134 228 444 0 58 38 24 19 50 61
subtotals 3953 3108
	
2000 44 361 703 1548 1217 56 39 23 16 51 65<entity-relati
436 352
	
256 0 30 66 150 0 49 34 19 10 59 7 3entityl
883 891
	
463 0 82 346 338 0 62 38 39 15 52 5 2entity2
266 85
	
43 0 13 29 210 97 85 79 34 23 16 50
rel-ent2-to-e 436 352
	
178 0 108 66 150 0 64 34 19 38 41 50status
431 352
	
195 0 90 67 146 0 61 34 19 32 45 5 5subtotals
2016 1680
	
879 0 293 508 844 97 65 42 30 25 44 5 2
ALL-OBJECTS 7635 6381
	
3845 44 94 1709 2963 1416 59 39 27 17 51 6 1
MATCHED-ONLY 5269 5078
	
3845 44 94 668 859 1416 35 16 13 12 73 7 6
TEXT-FILTERING 251 262
	
237 - - 25 14 6 - 6 10 - 94 9 0
Wrong Word-count
	
Error-rate
ERROR-RATE-PER-WORD 5477 92862 0.05 9
P&R 2P&R P&2RF-MEASURES
_ 55.18 58.31 52.3 7
36 5
EJV Summary Score s
GE/CMU Shogun System
1. Full-Template Tes t
SLOT
	
PUS ACT
(template>
	
282
	
282
doc-nr
	
282
	
282doc-date
	
275
	
275doc-source
	
276
	
282
content
	
348
	
389
subtotals
	
348
	
389
<tie-up-relati
	
348
	
389
status
	
348
	
389
entity
	
791
	
934
joint-venture
	
180
	
212
ownership
	
103
	
122
activity
	
387
	
367subtotals
	
1809
	
2024<entity>
	
976
	
1057
name
	
872
	
937aliases
	
359
	
389location
	
322
	
338nationality
	
265
	
212
type
	
976
	
1057
entity-rel
	
1153
	
1242
person
	
118
	
170facility
	
84
	
192subtotals
	
4149
	
4537
<entity-relati
	
437
	
436entityl
	
885
	
982
entity2
	
268
	
260
rel-ent2-to-e
	
437
	
436status
	
432
	
436subtotals
	
2022
	
2114
<activity>
	
387
	
366
industry
	
420
	
424
site
	
552
	
826
start
	
104
	
143
end
	
. 16
	
3
revenue
	
23
	
44
subtotals
	
1115
	
1440
<industry>
	
415
	
425
type
	
384
	
425
prod-sery
	
847
	
1193
subtotals
	
1231
	
1618<facility>
	
164
	
280
name
	
36
	
0
location
	
142
	
180
type
	
163
	
280
subtotals
	
341
	
460
<person>
	
118
	
162
name
	
118
	
162
entity
	
118
	
162
position
	
136
	
162
subtotals
	
372
	
486
<ownership>
	
103
	
122
time
	
21
	
2
owned
	
103
	
123
capital
	
28
	
40
owner%
	
376
	
450
subtotals
	
528
	
615
<revenue>
	
23
	
37
type
	
23
	
37
total
	
15
	
30
total-eta
	
11
	
14
rate
	
8
	
7
rate-eta
	
0
	
0
subtotals
	
57
	
88
<time>
	
149
	
137
after
	
16
	
13
before
	
27
	
23
during
	
91
	
74
on
	
19
	
32
subtotals
	
153
	
142
COIL PAR INC SPU MIS NON 00R UND OVG SUB REC PRE
282 0 0 0 0 0 0 0 0 0 100 100
282 0 0 0 0 0 0 0 0 0 100 100275 0
0 0 0 7 0 0 0 0 100 100276 0 0 6 0 0 2 0 2
0 100 98289 0 6 94 53 11 35 15 24 2 83 74
289 0 6 94 53 11 35 15 24 2 83 74
289 0 6 94 53 0 35 15 24 2 83 74
231 0 64 94 53 0 48 15 24 22 66 59
535 0 94 305 162 0 51 20 33 15 68 57
101 0 23 88 56 96 62 31 42 18 56 48
0 4 54
35
180 59 34 44 6 62 52
214 0 44 109 129 10 57 33 30 17 55 58
1145 0 229 650 435 286 53 24 32 17 63 56749 0
61 247 166 0 39 17 23 8 77 71
554 44 115 224 159 27 47 18 24 19 66 61
232 6 5 146 116 399 53 32 38 3 65 60140
19 23 156 140 443 69 43 46 18 46 44
100 0 11 101 154 509 73 58 48 10 38 47
716 0 94 247 166 0 41 17 23 12 73 68
747 0 117 378 289 0 51 25 30 14 65 60
58 0 0 112 60 637 75 51 66 0 49 34
37 0 10 145 37 650 84 44 76 21 44 192584 69 375 1509
1121 2665 54 27 33 14 63 58
311 0 24 101 102 0 42 23 23 7 71 71
568 0 97 317 220 0 53 25 32 14 64 58128 0 32 100 108
93 65 40 38 20 48 49
239 0 96 101 102 0 56 23 23 29 55 55
239 0 94 103 99 0 55 23 24 28 55 55
1174 0 319 621 529 93 56 26 29 21 58 56
262 0 20 84 105 0 44 27 23 7 68 72
228 0 35 161 157 5 61 37 38 13 54 54
217 3 71 535 261 83 80 47 65 25 40 26
27 0 22 94 55 126 86 53 66 45 26 191
0 0 2 15 251 94 94 67 0 6 33
9 0 3 32 11 235 84 48 73 25 39 20
482 3 131 824 499 700 75 45 57 22 43 34
271 0 22 132 122 0 50 29 31 8 65 64
227 0 42 156 115 0 58 30 37 16 59 53
287 54 220 632 286 1 79 34 53 44 37 26
514 54 262 788 401 1 73 32 49 35 44 33
104 0 19 157 41 0 68 25 56 15 63 37
0 0 0 0 36 91 100 100 * * 0
41 10 28 101 63 8 81 44 56 42 32 26
91 0 30 159 42 0 72 26 57 25 56 32
132 10 58 260 141 99 77 41 56 32 40 30
71 0 5 86 42 0 65 36 53 6 60 44
64 1 11 86 42 0 68 36 53 15 55 40
54 0 22 86 42 0 74 36 53 29 46 33
42 0 33 87 61 0 81 45 54 44 31 26
160 1 66 259 145 0 74 39 53 29 43 33
76 0 2 44 25 0 48 24 36 2 74 62
0 0 1 1 20 65 100 95 50 100 0 0
38 0 39 46 26 0 74 25 37 51 37 31
18 2 2 18 6 45 59 21 45 14 68 48
199 9 54 184 114 5 64 30 41 22 54 45255 11 96 249 166
115 66 31 40 28 49 4210 0 4 23
9 0 78 39 62 28 43 279
1 5 23 9 0 80 39 62 36 39 245 2 22 7 4 85 47 73 31 37 18
4 0 1 9 6 6 80 54 64 20 36 28
4 0 1 2 3 6 60 38 28 20 50 57
0 0 0 0 0 10 . * * * * *
22 1 9 56 25 26 80 44 64 30 39 26
36 0 31 70 82 0 84 55 51 46 24 26
0 0 3 10 13 0 100 81 77 100 0 0
6 0 1 16 20 0 86 74 70 14 22 26
25 0 7 42 59 0 81 65 57 22 27 34
5 0 0 27 14 0 89 74 84 0 26 16
36 0 11 95 106 0 85 69 67 23 24 25
366
ALL-OBJECTS T2125 139136793 149 124 5405 3621 3997 61 30 39 19 57 49
MATCHED-ONLY 9140 9729
	
6793 149 124 1624 1039 3996 36 11 17 15 75 70TEXT-FILTERING 251 262
	
242 - - 20 9 11 - 4 8 - 96 9 2
RICHNESS-NORMALIZED-ERROR Wrong10662.5 Req-Tills1181 3
ERROR-RATE-PER-WORD Wron g106625 Word-count92862
F-MEASURES P&R52.75 2P&R50.66
2.	 Core-Template Test
SLAT POS ACTCOR PAR INC SPU MIS NON ERR UND OVG SUB REC PRE
<template> 282 282
	
282 0 0 0 0 0 0 0 0 0 100 100content 348 389
	
289 0 6 94 53 11 35 15 24 2 83 74subtotals 348
389
	
289 0 6 94 53 11 35 15 24 2 83 74<tie-up-relati
348 389
	
289 0 6 94 53 0 35 15 24 2 83 74status 348 389
	
231 0 64 94 53 0 48 15 24 22 66 59
entity 791 934
	
535 0 94 305 162 0 51 20 33 15 68 57
joint-venture 180 212
	
101 0 23 88 56 96 62 31 42 18 56 4 8subtotals 1319 1535
	
867 0 181 487 271 96 52 20 32 17 66 56
<entity> 976 1057
	
749 0 61 247 166 0 39 17 23 8 77 7 1
name 872 937
	
554 43 116 224 159 27 47 18 24 19 66 61aliases
359 389
	
232 5 6 146 116 399 54 32 38 3 65 60location
322 338
	
140 19 23 156 140 443 69 43 46 18 46 44
nationality 265 212
	
100 0 11 101 154 509 73 58 48 10 38 4 7type
976 1057
	
716 0 94 247 166 0 41 17 23 12 73 68entity-rel
1153 1242
	
747 0 117 378 289 0 51 25 30 14 65 6 0subtotals
3947 4175
	
2489 67 367 1252 1024 1378 51 26 30 14 64 6 0<entity-relati 437 436
	
311 0 24 101 102 0 42 23 23 7 71 7 1entity'
885 982
	
568 0 97 317 220 0 53 25 32 14 64 5 8
entity2 268 260
	
128 0 32 100 108 93 65 40 38 20 48 49rel-ent2-to-e
437 436
	
239 0 96 101 102 0 56 23 23 29 55 5 5status
432 436
	
239 0 94 103 99 0 55 23 24 28 55 5 5
subtotals 2022 2114
	
1174 0 319 621 529 93 56 26 29 21 58 5 6
ALL-OBJECTS 7636 8213
	
4819 67 96 2454 1877 1578 52 24 30 16 64 5 9MATCHED-ONLY 6041 6122
	
4819 67 96 627 546 1578 27 9 10 12 80 79TEXT-FILTERING
251 262
	
242 -
	
- 20 9 11 4 8
	
- 96 92
Wrong Word-count
	
Error-rat e
ERROR-RATE-PER-WORD 5237.5
P&R
92862
2P&R
0.0564
P&2RF-MEASURES
61.23 59.93 62.6
	
All-fills
	
Mm-err
	
Max-err
	
12138
	
0.8784
	
0.9026
Error-rate
0.114 8
P&2R55 .02
367
EJV Summary Scores
MITRE Alembic Syste m
1. Full-Template TestSLOT PUS
	
AC f CUR PAR INC SPU MIS NON ERR UND OVG SUB REC PRE
<temp ate> J : 1 PI PI
doc-nr 282 282 282 0 0 0 0 0 0 0 0 0 100 100
doc-date 275 274 274 0 0 0 7 0 0 0 0 100 100
doc-source 276 282 276 0 0 6 0 0 2 0 2 0 100 9 8
content 348 327 151 0 104 72 93 12 64 27 22 41 43 46subtotals 348 327 151 0 104 72
93 .12 64 27 22 41 43 46<tie-up-relati 348 326 151 0 103 72 94 0 64
27 22 40 43 46status 348 326 123 0 131 72 94 0 71
27 22 52 35 3 8entity 791 275 188 0 42 45 561 0 78 71 16 18 24
68joint-venture 179 10 9 0 0 1 170 71
95 95 10 0 5 90ownership 103 1 0 0 0 102 115
99 99 0 0 1 10 0activity 386 64 37 0 11 16 338 9
91 88 25 23 10 5 8subtotals 1807 676 358 0 184 134 1265 195 82 70 20 34 20
5 3<entity> 976 281 229 0 30 22 717 0 77 73 8 12 23
8 1name 873 269 96 35 107 31 635 0 87 73 12 52 13 4
2aliases 359 43 32 0 1 10 326 149 91 91 23 3 9 7 4
location 323 25 18 0 4 3 301 162 94 93 12 18 6 72
nationality 265 38 24 0 5 9 236 160 91 89 24 17 9 63
t YPe 976 281 217 0 42 22 717 0 78 73 8 16 22 77entity-rel 1152 293 223 0 42 28 887 0 81 77 10 16 19 76
person 118 1 0 0 0 117 212 99 99 0 0 1 100
facility 81 0 0 0 0 0 81 215 100 100 * * 0
subtotals 4147 950 611 35 201 103 3300 898 85 80 11 26 15 66
<entity-relati 437 186 135 0 32 19 270 0 70 62 10 19 31 72
entityl 885 284 193 0 54 37 638 0 79 72 13 22 22 68
entity2 267 14 11 0 1 2 255 59 96 96 14 8 4 78
rel-ent2-to-e 437 186 70 0 97 19 270 0 85 62 10 58 16 38
status 432 186 106 0 58 22 268 0 77 62 12 35 24 57
subtotals 2021 670 380 0 210 80 1431 59 82 71 12 36 19 57
<activity> 386 64 42 0 11 11 333 0 89 86 17 21 11 66
industry 419 41 35 0 1 5 383 1 92 91 12 3 8 85
site 551 46 13 2 13 18 523 18 98 95 39 50 2 30
start 103 0 0 0 0 0 103 32 100 100 * * 0 *
end 16 0 0 0 0 0 16 42 100 100 * * 0 *
revenue 23 0 0 0 0 0 23 39 100 100 * * 0 *
subtotals 1112 87 48 2 14 23 1048 132 96 94 26 23 4 56
<industry> 412 41 35 0 1 5 376 0 92 91 12 3 8 85
type 379 41 34 0 2 5 343 0 91 90 12 6 9 83prod-sery 799 73 32 2 24
15 741 1 96 93 20 43 4 45subtotals 1178 114 66 2 26 20 1084 1
94 92 18 29 6 59<facility> 164 1 0 0 0 163 0
99 99 0 0 1 100name 35 0 0 0 0 0 35 1 100 100 * * 0 *
location 142 0 0 0 0 0 142 0 100 100 * * 0 *
type 163 1 0 0 0 162 0 99 99 0 0 1 100
subtotals 340 1 0 0 0 339 1 100 100 0 0 0 10 0
<person> 118 1 0 0 0 117 0 99 99 0 0 1 10 0
name 118 1 0 0 1 0 117 0 100 99 0 100 0 0
entity 118 1 0 0 0 117 0 99 99 0 0 1 10 0
position 136 1 0 0 1 0 135 0 100 99 0 100 0 0
subtotals 372 3 0 2 0 369 0 100 99 0 67 0 3 3
<ownership> 102 1 0 0 0 101 0 99 99 0 0 1 10 0
time 21 0 0 0 0 0 21 1 100 100 * * 0 *
owned 102 0 0 0 0 0 102 0 100 100 * * 0 *
capital 27 0 0 0 0 0 27 1 100 100 * * 0 *
owner% 376 4 4 0 0 0 372 0 99 99 0 0 1 100
subtotals 526 4 4 0 0 0 522 2 99 99 0 0 1 10 0
<revenue> 23 0 0 0 0 0 23 0 100 100 * * 0 *
type 23 0 0 0 0 0 23 0 100 100 * * 0
total 15 0 0 0 0 0 15 0 100 100 * * 0 *
total-eta 11 0 0 0 0 0 11 0 100 100 * * 0 *
rate 8 0 0 0 0 0 8 0 100 100 * * 0 *
rate-eta 0 0 0 0 0 0 0 0 * * * * * *
subtotals 57 0 0 0 0 0 57 0 100 100 * * 0 *
<time> 149 0 0 0 0 0 149 0 100 100 * * 0 *
after 16 0 0 0 0 0 16 0 100 100 * * 0 *
before 27 0 0 0 0 0 27 0 100 100 * * 0 *
during 91 0 0 0 0 0 91 0 100 100 * * 0 *
on 19 0 0 0 0 0 19 0 100 100 * * 0 *
subtotals 153 0 0 0 0 0 153 0 100 100 * * 0 *
368
MATCHED-ONLY
	
2967 221 6
TEXT-FILTERING	 251	 239
RICHNESS-NORMALIZED-ERROR
ERROR-RATE-PER-WORD
F-MEASURES
1620
	
39
	
100
	
144
	
895 1300
	
47
	
30
	
6
	
21
	
55
	
74
	220	 -	 -	 19	 31	 12	 -	 12	 8
	
-
	
88
	
9 2
	
Wrong
	
Req-fills
	
All-fills
	
Mm-err
	
Max-err
	
10853 .5
	
11802
	
12127
	
0.895
	
0.9196
	
Wrong
	
Word-count
	
Error-rat e
	
10853 .5
	
92862
	
0.1169
	
P&R
	
2P&R
	
P&2R
	
22.02
	
35.05
	
16.05
2. Core-Tem~ate Test
SLO1 POS ACT ZOR PAR INC SPU
	
MIS NON R UND OVG SUB RIiC PRE
<template> 282 282
	
282 0 0 0
	
0 0 0 0 0 0 100 10 0
content 348 327
	
151 0 104 72
	
93 12 64 27 22 41 43 4 6
subtotals 348 327
	
151 0 104 72
	
93 12 64 27 22 41 43 46
<tie-up-relati 348 326
	
151 0 103 72
	
94 0 64 27 22 40 43 46status 348 326
	
123 0 131 72
	
94 0 71 27 22 52 35 3 8
entity 791 275
	
188 0 42 45
	
561 0 78 71 16 18 24 68
joint-venture 179 10
	
9 0 0 1
	
170 71 95 95 10 0 5 90
subtotals 1318 611
	
320 0 173 118
	
825 71 78 62 19 35 24 5 2
<entity> 976 281
	
229 0 30 22
	
717 0 77 73 8 12 23 8 1
name 873 269
	
96 35 107 31
	
635 0 87 73 12 52 13 4 2
aliases 359 43
	
32 0 1 10
	
326 149 91 91 23 3 9 7 4
location 323 25
	
18 0 4 3
	
301 162 94 93 12 18 6 7 2
nationality 265 38
	
24 0 5 9
	
236 160 91 89 24 17 9 63
type 976 281
	
217 0 42 22
	
717 0 78 73 8 16 22 7 7
entity-rel 1152 293
	
223 0 42 28
	
887 0 81 77 10 16 19 7 6
subtotals 3948 949
	
610 35 201 103
	
3102 471 84 78 11 26 16 66
<entity-relati 437 186
	
135 0 32 19
	
270 0 70 62 10 19 31 7 2
entityl 885 284
	
193 0 54 37
	
638 0 79 72 13 22 22 68
entity2 267 14
	
11 0 1 2
	
255 59 96 96 14 8 4 7 8
rel-ent2-to-e 437 186
	
70 0 97 19
	
270 0 85 62 10 58 16 3 8
status 432 186
	
106 0 58 22
	
268 0 77 62 12 35 24 57
subtotals 2021 670
	
380 0 210 80
	
1431 59 82 71 12 36 19 5 7
ALL-OBJECTS 7635 2557
	
1461 35 95 373--545I 613 82---71F— AA 32 19 5 8
MATCHED-ONLY 2482 2010
	
1461 35 95 13~
	
607 613 44
	
24
	
7 21 60 7 4
TEXT-FILTERING 251 239
	
220 - - 19
	
31 12 -
	
12
	
8 - 88 92
Wrong Word-count
	
Error-rat e
ERROR-RATE-PER-WORD 6529 . 5
P&R
92862
2P&R
0.0703
P&2R
F-MEASURES 29.01 41 .38 22.34
369
EJV Summary Scores
NMSU/Brandeis Diderot Syste m
1. Full-Template Test
SLAT POS ACT COR PAR INC SPU M LS NON ERR UNl)
	
OVG
	
SUE REC PRE
<temp ate> 3; T: 1 1 1'1 1'1
doc-nr 282 282 282 0 0 0 0 0 0 0 100 10 0
doe-date 275 205 204 0 1 0 70 7 26 25 74 100
doe-source 276 282 276 0 0 6 0 0 2 0 100 98
content 348 275 189 0 30 56 129 15 53 37 54 69
subtotals 348 275 189 0 30 56 129 15 53 37 54 69
<tie-up-relati 348 275 189 0 30 56 129 0 53 37 54 69
status 348 275 156 0 63 56 129 0 61 37 45 57
entity 791 419 252 0 80 87 459 0 71 58 32 60
joint-venture 179 59 20 0 13 26 146 74 90 82 11 34
ownership 103 27 15 0 5 7 83 123 86 80 14 56
activity 387 136 75 0 33 28 279 6 82 72 19 55subtotals
1808 916 518 0 194 204 1096 203 74 61 29 56<entity>
975 490 381 0 50 59 544 0 63 56 39 78name 873
452 258 34 95 65 486 7 71 56 32 6 1aliases 359 216 136 2 6
72 215 165 68 60 38 63location 323 179 27 11
23 118 262 181 93 81 10 1 8nationality
265 143 53 0 12 78 200 225 84 75 20 37type 975 490 375 0
56 59 544 0 64 56 38 76etity-rel 1153
619 354 0 97 168 702 0 73 61 31 57person 118
63 23 0 1 39 94 314 85 80 19 36facility 81 16 2 0 1
13 78 354 98 96 2 1 2subtotals 4147 2178 1228 47 291 612 2581 1246 74 62 30 57
<entity-relati 437 350 201 0 51 98 185 0 62 42 46 57
entityl 885 567 291 0 90 186 504 0 73 57 33 5 1entity2
267 58 21 0 17 20 229 74 93 86 8 36
rel-ent2-to-e 437 350 114 0 138 98 185 0 79 42 26 32status
432 350 153 0 96 101 183 0 71 42 35 44subtotals
2021 1325 579 0 341 405 1101 74 76 54 29 44<activity>
387 135 87 0 28 20 272 0 .79 70 22 64industry 420
200 88 0 34 78 298 0 82 71 21 44site 552 70 17
1 27 25 507 18 97 92 3 25start 104
10 1 0 1 8 102 57 99 98 1 0end 16 0 0 0 0
0 16 84 100 100 0revenue 23 0 0 0 0 0
23 86 100 100 0 *subtotals 1115 280 106 1 62
111 946 245 91 85 10 38<industry> 413 201 113 0 35 53 265 0 76 64 27 56
type 382 201 104 0 34 63 244 0 77 64 27 52
prod-sery 818 216 52 8 88 68 670 0 94 82 7 26
subtotals 1200 417 156 8 122 131 914 0 88 76 13 38
<facility> 164 29 15 0 5 9 144 0 91 88 9 52
name 35 0 0 0 0 0 35 12 100 100 0 *
location 142 8 1 0 2 5 139 3 99 98 1 2type
163 29 15 0 5 9 143 0 91 88 9 52
subtotals 340 37 16 0 7 14 317 15 95 93 5 43
<person> 118 59 25 0 2 32 91 0 83 77 21 42name 118
24 9 1 3 11 105 0 93 89 8 40entity
118 59 23 0 4 32 91 0 85 77 19 39position 136
53 11 0 17 25 108 0 93 79 8 2 1subtotals 372 136 43 1 24 68 304 0
90 82 12 32<ownership> 103 27 20 0
5 2 78 0 81 76 19 74time 22 0 0 0 0 0 22 18 100 100 0 *
owned 103 22 7 0 14 1 82 0 93 80 7 32capital
28 17 8 0 8 1 12 1 72 43 28 47
owner% 376 25 13 1 8 3 354 4 96 94 4 54
subtotals 529 64 28 1 30 5 470 23 95 89 5 4 4
<revenue> 23 0 0 0 0 0 23 0 100 100 0 *
type 23 0 0 0 0 0 23 . 0 100 100 0 *total
15 0 0 0 0 0 15 0 100 100 0 *total-eta
11 0 0 0 0 0 11 0 100 100 0 srate
8 0 0 0 0 0 8 0 100 100 0 *rate-eta 0 0 0
0 0 0 0 0 * * * *subtotals
57 0 0 0 0 0 57 0 100 100 0 *<time> 149
10 1 0 3 6 145 0 99 97 1 0after 16 0
0 0 0 0 16 0 100 100 0 *before 27 0
0 0 0 0 27 0 100 100 0 *during 91 7
1 0 3 3 87 0 99 96 1 4on 19 0 0 0 0 0
19 0 100 100 0 *subtotals 153 7 1 0 3
3 149 0 99 97 14
370
ALL-OBJECTS
	
12090 5635 -2864
	
58
	
81 1709 8064 1821
	
/9
	
67
	
28
	
28
	
24
	
5 1MATCHED-ONLY
	
4882 4059 2864
	
58
	
81
	
495 1318 1821
	
46
	
27
	
12
	
19
	
59
	
7 1TEXT-FILTERING
	
251
	
208
	
192
	
16
	
59
	
15
	
24
	
8
	
76
	
92
Wrong
	
Reglills
	
All-fills
	
Mm-err
	
Max-er r
10806
	
11804
	
12129
	
0.8909
	
0 .9155
Wrong
	
Word-count
	
Error-rat e
10806
	
92862
	
0.1164
P&R
	
2P&R
	
P&2R
32.64
	
41 .77
	
26.79
2.	 Core-Template__ Test
SLOT POS ACT
	
COR PAR INC SPU MIS NON ERR UND OVG SUB RE(: PRE
<template> 282 282
	
282 0 0 0 0 0 0 0 0 0 100 100
content 348 275
	
189 0 30 56 129 15 53 37 20 14 54 69
subtotals 348 275
	
189 0 30 56 129 15 53 37 20 14 54 69
<tie-up-relati 348 275
	
189 0 30 56 129 0 53 37 20 14 54 69
status 348 275
	
156 0 63 56 129 0 61 37 20 29 45 57entity 791 419
	
252 0 80 87 459 0 71 58 21 24 32 60
joint-venture 179 59
	
20 0 13 26 146 74 90 82 44 39 11 34subtotals 1318 753
	
428 0 156 169 734 74 71 56 22 27 32 57
<entity> 975 490
	
381 0 50 59 544 0 63 56 12 12 39 78
name 873 452
	
258 34 95 65 486 7 71 56 14 29 32 6 1
aliases 359 216
	
136 2 6 72 215 165 68 60 33 5 38 63
location 323 179
	
27 11 23 118 262 181 93 81 66 47 10 1 8
nationality 265 143
	
53 0 12 78 200 225 84 75 54 18 20 37
type 975 490
	
375 0 56 59 544 0 64 56 12 13 38 76
entity-rel 1153 619
	
354 0 97 168 702 0 73 61 27 22 31 57
subtotals 3948 2099
	
1203 47 289 560 2409 578 73 61 27 20 3I 5 8
<entity-relati 437 350
	
201 0 51 98 185 0 62 42 28 20 46 57
entity! 885 567
	
291 0 90 186 504 0 73 57 33 24 33 5 1
entity2 267 58
	
21 0 17 20 229 74 93 86 34 45 8 36
rel-ent2-to-e 437 350
	
114 0 138 98 185 0 79 42 28 55 26 32
status 432 350
	
153 0 96 101 183 0 71 42 29 38 35 44
subtotals 2021 1325
	
579 0 341 405 1101 74 76 54 30 37 29 44
ALL-OBJECTS 7635 4452
	
2399 47 75 1190 4373 741 72 57 27 26 32 54
MATCHED-ONLY 3647 3277
	
2399 47 75 384 754 741 40 21 12 16 66 74
TEXT-FILTERING 251 208
	
192 - - 16 59 15 24 8 - 76 92
Wrong Word-count
	
Error-rate
ERROR-RATE-PER-WORD 6402 . 5P&R 92862
2P&R
0.0689
P&2R
F-MEASURES 40.08 47 .61 34.62
RICHNESS-NORMALIZED-ERRO R
ERROR-RATE-PER-WOR D
F-MEASURES
37 1
EJV Summary Scores
NYU Proteus System
1 . Full-Template Test
SLOT FOS AC1 COR PAR INC SPU MIS NON ERR UND OVG SUB Rt;C PRE
<template> 282 281 281 0 0 0 1 0 0 0 0 0 100 100
doc-nr 282 281 281 0 0 0 1 0 0 0 0 0 100 10 0
doc-date 275 203 203 0 0 0 72 7 26 26 0 0 74 10 0
doc-source 276 281 275 0 0 6 1 0 2 0 2 0 100 9 8
content 348 380 198 0 52 130 98 18 58 28 34 21 57 52
subtotals 348 380 198 0 52 130 98 18 58 28 34 21 57 52
<tie-up-relati 348 378 198 0 51 129 99 0 58 28 34 20 57 52
status 348 378 165 0 84 129 99 0 65 28 34 34 47 4 4
entity 791 366 247 0 42 77 502 0 72 63 21 14 31 67
joint-venture 179 178 44 0 18 116 117 63 85 65 65 29 24 25
ownership 102 28 20 0 0 8 82 137 82 80 28 0 20 7 1
activity 387 72 53 0 7 12 327 14 87 84 17 12 14 74
subtotals 1807 1022 529 0 151 342 1127 214 75 62 33 22 29 5 2
<entity> 975 504 373 0 86 45 516 0 63 53 9 19 38 74
name 873 445 263 25 112 45 473 14 70 54 10 31 32 62
aliases 359 155 120 3 5 27 231 203 68 64 17 5 34 7 8
location 323 51 34 4 4 9 281 242 89 87 18 14 11 70
nationality 264 63 23 0 9 31 232 271 92 88 49 28 9 36type
975 504 351 0 108 45 516 0 66 53 9 24 36 70
entity-rel 1152 453 320 0 81 52 751 0 73 65 11 20 28 7 1
person 118 6 3 0 0 3 115 329 98 97 50 0 2 50
facility 81 0 0 0 0 0 81 355 100 100 * * 0 *
subtotals 4145 1677 1114 32 319 212 2680 1414 74 65 13 23 27 67
<entity-relati 437 280 193 0 36 51 208 0 60 48 18 16 44 69entityl
885 381 270 0 50 61 565 0 71 64 16 16 30 7 1entity2
267 74 34 0 12 28 221 64 88 83 38 26 13 46
rel-ent2-to-e 437 280 107 0 122 51 208 0 78 48 18 53 24 38status
432 280 149 0 77 54 206 0 69 48 19 34 34 53subtotals
2021 1015 560 0 261 194 1200 64 75 59 19 32 28 55
<activity> 386 72 61 0 4 7 321 0 84 83 10 6 16 85industry
418 72 54 0 11 7 353 0 87 84 10 17 13 75
site 551 48 22 1 8 17 520 17 96 94 35 27 4 47
start 103 0 0 0 0 0 103 44 100 100 * * 0 *
end 16 0 0 0 0 0 16 57 100 100 * * 0
revenue 23 0 0 0 0 0 23 59 100 100 * * 0 *
subtotals 1111 120 76 1 19 24 1015 177 93 91 20 20 7 64
<industry> 411 72 58 0 9 5 344 0 86 84 7 13 14 80
type 379 53 44 0 1 8 334 1 89 88 15 2 12 83
prod-sery 808 154 59 8 57 30 684 1 92 85 19 49 8 4 1
subtotals 1187 207 103 8 58 38 1018 2 91 86 18 37 9 5 2
<facility> 164 0 0 0 0 0 164 0 100 100 * 0 *
name 35 0 0 0 0 0 35 0 100 100 * * 0 *
location 142 0 0 0 0 0 142 0 100 100 * * 0 *
type 163 0 0 0 0 0 163 0 100 100 * * 0 *
subtotals 340 0 0 0 0 0 340 0 100 100 * * 0 *
<person> 118 6 4 0 1 1 113 0 97 96 17 20 3 67
name 118 6 4 0 1 1 113 0 97 96 17 20 3 67
entity 118 6 3 0 2 1 113 0 97 96 17 40 2 5 0
position 136 6 3 0 2 1 131 0 98 96 17 40 2 50
subtotals 372 18 10 0 5 3 357 0 97 96 17 33 3 5 6
<ownership> 102 28 24 0 0 4 78 0 77 76 14 0 24 86
time 21 0 0 0 0 0 21 20 100 100 * * 0 *owned
102 15 6 0 6 3 90 0 94 88 20 50 6 40capital
27 0 0 0 0 0 27 19 100 100 * * 0 *owner%
376 70 47 0 15 8 314 0 88 84 11 24 12 67subtotals
526 85 53 0 21 11 452 39 90 86 13 28 10 62<revenue>
23 0 0 0 0 0 23 0 100 100 * * 0 *type
23 0 0 0 0 0 23 0 100 100 * * 0 *total
15 0 0 0 0 0 15 0 100 100 * * 0total-eta
11 0 0 0 0 0 11 0 100 100 * * 0 *rate
8 0 0 0 0 0 8 0 100 100 * * 0 *rate-eta
0 0 0 0 0 0 0 0 * * * * * *subtotals
57 0 0 0 0 0 57 0 100 100 * * 0 *<time>
149 0 0 0 0 0 149 0 100 100 * * 0 *after
16 0 0 0 0 0 16 0 100 100 * * 0before
27 0 0 0 0 0 27 0 100 100 * * 0 *during
91 0 0 0 0 0 91 0 100 100 * 0 *on
19 0 0 0 0 0 19 0 100 100 * * 0 *subtotals
153 0 0 0 0 0 153 0 100 100 * * 0 *
372
MATCHED-ONLY
	
4534 339 3
TEXT-FILTERING	 251	 217
RICHNESS-NORMALIZED-ERRO R
ERROR-RATE-PER-WORD
F-MEASURES
2643
	
41
	
57
	
258
	
1399 1928
	
44
	
31
	
8
	
15
	
59
	
78
	 204	 -	 -	 13	 47	 18	 -	 19	 6	 81	 94
	
Wrong
	
Req-fills
	
All-fills
	
Min-err
	
Max-er r
	
10357 .5
	
11797
	
12122
	
0.8544
	
0.87 8
	
Wrong
	
Word-count
	
Error-rat e
	
10357 .5
	
92862
	
0.111 5
	
P&R
	
2P&R
	
P&2R
	
32 .11
	
44 .15
	
25.23
2. Core-Tem plate Test
SLOT PGS ACP COR PAR INC SPU MIS NON ERR UND OVG SUB REC PRE
<template> 282 281
	
281 0 0 0 1 0 0 0 0 0 100 10 0content
348 380
	
198 0 52 130 98 18 58 28 34 21 57 52
subtotals 348 380
	
198 0 52 130 98 18 58 28 34 21 57 52
<tie-up-relati 348 378
	
198 0 51 129 99 0 58 28 34 20 57 5 2status 348
378
	
165 0 84 129 99 0 65 28 34 34 47 4 4entity
791 366
	
247 0 42 77 502 0 72 63 21 14 31 67
joint-venture 179 178
	
44 0 18 116 117 63 85 65 65 29 24 2 5
subtotals 1318 922
	
456 0 144 322 718 63 72 54 35 24 34 4 9
<entity> 975 504
	
373 0 86 45 516 0 63 53 9 19 38 74
name 873 445
	
258 21 121 45 473 14 71 54 10 33 31 60aliases
359 155
	
117 2 9 27 231 203 69 64 17 8 33 7 6location 323
51
	
34 4 4 9 281 242 89 87 18 14 11 7 0
nationality 264 63
	
23 0 9 31 232 271 92 88 49 28 9 3 6type 975 504
	
351 0 108 45 516 0 66 53 9 24 36 70
entity-rel 1152 453
	
320 0 81 52 751 0 73 65 11 20 28 7 1
subtotals 3946 1671
	
1103 27 332 209 2484 730 73 63 12 24 28 67
<entity-relati 437 280
	
193 0 36 51 208 0 60 48 18 16 44 69
entityl 885 381
	
270 0 50 61 565 0 71 64 16 16 30 7 1
entity2 267 74
	
34 0 12 28 221 64 88 83 38 26 13 46rel-ent2-to,e 437
280
	
107 0 122 51 208 0 78 48 18 53 24 38
status 432 280
	
149 0 77 54 206 0 69 48 19 34 34 53
subtotals 2021 1015
	
560 0 261 194 1200 64 75 59 19 32 28 55
ALL-OBJECTS 7633 3988
	
2317 27 68 -853 4500 875 72 59 21 26 30 58
MATCHED-ONLY 3630 2949
	
2317 27 68 223 904 875 40 25 8 14 64 79
TEXT-FILTERING 251 217
	
204 - - 13 47 18 - 19 6 - 81 94
Wrong Word-count
	
Error-rate
ERROR-RATE-PER-WORD 6157 .5 92862 0.0663P&R 2P&R P&2R
F-MEASURES 40.11 49.41 33 .76
373
EJV Summary Scores
PRC PAKTUS Syste m
	 1 .	 Full-Template Test
SLOT
	
PUS ACT
<temp ate>
	
: t
doc-nr
	
282
	
280doe-date
	
275
	
203doe-source
	
276
	
280content
	
348
	
237subtotals
	
348
	
237<tie-up-relati
	
348
	
236status
	
348
	
228entity
	
791
	
487joint-venture
	
179
	
158ownership
	
102
	
1 5activity
	
386
	
104
subtotals
	
1806
	
992
<entity>
	
975
	
647
name
	
873
	
51 3aliases
	
359
	
62location
	
323
	
77
nationality
	
264
	
55type
	
975
	
592entity-rel
	
1153
	
592person
	
118
	
1facility
	
81
	
0subtotals
	
4146
	
1892<entity-relati
	
436
	
263entityl
	
883
	
447entity2
	
266
	
140rel-ent2-to-e
	
436
	
263status
	
430
	
11 8subtotals
	
2015
	
968<activity>
	
386
	
103industry
	
419
	
13 1site
	
551
	
9start
	
103
	
1end
	
16
	
0revenue
	
23
	
0subtotals
	
1112
	
141<industry>
	
412
	
13 2type
	
381
	
12 9prod-sery
	
804
	
207
subtotals
	
1185
	
336<facility>
	
164
	
4name
	
35
	
0location
	
142
	
3type
	
163
	
4
subtotals
	
340
	
7<person>
	
118
	
1name
	
118
	
0
entity
	
118
	
1position
	
136
	
1
subtotals
	
372
	
2<ownership>
	
102
	
1 4
time
	
21
	
0owned
	
102
	
1 4
capital
	
27
	
1owner%
	
376
	
2 8subtotals
	
526
	
4 3
<revenue>
	
23
	
0
type
	
23
	
0total
	
15
	
0
total-eta
	
11
	
0
rate
	
8
	
0rate-eta
	
0
	
0subtotals
	
57
	
0<time>
	
149
	
1after
	
16
	
0before
	
27
	
0during
	
91
	
1
on
	
19
	
0
subtotals
	
153
	
1
COR PAR INC SPU MIS NON ERR UND OVG SUB REC PRE
280 0 0 0 2 0 1 1 0 0 Pt10 0
202 0 1 0 72 7 26 26 0 0 10 0
204 0 70 6 2 0 28 2 26 7 3
166 0 16 55 166 14 59 48 23 9 70
166 0 16 55 166 14 59 48 23 9 70
166 0 15 55 167 0 59 48 23 8 70
132 0 43 53 173 0 67 50 23 24 5 8
254 0 86 147 451 0 73 57 30 25 5 2
29 0 47 82 103 25 89 58 52 62 1 8
9 0 1 5 92 112 92 90 33 10 60
52 0 14 38 320 8 88 83 36 21 50476
0 191 325 1139 145 78 63 33 29 48
404 0 105 138 466 0 64 48 21 21 62
175 68 157 113 473 8 79 54 22 48 4 1
5 5 11 41 338 213 98 94 66 64 12
26 7 15 29 275 269 92 85 38 38 38
29 0 3 23 232 287 90 88 42 9 5 3
372 0 103 117 500 0 66 51 20 22 63
343 0 94 155 716 0 74 62 26 22 58
0 0 0 118 370 100 100 100 . 0
0 0 0 0 81 374 100 100 * * .
950 80 383 479 2733 1521 78 66 25 30 52
152 0 42 69 242 0 70 56 26 22 58
243 0 74 130 566 0 76 64 29 23 54
33 0 44 63 189 32 90 71 45 57 24
86 0 108 69 242 0 83 56 26 56 33
53 0 35 30 342 0 88 80 25 40 45
415 0 261 292 1339 32 82 66 30 39 43
65 0 10 28 311 0 84 80 27 13 63
64 0 12 55 343 0 86 82 42 16 49
5 0 2 2 544 14 99 99 22 28 5 6
1 0 0 0 102 50 99 99 0 0 100
0 0 0 0 16 61 100 100 * * .
0 0 0 0 23 63 100 100 * . .
70 0 14 57 1028 188 94 92 40 17 50
69 0 18 45 325 0 85 79 34 21 5 2
63 0 19 47 299 0 85 78 36 23 49
44 8 78 77 674 0 94 84 37 63 23
107 8 97 124 973 0 92 82 37 48 3 3
4 0 0 0 160 0 98 98 0 0 10 0
0 0 0 0 35 4 100 100 * * *
0 3 0 0 139 0 99 98 0 50 5 0
4 0 0 0 159 0 98 98 0 0 10 04
3 0 0 333 4 98 98 0 21 7 8
0 0 0 118 0 100 100 100 * 00
0 0 0 118 0 100 100 * * .
0 0 0 118 0 100 100 100 * 00
0 0 136 0 100 100 100 * 0
0 0 0 2 372 0 100 100 100 * 0
12 0 0 2 90 0 88 88 14 0 86
0 0 0 0 21 12 100 100 . * *
2 0 10 2 90 0 98 88 14 83 1 4
0 0 1 0 26 5 100 96 0 100 0
20 0 4 4 352 0 95 94 14 17 7 1
22 0 15 6 489 17 96 93 14 40 5 10
0 0 0 23 0 100 100 . * *
0 0 0 0 23 0 100 100 . .
0 0 0 0 15 0 100 100 . .0
0 0 0 11 0 100 100 * * .0
0 0 0 8 . 0 100 100 * . *0
0 0 0 0 0 * . * * .0
0 0 0 57 0 100 100 . * .1
0 0 0 148 0 99 99 0 0 1000
0 0 0 16 0 100 100 * * .0
0 0 0 27 0 100 100 * * *1
0 0 0 90 0 99 99 0 0 100
0 0 0 0 19 0 100 100 * . .1
0 0 0 152 0 99 99 0 0 100
374
•1a
	
s : 1 :I
MATCHED-ONLY 4330
	
3151
	
221 296 1475 1921 51 34 9 21 52 72
TEXT-FILTERING 251
	
185
	
168 - 17 83 14 - 33 9 - 67 9 1
Wrong Req-fills MI-fills Mm-err Max-er rRICHNESS-NORMALIZED-ERROR
11143 . 5Wrong 11794Word-count 1211 9Error-rate 0.9195 0.944 8
ERROR-RATE-PER-WORD 11143 .5 92862 0 .1 2
P&R 2P&R P&2RF-MEASURES 27.06 36.95 21 .34
2. Core-Tem plate Test
SLOT 1'JS ACT
	
C:OR PAR INC SPIT MIS NON ERR UNIY O~G SLB REC PRE
<template> 282 280
	
280 0 0 0 2 0 1 1 0 0 99 10 0
content 348 237
	
166 0 16 55 166 14 59 48 23 9 48 7 0subtotals
348 237
	
166 0 16 55 166 14 59 48 23 9 48 70
<tie-up-relati 348 236
	
166 0 15 55 167 0 59 48 23 8 48 70
status 348 228
	
132 0 43 53 173 0 67 50 23 24 38 5 8entity
791 487
	
254 0 86 147 451 0 73 57 30 25 32 5 2
joint-venture 179 158
	
29 0 47 82 103 25 89 58 52 62 16 1 8
subtotals 1318 873
	
415 0 176 282 727 25 74 55 32 30 31 4 8<entity>
975 647
	
404 0 105 138 466 0 64 48 21 21 41 62
name 873 513
	
175 68 157 113 473 8 79 54 22 48 24 4 1
aliases 359 62
	
5 5 11 41 338 213 98 94 66 64 2 1 2location
323 77
	
26 7 15 29 275 269 92 85 38 38 9 3 8
nationality 264 55
	
29 0 3 23 232 287 90 88 42 9 11 5 3
type 975 592
	
372 0 103 117 500 0 66 51 20 22 38 6 3
entity-rel 1153 592
	
343 0 94 155 716 0 74 62 26 22 30 5 8
subtotals 3947 1891
	
950 80 383 478 2534 777 78 64 25 30 25 5 2
<entity-relati 436 263
	
152 0 42 69 242 0 70 56 26 22 35 5 8
entityl 883 447
	
243 0 74 130 566 0 76 64 29 23 28 5 4
entity2 266 140
	
33 0 44 63 189 32 90 71 45 57 12 24rel-ent2-to-e 436 263
	
86 0 108 69 242 0 83 56 26 56 20 3 3
status 430 118
	
53 0 35 30 342 0 88 80 25 40 12 4 5
subtotals 2015 968
	
415 0 261 292 1339 32 82 66 30 39 20 43
ALL-OBJECTS 7628 3969
	
1946 80 118 1107 4766 848 77 62 28 31 26 50
MATCHED-ONLY 3473 2745
	
1946 80 118 264 992 848 47 28 10 20 57 7 2
TEXT-FILTERING 251 185
	
168 - - 17 83 14 - 33 9 - 67 9 1
Wrong Word-count
	
Error-rat e
ERROR-RATE-PER-WORD 6749 92862 0.0727
P&R 2P&R P&2RF-MEASURES
34.25 42.25 28 . 8
375
EJV Summary Scores
SRA SOLOMON System
1 . Full-Template Test
SLOT PC6 ACT COR PAR INC SPU MIS NON hltlt UND OVG SUB REC PRE
<template> 282 277 277 0 0 0 5 0 2 2 0 0 98 100
doc-nr 282 277 277 0 0 0 5 0 2 2 0 0 98 100
doc-date 275 198 198 0 0 0 77 6 28 28 0 0 72 100
doc-source 276 272 272 0 0 0 4 5 1 1 0 0 98 100
content 347 317 197 0 23 97 127 16 56 36 30 10 57 62
subtotals 347 317 197 0 23 97 127 16 56 36 30 10 57 62
<tie-up-relati 347 318 199 0 22 97 126 0 55 36 30 10 57 62
status 347 318 161 0 60 97 126 0 64 36 30 27 46 51
entity 788 478 245 0 104 129 439 0 73 56 27 30 31 51
joint-venture 179 99 30 0 30 39 119 68 86 66 39 50 17 30
ownership 102 0 0 0 0 0 102 136 100 100 * * 0 *
activity 386 291 128 0 65 98 193 1 74 50 34 34 33 44
subtotals 1802 1186 564 0 259 363 979 205 74 54 31 31 31 48
<entity> 975 544 371 0 84 89 520 0 65 53 16 18 38 68
name 873 407 180 33 114 80 546 18 79 62 20 40 22 48
aliases 359 158 81 5 15 57 258 180 80 72 36 17 23 53
location 323 76 22 8 9 37 284 231 93 88 49 33 8 34
nationality 265 132 57 0 5 70 203 239 83 77 53 8 22 43type 975 544 354
0 101 89 520 0 67 53 16 22 36 65entity-rel 1151 547 324 0
124 99 703 0 74 61 18 28 28 59person 118 40 18
0 1 21 99 326 87 84 52 5 15 45facility
81 1 0 0 0 1 81 355 100 100 100 * 0 0
subtotals 4145 1905 1036 46 369 454 2694 1349 77 65 24 27 26 56<entity-relati 437 295 181
0 46 68 210 0 64 48 23 20 41 61entityl 885 473
251 0 107 115 527 0 75 60 24 30 28 53entity2 267 92
33 0 28 31 206 60 89 77 34 46 12 36
rel-ent2-to-e 437 295 106 0 121 68 210 0 79 48 23 53 24 36
status 432 295 134 0 91 70 207 0 73 48 24 40 31 45
subtotals 2021 1155 524 0 347 284 1150 60 77 57 24 40 26 45
<activity> 386 221 146 0 39 36 201 0 65 52 16 21 38 66
industry 419 221 129 0 51 41 239 0 72 57 18 28 31 58
site 551 151 47 0 40 64 464 34 92 84 42 46 8 31
start 103 0 0 0 0 0 103 107 100 100 * * 0 *
end 16 0 0 0 0 0 16 136 100 100 * * 0 *
revenue 23 0 0 0 0 0 23 138 100 100 * * 0 *
subtotals 1112 372 176 0 91 105 845 415 86 76 28 34 16 47
<industry> 412 222 134 0 48 40 230 0 70 56 18 26 32 60
type 380 222 123 0 50 49 207 0 71 54 22 29 32 55
prod-sery 813 290 34 12 196 48 571 0 95 70 16 83 5 14
subtotals 1193 512 157 12 246 97 778 0 87 65 19 61 14 32
<facility> 164 5 1 0 3 1 160 0 99 98 20 75 1 20
name 35 3 0 0 1 2 34 0 100 97 67 100 0 0
location 142 1 1 0 0 0 141 0 99 99 0 0 1 100
type 163 5 0 0 4 1 159 0 100 98 20 100 0 0
subtotals 340 9 1 0 5 3 334 0 100 98 33 83 0 11
<person> 118 53 22 0 4 27 92 0 85 78 51 15 19 42
name 118 21 3 2 7 9 106 0 97 90 43 67 3 19
entity 118 32 14 0 1 17 103 0 90 87 53 7 12 44
position 136 53 13 0 13 27 110 0 92 81 51 50 10 24
subtotals 372 106 30 2 21 53 319 0 93 86 50 42 8 29
<ownership> 102 0 0 0 0 0 102 0 100 100 * 0
time 21 0 0 0 0 0 21 0 100 100 * 0 *
owned 102 0 0 0 0 0 102 0 100 100 * * 0
capital 27 0 0 0 0 0 27 0 100 100 * * 0 *
owner% 376 0 0 0 0 0 376 0 100 100 * * 0 *subtotals 526 0 0 0
0 0 526 0 100 100 * 0 *<revenue> 23 0 0
0 0 0 23 0 100 100 * * 0 *23 0 0
0 0 0 23 0 100 100 * * 0 *total 15 0 0
0 0 0 15 0 100 100 * * 0 *total-eta 11 0 0
0 0 0 11 0 100 100 * * 0 *rate 8 0 0
0 0 0 8 0 100 100 * * 0 *rate-eta 0 0 0
0 0 0 0 0 * * * * •subtotals 57 0 0
0 0 0 57 0 100 100 * * 0 *<time> 149 0 0
0 0 0 149 0 100 100 * * 0after 16 0 0
0 0 0 16 0 100 100 * * 0 *
before 27 0 0 0 0 0 27 0 100 100 * * 0
during 91 0 0 0 0 0 91 0 100 100 * * 0 *
on 19 0 0 0 0 0 19 0 100 100 * * 0 *
subtotals 153 0 0 0 0 0 153 0 100 100 * * 0 *
37 6
MATCHED-ONLY
	
4881 3825
TEXT-FILTERING	 251	 201	 187
RICHNESS-NORMALIZED-ERRO R
ERROR-RATE-PER-WORD
F-MEASURES
Wron g
10809
Wron g
10809
P&R30 .8
	
107
	
300
	
1356 2045
	
48
	
28
	
8
	
23
	
56
	
7 1
	 14	 64	 17
	
-	 25
	
7
	
-
	
74	 93
	
Req-fills
	
All-fills
	
Mtn-err
	
Max-err
	
11795
	
12119
	
0 .8919
	
0.9164
	
Word-count
	
Error-rat e
	
92862
	
0.116 4
	
2P&R
	
P&2R
	
39.56
	
25 .2 2
2. Core-Template Test
SLOT YOS ACT
	
COR PAR INC SIFT MIS NON ERR UND OVG SUB REC PRE
<template> 282 277
	
277 0 0 0 5 0 2 2 0 0 98 100content
347 317
	
197 0 23 97 127 16 56 36 30 10 57 62subtotals
347 317
	
197 0 23 97 127 16 56 36 30 10 57 62
<tie-up-relati 347 318
	
199 0 22 97 126 0 55 36 30 10 57 62
status 347 318
	
161 0 60 97 126 0 64 36 30 27 46 5 1
entity 788 478
	
245 0 104 129 439 0 73 56 27 30 31 5 1
joint-venture 179 99
	
30 0 30 39 119 68 86 66 39 50 17 3 0
subtotals 1314 895
	
436 0 194 265 684 68 72 52 30 31 33 49
<entity> 975 544
	
371 0 84 89 520 0 65 53 16 18 38 68
name 873 407
	
180 34 113 80 546 18 79 62 20 40 22 48
aliases 359 158
	
81 5 15 57 258 180 80 72 36 17 23 5 3location
323 76
	
22 8 9 37 284 231 93 88 49 33 8 3 4
nationality 265 132
	
57 0 5 70 203 239 83 77 53 8 22 43type
975 544
	
354 0 101 89 520 0 67 53 16 22 36 65
entity-rel 1151 547
	
324 0 124 99 703 0 74 61 18 28 28 5 9
subtotals 3946 1864
	
1018 47 367 432 2514 668 76 64 23 27 26 5 6
<entity-relati 437 295
	
181 0 46 68 210 0 64 48 23 20 41 6 1
entityl 885 473
	
251 0 107 115 527 0 75 60 24 30 28 5 3
entity2 267 92
	
33 0 28 31 206 60 89 77 34 46 12 3 6rel-ent2-to-e
437 295
	
106 0 121 68 210 0 79 48 23 53 24 3 6status
432 295
	
134 0 91 70 207 0 73 48 24 40 31 45
subtotals 2021 1155
	
524 0 347 284 1150 60 77 57 24 40 26 45la p s :
it t' ,r . t .
MATCHED-ONLY 3555 2955
	
2175 47 96 227 827 812 42 23 8 19 62 74
TEXT-FILTERING _
	
251 201
	
187 - - 14 64 17 - 25 7 - 74 93
Wrong Word-count
	
Error-rate
ERROR-RATE-PER-WORD 6507 .5 92862 0.0701
P&R 2P&R P&2R
F-MEASURES 37.08 44.77 31 .64
377
EJV Summary Scores
SRI FASTUS System
1 . Full-Template Test
SLOT POS ACT COR PAR INC SPU MIS NON ERR UND OVG SUB REC PRE
<template> 282 281 281 0 0 0 1 0 0 0 0 0 100 100
doc-nr 282 281 281 0 0 0 1 0 0 0 0 0 100 100
doe-date 275 281 273 0 1 7 1 0 3 0 2 0 99 97
doe-source 276 0 0 0 0 0 276 6 100 100 * * 0 *
content 348 319 223 0 32 64 93 6 46 27 20 12 64 70
subtotals 348 319 223 0 32 64 93 6 46 27 20 12 64 70
<tie-up-relati 348 318 223 0 31 64 94 0 46 27 20 12 64 70
status 348 318 192 0 62 64 94 0 53 27 20 24 55 60
entity 791 650 391 0 118 141 282 0 58 36 22 23 49 60
joint-venture 180 119 61 0 16 42 103 74 72 57 35 21 34 51
ownership 102 68 38 0 0 30 64 143 71 63 44 0 37 56
activity 387 304 134 0 32 138 221 9 74 57 45 19 35 44
subtotals 1808 1459 816 0 228 415 764 226 63 42 28 22 45 56
<entity> 976 807 538 0 108 161 330 0 53 34 20 17 55 67
name 872 518 352 35 58 73 427 23 61 49 14 17 42 71aliases
359 246 173 4 7 62 175 277 58 49 25 5 49 71location 323 87
33 5 9 40 276 348 90 85 46 24 11 41
nationality 265 53 27 1 2 23 235 383 90 89 43 8 10 52type
976 807 523 0 123 161 330 0 54 34 20 19 54 65
entity-rel 1152 837 520 0 124 193 508 0 61 44 23 19 45 62
person 118 12 10 0 0 2 108 480 92 92 17 0 8 83
facility 81 0 0 0 0 0 81 511 100 100 * * 0 *
subtotals 4146 2560 1638 45 323 554 2140 2022 65 52 22 17 40 65
<entity-relati 437 335 226 0 41 68 170 0 55 39 20 15 52 67
entityl 885 667 410 0 110 147 365 0 60 41 22 21 46 61
cntity2 268 170 71 0 36 63 161 64 78 60 37 34 26 42
rel-ent2-to-e 437 335 151 0 116 68 170 0 70 39 20 43 34 45
status 432 335 176 0 88 71 168 0 65 39 21 33 41 52
subtotals 2022 1507 808 0 350 349 864 64 66 43 23 30 40 54
<activity> 386 303 162 0 39 102 185 0 67 48 34 19 42 53
industry 419 303 160 0 40 103 219 0 69 52 34 20 38 53
site 551 32 8 0 6 18 537 31 98 97 56 43 1 25
start 103 21 0 0 9 12 94 104 100 91 57 100 0 0
end 16 0 0 0 0 0 16 156 100 100 • * 0 *
revenue 23 0 0 0 0 0 23 156 100 100 * * 0
subtotals 1112 356 168 0 55 133 889 447 86 80 37 25 15 47
<industry> 413 302 173 0 36 93 204 0 66 49 31 17 42 57
type 383 302 153 0 47 102 183 0 68 48 34 24 40 51
prod-sery 826 493 153 27 137 176 509 13 83 62 36 47 20 34
subtotals 1209 795 306 27 184 278 692 13 78 57 35 38 26 40
<facility> 164 4 3 0 0 1 161 0 98 98 25 0 2 75
name 35 0 0 0 0 0 35 3 100 100 * * 0 *
location 142 2 1 0 0 1 141 0 99 99 50 0 1 50type
163 4 3 0 0 1 160 0 98 98 25 0 2 75
subtotals 340 6 4 0 0 2 336 3 99 99 33 0 1 67
<person> 118 12 10 0 0 2 108 0 92 92 17 0 8 83
name 118 12 6 4 0 2 108 0 93 92 17 20 7 67
entity 118 12 10 0 0 2 108 0 92 92 17 0 8 83position
136 14 13 0 1 0 122 0 90 90 0 7 10 93subtotals 372
38 29 4 1 4 338 0 92 91 10 9 8 82<ownership> 102
68 41 0 0 27 61 0 68 60 40 0 40 60time 21 0 0
0 0 0 21 35 100 100 • * 0 *owned 102 68
22 0 19 27 61 0 83 60 40 46 22 32capital
27 16 13 0 2 1 12 22 54 44 6 13 48 8 1owner%
376 218 83 1 27 107 265 2 83 70 49 25 22 38subtotals
526 302 118 1 48 135 359 59 82 68 45 29 22 39<revenue> 23
0 0 0 0 0 23 0 100 100 * * 0 *
23 0 0 0 0 0 23 0 100 100 * * 0 *tottotal 15
0 0 0 0 0 15 0 100 100 * • 0 *total-eta 11
0 0 0 0 0 11 0 100 100 * 0rate 8 0
0 0 0 0 8 0 100 100 * * 0 •rate-eta 0 0
0 0 0 0 0 0 * * * * * nsubtotals
57 0 0 0 0 0 57 0 100 100 * * 0 *<time> 149 15
1 0 8 6 140 0 99 94 40 89 1 7after 16 0
0 0 0 0 16 0 100 100 * * 0 *before
27 1 0 0 0 1 27 0 100 100 100 * 0 0during 91
11 0 0 4 7 87 0 100 96 64 100 0 0on 19 2
1 0 0 1 18 0 95 95 50 0 5 50subtotals 153 14
1 0 4 9 148 0 99 97 64 80 1 7
378
law* : 11' '
MATCHED-ONLY
	
6345 5254
TEXT-FILTERING	 251	 252
RICHNESS-NORMALIZED-ERROR
ERROR-RATE-PER-WORD
F-MEASURES
4111
	
77
	
50
	
425
	
1516 2840
	
39
	
24
	
8
	
14
	
65
	
79
	227	 -	 -	 25	 24	 6	 -	 10	 10	 -	 90	 90
	
Wrong
	
Req-fills
	
All-tills
	
Min-err
	
Max-er r
	
9886.5
	
11804
	
12129
	
0.8151
	
0 .8376
	
Wrong
	
Word-count
	
Error-rate
	
9886.5
	
92862
	
0.1065
	
P&R
	
2P&R
	
P&2R
	
42.67
	
49 .97
	
37.23
2. Core-Template Test
SLOT FOR ACT tOR PAR INC SPU MIS NON ERR UND OVG SUB REC PRE
<template> 282 281
	
281 0 0 0 1 0 0 0 0 0 100 100
content 348 319
	
223 0 32 64 93 6 46 27 20 12 64 70
subtotals 348 319
	
223 0 32 64 93 6 46 27 20 12 64 70
<tie-up-relati 348 318
	
223 0 31 64 94 0 46 27 20 12 64 70status 348 318
	
192 0 62 64 94 0 53 27 20 24 55 60
entity 791 650
	
391 0 118 141 282 0 58 36 22 23 49 60
joint-venture 180 119
	
61 0 16 42 103 74 72 57 35 21 34 5 1subtotals 1319 1087
	
644 0 196 247 479 74 59 36 23 23 49 59
<entity> 976 807
	
538 0 108 161 330 0 53 34 20 17 55 67
name 872 518
	
352 35 58 73 427 23 61 49 14 17 42 7 1
aliases 359 246
	
173 4 7 62 175 277 58 49 25 5 49 7 1
location 323 87
	
33 5 9 40 276 348 90 85 46 24 11 4 1
nationality 265 53
	
27 1 2 23 235 383 90 89 43 8 10 5 2type
976 807
	
523 0 123 161 330 0 54 34 20 19 54 65
entity-rel 1152 837
	
520 0 124 193 508 0 61 44 23 19 45 62
subtotals 3947 2548
	
1628 45 323 552 1951 1031 63 49 22 17 42 65<entity-relati 437 335
	
226 0 41 68 170 0 55 39 20 15 52 67
entityl 885 667
	
410 0 110 147 365 0 60 41 22 21 46 6 1
entity2 268 170
	
71 0 36 63 161 64 78 60 37 34 26 42
rel-ent2-to-e 437 335
	
151 0 116 68 170 0 70 39 20 43 34 45
status 432 335
	
176 0 88 71 168 0 65 39 21 33 41 5 2
subtotals 2022 1507
	
808 0 350 349 864 64 66 43 23 30 40 5 4
ALL-OBJECTS 7636 5461
	
3303 45 43 1212 3387 1175 62 44 22 22 44 6 1
MATCHED-ONLY 4563 4053
	
3303 45 43 264 774 1175 31 17 6 12 73 82
TEXT-FILTERING 251 252
	
227 - - 25 24 6 - 10 10 - 90 90
Wrong Word-count
	
Error-rate
ERROR-RATE-PER-WORD 5522. 5P&R 928622P&R 0.059 5P&2R
F-MEASURES 50.78 56.4 46.18
379
EJV Summary Scores
TRW DEFT System
1 . Full-Template Test
SLOT POS ACT-BUR PAR INC SPU MIS NON ERR UND OVG SUB REC PR E
<template> 282 280 - 268 0 0 12 14 0 9 5 4 0 95 96
doc-nr 282 280 268 0 0 12 14 0 9 5 4 0 95 96
doc-date 275 200 192 0 0 8 83 7 32 30 4 0 70 96
doc-source 276 280 262 0 0 18 14 0 11 5 6 0 95 94
content 348 86 22 0 46 18 280 24 94 80 21 68 6 26
subtotals 348 86 22 0 46 18 280 24 94 80 21 68 6 26
<tie-up-relati 348 38 22 0 7 9 319 0 94 92 24 24 6 5 8
status 348 0 0 0 0 0 348 0 100 100 * * 0 *
entity 791 53 24 0 12 17 755 0 97 95 32 33 3 45
joint-venture 179 53 2 0 13 38 164 0 99 92 72 87 1 4
ownership 102 0 0 0 0 0 102 16 100 100 * * 0 *
activity 386 0 0 0 0 0 386 0 100 100 * * 0 *
subtotals 1806 106 26 0 25 55 1755 16 99 97 52 49 1 24
<entity> 975 49 28 0 13 8 934 0 97 96 16 32 3 5 7
name 873 49 17 7 17 8 832 0 98 95 16 50 2 42
aliases 359 0 0 0 0 0 359 11 100 100 * * 0 *location
323 27 0 4 1 22 318 7 99 98 81 60 1 7
nationality 264 27 1 0 3 23 260 10 100 98 85 75 0 4
type 975 0 0 0 0 0 975 0 100 100 * * 0 *entity-rel 1151 0 0 0
0 0 1151 0 100 100 * * 0 *person 118 31 0 0
1 30 117 18 100 99 97 100 0 0facility 81 0
0 0 0 0 81 26 100 100 * * 0 *subtotals 4144 134 18
11 22 83 4093 72 99 99 62 54 0 1 8<entity-relati 437 0 0 0
0 0 437 0 100 100 * * 0 *entityl 885 0 0 0
0 0 885 0 100 100 * * 0 *entity2 267 0 0 0
0 0 267 0 100 100 * * 0 *rel-ent2-to-e 437
0 0 0 0 0 437 0 100 100 * * 0 *status
431 0 0 0 0 0 431 0 100 100 * * 0 *subtotals
2020 0 0 0 0 0 2020 0 100 100 * * 0 *<activity> 386
0 0 0 0 0 386 0 100 100 * * 0industry 418
0 0 0 0 0 418 0 100 100 * * 0 *site 551 0 0 0
0 0 551 0 100 100 * * 0 *start 103 0 0 0
0 0 103 0 100 100 * * 0 *end 16 0 0 0
0 0 16 0 100 100 * * 0 *revenue 23 0 0 0
0 0 23 0 100 100 * * 0 *subtotals 1111 0 0 0
0 0 1111 0 100 100 * * 0 *<industry> 411
0 0 0 0 0 411 0 100 100 * * 0 *type 378 0 0 0
0 0 378 0 100 100 * * 0prod-sery 793 0 0 0
0 0 793 0 100 100 * 0 *subtotals 1171 0 0 0 0
0 1171 0 100 100 * * 0 *<facility> 164 0 0 0
0 0 164 0 100 100 * * 0 *name 35 0 0 0
0 0 35 0 100 100 * * 0location 142 0 0 0
0 0 142 0 100 100 * * 0 *type 163 0 0 0
0 0 163 0 100 100 * * 0subtotals 340 0 0
0 0 0 340 0 100 100 * * 0 *<person> 118
18 0 0 2 16 116 0 100 98 89 100 0 0name 118
18 0 0 2 16 116 0 100 98 89 100 0 0entity 118 28 0
0 0 28 118 0 100 100 100 * 0 0position 136 0 0
0 0 0 136 0 100 100 * * 0 *subtotals 372 46 0 0 2 44 370
0 100 99 96 100 0 0<ownership> 102 0 0 0 0 0 102 0
100 100 * * 0time 21 0 0 0 0 0 21 0 100
100 * * 0 *owned 102 0 0 0 0 0 102 0 100 100 * * 0
capital 27 0 0 0 0 0 27 0 100 100 * * 0 *
owner% 376 0 0 0 0 0 376 0 100 100 * * 0 *
subtotals 526 0 0 0 0 0 526 0 100 100 * * 0 *
<revenue> 23 0 0 0 0 0 23 0 100 100 * * 0 *type
23 0 0 0 0 0 23 0 100 100 * * 0 *total
15 0 0 0 0 0 15 0 100 100 * * 0 *total-eta
11 0 0 0 0 0 11 0 100 100 * * 0 *rate
8 0 0 0 0 0 8 0 100 100 * * 0 *rate-eta
0 0 0 0 0 0 0 0 * * * * * *subtotals
57 0 0 0 0 0 57 0 100 100 * 0
<time> 149 0 0 0 0 0 149 0 100 100 * * 0 *
after 16 0 0 0 0 0 16 0 100 100 * * 0 *before
27 0 0 0 0 0 27 0 100 100 * * 0 *during
91 0 0 0 0 0 91 0 100 100 * * 0 *on 19 0 0
0 0 0 19 0 100 100 n * 0subtotals 153 0 0 0 0
0 153 0 100 100 * * 0 *
380
111-• .
	
I k
MATCHED-ONLY
	
566 22 5
TEXT-FILTERING	 251	 63
RICHNESS-NORMALIZED-ERROR
ERROR-RATE-PER-WORD
F-MEASURES
66
	
11
	
4
	
83
	
424
	
112
	
89
	
75
	
37
	
50
	
13
	
32
58	 -	 -	 5	 193	 26	 -	 77	 8	 -	 23	 92
	
Wrong
	
Req-t lls
	
All-fills
	
Mm-err
	
Max-er r
	
12176.5
	
11797
	
12122
	
1 .0045
	
1 .032 2
12176. 5
	
Wrong
	
Word-count
	
Error-rate
	
92862
	
0.131 1
	
P&R
	
2P&R
	
P&2R
	
1 .15
	
2.64
	
0 .74
2. Core-Template Test
SLOT PUS ACT
	
COR PAR INC SPU MIS NON ERR UND OVG SUB REC PR E
<template> 282 280 268 0 0 12 14 0 9 — 4 - 0 95 96
content 348 86 22 0 46 18 280 24 94 80 21 68 6 26
subtotals 348 86 22 0 46 18 280 24 94 80 21 68 6 26
<tie-up-relati 348 38 22 0 7 9 319 0 94 92 24 24 6 58
status 348 0 0 0 0 0 348 0 100 100 * * 0
entity 791 53 24 0 12 17 755 0 97 95 32 33 3 45
joint-venture 179 53 2 0 13 38 164 0 99 92 72 87 1 4subtotals 1318 106 26 0 25 55 1267 0 98 96 52 49 2 24
<entity> 975 49 28 0 13 8 934 0 97 96 16 32 3 5 7
name 873 49 17 7 17 8 832 0 98 95 16 50 2 42
aliases 359 0 0 0 0 0 359 11 100 100 * * 0
location 323 27 0 4 1 22 318 7 99 98 81 60 1 7
nationality 264 27 1 0 3 23 260 10 100 98 85 75 0 4
type 975 0 0 0 0 0 975 0 100 100 * * 0 *
entity-rel 1151 0 0 0 0 0 1151 0 100 100 * * 0
subtotals 3945 103 18 11 21 53 3895 28 99 99 51 53 0 2 3<entity-relati 437 0 0 0 0 0 437 0 100 100 * * 0 *
entityl 885 0 0 0 0 0 885 0 100 100 * * 0 *
entity2 267 0 0 0 0 0 267 0 100 100 * * 0rel-ent2-to-e 437 0 0 0 0 0 437 0 100 100 * * 0 *
status 431 0 0 0 0 0 431 0 100 100 * * 0 *
subtotals 2020 0 0 0 0 0 2020 0 100 100 * * 0 *
ALL-OBJECTS 7631 295 66 11 4 126 7462 52 99 98 43 58 1 2 4
MATCHED-ONLY 525 212 66 11 4 70 383 52 88 73 33 50 14 3 4
TEXT-FILTERING 251 63 58 - - 5 193 26 - 77 8 - 23 92
Wrong Word-count
	
Error-rate
ERROR-RATE-PER-WORD 7685 .5
P&R
92862
2P&R
0.0828
P&2R
F-MEASURES 1 .8 4.06 1 .16
38 1
,
EJV Summary Scores
Umass/Hughes CIRCUS System
1 . Full-Template Test
SLOT POS ACT---COR PAR INC SPU MIS NON ERR UNI) OVU SUB REC PRE
<template> 282 281 281 --
	
0 0 0 1 0 0 0 0 0 100 100
doc-nr 282 281 281 0 0 0 1 0 0 0 0 0 100 100
doc-date 275 281 274 0 0 7 1 0 3 0 2 0 100 98
doc-source 276 281 275 0 0 6 1 0 2 0 2 0 100 98
content 349 223 169 0 10 44 170 21 57 49 20 6 48 76
subtotals 349 223 169 0 10 44 170 21 57 49 20 6 48 76
<tie-up-relati 349 237 182 0 10- 45 157 0 54 45 19 5 52 77
status 349 167 115 0 33 19 201 0 69 58 11 22 33 69
entity 793 509 304 0 66 139 423 0 67 53 27 18 38. 60
joint-venture 180 78 33 0 16 29 131 53 84 73 37 33 18 42
ownership 102 41 27 0 2 12 73 114 76 72 29 7 26 66
activity 388 103 45 0 24 34 319 15 89 82 33 35 12 44
subtotals 1812 898 524 0 141 233 1147 182 74 63 26 21 29 58<entity> 976 1017 611 0 128 278 237 0 51 24 27 17 63 60
name 873 873 320 84 215 254 254 16 68 29 29 42 41 41
aliases 359 92 66 3 4 19 286 358 82 80 21 8 19 73
location 325 223 91 8 23 101 203 393 78 62 45 22 29 43
nationality 265 190 87 1 9 93 168 418 76 63 49 10 33 46
type 976 878 545 0 120 213 311 0 54 32 24 18 56 62
entity-rel 1155 599 392 0 73 134 690 0 70 60 22 16 34 65
person 118 42 16 0 4 22 98 535 88 83 52 20 14 38
facility 81 0 0 0 0 0 81 577 100 100 * * 0 *
subtotals 4152 2897 1517 96 448 836 2091 2297 69 50 29 24 38 54
<entity-relati 436 237 175 0 29 33 232 0 63 53 14 14 40 74
entityl 883 518 304 0 76 138 503 0 70 57 27 20 34 59
entity2 266 78 38 0 19 21 209 53 87 78 27 33 14 49
rel-ent2-to-e 436 237 101 0 103 33 232 0 78 53 14 50 23 43
status 431 237 136 0 66 35 229 0 71 53 15 33 32 57
subtotals 2016 1070 579 0 264 227 1173 53 74 58 21 31 29 54
<activity> 387 101 66 0 19 16 302 0 84 78 16 22 17 65
industry 420 88 64 0 8 16 348 0 85 83 18 11 15 73
site 552 0 0 0 0 0 552 9 100 100 0 *
start 103 0 0 0 0 0 103 48 100 100 * • 0
end 16 0 0 0 0 0 16 61 100 100 * • 0 •
revenue 23 17 2 0 1 14 20 60 94 87 82 33 9 12
subtotals 1114 105 66 0 9 30 1039 178 94 93 28 12 6 63
<industry> 413 89 66 0 11 12 336 0 84 81 13 14 16 74
type 380 58 41 0 8 9 331 89 87 16 16 11 71prod-sery 807 113 21 26 50 16 710 11 96 88 14 65 4 30
subtotals 1187 171 62 26 58 25 1041 12 94 88 15 49 6 44
<facility> 164 4 1 0 1 2 162 0 99 99 50 50 1 25
name 35 0 0 0 0 0 35 100 100 0 *
location 142 4 1 0 1 2 140 0 99 98 50 50 1 25type 163 0 0 0 0 0 163 0 100 100 * * 0 *
subtotals 340 4 1 0 1 2 338 100 99 50 50 0 25
<person> 118 195 64 0 11 120 43 0 73 36 62 15 54 33
name 118 164 45 5 17 97 51 0 78 43 59 29 40 29
entity 118 42 16 0 8 18 94 0 88 80 43 33 14 38
position 136 112 46 0 7 59 83 0 76 61 53 13 34 41
subtotals 372 318 107 5 32 174 228 0 80 61 55 24 29 34
<ownership> 102 41 29 0 3 9 70 0 74 69 22 9 28 71
time 21 0 0 0 0 0 21 27 100 100 * * 0 *
owned 102 17 7 0 7 3 88 0 93 86 18 50 7 41
capital 27 8 4 0 1 3 22 15 87 81 38 20 15 50
owner% 376 124 48 10 30 36 288 1 87 76 29 40 14 43
subtotals 526 149 59 10 38 42 419 43 89 80 28 40 12 43
<revenue> 23 17 2 0 3 12 18 0 94 78 70 60 9 12
type 23 17 2 0 3 12 18 0 94 78 70 60 9 12
total 15 0 0 0 0 0 15 2 100 100 * * 0 *
total-eta 11 0 0 0 0 0 11 2 100 100 * * 0 *
rate 8 17 2 0 1 14 5 0 91 62 82 33 25 12
rate-eta 0 0 0 0 0 0 0 2 * * * * *
subtotals 57 34 4 0 4 26 49 6 95 86 76 50 7 12<time> 149 0 0 0 0 0 149 0 100 100 * * 0
after 16 0 0 0 0 0 16 0 100 100 * * 0 *
before 27 0 0 0 0 0 27 0 100 100 * * 0 *
during 91 0 0 0 0 0 91 0 100 100 * * 0 *
on 19 0 0 0 0 0 19 0 100 100 0 *
subtotals 153 0 0 0 0 0 153 0 100 100 * * 0 *
38 2
ALL-OBJECTS 12078 5869
	
3088 T37 T61 1639 7848 2793 77 65 28 25 26 54
MATCHED-ONLY 5772 4137
	
3088 137 161 313 1948 2793 48 34 8 I7 55 76TEXT-FILTERING
251 174 164 - - 10 87 21 - 35 6 - 65 94
RICHNESS-NORMALIZED-ERROR
Wrong
10560 .5
Req-Tills
11807
All-fills
12133
Mm-err
0.8704
Max-er r
0.8944
ERROR-RATE-PER-WORD
Wrong
10560 .5
Word-count
92862
Error-rate
0.1137
F-MEASURES
P&R
35.18
2P&R
44.39
P&2R
29 .13
2.
	
Core-Template Test
SLO FOS ACT
	
COR PAR INC SPLT MIS NON ERR UND OVG SUB REC PRE
<template> 282 281
	
281 0 0 0 1 0 0 0 0 0 100 100
content 349 223
	
169 0 10 44 170 21 57 49 20 6 48 76
subtotals 349 223
	
169 0 10 44 170 21 57 49 20 6 48 76
<tie-up-relati 349 237
	
182 0 10 45 157 0 54 45 19 5 52 77status 349
167
	
115 0 33 19 201 0 69 58 11 22 33 69entity 793 509
	
304 0 66 139 423 0 67 53 27 18 38 60
joint-venture 180 78
	
33 0 16 29 131 53 84 73 37 33 18 42
subtotals 1322 754
	
452 0 115 187 755 53 70 57 25 20 34 60<entity> 976 1017
	
611 0 128 278 237 0 51 24 27 17 63 60
name 873 873
	
320 84 215 254 254 16 68 29 29 42 41 4 1
aliases 359 92
	
66 3 4 19 286 358 82 80 21 8 19 73location 325 223
	
91 8 23 101 203 393 78 62 45 22 29 43
nationality 265 190
	
87 1 9 93 168 418 76 63 49 10 33 46
type 976 878
	
545 0 120 213 311 0 54 32 24 18 56 62entity-rel 1155 599
	
392 0 73 134 690 0 70 60 22 16 34 65
subtotals 3953 2855
	
1501 96 444 814 1912 1185 68 48 28 24 39 54
<entity-relati 436 237
	
175 0 29 33 232 0 63 53 14 14 40 74
entity! 883 518
	
304 0 76 138 503 0 70 57 27 20 34 59
entity2 266 78
	
38 0 19 21 209 53 87 78 27 33 14 49
rel-ent2-to-e 436 237
	
101 0 103 33 232 0 78 53 14 50 23 4 3
status 431 237
	
136 0 66 35 229 0 71 53 15 33 32 57
subtotals 2016 1070
	
579 0 264 227 1173 53 74 58 21 31 29 54
ALL-OBJECTS 7640 4902
	
2701 96 145 1272 4010 1312 32 26 24 36 56
MATCHED-ONLY 4513 3562
	
2701 96 145 284 1235 1312 43 27 8 16 61 7 7
TEXT-FILTERING 251 174
	
164 - - 10 87 21 - 35 6 - 65 94
Wrong Word-count
	
Error-rat e
ERROR-RATE-PER-WORD 6163 92862 0.066 4
P&R 2P&R P&2R
F-MEASURES 43.84 50 .44 38 .76
383
EJV Summary Scores
UNISYS CBAS System
1 . Full-Template Test
SLOT POS ACT COR PAR INC SPU MIS NON 1:RR UND OVG SUB REC PRE
<template> 282 280 280 0 0 0 2 0 1 1 0 0 99 100
doc-nr 282 280 280 0 0 0 2 0 1 1 0 0 99 100
doc-date 275 199 196 0 3 0 76 7 29 28 0 2 71 9 8
doc-source 276 280 274 0 0 6 2 0 3 1 2 0 99 9 8
content 348 293 223 0 9 61 116 19 45 33 21 4 64 76
subtotals 348 293 223 0 9 61 116 19 45 33 21 4 64 76<tie-up-relati
348 293 223 0 9 61 116 0 45 33 21 4 64 76status
348 293 181 0 51 61 116 0 56 33 21 22 52 62entity
791 1086 449 0 55 582 287 0 67 36 54 11 57 4 1joint-venture
179 40 29 0 5 6 145 95 84 81 15 15 16 72ownership
102 0 0 0 0 0 102 162 100 100 * * 0activity
387 0 0 0 0 0 387 17 100 100 * * 0 *subtotals
1807 1419 659 0 111 649 1037 274 73 57 46 14 36 46<entity> 975
941 589 0 54 298 332 0 54 34 32 8 60 62name 872
939 335 76 187 341 274 1 69 31 36 38 43 40aliases 359
123 76 6 10 31 267 342 80 74 25 14 22 6 4location 323
0 0 0 0 0 323 403 100 100 * * 0 *nationality 265
147 37 1 10 99 217 385 90 82 67 22 14 26type
975 941 547 0 96 298 332 0 57 34 32 15 56 58entity-rel
1152 1135 553 0 122 460 477 0 66 41 40 18 48 49person
118 0 0 0 0 0 118 517 100 100 * * 0 *facility
81 0 0 0 0 0 81 556 100 100 * * 0 *subtotals
4145 3285 1548 83 425 1229 2089 2204 70 50 37 23 38 4 8<entity-relati 437
329 239 0 16 74 182 0 53 42 22 6 55 7 3entityl 885
1121 457 0 56 608 372 0 69 42 54 11 52 4 1entity2 267
75 38 0 10 27 219 97 87 82 36 21 14 5 1rel-ent2-to-e 437
329 142 0 113 74 182 0 72 42 22 44 32 4 3status
432 329 184 0 69 76 179 0 64 41 23 27 42 56subtotals
2021 1854 821 0 248 785 952 97 71 47 42 23 41 44<activity>
386 0 0 0 0 0 386 0 100 100 * * 0 *industry
418 0 0 0 0 0 418 0 100 100 * * 0 *site
551 0 0 0 0 0 551 0 100 100 * * 0 *start
103 0 0 0 0 0 103 0 100 100 * * 0 *end
16 0 0 0 0 0 16 0 100 100 * * 0 *revenue
23 0 0 0 0 0 23 0 100 100 * + 0 *subtotals
1111 0 0 0 0 0 1111 0 100 100 * * 0 *
<industry> 411 0 0 0 0 0 411 0 100 100 * * 0 *
type 378 0 0 0 0 0 378 0 100 100 * * 0 *prod-sery
793 0 0 0 0 0 793 0 100 100 * * 0subtotals
1171 0 0 0 0 0 1171 0 100 100 * * 0 +<facility> 164
0 0 0 0 0 164 0 100 100 + * 0 *name 35 0 0
0 0 0 35 0 100 100 * + 0 *location 142 0 0
0 0 0 142 0 100 100 * a 0 *type 163 0 0
0 0 0 163 0 100 100 * * 0 *subtotals 340 0 0
0 0 0 340 0 100 100 * + 0 *<person> 118
0 0 0 0 0 118 0 100 100 * 0 *name 118 0 0
0 0 0 118 0 100 100 * + 0 *entity 118 0 0
0 0 0 118 0 100 100 * * 0 *position 136 0 0
0 0 0 136 0 100 100 + * 0 *subtotals 372 0 0 0
0 0 372 0 100 100 * * 0 +<ownership> 102 0 0
0 0 0 102 0 100 100 * * 0 *time 21 0 0
0 0 0 21 0 100 100 * * 0 *owned 102 0 0
0 0 0 102 0 100 100 * * 0 *capital 27
0 0 0 0 0 27 0 100 100 * * 0 *owner% 376
0 0 0 0 0 376 0 100 100 * + 0 *subtotals 526
0 0 0 0 0 526 0 100 100 + * 0 *<revenue> 23 0 0 0
0 0 23 0 100 100 * * 0 +type 23 0 0 0
0 0 23 0 100 100 * * 0 *total 15 0 0 0 0 0
15 0 100 100 * * 0total-eta 11 0 0 0 0 0 11 0 100
100 * * 0 *rate 8 0 0 0 0 0 8 0 100 100 * * 0 +
rate-eta 0 0 0 0 0 0 0 0 * a * + + *
subtotals 57 0 0 0 0 0 57 0 100 100 * * 0 +
<time> 149 0 0 0 0 0 149 0 100 100 * * 0 *
after 16 0 0 0 0 0 16 0 100 100 * * 0 *
before 27 0 0 0 0 0 27 0 100 100 * * 0
during 91 0 0 0 0 0 91 0 100 100 * 0 *
on 19 0 0 0 0 0 19 0 100 100 * * 0 *
subtotals 153 0 0 0 0 0 153 0 100 100 * * 0 *
384
.1a -a :
	
.: L
MATCHED-ONLY
	
5225 4896
TEXT-FILTERING
	
251
	
19 8
RICHNESS-NORMALIZED-ERRO R
ERROR-RATE-PER-WORD
F-MEASURES
2. Core-Template Test
ACI'COR
3251
	
83
	
168
	
1004
	
1333 2594
	
47
	
26
	
20
	
15
	
63
	
67
	186	 -	 -	 12	 65	 19	 -
	
26
	
6
	
-
	
74
	
9 4
	
Wrong
	
Req-Tills
	
All-fills
	
Non-err
	
Max-err
	
0 .9472
	
0.973 311482 .5 11797 12122
Wron g
11482.5
Word-count Error-rate
92862 0.123 7
PAR 2P&R P&2R
34.84 41 .72 29.9
PAR INC SPU MIS NON ERRSLOT P(3S
<template> 28 2
content 348
subtotals 348
<tie-up-relati 348
status 348
entity 79 1
joint-venture 179subtotals 131 8
<entity> 97 5
name 87 2
aliases 35 9
location 323
nationality 265
type 975
etiy-rel 1152
subtotals 394 6
<entity-relati 43 7
entityl 88 5
entity2 267
rel-ent2-to-e 43 7
status 43 2
subtotals 202 1
ALL-OBJECTS 763 3
MATCHED-ONLY 477 2
TEXT-FILTERING 251
ERROR-RATE-PER-WORD
F-MEASURES
280
	
280 0 0 0 2 0 1 1 0 0 99 100
293
	
223 0 9 116 19 45 33 21 4 64 76
293
	
223 0 9 61 116 19 45 33 21 4 64 76
293
	
223 0 9 61 116 0 45 33 21 4 64 76
293
	
181 0 51 61 116 0 56 33 21 22 52 62
1086
	
449 0 55 582 287 0 67 36 54 11 57 4 1
40
	
29 0 5 6 145 95 84 81 15 15 16 7 2
1419
	
659 0 111 649 548 95 66 42 46 14 50 4 6
941
	
589 0 54 298 332 0 54 34 32 8 60 62
939
	
335 77 186 341 274 1 69 31 36 38 43 40123
	
77 6 9 31 267 342 79 74 25 13 22 65
0
	
0 0 0 0 323 403 100 100 • * 0 .
147
	
37 1 10 99 217 .385 90 82 67 22 14 26
941
	
547 0 96 298 332 0 57 34 32 15 56 5 8
1135
	
552 0 123 460 477 0 66 41 40 18 48 49
3285
	
1548 84 424 1229 1890 1131 69 48 37 23 40 48
329
	
239 0 16 74 182 0 53 42 22 6 55 7 3
1121
	
457 0 56 608 372 0 69 42 54 11 52 4 1
75
	
38 0 10 27 219 97 87 82 36 21 14 5 1
329
	
142 0 113 74 182 0 72 42 22 44 32 43
329
	
184 0 69 76 179 0 64 41 23 27 42 5 6
1854
	
821 0 248 785 952 97 71 47 42 23 41 4 4
6851
	
3251 84 166 2724 3506 1342 68 46 40 20 43 48
4896
	
3251 84 166 1004 880 1342 43 18 20 15 69 67
198
	
186 - - 12 65 19 26 6 74 94
	
Wrong
	
Word-count
	
Error-rate
	
7064
	
92862
	
0.0761
	
P&R
	
2P&R
	
P&2R
	
45.47
	
46.99
	
44.04
UND OVG SUB REC PRE
38 5
EJV Summary Scores
USC SNAP System
1 . Full-Template Test
SLOT POS AC 1 COIL PAR INC SPU MIS NON ERR UND OVG SUB RbC PRE
<template> 282 275 275 0 0 0 7 0 2 2 0 0 98 100
doc-nr 282 275 275 0 0 0 7 0 2 2 0 0 98 100
doc-date 275 0 0 0 0 0 275 7 100 100 * * 0 *
doc-source 276 0 0 0 0 0 276 6 100 100 * * 0 *
content 348 332 178 0 36 118 134 24 62 38 36 17 51 54
subtotals 348 332 178 0 36 118 134 24 62 38 36 17 51 54
<tie-up-relati 348 331 178 0 36 117 134 0 62 38 35 17 51 5 4
status 348 331 152 0 62 117 134 0 67 38 35 29 44 46
entity 791 688 249 0 166 273 376 0 76 48 40 40 31 36
joint-venture 179 24 3 0 7 14 169 70 98 94 58 70 2 1 2
ownership 102 0 0 0 0 0 102 121 100 100 * * 0
activity 386 379 110 0 98 171 178 0 80 46 45 47 28 29
subtotals 1806 1422 514 0 333 575 959 191 78 53 40 39 28 3 6
<entity> 976 675 343 0 160 172 473 0 70 48 25 32 35 5 1
name 873 665 136 58 265 206 414 0 85 47 31 64 19 25
aliases 359 32 20 2 4 6 333 205 94 93 19 19 6 66
location 323 30 2 1 17 10 303 237 99 94 33 88 1 8
nationality 265 51 13 1 7 30 244 238 95 92 59 36 5 26
type 976 665 330 0 166 169 480 0 71 49 25 33 34 50
entity-rel 1152 630 290 0 176 164 686 0 78 60 26 38 25 46
person 118 1 0 0 0 1 118 316 100 100 100 * 0 0
facility 81 32 2 0 2 28 77 306 98 95 88 50 2 6
subtotals 4147 2106 793 62 637 614 2655 1302 83 64 29 45 20 39
<entity-relati 437 295 146 0 66 83 225 0 72 51 28 31 33 49
entityl 885 617 253 0 154 210 478 0 77 54 34 38 28 4 1
entity2 267 14 4 0 3 7 260 59 98 97 50 43 1 28
rel-ent2-to-e 437 295 67 0 145 83 225 0 87 51 28 68 15 23
status 432 295 111 0 98 86 223 0 78 52 29 47 26 38
subtotals 2021 1221 435 0 400 386 1186 59 82 59 32 48 22 36
<activity> 386 379 143 0 87 149 156 0 73 40 39 38 37 3 8
industry 418 379 143 0 82 154 193 0 75 46 41 36 34 38
site 551 0 0 0 0 0 551 16 100 100 * * 0 *
start 103 0 0 0 0 0 103 103 100 100 * * 0 *
end 16 0 0 0 0 0 16 133 100 100 * * 0 *
revenue 23 0 0 0 0 0 23 136 100 100 * * 0 *
subtotals 1111 379 143 0 82 154 886 388 89 80 41 36 13 38
<industry> 411 379 149 0 86 144 176 0 73 43 38 36 36 39
type 381 379 147 0 70 162 164 0 73 43 43 32 38 39prod-sery 814 305
5 9 96 152 704 1 99 86 50 91 1 3subtotals
1195 684 152 9 166 314 868 1 90 73 46 52 13 23<facility> 164
32 5 0 20 7 139 0 97 85 22 80 3 1 6name
35 32 0 0 2 30 33 0 100 94 94 100 0 0location 142 0 0
'0 0 0 142 1 100 100 * * 0 *type 163 32 5
0 20 7 138 0 97 85 22 80 3 1 6susubtotals 340 64 5
0 22 37 313 1 99 92 58 81 1 8<person> 118 0
0 0 0 0 118 0 100 100 * * 0name 118 '0 0
0 0 0 118 0 100 100 * * 0 *entity 118 0 0
0 0 0 118 0 100 100 * * 0 *position 136 0 0 0 0
0 136 0 100 100 * * 0 *subtotals 372 0 0 0 0 0 372 0 100 100
* * 0<ownership> 102 0 0 0 0 0 102 0 100 100 * * 0 *
time 21 0 0 0 0 0 21 0 100 100 * * 0 *owned
102 0 0 0 0 0 102 0 100 100 * * 0 *
capital 27 0 0 0 0 0 27 0 100 100 * * 0 *
owner% 376 0 0 0 0 0 376 0 100 100 * * 0 *
subtotals 526 0 0 0 0 0 526 0 100 100 * * 0 *
<revenue> 23 0 0 0 0 0 23 0 100 100 * * 0 *type
23 0 0 0 0 0 23 0 100 100 * * 0 *total
15 0 0 0 0 0 15 0 100 100 * * 0 *
total-eta 11 0 0 0 0 0 11 0 100 100 * * 0 *rate
8 0 0 0 0 0 8 0 100 100 * * 0 *rate-eta
0 0 0 0 0 0 0 0 * * * * * *
subtotals 57 0 0 0 0 0 57 0 100 100 * * 0 *<time>
149 0 0 0 0 0 149 0 100 100 * * 0 *after
16 0 0 0 0 0 16 0 . 100 100 * * 0 *
before 27 0 0 0 0 0 27 0 100 100 * * 0 *during
91 0 0 0 0 0 91 0 100 100 * * 0 *on
19 0 0 0 0 0 19 0 100 100 * * 0 *subtotals 153 0
0 0 0 0 153 0 100 100 * * 0 *
386
. 1m sir e :
	
1 ' .
	
. ll
MATCHED-ONLY
	
4406 328 6
TEXT-FILTERING
	
251
	
18 0
RICHNESS-NORMALIZED-ERROR 12018 . 5
Wrong
1180 0
Word-count
12125
Error-rate
0.9912 1 .018 5
ERROR-RATE-PER-WORD 12018 . 5
P&R
92862
2P&R
0.1294
P&2R
F-MEASURES 24.67 3036 20.69
2 . Core-Template Test
SLOT MS ACT FOR PAR INC SPU MIS NON ERR UND OVG SUB REC
	
PRE
<template> 282 275
	
275 0 0 0 7 0 2 2 0 0 9th— 100
content 348 332
	
178 0 36 118 134 24 62 38 36 17 51
	
5 4
subtotals 348 332
	
178 0 36 118 134 24 62 38 36 17 51
	
5 4
<tie-up-relati 348 331
	
178 0 36 117 134 0 62 38 35 17 51
	
5 4
status 348 331
	
152 0 62 117 134 0 67 38 35 29 44
	
46
entity 791 688
	
249 0 166 273 376 0 76 48 40 40 31
	
3 6
joint-venture 179 24
	
3 0 7 14 169 70 98 94 58 70 2
	
1 2
subtotals 1318 1043
	
404 0 235 404 679 70 76 52 39 37 31
	
3 9<entity>
976 675
	
343 0 160 172 473 0 70 48 25 32 35
	
5 1name
873 665
	
136 58 265 206 414 0 85 47 31 64 19
	
25aliases 359 32
	
20 2 4 6 333 205 94 93 19 19 6
	
66
location 323 30
	
2 1 17 10 303 237 99 94 33 88 1
	
8
nationality 265 51
	
13 1 7 30 244 238 95 92 59 36 5
	
26type 976 665
	
330 0 166 169 480 0 71 49 25 33 34
	
5 0
entity-rel 1152 630
	
290 0 176 164 686 0 78 60 26 38 25
	
4 6
subtotals 3948 2073
	
791 62 635 585 2460 680 82 62 28 45 21
	
40<entity-relati
437 295
	
146 0 66 83 225 0 72 51 28 31 33
	
49entity! 885 617
	
253 0 154 210 478 0 77 54 34 38 28
	
4 1entity2
267 14
	
4 0 3 7 260 59 98 97 50 43 1
	
28rel-ent2-to-e
437 295
	
67 0 145 83 225 0 87 51 28 68 15
	
23status 432 295
	
111 0 98 86 223 0 78 52 29 47 26
	
38
subtotals 2021 1221
	
435 0 400 386 1186 59 82 59 32 48 22
	
36
ALL-OBJECTS 7635 4669
	
1808 62 144 1493 4459 833 80 58 32 42 24
	
3 9
MATCHED-ONLY 3179 2603
	
1808 62 144 213 789 833 46 25 8 23 58
	
7 1
TEXT-FILTERING 251 180
	
173 - - 7 78 24 - 31 4 - 69
	
96
Wrong Word-count
	
Error-rate
ERROR-RATE-PER-WORD 7289 92862 0.078 5
P&R 2P&R P&2RF-MEASURES
29.89 34 .95 26.1 2
1•.
	
or.
	
: i
2220
	
71
	
144
	
292 1455 1966
	
52
	
33
	
9
	
24
	
51
	
69
	173	 -	 -	 7
	
78
	
24	 -	 31
	
4
	
-
	
69	 96
	
Wrong
	
Req-fills
	
All-fills
	
Mm-err
	
Max-er r
38 7
EJV Summary Scores
USussex SUSSEX System
1. Full-Template Test
SLOT NUS AC1' L:OR PAR INC SPU MIS NON ERR UND OVG SUB REC PRE
<template> 282 279 - 279 0 0 0 3 0 1 1 0 0 99 100doc-nr
282 279 279 0 0 0 3 0 1 1 0 0 99 100doc-date 275 272
272 0 0 0 3 7 1 1 0 0 99 100doc-source 276 273 273 0
0 0 3 6 1 1 0 0 99 100content 347 273 223 0
18 32 106 4 41 30 12 7 64 82subtotals 347 273 223 0 18 32 106 4
41 30 12 7 64 82<tie-up-relati 347 273 223 0 18 32 106 0
41 30 12 7 64 82status 347 270 183 0
56 31 108 0 52 31 11 23 53 68entity 788 780 332 0 176 272 280 0
69 36 35 35 42 42joint-venture 179 21 14 0 6
1 159 96 92 89 5 30 8 67ownership 102 0 0 0 0 0 102 159
100 100 * * 0 *activity 386 270 176 0 51 43 159 0 59
41 16 22 46 65subtotals 1802 1341 705 0 289 347 808 255 67 45 26 29 39 52
<entity> 976 814 448 0 205 161 323 0 60 33 20 31 46 55
name 873 464 162 52 168 82 491 24 80 56 18 51 22 40
aliases 360 0 0 0 0 0 360 258 100 100 * * 0 *
location 323 44 4 1 12 27 306 314 99 95 61 74 1 10
nationality 265 68 29 0 6 33 230 318 90 87 48 17 11 43
type 976 814 376 0 277 161 323 0 67 33 20 42 38 46
entity-rel 1154 795 405 0 235 155 514 0 69 44 19 37 35 5 1
person 118 28 0 0 7 21 111 394 100 94 75 100 0 0facility 81 38 4 0
1 33 76 392 96 94 87 20 5 10subtotals 4150 2251 980 53
706 512 2411 1700 78 58 23 42 24 45<entity-relati 436 267 169
0 70 28 197 0 64 45 10 29 39 63entityl 883 774 334 0
167 273 382 0 71 43 35 33 38 43entity2 266 21 15 0
5 1 246 69 94 92 5 25 6 71rel-ent2-to-e
436 267 87 0 152 28 197 0 81 45 10 64 20 32status
431 267 128 0 109 30 194 0 72 45 11 46 30 48subtotals 2016
1329 564 0 433 332 1019 69 76 50 25 43 28 42
<activity> 386 268 190 0 41 37 155 0 55 40 14 18 49 71
industry 419 268 170 0 54 44 195 0 63 46 16 24 40 63
site 551 201 87 0 45 67 419 49 86 76 33 34 16 43
start 103 0 0 0 0 0 103 134 100 100 * * 0 *
end 16 0 0 0 0 0 16 181 100 100 * * 0 *
revenue 23 0 0 0 0 0 23 181 100 100 * * 0 *
subtotals 1112 469 257 0 99 111 756 545 79 68 24 28 23 55
<industry> 412 268 173 0 52 43 187 0 62 45 16 23 42 64
type 382 275 152 0 53 70 177 0 66 46 25 26 40 55
prod-sery 816 482 49 18 227 179 522 7 94 64 37 80 7 12
subtotals 1198 757 201 18 280 249 699 7 85 58 33 58 18 28<facility>
164 139 66 0 15 58 83 0 70 51 42 18 40 47
name 35 0 0 0 0 0 35 59 100 100 * * 0 *location 142 34 4 3
17 10 118 7 96 83 29 77 4 16type 163 139 66 0
15 58 82 0 70 50 42 18 40 47subtotals 340 173
70 3 32 68 235 66 82 69 39 32 21 41<person> 118 0 0 0
0 0 118 0 100 100 * * 0 *name 118 0 0 0
0 0 118 0 100 100 * * 0 *entity 118 0 0
0 0 0 118 0 100 100 * * 0 *position 136 0 0 0 0 0 136 0 100 100 * * 0
subtotals 372 0 0 0 0 0 372 0 100 100 * 0
<ownership> 102 0 0 0 0 0 102 0 100 100 * * 0
time 21 0 0 0 0 0 21 0 100 100 * * 0 *
owned 102 0 0 0 0 0 102 0 100 100 * * 0
capital 27 0 0 0 0 0 27 0 100 100 * * 0 *
owner% 376 0 0 0 0 0 376 0 100 100 * 0 *subtotals
526 0 0 0 0 0 526 0 100 100 * 0 *
<revenue> 23 0 0 0 0 0 23 0 100 100 * • 0 *type 23 0 0 0 0 0 23 0 100 100 * * 0 *
total 15 0 0 0 0 0 15 0 100 100 * • 0 *
total-eta 11 0 0 0 0 0 11 0 100 100 * * 0
rate 8 0 0 0 0 0 8 0 100 100 * * 0 •
rate-eta 0 0 0 0 0 0 0 0 • • * * * *
subtotals 57 0 0 0 0 0 57 0 100 100 * * 0 *
<time> 149 0 0 0 0 0 149 0 100 100 * * 0 *
after 16 0 0 0 0 0 16 0 100 100 * * 0 *
before 27 0 0 0 0 0 27 0 100 100 * * 0 *
during 91 0 0 0 0 0 91 0 100 100 * • 0 *
on 19 0 0 0 0 0 19 0 100 100 * * 0 •
subtotals 153 0 0 0 0 0 153 0 100 100 • * 0 *
388
14
	
24
	
55
	
65
	10
	
-
	
95
	
90
Max-crr
0.9062
Wron g
10687
Wrong
10687
P&R
32.54
MATCHED-ONLY
	
5560 4646 3000
TEXT-FILTERING	 251	 266	 239
RICHNESS-NORMALIZED-ERRO R
ERROR-RATE-PER-WOR D
F-MEASURES
	
74
	
124
	
650
	
1575 2646
	
51
	
28
	
-
	
27
	
12
	
4
	
-
	
5
	
Rey-tills
	
All-fills
	
11793
	
1211 7
	
Word-count
	
Error-rate
	
92862
	
0.115 1
	
2P&R
	
P&2R
	
39 .5
	
27.67
Min-err
0.882
2. Core-Template Test
SLOT PUS ACT
	
COR 1,AR INC SPU MIS NON ERR UND OVG SUB REC PRE
<template> 282 279
	
279 0 0 0 3 0 1 1 0 0 99 100
content 347 273
	
223 0 18 32 106 4 41 30 12 7 64 8 2subtotals
347 273
	
223 0 18 32 106 4 41 30 12 7 64 8 2
<tie-up-relati 347 273
	
223 0 18 32 106 0 41 30 12 7 64 8 2
status 347 270
	
183 0 56 31 108 0 52 31 11 23 53 68entity 788 780
	
332 0 176 272 280 0 69 36 35 35 42 42
joint-venture 179 21
	
14 0 6 1 159 96 92 89 5 30 8 67subtotals
1314 1071
	
529 0 238 304 547 96 67 42 28 31 40 4 9<entity> 976 814
	
448 0 205 161 323 0 60 33 20 31 46 5 5
name 873 464
	
162 52 168 82 491 24 80 56 18 51 22 4 0
aliases 360 0
	
0 0 0 0 360 258 100 100 • * 0 *
location 323 44
	
4 1 12 27 306 314 99 95 61 74 1 1 0
nationality 265 68
	
29 0 6 33 230 318 90 87 48 17 11 43type 976 814
	
376 0 277 161 323 0 67 33 20 42 38 46
entity-rel 1154 795
	
405 0 235 155 514 0 69 44 19 37 35 5 1
subtotals 3951 2185
	
976 53 698 458 2224 914 77 56 21 42 25 4 6
<entity-relati 436 267
	
169 0 70 28 197 0 64 45 10 29 39 63
entityl 883 774
	
334 0 167 273 382 0 71 43 35 33 38 43
entity2 266 21
	
15 0 5 1 246 69 94 92 5 25 6 7 1
rel-dnt2-to-e 436 267
	
87 0 152 28 197 0 81 45 10 64 20 3 2
status 431 267
	
128 0 109 30 194 0 72 45 11 46 30 48
subtotals 2016 1329
	
564 0 433 332 1019 69 76 50 25 43 28 42
ALL-OBJECTS 7628 4858
	
2292 53 121 1126 3896 1083 74 51 23 38 30 48
MATCHED-ONLY 3879 3374
	
2292 53 121 441 946 1083 46 24 13 21 60 69
TEXT-FILTERING 251 266
	
239 - - 27 12 4 - 5 10 - 95 90
Wrong Word-count
	
Error-rate
ERROR-RATE-PER-WORD 6435 .5 92862 0.0693
P&R 2P&R P&2R
F-MEASURES 37 .14 42 .84 32.77
389
JJV Summary Score s
BBN PLUM System
1. Full-Template Test
SLOT
	
POS ACT COR PAR INC .SPU MIS NON ERR UND OVG SUB- REC PR E<template>
150 150 150 0 0 o 0 6 0 0 0 0 100 100 'doc-nr 150 150 150 0 0 0 0 0 0 0 0 0 100 100
doc-date 147 147 147 0 0 0 0 3 0 0 0 0 100 100doc-source 150 150 150
0 0 0 0 0 0 0 0 0 100 100content 205 246 158 0 8 80 39 1 44 19 32 5 77 64
subtotals 205 246 158 0 8 80 39 1 44 19 32 5 77 64<tie-up-relati 205 246 158 0 8 80
39 0 44 19 32 5 77 64status 205 246 151 0 15 80 39 0 47 19 32 9 74 61
entity 440 370 237 0 21 112 182 0 57 41 30 8 54 64joint-venture 23 21 5 0
2 14 16 138 86 70 67 28 22 24ownership 31 0 0 0 0 0 31 134 100 100 • • 0 *
activity 235 92 34 0 24 34 177 14 87 75 37 41 14 37subtotals 934 729 427 0 62
240 445 286 64 48 33 13 46 58<entity> 423 316 265 0 20 31 138 0 42 33 10 7 63 84
name 405 295 199 23 45 28 138 3 51 34 9 21 52 71aliases 174 118 68 3
9 38 94 127 67 54 32 13 40 59location 118 34 0 0 26 8 92 207 100 78 24 100 0 0
nationality 125 22 14 0 1 7 110 195 89 88 32 7 11 64type 423 316 262 0 23
31 138 0 42 33 10 8 62 83entity-rel 495 388 254 0 31 103 210 0 58 42 26 11 51 65
rson 75 9 8 0 0 1 67 227 89 89 11 0 11 89facility 108 52 13 0 11 28 84
206 90 78 54 46 12 25subtotals 1923 1234 818 26 146 244 933 965 62 48 20 16 43 67
<entity-relati 219 246 161 0 8 77 50 0 46 23 31 5 74 65entityl 454 370 242 0 18 110 194 0 57 43 30 7 53 65
entity2 37 21 6 0 3 12 28 138 88 76 57 33 16 28
rel-ent2-to-e 219 246 144 0 25 77 50 0 51 23 31 15 66 58status 218 246 152 0 16 78 50 0 49 23 32 10 70 62
subtotals 928 883 544 0 62 277 322 138 55 35 31 10 59 62
<activity> 235 65 38 0 19 8 178 0 84 76 12 33 16 58industry 275 64 38 0 19 7 218 0 86 79 11 33 14
59site 616 1 0 0 0 1 616 0 100 100 100 * 0 0
start 84 0 0 0 0 0 84 25 100 100 * * 0 *end 6 0 0 0 0 0 6 37 100 100 • • 0
revenue 18 0 0 0 0 0 18 33 100 100 * * 0
subtotals 999 65 38 0 19 8 942 95 96 94 12 33 4 58<industry> 255 64 42 0 15 7
198 0 84 78 11 26 16 66type 253 64 42 0 15 7 196" 0 84 77 11 26 17 66
prod-sery 566 0 0 0 0 0 566 0 100 100 * • 0 •subtotals 819 64 42 0 15 7 762 0 95
93 11 26 5 66<facility> 99 46 19 0 6 21 74 0 84 75 46 24 19 41
name 11 17 0 0 3 14 8 12 100 73 82 100 0 0location 51 10 2 1 4 3 44 5 95 86 30 64 5 25
type 98 46 18 0 7 21 73 0 85 74 46 28 18 39
subtotals 160 73 20 1 14 38 125 17 90 78 52 41 13 28<person> 75 9 8 0 1 0 66 0 89 88 0 11 11 89
name 75 9 8 0 1 0 66 0 89 88 0 11 11 89
entity 73 9 8 0 1 0 64 0 89 88 0 11 11 89position 75 9 8 0 1 0 66 0 89 88 0 11 11 89
subtotals 223 27 24 0 3 0 196 0 89 88 0 11 11 89
<ownership> 31 0 0 0 0 0 31 0 100 100 * * 0 •time 12 0 0 0 0 0 12 0 100 100 * * 0 •
owned 30 0 0 0 0 0 30 0 100 100 * * 0 •
capital 9 0 0 0 0 0 9 0 100 100 * * 0 •owner% 76 0 0 0 0 0
76 0 100 100 * * 0 *subtotals 127 0 0 0 0 0 127 0 100 100 * * 0 *
<revenue> 16 0 0 0 0 0 16 0 100 100 * * 0 *type 16 0 0 0 0 0 16 0 100 100 * * 0 *
total 10 0 0 0 0 0 10 0 100 100 * * 0total-eta
10 0 0 0 0 0 10 0 100 100 * 0 *rate 6 0 0 0 0 0 6 0 100 100 * 0 *
rate-eta 15 0 0 0 0 0 15 0 100 100 * * 0 *subtotals
57 0 0 0 0 0 57 0 100 100 * * 0 *<time> 107 0 0 0 0 0 107 0 100 100 * * 0 *
after 101 0 0 0 0 0 101 0 100 100 * * 0before
13 0 0 0 0 0 13 0 100 100 * * 0 *during 1 0 0 0 0 0 1 0 100 100 * * 0 *
on 0 0 0 0 0 0 0 0 * n * * *subtotals 115 0
0 0 0 0 115 0 100 100 * * 0 *
390
2071 27 46 272 853 1502 39 27 11 9 66 8 1
129 - - 4 16 1 11 3 - 89 97
Wrong Req-fills All-fills Min-err Max-err
5299.5 6633 0.799 0.81966466
Wrong
5299.5
Word-count Error-rat e
1771 2.9924
P&R 2P&R P&2R42 .49 52.71 35 .5 9
COR PAR INC SPU MIS NON
	
ERR UND OVG SUB REC PRE
150
	
150 0 0 0 0 0 0 0 0 0 100 100
246
	
158 0 8 80 39 1 44 19 32 5 77 64
246
	
158 0 8 80 39 1 44 19 32 5 77 6 4
246
	
158 0 8 80 39 0 44 19 32 5 77 64
246
	
151 0 15 80 39 0 47 19 32 9 74 6 1
370
	
237 0 21 112 182 0 57 41 30 8 54 6 4
21
	
5 0 1 15 17 137 87 74 71 17 22 24
637
	
393 0 37 207 238 137 55 36 32 9 59 62
316
	
265 0 20 31 138 0 42 33 10 7 63 8 4
295
	
199 23 45 28 138 3 51 34 9 21 52 7 1
118
	
68 3 9 38 94 127 67 54 32 13 40 5 9
34
	
0 0 26 8 92 207 100 78 24 100 0 0
22
	
14 0 1 7 110 195 89 88 32 7 11 64
316
	
262 0 23 31 138 0 42 33 10 8 62 83
388
	
254 0 31 103 210 0 58 42 26 11 51 65
1173
	
797 26 135 215 782 532 58 45 18 15 46 69
246
	
161 0 8 77 50 0 46 23 31 5 74 65
370
	
242 0 18 110 194 0 57 43 30 7 53 65
21
	
6 0 3 12 28 138 88 76 57 33 16 2 8
246
	
144 0 25 77 50 0 51 23 31 15 66 5 8
246
	
152 0 16 78 50 0 49 23 32 10 70 62
883
	
544 0 62 277 322 138 55 35 31 10 59 62
2939
	
1892 26 44 779 1381 808 56 39 26 12 54 65
2299
	
1892 26 44 228 386 808 29 16 10 8 78 83
133
	
129 - 4 16 1 11 3 - 89 97
	
Wrong
	
Word-count
	
Error-rate
	
2415
	
1771
	
1 .363 6
	
P&R
	
2P&R
	
P&2R
	
58 .8
	
62 .27
	
55.69
MATCHED-ONLY
	
3146 2565
TEXT-FILTERING
	
145
	
133
RICHNESS-NORMALIZED-ERRO R
ERROR-RATE-PER-WORD
F-MEASURES
2. Core-Template Test
SLAT
	
K
	
ACT
<template>
	
150
content
	
205
subtotals
	
205
<tie-up-relati
	
205
status
	
205
entity
	
44 0
joint-venture
	
23
subtotals
	
668
<entity>
	
423
name
	
405
aliases
	
174
location
	
11 8
nationality
	
125
type
	
423
entity-rel
	
495
subtotals
	
1740
<entity-relati
	
219
entityl
	
454
entity2
	
3 7
rel-ent2-to-e
	
21 9
status
	
21 8
subtotals
	
928
ALL-OBJECTS
	
354 1
MATCHED-ONLY
	
2457
TEXT-FILTERING
	
145
ERROR-RATE-PER-WORD
F-MEASURES
39 1
JJV Summary Scores
GE/CMU Shogun System
1. Full-Template Test
SLOT POS ACC--COR PAR INC SPU bIIS NON 1 :RR UND OVG SUB REC PRE
<template> 150 150 150 0 0 0 0 0 ' 0 0 0 0 100 100
doc-nr 150 150 150 0 0 0 0 0 0 0 0 0 100 100
doc-date 147 147 147 0 0 0 0 3 0 0 0 0 100 100
doe-source 150 150 119 31 0 0 0 0 10 0 0 10 90 90
content 204 197 170 0 3 24 31 2 25 15 12 2 83 86
subtotals 204 197 170 0 3 24 31 2 25 15 12 2 83 86
<tie-up-relati 204 197 170 0 3 24 31 0 25 15 12 2 83 86
status 204 197 164 0 9 24 31 0 28 15 12 5 80 83entity
438 458 325 0 24 109 89 0 40 20 24 7 74 7 1joint-venture 22 22
9 0 2 11 11 146 73 50 50 18 41 4 1ownership 32 16
10 0 1 5 21 141 73 66 31 9 31 62activity 236 206
124 0 11 71 101 13 60 43 34 8 52 60subtotals
932 899 632 0 47 220 253 300 45 27 24 7 68 70<entity> 423 433
357 0 10 66 56 0 27 13 15 3 84 82name 405 417 296
22 37 62 50 2 34 12 15 14 76 74aliases 177 226
126 9 6 85 36 171 50 20 38 7 74 5 8location 118 101
65 7 3 26 43 240 52 36 26 9 58 68nationality 126 61 50
0 1 10 75 243 63 60 16 2 40 82type 423
433 351 0 16 66 56 0 28 13 15 4 83 8 1entity-rel
494 502 351 0 32 119 111 0 43 22 24 8 71 70
person 75 71 50 0 0 21 25 285 48 33 30 0 67 70
facility 108 93 23 0 10 60 75 258 86 69 64 30 21 25
subtotals 1926 1904 1312 38 105 449 471 1199 44 24 24 8 69 70
<entity-relati 219 208 174 0 5 29 40 0 30 18 14 3 79 84
entityl 454 469 331 0 24 114 99 0 42 22 24 7 73 70
entity2 37 33 14 0 1 18 22 147 74 59 54 7 38 42
rel-ent2-to-e 219 208 162 0 17 29 40 0 35 18 14 9 74 78status
218 208 167 0 11 30 40 0 33 18 14 6 77 80
subtotals 928 918 674 0 53 191 201 147 40 22 21 7 73 73
<activity> 234 206 146 0 3 57 85 0 50 36 28 2 62 7 1
industry 274 208 131 0 18 59 125 0 61 46 28 12 48 63
site 615 536 217 0 29 290 369 154 76 60 54 12 35 40
start 84 49 12 0 8 29 64 80 89 76 59 40 14 24
end 6 0 0 0 0 0 6 144 100 100 * * 0 *
revenue 18 0 0 0 0 0 18 140 100 100 * * 0
subtotals 997 793 360 0 55 378 582 518 74 58 48 13 36 45
<industry> 255 208 160 0 2 46 93 0 47 36 22 1 63 77
type 253 208 135 0 26 47 92 0 55 36 22 16 53 65
prod-sery 568 417 140 36 120 121 272 22 77 48 29 47 28 38
subtotals 821 625 275 36 146 168 364 22 70 44 27 36 36 47
<facility> 99 76 28 0 8 40 63 0 80 64 53 22 28 37
name 11 16 3 3 1 9 4 19 78 36 56 36 41 28
location 51 35 6 2 10 17 33 6 90 65 48 61 14 20
type 98 76 28 0 8 40 62 0 80 63 53 22 28 37
subtotals 160 127 37 5 19 66 99 25 82 62 52 35 25 3 1
<person> 75 70 55 0 0 15 20 0 39 27 21 0 73 78
name 75 70 47 7 1 15 20 0 44 27 21 8 67 72
entity 73 70 50 0 5 15 18 0 43 25 21 9 68 7 1
position 75 70 46 0 9 15 20 0 49 27 21 16 61 66
subtotals 223 210 143 7 15 45 58 0 45 26 21 11 66 70
<ownership> 32 16 10 0 2 4 20 0 72 62 25 17 31 62
time 12 0 0 0 0 0 12 5 100 100 * * 0 *
owned 31 4 3 0 0 1 28 0 91 90 25 0 10 75
capital 10 6 0 0 4 2 6 4 100 60 33 100 0 0
owner% 80 32 18 1 3 10 58 1 79 72 31 16 23 5 8
subtotals 133 42 21 1 7 13 104 10 85 78 31 26 . 16 5 1
<revenue> 16 0 0 0 0 0 16 0 100 100 * * 0 *
t 16 0 0 0 0 0 16 0 100 100 * * 0 *
totalot 10 0 0 0 0 0 10 0 100 100 * * 0 *total-eta
10 0 0 0 0 0 10 0 100 100 * 0rate 6 0
0 0 0 0 6 0 100 100 * * 0 *rate-eta 15 0 0
0 0 0 15 0 100 100 * * 0 *subtotals 57 0 0 0
0 0 57 0 100 100 * * 0 *<time> 107 49 15 0 15 19 77
0 88 72 39 50 14 3 1after 101 47 15 0 11 21 75 0 88 74 45 42 15 3 2
before 13 1 0 0 0 I 13 0 100 100 100 * 0 0
during I 1 0 0 0 1 1 0 100 100 100 * 0 0
on 0 0 0 0 0 0 0 0 * * * * * *
'subtotals 115 49 15 0 11 23 89 0 89 77 47 42 13 31
392
MATCHED-ONLY
	
4783 4676 3639
	
87
	
48
	
584
	
691 2223
	
31
	
14
	
12
	
10
	
77
	
79TEXT-FILTERING
	
145
	
146
	
143
	
-
	
-
	
3
	
2
	
2
	
-
	
1
	
2
	
-
	
99
	
98
	
Wrong
	
Req-fills
	
All-fills
	
Min-err
	
Max-err
	
4390 .5
	
6462
	
6628
	
0.6624
	
0.6794
Wron
	
Word-count
	
Error-rat e
	
43905
	
1771
	
2.479 1
	
P&R
	
2P&R
	
P&2R
	
60.07
	
62 .31
	
5 8
2. Core-Tem plate Test
SLOT POS ACT
	
COR PAR INC SPU MIS NON ERR UND OVG SUB REC PRE
<template> 150 150
	
150 0 0 0 0 0 0 0 0 0 100 100
content 204 197
	
170 0 3 24 31 2 25 15 12 2 83 86
subtotals 204 197
	
170 0 3 24 31 2 25 15 12 2 83 86
<tie-up-relati 204 197
	
170 0 3 24 31 0 25 15 12 2 83 86
status 204 197
	
164 0 9 24 31 0 28 15 12 5 80 83entity 438 458
	
325 0 24 109 89 0 40 20 24 7 74 7 1
joint-venture 22 22
	
9 0 2 11 11 146 73 50 50 18 41 4 1
subtotals 664 677
	
498 0 35 144 131 146 38 20 21 6 75 74
<entity> 423 433
	
357 0 10 66 56 0 27 . 13 15 3 84 82
name 405 417
	
296 22 37 62 50 2 34 12 15 14 76 74
aliases 177 226
	
126 9 6 85 36 171 50 20 38 7 74 5 8
location 118 101
	
65 7 3 26 43 240 52 36 26 9 58 68
nationality 126 61
	
50 0 1 10 75 243 63 60 16 2 40 82
type 423 433
	
351 0 16 66 56 0 28 13 15 4 83 8 1
entity-rel 494 502
	
351 0 32 119 111 0 43 22 24 8 71 70
subtotals 1743 1740
	
1239 38 95 368 371 656 40 21 21 8 72 7 2
<entity-relati 219 208
	
174 0 5 29 40 0 30 18 14 3 79 84
entityl 454 469
	
331 0 24 114 99 0 42 22 24 7 73 70
entity2 37 33
	
14 0 1 18 22 147 74 59 54 7 38 42
rel-ent2-to-e 219 208
	
162 0 17 29 40 0 35 18 14 9 74 78
status 218 208
	
167 0 11 30 40 0 33 18 14 6 77 80
subtotals 928 918
	
674 0 53 191 201 147 40 22 21 7 73 73
ALL-OBJECTS 3539 3532
	
2581 38 42 727 734 951 39 21 20 7 73 74
MATCHED-ONLY 2990 3018
	
2581 38 42 268 240 951 20 8 9 5 87 86
TEXT-FILTERING 145 146
	
143 - - 3 2 2 1 2 - 99 98
Wrong Word-count
	
Error-rat e
ERROR-RATE-PER-WORD 1666
P&R
177 1
2P&R
0.9407
P&2R
F-MEASURES 73.54 73 .58 73.5
RICHNESS-NORMALIZED-ERROR
ERROR-RATE-PER-WORD
F-MEASURES
393
JJV Summary Scores
GE/CMU TEXTRACT System (Optional Test Run )
1 . Full-Template Test
SLOT PCS ACT COR PAR INC SPOT MIS NON IAfR UND OVO SUB REC PRE
<template> 150 150 150 0 0 0 0 0 0 0 0 0 100 100
doc-nr 150 150 150 0 0 0 0 0 0 0 0 0 100 100
doc-date 147 148 147 0 0 1 0 2 1 0 1 0 100 99
doc-source 150 150 150 0 0 0 0 0 0 0 0 0 100 100
content 205 188 173 0 1 14 31 3 21 15 7 0 84 92
subtotals 205 188 173 0 1 14 31 3 21 15 7 0 84 92
<tie-up-relati 205 188 173 0 1 14 31 0 21 15 7 - 0 84 92
status 205 188 165 0 9 14 31 0 25 15 7 5 80 88
entity 440 439 345 0 10 84 85 0 34 19 19 3 78 78
joint-venture 23 17 6 0 1 10 16 148 82 70 59 14 26 35
ownership 32 28 21 0 0 7 11 139 46 34 25 0 66 75.
activity 235 191 150 0 6 35 79 0 44 34 18 4 64 78
subtotals 935 863 687 0 26 150 222 287 37 24 17 4 73 80
<entity> 423 410 352 0 8 50 63 0 26 15 12 2 83 86name
405 398 317 18 18 45 52 2 28 13 11 8 80 82aliases
176 155 123 9 2 21 42 191 35 24 14 5 72 82location 118
89 69 6 3 11 40 254 44 34 12 8 61 81nationality 126
84 66 1 3 14 56 232 52 44 17 5 53 79type 423 410
349 0 11 50 63 0 26 15 12 3 82 85entity-rel 495 456
354 0 23 79 118 0 38 24 17 6 72 78person 75 73 54 0
0 19 21 283 42 28 26 0 72 74facility 106
99 22 0 15 62 69 259 87 65 63 40 21 22subtotals 1924
1764 1354 34 75 301 461 1221 38 24 17 6 71 78<entity-relati 219
188 171 0 2 15 46 0 27 21 8 1 78 91entityl 454
439 345 0 8 86 101 0 36 22 20 2 76 78entity2 37
17 6 0 0 11 31 148 88 84 65 0 16 35rel-ent2-to-e 219 188
153 0 20 15 46 0 35 21 8 12 70 81status 218
188 162 0 10 16 46 0 31 21 8 6 74 86subtotals
928 832 666 0 38 128 224 148 37 24 15 5 72 80
<activity> 235 191 162 0 5 24 68 0 37 29 12 3 69 85industry
275 206 124 0 21 61 130 2 63 47 30 14 45 60site
616 532 264 4 49 215 299 160 68 48 40 16 43 50start
85 43 25 0 7 11 53 91 74 62 26 22 29 58end
6 0 0 0 0 0 6 157 100 100 • • 0 •revenue 18 10
5 0 4 1 9 151 74 50 10 44 28 50subtotals 1000
791 418 4 81 288 497 561 67 50 36 16 42 53<industry>
255 206 155 0 7 44 93 0 48 36 21 4 61 75type
253 206 127 0 34 45 92 0 57 36 22 21 50 62prod-sery
565 528 126 37 151 214 251 8 81 44 40 54 26 27
subtotals 818 734 253 37 185 259 343 8 75 42 35 43 33 37
<facility> 99 99 39 0 8 52 52 0 74 52 52 17 39 39name
13 11 6 0 0 5 7 29 67 54 45 0 46 54location
51 56 9 4 10 33 28 9 87 55 59 52 22 20type
98 99 38 0 9 52 51 0 75 52 52 19 39 38subtotals
162 166 53 4 19 90 86 38 78 53 54 28 34 33
<person> 75 73 60 0 0 13 15 0 32 20 18 0 80 82
name 75 73 53 5 2 13 15 0 37 20 18 8 74 76
entity 73 73 53 0 5 15 15 0 40 20 20 9 73 73
position 75 73 55 0 5 13 15 0 38 20 18 8 73 75
subtotals 223 219 161 5 12 41 45 0 38 20 19 8 73 75
<ownership> 32 28 22 0 0 6 10 0 42 31 21 0 69 78
time 12 4 0 0 2 2 10 14 100 83 50 100 0 0
owned 31 28 12 0 9 7 10 0 68 32 25 43 39 43
capital 10 6 4 0 0 2 6 16 67 60 33 0 40 67
owner% 80 56 41 0 7 8 32 0 53 40 14 14 51 73
subtotals 133 94 57 0 18 19 58 30 62 44 20 24 . 43 61
<revenue> 16 10 5 0 5 0 6 0 69 38 0 50 31 50type
16 10 5 0 5 0 6 0 69 38 0 50 31 50total
10 5 5 0 0 0 5 0 50 50 0 0 50 100total-eta
10 3 0 0 3 0 7 0 100 70 0 100 0 0rate
6 5 0 0 2 3 4 5 100 67 60 100 0 0rate-eta
15 3 0 0 3 0 12 0 100 80 0 100 0 0subtotals
57 26 10 0 13 3 34 5 83 60 12 56 18 38<time>
108 49 29 0 13 7 66 0 75 61 14 31 27 59after
102 46 29 0 10 7 63 0 73 62 15 26 28 63
before 13 3 0 0 0 3 13 0 100 100 100 • 0 0during
1 0 0 0 0 0 1 0 100 100 • • 0 •
on 0 0 0 0 0 0 0 0 • • • • • fsubtotals
116 49 29 0 10 10 77 0 77 66 20 26 25 59
394
.1arre :
	
511
MATCHED-ONLY
	
4939 4939 3861
	
84
	
28
	
606
	
606 2301
	
30
	
12
	
12
	
10
	
79
	
7 9TEXT-FILTERING
	
145
	
145
	
143
	
-
	
-
	
2
	
2
	
3
	
-
	
1
	
1
	
-
	
99
	
99
	
Wrong
	
Req-tills
	
All-fills
	
Min-err
	
Max-err
	
3901
	
6471
	
6638
	
0 .5877
	
0.6028
	
Wrong
	
Word-count
	
Error-rat e
	
3901
	
1771
	
2 .2027
	
P&R
	
2P&R
	
P&2R
	
63 .84
	
66.37
	
61 .5
2. Core-Template Test
SLOT PUS ACT
	
COR PAR INC
	
SPU MIS NON ERR UND OVG SUB RIC PRE
<template> 150 150
	
150 0 0 0 0 0 0 0 0 0 100 100
content 205 188
	
173 0 1 14 31 3 21 15 7 0 84 92
subtotals 205 188
	
173 0 1 14 31 3 21 15 7 0 84 92
<tie-up-relati 205 188
	
173 0 1 14 31 0 21 15 7 0 84 92
status 205 188
	
165 0 9 14 31 0 25 15 7 5 80 88
entity 440 439
	
345 0 10 84 85 0 34 19 19 3 78 78
joint-venture 23 17
	
6 0 1 10 16 148 82 70 59 14 26 35
subtotals 668 644
	
516 0 20 108 132 148 34 20 17 4 77 80
<entity> 423 410
	
352 0 8 50 63 0 26 15 12 2 83 86
name 405 398
	
317 18 18 45 52 2 28 13 11 8 80 82
aliases 176 155
	
123 9 2 21 42 191 35 24 14 5 72 82
location 118 89
	
69 6 3 11 40 254 44 34 12 8 61 8 1
nationality 126 84
	
66 1 3 14 56 232 52 44 17 5 53 79
type 423 410
	
349 0 11 50 63 0 26 15 12 3 82 85
entity-rel 495 456
	
354 0 23 79 118 0 38 24 17 6 72 78
subtotals 1743 1592
	
1278 34 60 220 371 679 34 21 14 6 74 8 1
<entity-relati 219 188
	
171 0 2 15 46 0 27 21 8 1 78 9 1
entity! 454 439
	
345 0 8 86 101 0 36 22 20 2 76 7 8
entit}2 37 17
	
6 0 0 11 31 148 88 84 65 0 16 35
rel-ent2-to-e 219 188
	
153 0 20 15 46 0 35 21 8 12 70 8 1status
218 188
	
162 0 10 16 46 0 31 21 8 6 74 86
subtotals 928 832
	
666 0 38 128 224 148 37 24 15 5 72 80
ALL-OBJECtS 3544 3256
	
2633 34 19 470 758 978 34 21 14 5 75 8 1
MATCHED-ONLY 2954 2943
	
2633 34 19 190 201 978 16 7 6 4 90 90
TEXT-FILTERING 145 145
	
143 - - 2 2 3 - 1 1 - 99 99
Wrong Word-count
	
terror-rate
ERROR-RATE-PER-WORD 1364
P&R
1771
	
0.7702
2P&R
	
P&2R
F-MEASURES 77.94 79.97
	
76 .01
RICHNESS-NORMALIZED-ERROR
ERROR-RATE-PER-WORD
F-MEASURES
395
JJV Summary Score s
NMSUBrandeis Diderot System
1. Full-Template Test
SLOT POS
	
ACI— COR PAR INC SPU -MIS NON ERR UND OVG SUB REC PRE<template>
150 150 150 0 0 0 0 0 0 0 0 0 100 100doc-nr 150 150 150 0 0 0 0 0 0 0 0 0 100
100doc-date 147 147 147 0 0 0 0 3 0 0 0 0 100 100
doe-source 150 150 150 0 0 0 0 0 0 0 0 0 100 100content 203 219 159 0 5 55
39 3 38 19 25 3 78 73subtotals 203 219 159 0 5 55 39 3 38 19 25 3 78 73
<tie-up-relati 203 218 159 0 5 54 39 0 38 19 25 3 78 73status 203 218 153 0
11 54 39 0 40 19 25 7 75 70entity 436 382 281 0 26 75 129 0 45 30 20 8 64 74
joint-venture 22 15 2 0 3 10 17 143 94 77 67 60 9 13ownership 31 0 0 0 0 0 31 136 100 100 • * 0 •
activity 235 134 88 0 13 33 134 10 67 57 25 13 37 66subtotals 927 749 524 0 53 172 350 289 52 38 23
9 56 70<entity> 423 345 297 0 5 43 121 0 36 29 12 2 70 86
name 405 341 239 23 35 44 108 0 44 27 13 16 62 73aliases 174 73 57 6 2 8
109 161 67 63 11 8 34 82location 118 98 50 3 7 38 58 185 67 49 39 14 44 52
nationality 125 44 35 0 2 7 88 206 73 70 16 5 28 80
tYPe 423 345 292 0 10 43 121 0 37 29 12 3 69 85entity-rel 494 405 290 0 23 92 181 0 50 37 23 7 59 72
rson 75 43 37 0 0 6 38 243 54 51 14 0 49 86flinty 106 4 0 0 2 2 104 248 100
98 50 100 0 0subtotals 1920 1353 1000 32 81 240 807 1043 53 42 18 9 53 75
<entity-relati 219 206 154 0 6 46 59 0 42 27 22 4 70 75entityl 454 389 283 0 23
83 148 0 47 32 21 8 62 73entity2 37 6 2 0 1 3 34 142 95 92 50 33 5 33
rel-ent2-to-e 219 206 144 0 16 46 59 0 46 27 22 10 66 70status 218 206
145 0 14 47 59 0 45 27 23 9 66 70subtotals 928 807 574 0 54 179 300 142 48 32 22 8 62 71
<activity> 235 134 116 0 2 16 117 0 54 50 12 2 49 86industry 274
198 87 0 38 73 149 0 75 54 37 30 32 44site 617 356 164 4 46 142 403 81 78 65 40 22 27 4
7dart 84 7 4 0 0 3 80 69 95 95 43 0 5 57
end 6 0 0 0 0 0 6 113 100 100 * * 0 *revenue 18 0 0 0 0 0 18 107 100
100 * * 0 nsubtotals 999 561 255 4 84 218 656 370 79 66 39 25 26 46
<industry> 255 198 110 0 40 48 105 0 64 41 24 27 43 56type 253
198 102 0 47 49 104 0 66 41 25 32 40 52prod-sery 569 58 11 1 35 11 522 0 98 92 19 76 2 20
subtotals 822 256 113 1 82 60 626 0 87 76 23 42 14 44<facility> 99 8 4 0
2 2 93 0 96 94 25 33 4 50name 9 8 0 0 0 8 9 0 100 100 100 * 0 0
location 51 2 0 0 0 2 51 2 100 100 100 * 0 0
type 98 8 4 0 2 2 92 0 96 94 25 33 4 50158 18 4 0 2 12 152 2 98 96 67 33 2 22
<person> 75 43 40 0 0 3 35 0 49 47 7 0 53 93name 75 38 33 0 2
3 40 0 58 53 8 6 44 87entity 73 43 37 0 3 3 33 0 51 45 7 8 51 86
position 75 42 34 0 5 3 36 0 56 48 7 13 45 81subtotals 223 123 104 0 10
9 109 0 55 49 7 9 47 84<ownership> 31 0 0 0 0 0 31 0 100 100 * • 0 *
time 12 0 0 0 0 0 12 0 100 100 * * 0 *owned
30 0 0 0 0 0 30 0 100 100 * * 0capital 9 0 0 0 0 0 9 0 100 100 * * 0 •
owner% 76 0 0 0 0 0 76 0 100 100 • • 0 •subtotals 127 0 0 0
0 0 127 0 100 100 * * 0 •<revenue> 16 0 0 0 0 0 16 0 100 100 * *' 0
type 16 0 0 0 0 0 16 0 100 100 * • 0total 10 0 0 0 0 0
10 0 100 100 * * 0total-eta 10 0 0 0 0 0 10 0 100 100 * * 0 *
rate 6 0 0 0 0 0 6 0 100 100 * * 0 *rate-eta 15 0
0 0 0 0 15 0 100 100 * * 0 *subtotals 57 0 0 0 0 0 57 0 100 100 * * 0 *
<time> 107 7 4 0 2 1 101 0 96 94 14 33 4 57after 101 5 4 0
1 0 96 0 96 95 0 20 4 80before 13 2 0 0 0 2 13 0 100 100 100 * 0 0
during 1 0 0 0 0 0 1 0 100 100 * * 0 *on 0 0 0 0 0
0 0 0 * * * * * *subtotals 115 7 4 0 1 2 110 0 96 96 28 20 3 57
39 6
MATCHED-ONLY
	
3968 336 4
TEXT-FILTERING
	
145
	
130
RICHNESS-NORMALIZED-ERROR
ERROR-RATE-PER-WORD
F-MEASURES
2737
	
37
	
43
	
315
	
919 1849
	
36
	
23
	
9
	
10
	
69
	
8 2128
	
-
	
-
	
2
	
17
	
3
	
-
	
12
	
2
	
-
	
88	 9 8
	
Wrong
	
Req-tills
	
All-fills
	
Min-err
	
Max-err
	
4670.5
	
6454
	
6619
	
0.7056
	
0 .7237
	
Wrong
	
Word-count
	
Error-rat e
	
4670.5
	
1771
	
2 .6372
	
MR
	
2P&R
	
P&2R
	
52 .13
	
60.29
	
45 .9 1
2.	 Core-Template Test
SIA ~S ACT
	
COR PAR INC SPU MIS NON ERR UND OVG SUB REC PRE
<template> 150 150
	
150 0 0 0 0 0 0 0 0 0 100 100content 203 219
	
159 0 5 55 39 3 38 19 25 3 78 73subtotals 203 219
	
159 0 5 55 39 3 38 19 25 3 78 73
<tie-up-relati 203 218
	
159 0 5 54 39 0 38 19 25 3 78 73
status 203 218
	
153 0 11 54 39 0 40 19 25 7 75 70
entity 436 382
	
281 0 26 75 129 0 45 30 20 8 64 74
joint-venture 22 15
	
2 0 3 10 17 143 94 77 67 60 9 1 3
subtotals 661 615
	
436 0 40 139 185 143 46 28 23 8 66 7 1
<entity> 423 345
	
297 0 5 43 121 0 36 29 12 2 70 86
name 405 341
	
239 23 35 44 108 0 44 27 13 16 62 73
aliases 175 73
	
58 6 2 7 109 161 66 62 10 8 35 84
location 118 98
	
50 3 7 38 58 185 67 49 39 14 44 52
nationality 125 44
	
35 0 2 7 88 206 73 70 16 5 28 80
type 423 345
	
292 0 10 43 121 0 37 29 12 3 69 85
entity-rel 494 405
	
290 0 23 92 181 0 50 37 23 7 59 72
subtotals 1740 1306
	
964 32 79 231 665 552 50 38 18 9 56 75
<entity-relati 219 206
	
154 0 6 46 59 0 42 27 22 4 70 75
entityl 454 389
	
283 0 23 83 148 0 47 32 21 8 62 73
entity/2 37 6
	
2 0 1 3 34 142 95 92 50 33 5 33
rel-ent2-to-e 219 206
	
144 0 16 46 59 0 46 27 22 10 66 70
status 218 206
	
145 0 14 47 59 0 45 27 23 9 66 70
subtotals 928 807
	
574 0 54 179 300 142 48 32 22 8 62 7 1
ALL-0B3E0S 3532 2947
	
2133 32 38 604 1189 840 48 34 20 8 61 73
MATCHED-ONLY 2599 2447
	
2133 32 38 150 302 840 22 12 6 6 83 8 8
TEXT-FILTERING 145 130
	
128 - - 2 17 3 12 2 - 88 9 8
Wrong Word-count
	
Error-rate
ERROR-RATE-PER-WORD 1987
P&R
177 1
2P&R
1 .122P&2R
F-MEASURES 66.34 70 .14 62.93
397
JJV Summary Scores
SRA SOLOMON Syste m
1. Full-Template Test
SLO'i` MIS AC I COR PAR INC SPU MIS NON ERR UND OVG SUB REC PRE
<template> 150 149 149 0 0 0 1 0 1 1 0 0 99 10 0
doc-nr 150 149 149 0 0 0 1 0 1 1 0 0 99 100doc-date 147 146
146 0 0 0 1 3 1 1 0 0 99 100doc-source 150 149 149
0 0 0 1 0 1 1 0 0 99 100content 204 365 174 0 12 179
18 3 54 9 49 6 85 48subtotals 204 365 174 0 12 179 18 3 54 9 49 6 85 48
<tie-up-relati 204 367 176 0 11 180 17 0 54 8 49 6 86 48
status 204 367 169 0 18 180 17 0 56 8 49 10 83 46
entity 438 398 210 0 39 149 189 0 64 43 37 16 48 53
joint-venture 23 39 8 0 5 26 10 153 84 43 67 38 35 20
ownership 31 0 0 0 0 0 31 149 100 100 0 *
activity 234 268 95 0 26 147 113 9 75 48 55 21 40 35
subtotals 930 1072 482 0 88 502 360 311 66 39 47 15 52 45
<entity> 423 421 292 0 18 111 113 0 45 27 26 6 69 69
name 405 329 204 24 43 58 134 6 53 33 18 20 53 66aliases 174
35 27 0 2 6 145 159 85 83 17 7 16 77location
118 75 38 1 4 32 75 205 74 64 43 10 33 5 1
nationality 125 92 48 0 2 42 75 191 71 60 46 4 38 52
type 423 420 286 0 24 110 113 0 46 27 26 8 68 6 8entity-rel
494 435 238 0 71 126 185 0 62 37 29 23 48 55
person 75 40 25 0 0 15 50 241 72 67 38 0 33 62facility 106 1 1
0 0 0 105 251 99 99 0 0 1 100subtotals 1920 1427 867
25 146 389 882 1053 62 46 27 15 46 62<entity-relati
219 294 168 0 16 110 35 0 49 16 37 9 77 57entityl 454 394 219 0 35 140
200 0 63 44 36 14 48 56entity2 37 40 8
0 7 25 22 143 87 59 62 47 22 20
rel-ent2-to-e 219 294 153 0 31 110 35 0 53 16 37 17 70 52status
218 294 161 0 22 111 35 0 51 16 38 12 74 55subtotals
928 1022 541 0 95 386 292 143 59 31 38 15 58 5 3
<activity> 235 155 117 0 16 22 102 0 54 43 14 12 50 7 5industry 274
155 82 0 51 22 141 0 72 51 14 38 30 5 3site
616 258 112 2 78 66 424 103 83 69 26 41 18 44
start 84 0 0 0 0 0 84 78 100 100 * * 0 *end 6 0
0 0 0 0 6 112 100 100 * * 0revenue
18 0 0 0 0 0 18 107 100 100 0 *
subtotals 998 413 194 2 129 88 673 400 82 67 21 40 20 47<industry>
254 155 97 0 40 18 117 0 64 46 12 29 38 62type
252 155 87 0 49 19 116 0 68 46 12 36 34 5 6prod-sery 567 76
14 7 48 7 498 0 97 88 9 75 3 23subtotals 819 231 101 7 97 26
614 0 88 75 11 49 13 45<facility> 99 5 3 0
1 1 95 0 97 96 20 25 3 60name 9 5 0 0 0
5 9 0 100 100 100 * 0 0location 51 1 0 0 1 0
50 1 100 98 0 100 0 0type 98 5 3 0 1 1
94 0 97 96 20 25 3 60subtotals 158 11
3 0 2 6 153 1 98 97 54 40 2 27<person> 75 37
28 0 3 6 44 0 65 59 16 10 37 76name 75 30
15 4 8 3 48 0 78 64 10 37 23 5 7entity 73 38 24 0
2 12 47 0 72 64 32 8 33 63position 75 37 23 0 8 6
44 0 72 59 16 26 31 62subtotals 223 105 62 4 18
21 139 0 74 62 20 24 29 6 1<ownership> 31 0 0 0 0
0 31 0 100 100 * 0 *time 12 0 0
0 0 0 12 0 100 100 * * 0owned 30 0 0
0 0 0 30 0 100 100 * * 0 *capital
9 0 0 0 0 0 9 0 100 100 * * 0 *owner% 76
0 0 0 0 0 76 0 100 100 * * 0 *subtotals 127 0
0 0 0 0 127 0 100 100 0<revenue> 16 0
0 0 0 0 16 0 100 100 * * 0 *type 16 0 0 0
0 0 16 0 100 100 * 0 *total l0 0 0 0 0 0
10 0 100 100 * * 0 *total-eta 10 0 0 0 0 0
10 0 100 100 * 0rate 6 0 0 0 0 0 6
0 100 100 * 0 *rate-eta 15 0 0 0 0
0 15 0 100 100 * 0subtotals
57 0 0 0 0 0 57 0 100 100 * 0 *<time> 107 0 0
0 0 0 107 0 100 100 * 0 *after 101 0 0
0 0 0 101 0 100 100 * 0 *before 13 0 0
0 0 0 13 0 100 100 * * 0during 1 0 0
0 0 0 1 0 100 100 * 0 *
01 0 0 0 0 0 0 0 0 * * * *
subtotals 115 0 0 0 0 0 115 0 100 100 * * 0 *
398
MATCHED-ONLY
	
3982 3139 2424
	
38
	
43
	
294 1137 1911
	
43
	
28
	
9
	
14
	
61
	
7 8
TEXT-FILTERING	 145	 139	 137	 -	 -	 2	 8
	
3
	
-
	
6
	
I	 -	 94
	
98
	
Wrong
	
Req-fills
	
All-tills
	
Mm-err
	
Max-err
RICHNESS-NORMALIZED-ERROR
	
5633
	
6456
	
6622
	
0.8506
	
0.8725
	
Wrong
	
Word-count
	
Error-rate
ERROR-RATE-PER-WORD
	
5633
	
1771
	
3 .1807
	
P&R
	
2P&R
	
P&2R
F-MEASURES
	
43.92
	
48 .74
	
39.97
2. Core-Template Test
SLO ACT
	
COR PAR INC SPU MIS NON ERR UND OVG SUB REC PR E
<template> 150 149
	
149 0 0 0 1 0 1 1 0 0 99 100
content 204 365
	
174 0 12 179 18 3 54 9 49 6 85 48
subtotals 204 365
	
174 0 12 179 18 3 54 9 49 6 85 4 8
<tie-up-relati 204 367
	
176 0 11 180 17 0 54 8 49 6 86 4 8
status 204 367
	
168 0 19 180 17 0 56 8 49 10 82 46
entity 438 398
	
211 0 39 148 188 0 64 43 37 16 48 5 3
joint-venture 23 39
	
8 0 5 26 10 153 84 43 67 38 35 20
subtotals 665 804
	
387 0 63 354 215 153 62 32 44 14 58 4 8
<enti ty> 423 421
	
292 0 18 111 113 0 45 27 26 6 69 69
name 405 329
	
204 24 43 58 134 6 53 33 18 20 53 6 6
aliases 175 35
	
27 0 1 7 147 159 85 84 20 4 15 77
location 118 75
	
38 1 4 32 75 206 74 64 43 10 33 5 1
nationality 125 92
	
48 0 3 41 74 191 71 59 44 6 38 5 2
type 423 420
	
286 0 24 110 113 0 46 27 26 8 68 68
entity-rd 494 435
	
238 0 71 126 185 0 62 37 29 23 48 5 5
subtotals 1740 1386
	
841 25 146 374 728 562 60 42 27 16 49 62
<enti ty-relati 219 294
	
169 0 15 110 35 0 49 16 37 8 77 57
entityl 454 394
	
221 0 33 140 200 0 63 44 36 13 49 56
entity2 37 40
	
8 0 7 25 22 144 87 59 62 47 22 20
rel-ent2-to-e 219 294
	
154 0 30 110 35 0 53 16 37 16 70 52
status 218 294
	
161 0 22 111 35 0 51 16 38 12 74 55
subtotals 928 1022
	
544 0 92 386 292 144 58 31 38 14 59 53
ALL-OBJECTS -3537 3577
	
1946 25 37 1293 1253 862 59 35 36 14 55 55
MATCHED-ONLY 2717 2406
	
1946 25 37 234 545 862 34 20 10 10 72 8 1
TEXT-FILTERING 145 139
	
137 - - 2 8 3 6 1 - 94 98
Wrong Word-count
	
Error-rate
ERROR-RATE-PER-WORD 2871 5P&R 177 12P&R 1 .621 4P&2R
F-MEASURES 55 .06 54 .88 55.25
,r
399
JJV Summary Scores
SRI FASTUS System
1 . Full-Template Test
SLOT POS ACT COR PAR INC SPU MIS NON ERR UND OVG SUB REC PRE
<template> 150 150 150 0 0 0 0 0 0 0 0 0 100 100
doc-nr 150 150 150 0 0 0 0 0 0 0 0 0 100 100
doc-date 149 150 147 0 2 1 0 0 2 0 1 1 99 98
doc-source 150 0 0 0 0 0 150 0 100 100 * * 0 *
content 202 152 136 0 3 13 63 4 37 31 8 2 67 89
subtotals 202 152 136 0 3 13 63 4 37 31 8 2 67 89
<tie-up-relati 202 152 136 0 3 13 63 0 37 31 8 2 67 89
status 202 124 111 0 6 7 85 0 47 42 6 5 55 90
entity 434 426 250 0 28 148 156 0 57 36 35 10 58 59
joint-venture 22 0 0 0 0 0 22 129 100 100 * * 0 *ownership
31 1 1 0 0 0 30 115 97 97 0 0 3 100activity
232 88 71 0 7 10 154 13 71 66 11 9 31 8 1subtotals
921 639 433 0 41 165 447 257 60 48 26 9 47 68<entity>
423 373 270 0 12 91 141 0 47 33 24 4 64 7 2name
405 359 156 32 78 93 139 0 65 34 26 35 42 48aliases
175 142 44 10 9 79 112 133 81 64 56 22 28 3 4location
118 106 40 14 8 44 56 172 71 47 42 24 40 4 4nationality
126 91 33 0 3 55 90 152 82 71 60 8 26 3 6type
423 373 266 0 16 91 141 0 48 33 24 6 63 7 1entity-rel
494 427 260 0 18 149 216 0 60 44 35 6 53 61person
75 16 16 0 0 0 59 218 79 79 0 0 21 10 0facility
106 0 0 0 0 0 106 230 100 100 * * 0 *subtotals
1922 1514 815 56 132 511 919 905 65 48 34 16 44 56<entity-relati
219 152 134 0 4 14 81 0 42 37 9 3 61 8 8entityl
454 426 255 0 21 150 178 0 58 39 35 8 56 60entity2
37 1 0 0 0 1 37 126 100 100 100 * 0 0
rel-ent2-to-e 219 152 128 0 10 14 81 0 45 37 9 7 58 84status
218 152 128 0 9 15 81 0 45 37 10 6 59 84subtotals
928 731 511 0 40 180 377 126 54 41 25 7 55 70<activity>
232 88 78 0 3 7 151 0 67 65 8 4 34 89industry
271 122 73 0 14 35 184 0 76 68 29 16 27 60site
612 88 54 0 27 7 531 54 91 87 8 33 9 6 1start
84 0 0 0 0 0 84 57 100 100 * * 0 *end
6 0 0 0 0 0 6 76 100 100 * * 0 *revenue
18 0 0 0 0 0 18 75 100 100 * * 0 *subtotals
991 210 127 0 41 42 823 262 88 83 20 24 13 60<industry>
252 118 91 0 10 17 151 0 66 60 14 10 36 77type
250 118 70 0 30 18 150 0 74 60 15 30 28 59prod-sery
555 161 36 29 66 30 424 10 91 76 19 61 9 3 1subtotals 805
279 106 29 96 48 574 10 86 71 17 48 15 43<facility>
99 0 0 0 0 0 99 0 100 100 * * 0 *name
9 0 0 0 0 0 9 0 100 100 * * 0 *location
51 0 0 0 0 0 51 0 100 100 * * 0 *type
98 0 0 0 0 0 98 0 100 100 * * 0 *subtotals
158 0 0 0 0 0 158 0 100 100 * * 0 *<person> 75
16 16 0 0 0 59 0 79 79 0 0 21 100name 75
16 15 1 0 0 59 0 79 79 0 3 21 97entity
73 15 15 0 0 0 58 0 79 79 0 0 20 10 0position 75
16 14 0 2 0 59 0 81 79 0 12 19 8 8subtotals 223 47
44 1 2 0 176 0 80 79 0 5 20 9 5<ownership>
31 1 1 0 0 0 30 0 97 97 0 0 3 100time
12 0 0 0 0 0 12 0 100 100 * * 0 *owned 30
1 0 0 0 1 30 0 100 100 100 * 0 0capital
9 0 0 0 0 0 9 0 100 100 * * 0 *owner%
76 2 1 0 1 0 74 0 99 97 0 50 1 50subtotals 127
3 1 0 1 1 125 0 99 98 33 50 1 3 3<revenue>
16 0 0 0 0 0 16 0 100 100 * * 0 *type 16
0 0 0 0 0 16 0 100 100 * * 0 *total 10
0 0 0 0 0 10 0 100 100 * * 0 *total-eta 10
0 0 0 0 0 10 0 100 100 * * 0 *rate 6
0 0 0 0 0 6 0 100 100 * * 0 *rate-eta 15 0
0 0 0 0 15 0 100 100 * * 0 *subtotals 57
0 0 0 0 0 57 0 100 100 * * 0 *<time> 107 0 0
0 0 0 107 0 100 100 * * 0 *after 101 0 0 0
0 0 101 0 100 100 * * 0 *before 13 0 0 0 0
0 13 0 100 100 * * 0 *during 1 0 0 0 0
0 1 0 100 100 * * 0 *on 0 0 0 0 0
0 0 0 * * * * * *subtotals 115 0 0 0 0 0
115 0 100 100 * * 0 *
400
MATCHED-ONLY 3253
	
2961
	
2173 86 89
	
421 713 1564 40 22 14 13 68 75
TEXT-FILTERING 145
	
118
	
117 - -
	
1 28 4 - 19 1 - 81 99
Wrong Req-fills All-tills 1Ntn-err Max-err
RICHNESS-NORMALIZED-ERROR 519 3
Wrong
643 6
Word-count
660 0
Error-rate
0 .7868 0.8069
ERROR-RATE-PER-WORD 5193 1771 2 .932 2
P&R 2P&R P&2RF-MEASURES
44.21 53.4 37 .7 2
2 .
	
Core-Template Test
SLOT RJS ACT`
	
COR PAR INC— SPU MIS- NON ERR UND OVG SUB
	
REC PRE
<temp ate> t t
	
t t t t t t t t t t
	
t9 1 ftcontent
202 152
	
136 0 3 13 63 4 37 31 8 2
	
67 89subtotals 202 152
	
136 0 3 13 63 4 37 31 8 2
	
67 89<tie-up-relati
202 152
	
136 0 3 13 63 0 37 31 8 2
	
67 8 9status 202 124
	
111 0 6 7 85 0 47 42 6 5
	
55 9 0entity 434 426
	
250 0 28 148 156 0 57 36 35 10
	
58 5 9joint-venture
22 0
	
0 0 0 0 22 129 100 100 • $
	
0subtotals 658 550
	
361 0 34 155 263 129 56 40 28 9
	
55 66<entity> 423 373
	
270 0 12 91 141 0 47 33 24 4
	
64 7 2name 405 359
	
155 5 106 93 139 0 68 34 26 41
	
39 4 4aliases 176 142
	
45 1 17 79 113 133 82 64 56 28
	
26 3 2location
118 106
	
35 12 16 43 55 172 74 47 40 35
	
35 3 9
nationality 126 91
	
0 0 37 54 89 152 100 71 59 100
	
0 0type 423 373
	
266 0 16 91 141 0 48 33 24 6
	
63 7 1entity-rel
494 427
	
260 0 18 149 216 0 60 44 35 6
	
53 6 1
subtotals 1742 1498
	
761 18 210 509 753 457 66 43 34 22
	
44 5 1
<entity-relati 219 152
	
134 0 4 14 81 0 42 37 9 3
	
61 8 8entityl
454 426
	
255 0 21 150 178 0 58 39 35 8
	
56 6 0
entity2 37 1
	
0 0 0 1 37 126 100 100 100 *
	
0 0
rel-ent2-to-e 219 152
	
128 0 10 14 81 0 45 37 9 7
	
58 8 4status
218 152
	
128 0 9 15 81 0 45 37 10 6
	
59 8 4
subtotals 928 731
	
511 0 40 180 377 126 54 41 25 7
	
55 70
ALL-OBJECTS 3530 2931
	
1769 18 166 857 1456 716 59 41 29 14
	
50 61MATCHED-ONLY 2304 2406
	
1769 18 166 381 279 716 34 12 16 12
	
77 7 4
TEXT-FILTERING 145 118
	
117 - - 1 28 4 - 19 1 -
	
81 9 9
Wrong Word-count
	
Error-rat e
ERROR-RATE-PER-WORD 2609 1771 1 .4732
P&R 2P&R P&2R
F-MEASURES 55 .04 58 .28 52 .1 4
40 1
EME Summary Score s
BBN PLUM System
SLOT POS ACT
	
COR PAR INC SPU MIS NON ERR UND OVG SUB REC PRE
<template> 300 300
	
300 0 0 0 0 0 0 0 0 0 100 100
content 609 590
	
436 0 7 147 166 39 42 27 25 2 72 74
subtotals 609 590
	
436 0 7 147 166 39 42 27 25 2 72 74
<entity> 554 494
	
326 0 65 103 163 0 50 29 21 17 59 66
name 499 494
	
285 26 63 120 125 0 52 25 24 20 60 60
location 231 59
	
26 11 6 16 188 216 87 81 27 27 14 53
nationality 76 55
	
20 2 7 26 47 270 79 62 47 28 28 38
type 554 494
	
325 0 66 103 163 0 50 29 21 17 59 66
subtotals 1360 1102
	
656 39 142 265 523 486 58 38 24 19 50 61
<microelectron . 610 590
	
437 0 7 146 166 0 42 27 25 2 72 74
process 610 590
	
373 0 71 146 166 0 51 27 25 16 61 63
developer 288 107
	
30 0 8 69 250 271 92 87 64 21 10 28
manufacturer 239 430
	
119 0 57 254 63 191 76 26 59 32 50 28
distributor 233 106
	
48 0 18 40 167 242 82 72 38 27 21 45
purchaser 425 236
	
87 0 43 106 295 158 84 69 45 33 20 37
subtotals 1795 1469
	
657 0 197 615 941 862 73 52 42 23 37 45
<layering> 145 127
	
94 0 3 30 48 0 46 33 24 3 65 74
type 145 127
	
89 1 7 30 48 0 49 33 24 8 62 70
film 85 53
	
30 0 0 23 55 55 72 65 43 0 35 57
temperature 4 2
	
0 0 1 1 3 91 100 75 50 100 0 0
device 62 26
	
8 0 3 15 51 61 90 82 58 27 13 3 1
equipment 98 74
	
33 0 22 19 43 16 72 44 26 40 34 44
subtotals 394 282
	
160 1 33 88 200 223 67 51 31 17 41 57
<lithography> 213 220
	
153 0 14 53 46 0 42 22 24 8 72 70
type 213 220
	
122 5 40 53 46 0 53 22 24 25 58 56
granularity 45 7
	
0 0 1 6 44 118 100 98 86 100 0 0
device 92 99
	
46 0 6 47 40 80 67 43 47 12 50 46
equipment 177 225
	
75 0 55 95 47 12 72 26 42 42 42 3 3
subtotals 527 551
	
243 5 102 201 177 210 66 34 36 30 46 4 4
<etching> 60 73
	
42 0 0 31 18 0 54 30 42 0 70 58
type 60 73
	
37 0 5 31 18 0 59 30 42 12 62 5 1
etchant 6 3
	
0 0 0 3 6 37 100 100 100 * 0 0
film 27 22
	
6 2 0 14 19 25 83 70 64 12 26 3 2
granularity 11 14
	
1 0 2 11 8 32 95 73 78 67 9 7
temperature 1 1
	
0 0 0 1 1 41 100 100 100 * 0 0
device 40 26
	
12 0 3 11 25 20 76 62 42 20 30 46
equipment 46 43
	
19 0 6 18 21 9 70 46 42 24 41 44
subtotals 191 182
	
75 2 16 89 98 164 73 51 49 18 40 42
<packaging> 193 170
	
120 0 8 42 65 0 49 34 25 6 62 70
type 193 170
	
100 5 23 42 65 , 0 56 34 25 20 53 60
pitch 14 15
	
0 0 3 12 11 103 100 78 80 100 0 0
material 92 100
	
44 0 3 53 45 51 70 49 53 6 48 44
p-l-count 53 52
	
32 0 2 18 19 82 55 36 35 6 60 62
units 58 33
	
20 2 2 9 34 84 69 59 27 12 36 6 4
bonding 33 45
	
16 0 4 25 13 94 72 39 56 20 48 36
device 173 124
	
46 0 24 54 103 11 80 60 44 34 26 37
equipment 0 4
	
0 0 0 4 0 116 100 * 100 * * 0
subtotals 616 543
	
258 7 61 217 290 541 69 47 40 20 42 48
<device> 194 187
	
105 0 19 63 70 0 59 36 34 15 54 5 6
function 194 187
	
101 3 20 63 70 0 60 36 34 17 53 55
size 112 98
	
48 0 5 45 59 53 69 53 46 9 43 49
device-speed 25 19
	
6 0 2 11 17 91 83 68 58 25 24 3 2
subtotals 331 304
	
155 3 27 119 146 144 65 44 39 15 47 5 1
<equipment> 350 199
	
160 0 16 23 174 0 57 50 12 9 46 80
name 132 56
	
18 5 14 19 95 79 86 72 34 44 16 37
manufacturer 268 141
	
58 0 50 33 160 17 81 60 23 46 22 4 1
modules 45 11
	
3 0 0 8 42 140 94 93 73 0 7 27
type 350 199
	
145 3 28 23 174 0 61 50 12 17 42 74
throughput 17 1
	
0 0 0 1 17 152 100 100 100 * 0 0
status 373 199
	
151 0 25 23 197 0 62 53 12 14 40 76
wafer-size 49 7
	
5 0 0 2 44 146 90 90 28 0 10 7 1
subtotals 1234 614
	
380 8 117 109 729 534 71 59 18 24 31 62
ALL-OBJECTS 7057 5637
	
3020 65 33 1850 3270 3203 66 46 33 19 43 54
MATCHED-ONLY 4686 4292
	
3020 65 33 739 1133 3203 44 24 17 14 65 7 1
TEXT-FILTERING 220 244
	
203 - - 41 17 _ 39 - 8 17 - 92 8 3
Wrong Req-fills MI-fills Min-err Max-er r
RICHNESS-NORMALIZED-ERROR 5854 . 5
Wrong
748 5
Word-count
7822
Error-rate
0.7485 0.7822
ERROR-RATE-PER-WORD 5854 .5 125680 0.0466
P&R 2P&R P&2R
F-MEASURES 48 .09 51 .55 45.07
40 2
EME Summary Scores
GE/CMU Shogun System
SLOT PUS ACT
	
COR FAR INC SPU MIS NON ERR UND OVG SUB RhC PRE<template> 300 X00
	
300 0 0 0 0 a" 0 0 0 0 100 100content 608 633
	
440 0 4 189 164 32 45 27 30 1 72 70subtotals 608 633
	
440 0 4 189 164 32 45 27 30 1 72 70<entity> 553 560
	
320 0 90 150 143 0 54 26 27 22 58 57name 497 553
	
284 22 84 163 107 0 55 22 29 24 59 53location 231 179
	
57 11 19 92 144 177 81 62 51 28 27 35nationality 78 11
	
4 2 0 5 72 285 94 92 45 17 6 45553 560
	
320 0 90 150 143 0 54 26 27 22 58 571359 1303
	
665 35 193 410 466 462 61 34 31 24 50 52<microelectron 609 633
	
441 0 3 189 165 0 45 27 30 1 72 70process 609 633
	
372 0 72 189 165 0 53 27 30 16 61 59developer 294 201
	
56 0 25 120 213 268 86 72 60 31 19 28manufacturer 240 476
	
120 0 55 301 65 148 78 27 63 31 50 25distributor 236 359
	
96 0 28 235 112 187 80 47 65 22 41 27purchaser 426 461
	
124 0 78 259 224 131 82 52 56 39 29 27subtotals 1805 2130
	
768 0 258 1104 779 734 74 43 52 25 42 36<layering> 145 166
	
109 0 4 53 32 0 45 22 32 4 75 66type 145 166
	
104 1 8 53 32 0 47 22 32 8 72 63film 84 93
	
25 5 8 55 46 49 80 55 59 28 33 30temperature 4 6
	
2 0 0 4 2 105 75 50 67 0 50 33device 63 56
	
27 0 3 26 33 69 70 52 46 10 43 48equipment 98 77
	
49 0 15 13 34 25 56 35 17 23 50 64subtotals 394 398
	
207 6 34 151 147 248 61 37 38 15 53 53<lithography> 212 203
	
147 0 7 49 58 0 44 27 24 4 69 72type 212 203
	
109 4 41 49 58 0 57 27 24 28 52 55granularity 45 80
	
25 2 2 51 16 97 73 36 64 10 58 32device 92 183
	
50 0 11 122 31 50 77 34 67 18 54 27equipment 176 172
	
101 0 24 47 51 11 55 29 27 19 57 59subtotals 525 638
	
285 6 78 269 156 158 64 30 42 22 55 45<etching> 60 58
	
39 0 0 19 21 0 51 35 33 0 65 67type 60 58
	
36 0 3 19 21 0 54 35 33 8 60 62etchant 6 2
	
1 0 0 1 5 36 86 83 50 0 17 50Am 27 28
	
8 2 0 18 17 24 80 63 64 10 33 32granularity 11 16
	
3 0 1 12 7 33 87 64 75 25 27 19temperature 1 2
	
0 0 1 1 0 37 100 0 50 100 0 0device 40 31
	
14 0 1 16 25 19 75 62 52 7 35 45eq pment
ctals 45 30
	
22 0 3 5 20 7 56 44 17 12 49 73sub 190 167
	
84 2 9 72 95 156 68 50 43 10 45 51<packaging> 193 206
	
120 0 10 76 63 0 55 33 37 8 62 58type 193 206
	
92 9 29 76 63 0 64 33 37 26 50 47pitch 14 7
	
2 0 2 3 10 113 88 71 43 50 14 28material 92 58
	
35 0 2 21 55 63 69 60 36 5 38 60p-l-count 53 62
	
37 0 0 25 16 82 52 30 40 0 70 60units 57 42
	
19 2 1 20 35 82 74 61 48 9 35 48bonding 33 29
	
17 0 1 11 15 99 61 45 38 6 52 59device 173 173
	
61 0 26 86 86 18 76 50 50 30 35 35equipment 0 0
	
0 0 0 0 0 120 * * * * * *subtotals 615 577
	
263 11 61 242 280 577 69 46 42 20 44 46<device> 193 240
	
127 0 23 90 43 0 55 22 38 15 66 53function 193 240
	
122 5 23 90 43 0 56 22 38 17 64 52size 112 96
	
62 0 3 31 47 66 57 42 32 5 55 64device-speed 25 162
	
19 0 1 142 5 85 89 20 88 5 76 12subtotals 330 498
	
203 5 27 263 95 151 65 29 53 12 62 41<equipment> 352 297
	
228 0 14 55 110 0 44 31 18 6 65 77name 133 153
	
48 15 14 76 56 94 73 42 50 28 42 36manufacturer 273 286
	
119 0 71 96 83 6 68 30 34 37 44 42modules 45 32
	
8 0 2 22 35 199 88 78 69 20 18 25type 352 297
	
198 19 25 55 110 0 49 31 18 14 59 70throughput 17 28
	
7 0 5 16 5 202 79 29 57 42 41 25status 375 297
	
211 0 31 55 133 0 51 35 18 13 56 71wafer-size 51 41
	
22 0 0 19 29 200 68 57 46 0 43 54subtotals 1246 1134
	
613 34 148 339 451 701 60 36 30 21 50 56
ALL-OBJECTS 7072 7478
	
3528 99 26 3039 2633 3219 65 37 41 19 50 48MATCHED-ONLY 4993 5303
	
3528 99 26 1136 826 3219 42 16 21 14 72 67TEXT-FILTERING 219 257
	
208 - - 49 11 32 - 5 19 - 95 8 1
Wrong Req-fills All-fills Min-err Max-errRICHNESS-NORMALIZED-ERROR 6533.5
Wrong 7489Word-count 7821Error-rate 0.8354 0.8724ERROR-RATE-PER-WORD 6533.5 125680
0.052P&R 2P&R P&2R
F-MEASURES 49.18 48.37 50.01
403
EME Summary Score s
LSI DBG System
r• °•: ,, . : . '1 •1 tr' •1• •' • : "1
<template >
content
subtotals<entity>
name
location
nationality
type
subtotals
<microelectron
proces sdeveloper
manufacture r
distributor
purchase r
subtotals
<layering >
typ e
film
temperature
devic e
equipment
subtotals
<lithography>
type
granularity
devic e
equipment
subtotals
<etching>
type
etchant
film
granularit y
temperature
device
equipment
subtotals
<packaging>
type
pitc h
material
p-l-coun t
units
bonding
device
equipmen t
subtotal s
<device >
function
size
device-speed
subtotal s
<equipment>
name
manufacturer
modules
type
throughputstatus
wafer-siz e
subtotals
30 0
608
60855
149 7
230
76
55 1
135 4
60 8
60828
923 3
232 '
41 8
1780
145
145
83
4
62
98
392
21 0
21 0
45
89
17 3
51 76 1
616
27
1 1
14
146
19 3
192192
14
89
5 1
56
3 4
17 1
0
60 7
19 1
19 1
11 4
25
33 0
35 0
13 2
266
44
35 0
1 737
3
49
1231
292
	
29 2
554
	
365
554
	
365559
	
325
559
	
27 4
60
	
20
14
	
8
559
	
325
1192
	
627
553
	
365
553
	
31 4
1
	
137
	
3 0
37
	
3 0
0
	
0
628
	
375
136
	
80
129
	
73
39
	
1 1
0
	
0
9
	
3
36
	
20
213
	
107
172
	
11 3
172
	
87
18
	
3
17
	
1 0
109
	
7 0
316
	
170
92
	
40
92
	
370
	
0
20
	
6
9
	
10
	
0
9
	
3
11
	
7
141
	
5 4
154
	
90154
	
8 4
0
	
0
22
	
1 9
0
	
0
15
	
1 1
0
	
035
	
1 4
0
	
0
226
	
12 8
220
	
11 9
220
	
11 7
51
	
3 6
6
	
2
277
	
15 5
191
	
145
65
	
34
55
	
43
0
	
0
191
	
133
0
	
0
191
	
134
3
	
2
505
	
346
0
00
0
3 8
4
1
0
43
0
0
00
0
0
0
0
7
0
0
0
0
7
0
4
0
0
0
4
0
00
0
10
0
01
0
2
0
0
0
0
0
0
0
2
0
2
0
0
2
0
9
0
0
7
0
0
0
16
0
9
96
349
7
2
63
121
8
59
03
3
0
65
5
5
7
0
0
8
20
1 6
3 8
4
0
1 4
56
5
8
03
0
0
0
01
11 6
20
0
0
0
2
0
3
0
25
1 3
1 3
3
2
1 8
17
6
6
0
22
0
2 8
1
63
0
180180
171
198
29
3
17 1
40 1
180
180
04
4
0
188
5 1
44
21
0
68
79
4 3
4 3
1 1
7
2 5
86
47
47
011
7
0
6
47
548
48
0
3
0
2
0
1 80
7 1
88
88
12
2
102
29
16
6
0
29
0
29
0
80
8
23 423 4
16313 6
19 9
65
163
563
23 5
23 5
288200
199
41 8
1340
60
60
65
4
5970
258
8 1
8 1
3 8
79
89
28 7
1 6
16
61 8
9
1
3 8
3 9
12 786
86
1 4
7 0
5 1
4 3
3 415 4
0
45 2
5 9
5 9
7 5
21
15 5
188
83
21 7
44
188
1 7
21 1
46
806
0 '
2 22 2
00
222
282
0
504
0
0
25 2208
21 7
160
837
0
0
43
76
62
1 8
199
0
0
8 1
78
1 5
174
0
0
3724
3 3
39
2 1
1 1
1650
0
8 5
5 5
6 1
70
8 1
2 2
9 0
464
0
0
5 7
109
166
0
72
27
129
0
137
0
13 2
497
3
54
5455
58
92
89
55
63
54
60
1008 7
8 7
100
8 1
59
60
89
100
968 1
76
55
65
95
90
65
7 1
63
66
10084
92
100
94
86
8062
6 4
100
79
100
8 1
10 0
92*
8 1
5 7
5 8
7 1
9 2
64
62
7 4
8 4
10 0
64
100
67
96
73
3
3 8
3 830
27
86
8 6
3 0
42
3 9
3 9
10086
86
100
75
4 1
4 1
78
100
957 1
66
38
38
84
89
5 1
56
26
26
10067
82
100
93
85
6645
45
10 0
79
10 077
100
9 0*
74
3 1
3 1
66
84
47
54
63
82
100
54
100
56
94
65
0
3 2
3 23 0
3 5
48
2 1
30
34
32
32
01 1
11
*
30
38
34
54
*
67
22
37
2 5
25
6 1
4 1
2 3
2 7
5 1
51
*5 5
78
*
6 7
3 6
5 33 1
31
*
14
*
13
*51
*
3 1
40
40
24
3 3
3 7
1 5
25
11
*
15
*
15
0
16
0
2
216
1 9
29
2 3
1 6
1 8
2
1 60
9
9
*
15
6
1 0
39
*
0
28
1 8
1 2
3 1
57
0
1 7
25
1 1
18
*33
25
*
0
0
1 71 5
20
*
0*
15
*18
*
1 7
1 0
11
8
50
1 1
10
2 1
12
*
16
*
1 7
3 3
17
9 7
606
05 9
5 9
1 0
1 1
59
48
60
520
1 3
13
0
2 1
55
53
13
0
520
2 8
5 4
4 2
7
1 1
4 0
3 3
66
610
2 2
14
0
7
1 5
2 84 7
44
0
21
0
200
8
*
2 1
62
62
32
8
47
4 1
29
16
0
39
0
3 6
4
29
100
6 666
5 8
5 2
3 7
6 1
58
54
66
57
1008 1
81
*
60
59
59
28*
33
56
52
66
52
17
59
64
54
4 3
40*
30
17
*
33
6 4
39
5 8
55*
86
*
73*
40*
5 7
54
54
70
3 3
5 6
76
59
78
*
71
*
70
67
70
ALL-OBJECTS
MATCHED-ONLY
TEXT-FILTERING
701 2
411 2
220
4052
	
2327
2837
	
2327
252
	
196
75
75
-
1 5
15
-
1262
260
56
4222
1535
24
3028
3028
24
7 1
46
-
60
37
11
3 1
9
22
15
8
-
34
5 8
89
58
83
78
Wrong Req-fills All-fills Min-err Max-errRICHNESS-NORMALIZED-ERROR
59093Wrong 7459
Word-count
7794
Error-rate
0.7582 0.7923
ERROR-RATE-PER-WORD 5909.5 125680 0.047
P&R 2P&R P&2R
F-MEASURES 42.74 50.92 36 .83
404
EME Summary Scores
NMSUBrandeis Diderot System
r• •; • . ~ . .t To.
	
'•' ~r t n • •' t : 't~ 1
<template> -300 299
	
299 0 0 0 1
	
it 0 0 0 0 100 10 0
content 606 425
	
308 0 11 106 287
	
46 57 47 25 3 51 7 2
subtotals 606 425
	
308 0 11 106 287
	
46 57 47 25 3 51 7 2
<entity> 553 403
	
243 0 85 75 225
	
0 61 41 19 26 44 60
name 498 403
	
206 26 76 95 190
	
0 63 38 24 29 44 5 4
location 231 118
	
32 3 16 67 180
	
119 89 78 57 34 14 28
nationality 77 0
	
0 0 0 0 77
	
216 100 100 * * 0
type 553 400
	
243 0 82 75 228
	
0 61 41 19 25 44 61
suubbtt 1359 921
	
481 29 174 237 675
	
335 69 50 26 28 36 5 4
<microelectron 607 425
	
309 0 11 105 287
	
0 57 47 25 3 51 73
process 607 425
	
245 0 75 105 287
	
0 66 47 25 23 40 5 8
developer 294 156
	
53 0 16 87 225
	
152 86 76 56 23 18 3 4
manufacturer 237 253
	
49 0 45 159 143
	
100 88 60 63 48 21 1 9
distributor 235 96
	
34 0 30 32 171
	
164 87 73 33 47 14 3 5
purchaser 421 293
	
64 0 76 153 281
	
76 89 67 52 54 15 22
subtotals 1794 1223
	
445 0 242 536 1107
	
492 81 62 44 35 25 36
<layering> 145 103
	
69 0 4 30 72
	
0 60 50 29 5 48 67
type 145 103
	
61 5 7 30 72
	
0 64 50 29 13 44 62
film 85 31
	
18 0 0 13 67
	
36 82 79 42 0 21 58
temperature 4 0
	
0 0 0 0 4
	
68 100 100 * * 0
device 63 25
	
12 0 4 9 47
	
46 83 75 36 25 19 48
eeqqu ipment 98 42
	
31 0 4 7 63
	
13 70 64 17 11 32 74
subtotals 395 201
	
122 5 15 59 253
	
163 72 64 29 12 32 62
<ithography> 211 172
	
117 0 15 40 79
	
0 53 37 23 11 55 68
type 211 138
	
74 0 34 30 103
	
0 69 49 22 31 35 54
granularity 45 3
	
0 0 2 1 43
	
91 100 96 33 100 0 0
device 91 58
	
25 0 8 25 58
	
66 78 64 43 24 27 43
equipment 175 125
	
72 0 15 38 88
	
15 66 50 30 17 41 58
subtotals 522 324
	
171 0 59 94 292
	
172 72 56 29 26 33 53
<etching> 60 43
	
27 0 2 14 31
	
0 64 52 32 7 45 63
type 60 26
	
15 0 1 10 44
	
0 78 73 38 6 25 58
etchant 6 0
	
0 0 0 0 6
	
24 100 100 * * 0
film 27 10
	
4 1 1 4 21
	
18 85 78 40 25 17 45
granularity 11 1
	
0 0 0 1 11
	
22 100 100 100 * 0 0
temperature 1 0
	
0 0 0 0 1
	
27 100 100 * 0 *
device 40 11
	
6 0 2 3 32
	
17 86 80 27 25 15 54
equipment 45 23
	
13 0 2 8 30
	
7 75 67 35 13 29 56
subtotals 190 71
	
38 1 6 26 145
	
115 82 76 37 14 20 54
<packaging> 191 88
	
57 0 8 23 126
	
0 73 66 26 12 30 65
type 191 51
	
26 5 11 9 149
	
0 86 78 18 32 15 56
pitch 14 0
	
0 0 0 0 14
	
53 100 100 * * 0 *
material 92 44
	
25 0 1 18 66
	
26 77 72 41 4 27 57
p-l-count 52 24
	
18 0 1 5 33
	
36 68 63 21 5 35 75
units 57 0
	
0 0 0 0 57
	
43 100 100 * * 0 *
bonding 34 0
	
0 0 0 0 34
	
48 100 100 * * 0 *
device 172 93
	
35 0 12 46 125
	
9 84 73 49 26 20 38
equipment 1 8
	
1 0 0 7 0
	
52 88 0 88 0 100 1 2
subtotals 613 220
	
105 5 25 85 478
	
267 84 78 39 20 18 49
<device> 194 184
	
97 0 16 71 81
	
0 63 42 38 14 50 5 3
function 194 184
	
91 6 16 71 81
	
0 64 42 38 17 48 5 1
size 114 58
	
29 0 3 26 82
	
49 79 72 45 9 25 50
device-speed 25 7
	
2 0 1 4 22
	
91 93 88 57 33 8 2 8
subtotals 333 249
	
122 6 20 101 185
	
140 71 56 40 16 38 50
<equipment> 349 198
	
153 0 9 36 187
	
0 60 54 18 6 44 77
name 132 49
	
23 3 4 19 102
	
89 84 77 39 18 18 5 0
manufacturer 267 153
	
52 0 40 61 175
	
15 84 66 40 43 19 3 4
modules 44 0
	
0 0 0 0 44
	
143 100 100 * * 0
type 349 198
	
128 19 15 36 187
	
0 64 54 18 15 39 69
throughput 17 0
	
0 0 0 0 17
	
144 100 100 * * 0
status 372 198
	
140 0 22 36 210
	
0 66 56 18 14 38 7 1
wafer-size 49 1
	
1 0 0 0 48
	
138 98 98 0 0 2 100
subtotals 1230 599
	
344 22 81 152 783
	
529 74 64 25 20 29 5 9
ALL-OBJECTS 7042 4233
	
2136 68 17 1396 4205
	
2259 74 60 33 24 31 5 1
MATCHED-ONLY 3630 3142
	
2136 68 17 537 1025
	
2259 48 28 17 17 60 69
TEXT-FILTERING 220 206
	
172 - - 34 48
	
46 - 22 16 - 78 83
Wrong Req-fills All-fills Min-err Max-err
RICHNESS-NORMALIZED-ERROR 626 8Wrong 7469 7802Error-rate 0 .8034 0.8392Word-count
ERROR-RATE-PER-WORD 6268 125680 0.0499P&R 2P&R P&2R
F-MEASURES 38 .49 45 .26 33.49
405
EME Summary Scores
UManitoba NUBA System
SLOT POS ACT
	
COR PAR INC
	
SPU MIS NON ERR UND OVG
	
SUSIIPRE<temp ate> Pt
Pt
	
t9 t t
	
t t tcontent
610 716
	
441 0 8
	
267 161 46 50 26 37
	
2 72 62subtotals 610 716
	
441 0 8
	
267 161 46 50 26 37
	
2 72 62<entity> 546 488
	
305 0 65
	
118 176 0 54 32 24
	
18 56 62name 492 488
	
268 31 61
	
128 132 0 54 27 26
	
21 58 58location 229 116
	
46 3 19
	
48 161 188 83 70 41
	
30 21 4 1nationality 76 0
	
0 0 0
	
0 76 267 100 100 *
	
* 0 *type 546 488
	
304 0 66
	
118 176 0 54 32 24
	
18 56 62subtotals 1343 1092
	
618 34 146
	
294 545 455 61 40 27
	
20 47 58<microelectron 611 716
	
442 0 8
	
266 161 0 50 26 37
	
2 72 62process 611 716
	
346 0 104
	
266 161 0 60 26 37
	
23 57 48developer 293 133
	
50 0 13
	
70 230 277 86 78 53
	
21 17 38manufacturer 237 245
	
71 0 30
	
144 136 211 81 57 59
	
30 30 29distributor 236 222
	
67 0 27
	
128 142 226 82 60 58
	
29 28 3 0purchaser 424 419
	
112 0 82
	
225 230 106 83 54 54
	
42 26 27subtotals
	
' 1801 1735
	
646 0 256
	
833 899 820 75 50 48
	
28 36 3 7<layering> 144 154
	
92 0 3
	
59 49 0 55 34 38
	
3 64 60type 144 154
	
79 8 8
	
59 49 0 59 34 38
	
13 58 5 4film 83 36
	
24 0 2
	
10 57 47 74 69 28
	
8 29 67temperature 4 0
	
0 0 0
	
0 4 91 100 100 *
	
* 0 *device 61 3
	
3 0 0
	
0 58 73 95 95 0
	
0 5 10 0equipment 97 47
	
27 0 12
	
8 58 24 74 60 17
	
31 28 5 7subtotals 389 240
	
133 8 22
	
77 226 235 71 58 32
	
16 35 5 7<lithography> 214 285
	
160 0 13
	
112 41 0 51 19 39
	
8 75 5 6type 214 285
	
124 2 47
	
112 41 0 62 19 39
	
28 58 4 4granularity 45 0
	
0 0 0
	
0 45 127 100 100 *
	
* 0 *device 90 42
	
20 0 13
	
9 57 107 80 63 21
	
39 22 48equipment 178 237
	
109 0 26
	
102 43 14 61 24 43
	
19 61 46subtotals 527 564
	
253 2 86
	
223 186 248 66 35 40
	
26 48 4 5<etching> 61 46
	
31 0 2
	
13 28 0 58 46 28
	
6 51 67type 61 46
	
22 0 11
	
13 28 0 70 46 28
	
33 36 4 8etchant 6 6
	
2 0 0
	
4 4 26 80 67 67
	
0 33 3 3film 27 11
	
3 0 3
	
5 21 19 91 78 45
	
50 11 27granularity 11 0
	
0 0 0
	
0 11 25 100 100 *
	
* 0 *temperature 1 0
	
0 0 0
	
0 1 31 100 100 *
	
* 0 *device 41 3
	
2 0 0
	
1 39 18 95 95 33
	
0 5 67equipment 46 37
	
23 0 5
	
9 18 3 58 39 24
	
18 50 62subtotals 193 103
	
52 0 19
	
32 122 122 77 63 31
	
27 27 50<packaging> 193 231
	
133 0 5
	
93 55 0 53 28 40
	
4 69 58type 193 231
	
112 5 21
	
93 55 0 60 28 40
	
17 59 50pitch 14 0
	
0 0 0
	
0 14 124 100 100 *
	
* 0 *material 92 51
	
30 0 2
	
19 60 65 73 65 37
	
6 33 59p-l-count 52 25
	
17 0 0
	
8 35 95 72 67 32
	
0 33 68units 57 47
	
29 4 0
	
14 24 88 56 42 30
	
6 54 66bonding 33 27
	
14 0 0
	
13 19 116 70 58 48
	
0 42 52device 171 40
	
21 0 7
	
12 143 42 88 84 30
	
25 12 52equipment 0 0
	
0 0 0
	
0 0 133 * * *
	
n * *
subtotals 612 421
	
223 9 30
	
159 350 663 70 57 38
	
13 37 54<device> 190 88
	
52 0 9
	
27 129 0 76 68 31
	
15 27 59function 190 88
	
50 2 9
	
27 129 0 76 68 31
	
16 27 58size 111 0
	
0 0 0
	
0 111 25 100 100 *
	
* 0 *device-speed 25 0
	
0 0 0
	
0 25 46 100 100 *
	
* 0 *subtotals 326 88
	
50 2 9
	
27 265 71 86 81 31
	
16 16 58<equipment> 351 320
	
206 0 24
	
90 121 0 53 34 28
	
10 59 64name 132 93
	
31 8 9
	
45 84 112 80 64 48
	
27 26 3 8manufacturer 268 0
	
0 0 0
	
0 268 51 100 100 *
	
* 0 *modules 45 0
	
0 0 0
	
0 45 192 100 100 *
	
* 0type 351 320
	
199 3 28
	
90 121 0 54 34 28
	
13 57 63throughput 17 0
	
0 0 0
	
0 17 194 100 100 *
	
* 0 *status 374 320
	
183 0 47
	
90 144 0 60 38 28
	
20 49 5 7wafer-size 51 0
	
0 0 0
	
0 51 188 100 100 *
	
* 0 *subtotals 1238 733
	
413 11 84
	
225 730 737 71 59 31
	
18 34 5 7ALL-OBJECTS 7039 5692
	
2829 66 14
	
2137 3484 3397 69 49 38
	
19 41 5 0MATCHED-ONLY 4708 3828
	
2829 66 14
	
506 1386 3397 45 29 13
	
14 61 7 5TEXT-FILTERING 219 225
	
190 - -
	
35 29 46 13 16
	
- 87 8 4
Wrong Req-fills All-fills Min-err Max-er rRICHNESS-NORMALIZED-ERROR 631 4
Wrong 7474Word-count 7808Error-rate 0.8087 0 .844 8ERROR-RATE-PER-WORD
6314 125680 0.0502
P&R 2P&R P&2RF-MEASURES 44
.96 48.01 42.28
40 6
EME Summary Scores
: UMass/Hughes CIRCUS System
V . •; r•, • . .t . " s 'IL
	
. •' r t' •
	
• 1 s : •1 y
<template> 300 300
	
300 0 0 0 0
	
0— 0 0 0 0 100 100
content 605 318
	
225 0 4 89 376
	
24 68 62 28 2 37 7 1
subtotals 605 318
	
225 0 4 89 376
	
24 68 62 28 2 37 7 1
<entity> 549 647
	
256 0 129 262 164
	
0 68 30 40 34 47 40
name 495 630
	
203 38 119 270 135
	
2 71 27 43 38 45 35
location 231 52
	
18 3 8 23 202
	
157 92 87 44 33 8 38
nationality 78 34
	
12 0 1 21 65
	
225 88 83 62 8 15 35
type 549 631
	
251 0 128 252 170
	
0 69 31 40 34 46
	
- 40
subtotals 1353 1347
	
484 41 256 566 572
	
384 74 42 42 35 37 37
<microelectron . 605 318
	
225 0 4 89 376
	
0 68 62 28 2 37 7 1
process 605 347
	
190 0 39 118 376
	
0 74 62 34 17 31 55
developer 289 144
	
29 0 15 100 245
	
124 92 85 69 34 10 20
manufacturer 235 363
	
56 0 31 276 148
	
89 89 63 76 36 24 15
distributor 232 347
	
52 0 29 266 151
	
86 90 65 77 36 22 1 5
purchaser 418 226
	
69 0 20 137 329
	
77 88 79 61 22 16 30
subtotals 1779 1427
	
396 0 134 897 1249
	
376 85 70 63 25 22 2 8
<layering> 145 149
	
93 0 5 51 47
	
0 52 32 34 5 64 62
type 145 149
	
85 5 8 51 47
	
0 55 32 34 11 60 59
film 84 51
	
24 0 7 20 53
	
49 77 63 39 22 28 4 7
temperature 4 0
	
0 0 0 0 4
	
90 100 100 • • 0 •
device 63 67
	
19 0 6 42 38
	
54 82 60 63 24 30 2 8
eeqq 98 30
	
20 0 5 5 73
	
29 80 74 17 20 20 67
subttotalsnt 394 297
	
148 5 26 118 215
	
222 71 54 40 16 38 5 1
<lithography> 212 134
	
93 0 7 34 112
	
0 62 53 25 7 44 69
type 212 132
	
64 8 26 34 114
	
0 72 54 26 31 32 52
granularity 45 13
	
2 1 6 4 36
	
66 95 80 31 72 6 1 9
device 90 113
	
44 0 3 66 43
	
30 72 48 58 6 49 3 9
equipment 175 129
	
50 0 27 52 98
	
6 78 56 40 35 28 3 9
subtotals 522 387
	
160 9 62 156 291
	
102 76 56 40 29 32 42
<etching> 60 63
	
24 0 9 30 27
	
0 73 45 48 27 40 3 8
type 60 54
	
14 0 15 25 31
	
0 84 52 46 52 23 26
etchant 6 0
	
0 0 0 0 6
	
19 100 100 • • 0 *
film
	
' 27 32
	
8 0 5 19 14
	
12 83 52 59 38 30 25
granularity 11 0
	
0 0 0 0 11
	
17 100 100 * • 0 •
temperature 1 0
	
0 0 0 0 1
	
24 100 100 4, * 0
device 41 29
	
14 0 1 14 26
	
9 74 63 48 7 34 48
eeqquipment 45 12
	
7 0 2 3 36
	
4 85 80 25 22 16 5 8
subtculs 191 127
	
43 0 23 61 125
	
85 83 65 48 35 22 34
<packaging> 192 197
	
120 0 11 66 61
	
0 53 32 34 8 62 6 1
type 192 197
	
92 1 38 66 61
	
0 64 32 34 29 48 47
pitch 14 0
	
0 0 0 0 14
	
114 100 100 * • 0 *
material 91 37
	
26 0 1 10 64
	
62 74 70 27 4 28 70
p-l-count 51 0
	
0 0 0 0 51
	
88 100 100 • • 0
units 57 0
	
0 0 0 0 57
	
93 100 100 * • 0 •
bonding 34 0
	
0 0 0 0 34
	
105 100 100 • * 0 *
device 172 249
	
89 0 17 143 66
	
16 72 38 57 16 52 36
equipment 0 1
	
0 0 0 1 0
	
119 100 • 100 * • 0
subtotals 611 484
	
207 1 56 220 347
	
597 75 57 45 21 34 43
<device> 192 215
	
119 0 19 77 54
	
0 56 28 36 14 62 55
function 192 198
	
118 1 12 67 61
	
0 54 32 34 10 62 60
size 114 70
	
36 0 7 27 71
	
44 74 62 38 16 32 5 1
device-speed 25 11
	
2 0 0 9 23
	
105 94 92 82 0 8 1 8
subtotals 331 279
	
156 1 19 103 155
	
149 64 47 37 11 47 56
<equipment> 348 175
	
136 0 10 29 202
	
0 64 58 16 7 39 78
name 132 80
	
17 8 16 39 91
	
52 88 69 49 49 16 26
manufacturer 269 484
	
80 0 44 360 145
	
1 87 54 74 35 30 1 6
modules 44 0
	
0 0 0 0 44
	
126 100 100 • • 0
type 348 163
	
111 7 17 28 213
	
0 70 61 17 15 33 70
throughput 17 0
	
0 0 0 0 17
	
129 100 100 * * 0 *
status 371 175
	
129 0 17 29 225
	
0 68 61 16 12 35 74
wafer-size 49 0
	
0 0 0 0 49
	
125 100 100 e * 0 *
subtotals 1230 902
	
337 15 94 456 784
	
433 80 64 50 23 28 38
ALL-OBJECTS 7016 5568
	
2156 72 22 2666 4114
	
2372 77 59 48 24 31 39
MATCHED-ONLY 3663 3722
	
2156 72 22 1140 1081
	
2372 54 30 31 15 60 59
TEXT-FILTERING 219 243
	
186 - - 57 33
	
24 15 23 - 85 76
Wrong Req-fills All-fills Min-err Max-er r
RICHNESS-NORMALIZED-ERROR 7490 7449 7779 0 .9628 1 .005 5
Wrong Word-count Error-rate
ERROR-RATE-PER-WORD 7490 125680 0.0596P&R 2P&R P&2R
F-MEASURES 34.84 37 .42 32.59
407
EME Summary Score s
UMichigan LINK System
SLOT POS ACT
	
COR PAR INC SPU MIS NON ERR UND OVG SUB REC PRE
<template> 300 300
	
300 0 0 0 0 0 0 0 0 0 100 10 0content
606 489
	
254 0 60 175 292 9 67 48 36 19 42 5 2subtotals
606 489
	
254 0 60 175 292 9 67 48 36 19 42 5 2<entity>
546 291
	
124 0 77 90 345 0 80 63 31 38 23 43name
491 236
	
74 37 51 74 329 2 84 67 31 43 19 3 9location
231 72
	
14 4 14 40 199 76 94 86 56 50 7 22nationality
75 0
	
0 0 0 0 75 113 100 100 * * 0 *type
546 291
	
124 0 77 90 345 0 80 63 31 38 23 43subtotals 1343
599
	
212 41 142 204 948 191 85 70 34 41 17 3 9<microelectron 606 488
	
254 0 60 174 292 0 67 48 36 19 42 5 2
process 606 455
	
166 0 135 154 305 0 78 50 34 45 27 3 6developer
290 67
	
21 0 11 35 258 148 94 89 52 34 7 3 1
manufacturer 235 173
	
23 0 27 123 185 102 94 79 71 54 10 1 3distributor
232 156
	
23 0 21 112 188 109 93 81 72 48 10 1 5purchaser
419 40
	
10 0 15 15 394 99 98 94 38 60 2 25
subtotals 1782 891
	
243 0 209 439 1330 458 89 75 49 46 14 27<layering>
145 193
	
56 0 20 117 69 0 79 48 61 26 39 29type
145 77
	
43 3 5 26 94 0 74 65 34 13 31 58
film 83 68
	
17 0 10 41 56 24 86 67 60 37 20 25temperature
4 1
	
0 0 0 1 4 53 100 100 100 * 0 0device
63 79
	
7 0 10 62 46 34 94 73 78 59 11 9equipment
99 65
	
6 0 23 36 70 15 96 71 55 79 6 9subtotals
394 290
	
73 3 48 166 270 126 87 68 57 40 19 26<lithography>
209 101
	
55 0 10 36 144 0 78 69 36 15 26 54
type 209 76
	
34 2 11 29 162 0 85 78 38 26 17 4 6
granularity 45 2
	
0 0 2 0 43 43 100 96 0 100 0 0device 89 39
	
18 0 5 16 66 26 83 74 41 22 20 4 6
equipment 174 55
	
24 0 16 15 134 9 87 77 27 40 14 44subtotals
517 172
	
76 2 34 60 405 78 87 78 35 31 15 45
<etching> 61 77
	
16 0 20 41 25 0 84 41 53 56 26 2 1
type 61 37
	
11 0 3 23 47 0 87 77 62 21 18 3 0
etchant 6 0
	
0 0 0 0 6 14 100 100 * * 0 *
film 27 18
	
2 1 2 13 22 7 94 81 72 50 9 14granularity 11 0
	
0 0 0 0 11 12 100 100 * * 0 *
temperature 1 0
	
0 0 0 0 1 15 100 100 * * 0 *
device 42 20
	
5 0 4 11 33 5 90 78 55 44 12 25
equipment 47 25
	
1 0 13 11 33 4 98 70 44 93 2 4
subtotals 195 100
	
19 1 22 58 153 57 92 78 58 54 10 20
<packaging> 192 85
	
54 0 12 19 126 0 74 66 22 18 28 64type 192 56
	
40 1 4 11 147 0 80 76 20 10 21 7 2pitch
14 0
	
0 0 0 0 14 52 100 100 * * 0 *
material 89 0
	
0 0 0 0 89 33 100 100 * * 0 *
p-l-count 51 0
	
0 0 0 0 51 35 100 100 * * 0
units 56 0
	
0 0 0 0 56 46 100 100 * * 0 *bonding
33 8
	
5 0 0 3 28 45 86 85 38 0 15 62device
169 36
	
18 0 10 8 141 9 90 83 22 36 11 5 0equipment 0
9
	
0 0 0 9 0 50 100 * 100 * * 0subtotals 604 109
	
63 1 14 31 526 270 90 87 28 18 10 5 8<device> 191
169
	
79 0 27 63 85 0 69 44 37 25 41 47function 191 121
	
76 3 8 34 104 0 66 54 28 11 40 64
size 114 37
	
25 0 0 12 89 30 80 78 32 0 22 68device-speed
25 0
	
0 0 0 0 25 68 100 100 * * 0 *subtotals 330 158
	
101 3 8 46 218 98 73 66 29 8 31 65<equipment> 349
151
	
59 0 50 42 240 0 85 69 28 46 17 39name
133 19
	
5 0 5 9 123 33 96 92 47 50 4 26manufacturer 269
4
	
2 0 2 0 265 11 99 98 0 50 1 5 0modules
45 0
	
0 0 0 0 45 52 100 100 * ,p 0 *type 349
72
	
50 7 4 11 288 0 85 82 15 12 15 7 4throughput 17
0
	
0 0 0 0 17 55 100 100 * * 0 *status 373
0
	
0 0 0 0 373 0 100 100 * * 0 *wafer-size
49 0
	
0 0 0 0 49 52 100 100 * * 0 *subtotals
1235 95
	
57 7 11 20 1160 203 95 94 21 19 5 64
ALL-OBJECTS 7006 2903
	
1098 58 15 1199 5302 1490 86 76 41 34 16 39MATCHED-ONLY 2606 1786
	
1098 58 15 337 1157 1490 62 44 19 22 43 63
TEXT-FILTERING 220 288
	
217 - - 71 3 9 - 1 25 - 99 7 5
Wrong Req-fills All-fills Min-err Max-er rRICHNESS-NORMALIZED-ERROR 7078
Wrong
7441
	
7775
Word-count
	
Error-rate
0 .9104 0.951 2
ERROR-RATE-PER-WORD 7078 125680 0.056 3
P&R 2P&R P&2RI~-MEASURES
22 .75 30.27 18 .22
408
JME Summary Score s
BBN PLUM System
SLOT POS AC!
	
COR PAR INC SPU MIS NON ERR UND OVG SUB Rh(. PRh
<template> 100 100
	
1 0 0 0 if 0 0 0 0 100 100
content 123 109
	
77 0 0 32 46 23 50 37 29 0 63 7 1
subtotals 123 109
	
77 0 0 32 46 23 50 37 29 0 63 7 1
<entity> 121 111
	
71 0 8 32 42 0 54 35 29 10 59 6 4name
120 111
	
64 2 13 32 41 0 57 34 29 18 54 58
location 58 0
	
0 0 0 0 58 36 100 100 • • 0 *
nationality 36 0
	
0 0 0 0 36 50 100 100 * • 0 *
type 121 111
	
71 0 8 32 42 0 54 35 29 10 59 64
subtotals 335 222
	
135 2 21 64 177 86 66 53 29 14 40 6 1
vnicroelectron 124 109
	
78 0 0 31 46 0 50 37 28 0 63 72
process 124 109
	
68 0 10 31 46 0 56 37 28 13 55 62
developer 63 56
	
16 0 7 33 40 44 83 63 59 30 25 28
manufacturer 85 169
	
42 0 9 118 34 10 79 40 70 18 49 25
distributor 80 83
	
31 0 6 46 43 26 75 54 55 16 39 37
purchaser 25 11
	
6 0 0 5 19 61 80 76 45 0 24 54
subtotals 377 428
	
163 0 32 233 182 141 73 48 54 16 43 38
<layering> 44 30
	
23 0 1 6 20 0 54 45 20 4 52 77
type 44 30
	
21 2 1 6 20 0 56 45 20 8 50 73
film 13 0
	
0 0 0 0 13 20 100 100 * • 0 •
temperature 5 0
	
0 0 0 0 5 21 100 100 * • 0 *
device 13 2
	
1 0 0 1 12 15 93 92 50 0 8 50
equipment 39 34
	
10 0 6 18 23 1 82 59 53 38 26 29subtotals 114 66
	
32 2 7 25 73 57 76 64 38 20 29 50
<lithography> 51 53
	
36 0 3 14 12 0 45 24 26 8 70 6 8
type 51 53
	
33 0 6 14 12 0 49 24 26 15 65 62granularity 23 0
	
0 0 0 0 23 20 100 100 * • 0 •
device 35 7
	
4 0 1 2 30 12 89 86 28 20 11 57
equipment 51 74
	
30 0 9 35 12 0 65 24 47 23 59 40
subtotals 160 134
	
67 0 16 51 77 32 68 48 38 19 42 50<etching>
17 11
	
5 0 2 4 10 0 76 59 36 28 29 4 5
type 17 11
	
3 0 4 4 10 0 86 59 36 57 18 2 7etchant
0 0
	
0 0 0 0 0 5 • * • • • •film 4
0
	
0 0 0 0 4 4 100 100 * * 0 •
granularity 2 0
	
0 0 0 0 2 4 100 100 • • 0 •temperature
0 0
	
0 0 0 0 0 5 • • • • • •
device 6 3
	
2 0 0 1 4 2 71 67 33 0 33 67
equipment 10 14
	
4 0 1 9 5 0 79 50 64 20 40 2 8
subtotals 39 28
	
9 0 5 14 25 20 83 64 50 36 23 32<packaging>
12 15
	
8 0 0 7 4 0 58 33 47 0 67 5 3
type 12 15
	
8 0 0 7 4 0 58 33 47 0 67 5 3pitch 0 0
	
0 0 0 0 0 8 • • * * • •
material 4 13
	
3 0 0 10 1 4 78 25 77 0 75 2 3
p-l-count 6 5
	
5 0 0 0 1 3 17 17 0 0 83 10 0units 0 0
	
0 0 0 0 0 8 • • • * * *
bonding 0 0
	
0 0 0 0 0 8 * • • • * *
device 14 15
	
5 0 3 7 6 0 76 43 47 38 36 3 3equipment 0 0
	
0 0 0 0 0 8 * • • • * •
subtotals 36 48
	
21 0 3 24 12 39 65 33 50 12 58 4 4<device>
49 19
	
12 0 3 4 34 0 77 69 21 20 24 63function 49
19
	
12 0 3 4 34 0 77 69 21 20 24 63
size 34 0
	
0 0 0 0 34 5 100 100 • * 0 •
device-speed 4 0
	
0 0 0 0 4 10 100 100 • * 0 *subtotals 87
19
	
12 0 3 4 72 15 87 83 21 20 14 63<equipment>
110 78
	
55 0 4 19 51 0 57 46 24 7 50 7 0name
27 10
	
6 1 1 2 19 38 78 70 20 19 24 65manufacturer
91 89
	
31 0 10 48 50 1 78 55 54 24 34 3 5modules
15 0
	
0 0 0 0 15 51 100 100 • • 0 *
type 110 78
	
53 0 6 19 51 0 59 46 24 10 48 68throughput 3 0
	
0 0 0 0 3 55 100 100 • * 0 *
status 110 78
	
49 0 10 19 51 0 62 46 24 17 44 63
wafer-size 12 0
	
0 0 0 0 12 49 100 100 • * 0 •subtotals 368 255
	
139 1 27 88 201 194 69 55 34 16 38 5 5
ALL-OBJECTS 1639 1309
	
655 5 6 535 865 607 70 53 41 15 40 50
MATCHED-ONLY 987 955
	
655 5 6 220 252 607 46 26 23 10 67 69TEXT-FILTERING 68 66
	
57 - - 9 11 23 - 16 14 - 84 86
Wrong Req-fills All-fills Min-err Max-er rRICHNESS-NORMALIZED-ERROR 1516. 5
Wrong
1664
	
1749
Word-count
	
Error-rate
0.8671 0.9114
ERROR-RATE-PER-WORD 1516.5 1784 0.850 1P&R 2P&R P&2R
F-MEASURES 44.61 47.82 41 .8
409
JME Summary Score s
GE/CMU Shogun System
SLOT AZ'1~
	
COR PAR INC SPU 11318 NON ERR END OVO SUB-- REC PR E
<template> 100 100
	
100 0 6 0 0 0 0 0
	
0 0 100 100content
123 134
	
94 0 2 38 27 21 42 22
	
28 2 76 70subtotals 123
134
	
94 0 2 38 27 21 42 22
	
28 2 76 70<entity> 121
131
	
91 0 9 31 21 0 40 17
	
24 9 75 69name 121
131
	
77 3 20 31 21 0 48 17
	
24 22 65 60location
58 47
	
25 4 4 14 25 42 62 43
	
30 18 46 5 7nationality
36 19
	
14 0 4 1 18 61 62 50
	
5 22 39 74tyyppee
121 131
	
91 0 9 31 21 0 40 17
	
24 9 75 69subotsts 336 328
	
207 7 37 77 85 103 49 25
	
23 16 63 64<microelectron 124 134
	
94 0 2 38 28 0 42 22
	
28 2 76 70process 124 134
	
84 0 12 38 28 0 48 22
	
28 12 68 63developer 63 91
	
23 0 9 59 31 43 81 49
	
65 28 36 25
manufacturer 83 141
	
43 0 15 83 25 12 74 30
	
59 26 52 30distributor 80 138
	
45 0 9 84 26 20 72 32
	
61 17 56 33
purchaser 25 36
	
13 0 1 22 11 62 72 44
	
61 7 52 36subtotals 375 540
	
208 0 46 286 121 137 68 32
	
53 18 55 38
<layering> 44 57
	
36 0 1 20 7 0 44 16
	
35 3 82 63
type 44 57
	
32 2 3 20 7 0 48 16
	
35 11 75 58
film 13 2
	
0 0 1 1 12 28 -
	
100 92
	
50 100 0 0
temperature 5 5
	
2 0 0 3 3 31 75 60
	
60 0 40 40
device 13 9
	
6 0 0 3 7 24 62 54
	
33 0 46 67
equipment 39 57
	
20 0 13 24 6 0 - -
	
68 15
	
42 39 51 35
subtotals 114 130
	
60 2 17 51 35 83 63 31
	
39 23 54 47
<lithography> 51 47
	
35 0 1 11 15 0 44 29
	
23 3 69 74
type 51 47
	
30 0 6 11 15 0 52 29
	
23 17 59 6 4
granularity 24 36
	
8 5 2 21 9 15 77 38
	
58 30 44 29
device 35 39
	
20 0 2 17 13 7 62 37
	
44 9 57 5 1
equipment 51 43
	
32 0 2 9 17 0 47 33
	
21 6 63 74
subtotals 161 165
	
90 5 12 58 54 22 58 34
	
35 14 57 56
<etching> 17 15
	
9 0 1 5 7 0 59 41
	
33 10 53 60
type 17 15
	
6 0 4 5 7 0 73 41
	
33 40 35 40
etchant 0 0
	
0 0 0 0 0 9 * *
	
* * * *
film 4 1
	
1 0 0 0 3 8 75 75
	
0 0 25 10 0
granularity 2 7
	
0 2 0 5 0 5 86 0
	
71 50 50 14
temperature 0 0
	
0 0 0 0 0 9 * *
	
* * * *
device 6 3
	
2 0 0 1 4 4 71 67
	
33 0 33 67
equipment 10 11
	
7 0 1 3 2 0 46 20
	
27 12 70 64
subtotals 39 37
	
16 2 5 14 16 35 68 41
	
38 26 44 46
<packaging> 12 15
	
10 0 0 5 2 0 41 17
	
33 0 83 67
type 12 15
	
10 0 0 5 2 0 41 17
	
33 0 83 67
pitch 0 3
	
0 0 0 3 0 10 100 *
	
100 * * 0
material 4 6
	
3 0 0 3 1 7 57 25
	
50 0 75 50
p-I-count 5 5
	
5 0 0 0 0 5 0 0
	
0 0 100 100
units 0 0
	
0 0 0 0 0 10 * *
	
* * * *
bonding 0 0
	
0 0 0 0 0 10 * *
	
* •
device 14 11
	
7 0 0 4 7 0 61 50
	
36 0 50 64
equipment 0 0
	
0 0 0 0 0 10 * *
	
* * * *
subtotals 35 40
	
25 0 0 15 10 52 50 28
	
38 0 71 62
<device> 49 56
	
37 0 1 18 11 0 45 22
	
32 3 76 66
function 49 56
	
34 3 1 18 11 0 47 22
	
32 6 72 63
size 33 31
	
22 0 2 7 9 11 45 27
	
22 8 67 7 1
device-speed 4 11
	
3 0 1 7 0 31 73 0
	
64 25 75 27
subtotals 86 98
	
59 3 4 32 20 42 49 23
	
33 8 70 62
<equipment> 110 115
	
79 0 2 34 29 0 45 26
	
30 2 72 69
name 27 21
	
16 0 0 5 11 54 50 41
	
24 0 59 76
manufacturer 89 112
	
44 0 20 48 25 0 68 28
	
43 31 49 39
modules 15 4
	
3 0 0 1 12 74 81 80
	
25 0 20 75
type 110 115
	
74 0 7 34 29 0 49 26
	
30 9 67 64
throughput 3 8
	
1 0 0 7 2 73 90 67
	
88 0 33 1 2
status 110 115
	
74 0 7 34 29 0 49 26
	
30 9 67 64
wafer-size 12 4
	
1 0 0 3 11 68 93 92
	
75 0 8 25
subtotals 366 379
	
213 0 34 132 119 269 57 32
	
35 14 58 5 6
ALL-OBJECTS 1635 1851
	
972 19 17 703 487 764 58 30----38 14 60 5 3
MATCHED-ONLY 1261 1345
	
972 19 17 231 147 764 34 12
	
17 12 78 73
TEXT-FILTERING 68 77
	
66 - - 11 2 21 - 3
	
14 - 97 86
Wrong Req-fills A11-fills Min-err Max-er r
RICHNESS-NORMALIZED-ERROR 1356.5
Wrong
1664
	
174 9
Word-count
	
Error-rate
0.7756 0 .8152
ERROR-RATE-PER-WORD 1356 .5 1784 0.7604
P&R 2P&R P&2R
F-MEASURES 56.31 54.29 58 .49
41 0
JME Summary Score s
GE/CMU TEXTRACT System (Optional Test Run )
<template> 100 I00 100 0 0 0 0 0 0 0 0 0 100 100
content 122 85 70 0 0 15 52 22 49 43 18 0 57 82
subtotals 122 85 70 0 0 15 52 22 49 43 18 0 57 82
<entity> 117 112 81 0 6 25 30 0 43 26 22 7 69 7 2
name 116 112 70 4 13 25 29 0 49 25 22 17 62 64
location 56 39 21 5 4 9 26 40 64 46 23 22 42 60
nationality 36 31 20 0 4 7 12 57 53 33 22 17 56 64
typ 117 112 81 0 6 25 30 0 43 26 22 7 69 72subttotals
325 294 192 9 27 66 97 97 50 30 22 14 60 67
anieroelectron 123 85 70 0 0 15 53 0 49 43 18 0 57 82process
123 85 68 0 2 15 53 0 51 43 18 3 55 80developer
62 52 18 0 3 31 41 37 81 66 60 14 29 3 5
manufacturer 83 101 44 0 7 50 32 11 67 38 50 14 53 4 4
distributor 80 77 42 0 1 34 37 20 63 46 44 2 52 5 4purchaser
23 42 10 0 3 29 10 46 81 43 69 23 43 24
subtotals 371 357 182 0 16 159 173 114 66 47 44 8 49 5 1
<layering> 44 30 24 0 1 5 19 0 51 43 17 4 54 8 0
type 44 30 23 0 2 5 19 0 53 43 17 8 52 7 7
film 13 10 0 0 4 6 9 15 100 69 60 100 0 0
temperature 5 6 0 0 2 4 3 19 100 60 67 100 0 0
device 13 11 4 0 3 4 6 13 76 46 36 43 31 36
equipment 39 24 „ 17 0 3 4 19 3 60 49 17 15 44 7 1
subtotals 114 81 44 0 14 23 56 50 68 49 28 24 38 54
<lithography> 51 34 27 0 0 7 24 0 53 47 20 0 53 7 9
type 51 34 22 0 5 7 24 0 62 47 20 18 43 65granularity
24 14 10 0 2 2 12 13 62 50 14 17 . 42 7 1device
35 23 15 0 4 4 16 5 62 46 17 21 43 65equipment
51 33 25 0 2 , 6 24 0 56 47 18 7 49 76subtotals
161 104 72 0 13 19 76 18 60 47 18 15 45 69<etching>
16 10 9 0 0 1 7 0 47 44 10 0 56 90
type 16 10 7 1 1 1 7 0 56 44 10 17 47 75
etchant 0 2 0 0 0 2 0 7 100 * 100 * * 0
film 4 0 0 0 0 0 4 7 100 100 * * 0 •
granularity 2 1 1 0 0 0 1 8 50 50 0 0 50 100
temperature 0 0 0 0 0 0 0 9 * * • * * *device
5 3 2 0 0 1 3 5 67 60 33 0 40 67
equipment 9 6 6 0 0 0 3 2 33 33 0 0 67 100
subtotals 36 22 16 1 1 4 18 38 59 50 18 8 46 75
<packaging> 12 11 8 0 0 3 4 0 47 33 27 0 67 73
type 12 11 7 0 1 3 4 0 53 33 27 12 58 64
pitch 0 0 0 0 0 0 0 8 • • * * • *
material 4 6 3 0 0 3 1 5 57 25 50 0 75 50
p-I-count 7 4 4 0 0 0 3 3 43 43 0 0 57 100units
0 0 0 0 0 0 0 8 • • * * * *bonding
0 0 0 0 0 0 0 8 * • • • • •
device 14 5 5 0 0 0 9 0 64 64 0 0 36 100eeqq pment
0 0 0 0 0 0 0 8 * * * • •
subtotals 37 26 19 0 1 6 17 40 56 46 23 5 51 73
<device> 48 39 27 0 7 5 14 0 49 29 13 20 56 69
function 48 39 24 3 7 5 14 0 52 29 13 25 53 65
size 34 29 22 0 3 4 9 2 42 26 14 12 65 76
device-speed 4 4 3 0 1 0 0 23 25 0 0 25 75 75
subtotals 86 72 49 3 11 9 23 25 47 27 12 20 59 70
<equipment> 109 63 53 0 2 8 54 0 55 50 13 4 49 8 4
name 27 17 15 0 0 2 12 33 48 44 12 0 56 8 8
manufacturer 89 91 43 0 7 41 39 1 67 44 45 14 48 4 7
modules 15 0 0 0 0 0 15 49 100 100 * * 0 *
type 109 63 52 0 3 8 54 0 56 50 13 5 48 82
ghput 3 3 2 0 0 1 1 50 50 33 33 0 67 67
status 109 63 49 0 6 8 54 0 58 50 13 11 45 7 8
wafer-size 12 11 7 0 2 2 3 43 50 25 18 22 58 6 4subtotals
364 248 168 0 18 62 178 176 60 49 25 10 46 68
ALL-OBJECTS 1616 1289 812 13 10 363 690 580 59 43 28 12- — 6 3
MATCHED-ONLY 1015 1058 812 13 10 160 117 580 30 12 15 9 81 77
TEXT-FILTERING 67 75 64 - - 11 3 22 - 4 15 - 96 85
Wrong Req-fills All-fills Min-err Max-errRICHNESS-NORMALIZED-ERROR
1160.5
Wrong
164 1
Word-count 172 5Error-rate 0.6728 0 .707 2
ERROR-RATE-PER-WORD 1160.5 1784 0.6505
P&R 2P&R P&2RF-MEASURES
56.35 60.43 52.79
41 1
JME Summary Scores
NEC VENIEX System
SLO1 POS ACT
	
COR PAR INC SPU MIS NON ERR UND OVG SUS- REC PRE
<template> 100 100 100 0 0 0 0 0 0 0 0 0 100 100
content 123 92 66 0 0 26 57 23 56 46 28 0 54 72
subtotals 123 92 66 0 0 26 57 23 56 46 28 0 54 7 2
<entity> 121 81 61 0 8 12 52 0 54 43 15 12 50 75
name 121 79 48 7 12 12 54 0 61 45 15 23 42 65
location 58 34 21 2 5 6 30 25 66 52 18 21 38 65
nationality 36 16 15 0 0 1 21 40 59 58 6 0 42 94
type 121 81 61 0 8 12 52 0 54 43 15 12 50 75
subtotals 336 210 145 9 25 31 157 65 59 47 15 16 44 7 1
<microelectron 124 92 66 0 0 26 58 0 56 47 28 0 53 72
process 124 92 60 0 6 26 58 0 60 47 28 9 48 65
developer 64 45 15 0 6 24 43 31 83 67 53 28 23 3 3
manufacturer 82 72 35 0 10 27 37 13 68 45 38 22 43 49distributor 80
49 28 0 4 17 48 17 71 60 35 12 35 57purchaser 25
17 6 0 2 9 17 51 82 68 53 25 24 35
subtotals 375 275 144 0 28 103 203 112 70 54 37 16 38 52
<layering> 44 31 25 0 2 4 17 0 48 39 13 7 57 8 1
type 44 31 24 1 2 4 17 0 49 39 13 9 56 79
film 13 3 1 0 0 2 12 20 93 92 67 0 8 3 3
temperature 5 0 0 0 0 0 5 21 100 100 * * 0 *
device 13 0 0 0 0 0 13 17 100 100 * * 0 *
equipment 39 31 19 0 6 6 14 0 58 36 19 24 49 6 1subtotals 114
65 44 1 8 12 61 58 65 54 18 16 39 68<lithography>
51 34 23 0 0 11 28 0 63 55 32 0 45 68
type 51 34 18 0 5 11 28 0 71 55 32 22 35 5 3
granularity 24 2 0 0 2 0 22 12 100 92 0 100 0 0
device 35 4 3 0 0 1 32 8 92 91 25 0 8 75
equipment 51 34 23 0 0 11 28 0 63 55 32 0 45 68
subtotals 161 74 44 0 7 23 110 20 76 68 31 14 27 5 9<etching> 17
13 10 0 0 3 7 0 50 41 23 0 59 7 7
type 17 13 8 0 2 3 7 0 60 41 23 20 47 62etchant 0 0 0 0
0 0 0 10 * * * * *
film 4 0 0 0 0 0 4 9 100 100 * * 0 *
granularity 2 1 0 0 0 1 2 9 100 100 100 * 0 0
temperature 0 0 0 0 0 0 0 10 * * * * * *
device 6 0 0 0 0 0 6 6 100 100 * * 0 *
equipment 10 13 7 0 3 3 0 0 46 0 23 30 70 54
subtotals 39 27 15 0 5 7 19 44 67 49 26 25 38 56
<packaging> 12 14 6 0 0 8 6 0 70 50 57 0 50 43
type 12 14 •
	
6 0 0 8 6 0 70 50 57 0 50 43
pitch 0 0 0 0 0 0 0 6 * * * * * *
material 4 7 1 0 0 6 3 4 90 75 86 0 25 1 4
p-l-count 5 1 1 0 0 0 4 3 80 80 0 0 20 100
units 0 0 0 0 0 0 0 6 * * * * * *
bonding 0 0 0 0 0 0 0 6 * * * * * *
device 14 6 5 0 0 1 9 0 67 64 17 0 36 83
equipment 0 1 0 0 0 1 0 6 100 * 100 * * 0subtotals
35 29 13 0 0 16 22 31 74 63 55 0 37 45
<device> 49 8 8 0 0 0 41 0 84 84 0 0 16 100function 49 8 6 2 0 0 41 0 86 84 0 12 14 88
size 33 3 3 0 0 0 30 2 91 91 0 0 9 100
device-speed 3 0 0 0 0 0 3 6 100 100 * * 0 *
subtotals 85 11 9 2 0 0 74 8 88 87 0 9 12 9 1
<equipment> 110 84 55 0 6 23 49 0 59 44 27 10 50 65
name 27 4 2 1 1 0 23 40 91 85 0 38 9 62
manufacturer 89 94 41 0 11 42 37 1 69 42 45 21 46 4 4
modules 15 10 2 0 0 8 13 50 91 87 80 0 13 20
type 110 70 51 0 2 17 57 0 60 52 24 4 46 7 3
throughput 3 0 0 0 0 0 3 54 100 100 * * 0 *
status 110 84 52 0 9 23 49 0 61 44 27 15 47 62
wafer-size 12 2 1 0 0 1 11 46 92 92 50 0 8 5 0
subtotals 366 264 149 1 23 91 193 191 67 53 34 14 41 5 7
ALL-OBJECTS 1634 1047 629 13 7 309 896 552 67 55 30 14 39 6 1
MATCHED-ONLY 903 781 629 13 7 75 197 552 35 22 10 10 70 8 1
TEXT-FILTERING 68 54 45 - - 9 23 23 - 34 17 - 66 83
Wrong Req-fills All-fills Min-err Max-er rRICHNESS-NORMALIZED-ERROR 1307.5
Wrong
1664
	
1749
Word-count
	
Error-rate
0 .7476 0.7858
ERROR-RATE-PER-WORD 1307 .5 1784 0.7329
P&R 2P&R P&2R
F-MEASURES 47 .41 54.58 41 .9
41 2
JME Summary Scores
NMSUBrandeis Diderot System
SLAT POS ACT
	
COR PAR INC SPU MIS NON ERR UND OVG SUB REC PRE
<template> 100 100
	
MO 0 0 0 -0 0 0 0 0 0 100 100content 122
92
	
69 0 3 20 50 27 51 41 22 4 56 75subtotals 122
92
	
69 0 3 20 50 27 51 41 22 4 56 75
<entity> 117 91
	
71 0 6 14 40 0 46 34 15 8 61 78name 116 91
	
61 4 12 14 39 0 52 34 15 18 54 69
location 56 34
	
20 1 4 9 31 30 68 55 26 18 37 60
nationality 36 8
	
4 0 0 4 32 48 90 89 50 0 11 50
type 117 91
	
71 0 6 14 4S 0 46 34 15 8 61 7 8
subtotals 325 224
	
156 5 22 41 142 78 57 44 18 13 49 7 1
<microelectron 123 92
	
70 0 2 20 51 0 51 41 22 3 57 76
process 123 92
	
65 0 7 20 51 0 54 41 22 10 53 7 1
developer 62 29
	
11 0 1 17 50 38 86 81 59 8 18 3 8
manufacturer 81 83
	
34 0 5 44 42 11 73 52 53 13 42 41
distributor 79 28
	
19 0 1 8 59 25 78 75 28 5 24 68
purchaser 23 19
	
11 0 1 7 11 50 63 48 37 8 48 5 8
subtotals 368 251
	
140 0 15 96 213 124 70 58 38 10 38 56
<layering> 44 28
	
23 0 1 4 20 0 52 45 14 4 52 82
type 44 28
	
22 1 1 4 20 0 53 45 14 6 51 8 0
film 13 1
	
0 0 0 1 13 17 100 100 100 * 0 0
temperature 5 0
	
0 0 0 0 5 21 100 100 • * 0
device 13 1
	
1 0 0 0 12 16 92 92 0 0 8 10 0
equipment 39 21
	
14 0 3 4 22 1 67 56 19 18 36 67
subtotals 114 51
	
37 1 4 9 72 55 70 63 18 11 33 74
<lithography> 51 37
	
29 0 0 8 22 0 51 43 22 0 57 78
type 51 37
	
23 0 6 8 22 0 61 43 22 21 45 62
granularity 23 0
	
0 0 0 0 23 18 100 100 * * 0 *
device 35 9
	
4 0 2 3 29 9 89 83 33 33 11 44
equipment 51 40
	
27 0 1 12 23 0 57 45 30 4 53 68
subtotals 160 86
	
54 0 9 23 97 27 70 61 27 14 34 63
<etching> 16 17
	
8 0 0 9 8 0 68 50 53 0 50 47
type 16 17
	
4 1 3 9 8 0 82 50 53 44 28 26
etchant 0 0
	
0 0 0 0 0 8 * * * * • •
film 4 2
	
1 0 0 1 3 7 80 75 50 0 25 50
granularity 2 0
	
0 0 0 0 2 6 100 100 * * 0
temperature 0 0
	
0 0 0 0 0 8 * • * • • •
device 5 1
	
1 0 0 0 4 4 80 80 0 0 20 100
equipment 9 8
	
5 0 0 3 4 2 58 44 38 0 56 62
subtotals 36 28
	
11 1 3 13 21 35 76 58 46 23 32 4 1
<packaging> 12 10
	
8 0 0 2 4 0 43 33 20 0 67 80
type 12 10
	
8 0 0 2 4 0 43 33 20 0 67 80
pitch 0 0
	
0 0 0 0 0 8 * * • • * *
material 4 4
	
2 0 0 2 2 5 67 50 50 0 50 50
p-l-count 5 3
	
3 0 0 0 2 4 40 40 0 0 60 100
units 0 0
	
0 0 0 0 0 8 • * * * * *
bonding 0 0
	
0 0 0 0 0 8 • * *
device 14 5
	
5 0 0 0 9 0 64 64 0 0 36 100
equipment 0 0
	
0 0 0 0 0 8 • * * * * •
subtotals 35 22
	
18 0 0 4 17 41 54 48 18 0 51 82
<device> 48 16
	
14 0 0 2 34 0 72 71 12 0 29 8 8
function 48 16
	
11 3 0 2 34 0 75 71 12 11 26 7 8
size 33 5
	
5 0 0 0 28 2 85 85 0 0 15 100
device-speed 3 0
	
0 0 0 0 3 12 100 100 • • 0 •
subtotals 84 21
	
16 3 0 2 65 14 80 77 10 8 21 8 3
<equipment> 109 69
	
56 0 4 9 49 0 52 45 13 7 51 8 1
name 27 9
	
8 0 1 0 18 40 70 67 0 11 30 89
manufacturer 87 57
	
34 0 9 14 44 2 66 50 24 21 39 60
modules 15 0
	
0 0 0 0 15 53 100 100 * * 0 •
type 109 69
	
51 0 9 9 49 0 57 45 13 15 47 74
throughput 3 0
	
0 0 0 0 3 55 100 100 * • 0
status 109 69
	
52 0 8 9 49 0 56 45 13 13 48 75
wafer-size 12 .0
	
0 0 0 0 12 48 100 100 * * 0
subtotals 362 204
	
145 0 27 32 190 198 63 52 16 16 40 7 1
ALL-OSJECIS 1606 979
	
646 10 8 240 867 599 65 54 24 12 40 6 6
MATCHED-ONLY 949 793
	
646 10 8 82 238 599 37 25 10 8 68 8 2
TEXT-FILTERING 67 62
	
56 - - 6 11 27 - 16 10 - 84 9 0
Wrong Req-fills All-fills Min-err Max-err
RICHNESS-NORMALIZED-ERROR 1195
Wrong
1641
	
1725
Word-count
	
Error-rate
0.6928 0.7282
ERROR-RATE-PER-WORD 1195 1784 0.669 8P&R 2P&R P&2R
F-MEASURES 50.37 58.95 43 .97
41 3
