APPENDIX G :
FINAL TEST SCORE SUMMARIES
The tables in the first four sections of this appendix are summary scor e
reports produced by the scoring program . For information on the format o f
these reports, see the paper by N. Chinchor on the evaluation metrics in Part I
of this proceedings . For information on the testing, see appendix B on the tes t
procedures and the overview paper by B . Sundheim in Part I .
	
For specific
information on the test setups for the optional runs, see the papers in Part I I
by the individual sites . The last section of this appendix contains the F -
measure scores as calculated for statistical significance testing ; for furthe r
information, see the paper by N . Chinchor on statistical significance testing i n
Part I.
SECTION 1 : TST3
Note : The text filtering row has
pass2. All other rows are from
been inserted from the site-scored file named scores .tst3 -
the NRaD-scored file named scores .tst3-passl .nrad .
SLOT POS ACT COR PAR INC ICR
	
IPA SPU MIS NON REC PRE OVG FA L
template-id 104 95 59 0 0 0
	
0 36 45 27 57 62 3 8
inc-date 100 91 38 10 9 0
	
10 34 43 6 43 47 3 7
inc-loc 104 88 24 31 1 0
	
9 32 48 4 38 45 3 6inc-type
104 95 55 4 0 0
	
0 36 45 0 55 60 38 4
inc-stage 104 95 56 0 3 0
	
0 36 45 0 54 59 38 1 2inc-instr-id
33 19 6 1 1 0
	
1 11 25 104 20 34 5 8
inc-instr-type 51 53 16 3 1 0
	
1 33 31 66 34 33 62 1perp-inc-cat
63 58 22 0 7 0
	
0 29 34 48 35 38 50 1 7
perp-iad-id 79 46 14 2 lI 2
	
2 19 52 58 19 33 4 1
perp-org-id 45 28 14 0 1 0
	
0 13 30 84 31 SO 4 6
perp-org-conf 45 28 10 1 4 0
	
1 13 30 84 23 38 46 2
phys-tgt-id 64 35 13 3 1 0
	
3 18 47 79 23 41 5 1phys-tgt-type 64
34 12 0 4 1
	
0 18 48 79 19 35 53 1
phys-tgt-num 65 34 14 1 1 1
	
0 18 49 79 22 43 5 3
phys-tgt-nation 2 0 0 0 0 0
	
0 0 2 139 0 . ' 0
phys-tgt-effect 40 15 6 3 1 1
	
1 5 30 107 19 50 33 1phys-tgt-total-num
0 0 0 0 0 0
	
0 0 0 14 0
hum-tgt-name 54 25 14 1 1 0
	
1 9 38 90 27 58 3 6hum-tgt-desc
119 68 23 7 12 3
	
6 26 77 33 22 39 3 8
hum-tgt-type 129 78 23 10 14 0
	
9 31 82 23 22 36 40 3
hum-tgt-num 130 79 28 12 8 0
	
11 31 82 23 26 43 3 9
hum-tgt-nation 15 0 0 0 0 0
	
0 0 15 129 0 • ' 0
hum-tgt-effect 111 72 21 16 1 0
	
15 34 73 30 26 40 47 2hum-tgt-total-num 1 0
0 0 0 0
	
0 0 1 139 0 • •
inc-total 496 441 195 49 15 0
	
21 182 237 180 44 50 4 1perp-total 232 160 60
3 23 2
	
3 74 146 274 26 38 4 6
phys-tgt-total 235 118 45 7 7 3
	
4 59 176 623 21 41 5 0
hum-tat-total 559 322 109 46 36 3
	
42 131 368 467 24 41 4 1
MATCHED/MISSING 1522 665 409 105 81 8
	
70 70 927 1092 30 69 1 0MATCHED/SPURIOUS
897 1041 409 105 81 8
	
70 A46 302 1046 51 44 4 3
MATCHED ONLY 897 665 409 105 81 8
	
70 70 302 594 51 69 1 0ALL TEMPLATES
1522 1041 409 105 81 8
	
70 446 927 1544 30 44 4 3
SET FILLS ONLY 728 339 221 37 35 2
	
27 46 435 498 33 71 14 0STRING FILLS ONLY
394 143 84 14 27 5
	
13 18 269 310 23 64 1 2
TEXT FILTERING 65 62 54 ' • • 8 11 27 83 87 13 2 3
P&R 2P&R
	
P&2 R
F-MEASURES 35 .68 40 .24 32 .04
Table G1. TST3 Scores for BB N
SLOT POS ACT COR PAR INC - ICR
	
IP. SPU MIS NON REC PRE OVG FAL
template-id 114 122 90 0 0 0
	
0 32 24 23 79 74 2 6
inc-date 111 119 60 15 13 0
	
15 31 23 4 61 57 2 6inc-loc
114 122 57 29 4 0
	
1 32 24 0 63 59 2 6
inc-type 114 122 87 3 0 0
	
0 32 24 0 78 72 26 4
lac-stage 114 122 87 0 3 0
	
0 32 24 0 76 71 26 1 1
inc-instr-id 33 36 20 4 0 0
	
4 12 9 115 67 61 3 3
inc-instr-type 53 53 28 6 1 0
	
0 18 18 90 58 58 34 1
perp-inc-cat 68 73 41 0 3 0
	
0 29 24 49 60 56 40 1 4perp-ind-id
86 75 37 6 5 4
	
6 27 38 62 46 53 3 6
perp-org-id 52 43 23 0 2 1
	
0 18 27 80 44 53 4 2
perp-org-conf 52 42 12 1 12 0
	
0 17 27 80 24 30 40 3
phys-tgt-id 68 77 30 4 4 2
	
4 39 30 77 47 42 5 1
phys-tgt-type 69 77 20 4 14 1
	
2 39 31 77 32 28 51 2phys-tgt-num
69 77 31 5 2 1
	
2 39 31 77 48 44 5 1
phys-tgt-nation 2 0 0 0 0 0
	
0 0 2 145 0 ' • 0
phys-tgt-effect 41 38 14 2 3 1
	
2 19 22 106 36 39 50 2
phys-tgt-total-num 0 0 0 0 0 0
	
0 0 0 14 6hum-tat-name
57 42 31 1 2 1
	
I 8 23 96 55 75 1 9
hum-tgt-desc 133 165 76 20 8 7
	
20 61 29 34 65 52 3 7hum-tat-type
142 166 77 15 13 2
	
11 61 37 31 60 51 37 4
hum-tgt-num 145 172 88 15 8 6
	
12 61 34 31 66 56 3 5
hum-tat-nation 16 3 1 0 0 0
	
0 2 15 132 6 33 67 0
hum -tat-effect 121 145 69 13 3 2
	
11 60 36 47 62 52 41 4
hum-tgt-total-num 1 0 0 0 0 0
	
0 0 1 145 0
inc-total 539 574 339 57 21 0
	
20 157 122 209 68 64 2 7
perp-total 258 233 113 7 22 5
	
6 91 116 271 45 50 3 9
phys-tgt-total 249 269 95 15 23 5
	
10 136 116 628 41 38 5 0
hum-tat-total 615 693 342 64 34 18
	
55 253 175 516 61 54 3 6
MATCHED/MISSING 1661 1363 889 143 100 28
	
91 231 529 1205 58 70 1 7MATCHED/SPURIOUS 1369 1769 889
143 100 28
	
91 637 237 1340 70 54 3 6
MATCHED ONLY 1369 1363 889 143 100 28
	
91 231 237 921 70 70 1 7ALL TEMPLATES 1661 1769
889 143 100 28
	
91 637 529 1624 58 54 3 6
SET FILLS ONLY 792 643 436 44 52 6
	
26 111 260 562 58 71 17 0
STRING FILLS ONLY 429 350 217 35 21 15
	
35 77 156 334 55 67 2 2
TEXT FILTERING 68 71 62 •
	
,' ' 9 6 23 91 87 13 2 8
P&R 2P&R
	
P&2 R
F-MEASURES 55 .93 54 .76
	
57 .15
Table G2. TST3 Scores for GE
G- 2
SLOT POS ACT COR
	
PAR INC ICR IPA SPU
	
MIS NON REC
	
PRE OVG FA L
template-id 115 105 81
	
0 0 0 0 24
	
34 25 70
	
77 2 3
inc-date 112 104 52
	
14 14 0 14 24
	
32 3 53
	
57 2 3
inc-loc 115 105 40
	
39 2 0 3 24
	
34 0 52
	
57 2 3
inc-type 115 105 80 0 0 0 24
	
34 0 70
	
77 23 3
inc-stage 115 105 79
	
0 2 0 0 24
	
34 0 69
	
75 23 9
inc-instr-id 33 32 20
	
2 0 0 2 10
	
11 108 64
	
66 3 1
inc-instr-type 53 54 29
	
5 8 2 19
	
18 82 59
	
58 35 1
perp-inc-cat 69 61 38
	
0 4 0 0 19
	
27 51 55
	
62 31 1 1
perp-ind-id 83 53 22
	
4 6 3 4 21
	
51 60 29
	
45 4 0
perp-org-id 53 39 21
	
0 4 0 0 14
	
28 74 40
	
54 3 6
perp-org-conf 53 39 15 9 0 1 14
	
28 74 29
	
40 36 3
phys-tgt-id 67 67 28
	
3 9 3 3 27
	
27 81 44
	
44 4 0
phys-tgt-type 68 67 23
	
4 13 1 3 27
	
28 81 37
	
37 40 2
phys-tgt-num 68 67 28
	
11 1 9 27
	
28 81 49
	
50 4 0
phys-tgt-nation 2 0 0
	
0 0 0 0 0
	
2 138 0
	
• • 0
phys-tgt-effect 41 40 13
	
6 4 1 6 17
	
18 103 39
	
40 42 2
phys-tgt-total-num 0 0 0
	
0 0 0 0 0
	
0 13 9
hum-tgt-name 57 37 25 1 1 10
	
30 88 45
	
69 2 7
hum-tgt-desc 133 123 56
	
15 9 8 15 43
	
53 31 48
	
52 3 5
hum-tgt-type 142 125 57
	
11 10 2 9 47
	
64 26 44
	
50 38 3
hum-tgt-num 143 129 62
	
14 6 4 12 47
	
61 26 48
	
53 3 6
hum-tgt-nation 16 4 2
	
0 0 0 0 2
	
14 125 12
	
50 50 0
bum-tgt-effect 121 116 53
	
11 5 2 10 47
	
52 37 48
	
50 40 3
hum-tgt-total-num 1 0 0
	
0 0 0 0 0
	
1 138 0
	
.
inc-total 543 505 300
	
61 19 8 21 125
	
163 193 61
	
65 2 5
perp-total 258 192 96
	
5 23 3 5 68
	
134 259 38
	
51 3 5
phys-tgt-total 246 241 92
	
24 27 6 21 98
	
103 623 42
	
43 4 1
hum-tat-total 613 534 255
	
52 31 17 47 196
	
275 471 46
	
53 3 7
MATCHED/MISSING 1660 1135 743
	
142 100 34 94 150
	
675 1252 49
	
72 1 3
MATCHED/SPURIOUS 1225 1472 743
	
142 100 34 94 487
	
240 1135 66
	
5S 3 3
MATCHED ONLY 1225 1135 743
	
142 100 34 94 150
	
240 841 66
	
72 1 3
ALL TEMPLATES 1660 1472 743
	
142 100 34 94 487
	
675 1546 49
	
55 3 3
SET FILLS ONLY 795 551 389
	
39 48 14 31 75
	
319 582 51
	
74 14 0
STRING FILLS ONLY 426 274 172
	
25 29 15 25 48
	
200 352 43
	
67 1 8
TEXT FILTERING 6 8
P&R
6 6
2P&R
5 9
P&2R
7
	
9 25 87
	
89 11 2 2
F-MEASURES 51 .83 53 .69
	
50 .09
Table G3. TST3 Scores for GE-CMU
SLOT POS ACT COR
	
PAR INC ICR IPA SPU
	
MIS NON REC
	
PRE OVG FA L
template-id 112 106 63
	
0 0 0 0 43
	
49 0 56
	
59 4 0
inc-date 109 101 22
	
15 24 22 15 40
	
48 6 27
	
29 4 0
inc-loc 112 87 11
	
39 4 0 17 33
	
58 10 27
	
35 3 8
inc-type 112 106 55
	
8 0 0 0 43
	
49 0 53
	
56 40 4
inc-stage 112 106 59
	
0 4 0 0 43
	
49 0 53
	
56 40 1 3
inc-instr-id 33 14 5 0 1 1 8
	
27 127 17
	
39 5 7
inc-instr-type 52 14 4
	
0 2 0 0 8
	
46 109 8
	
28 57 0
perp-inc-cat 69 101 28
	
0 10 0 0 63
	
31 23 40
	
28 62 3 0
perp-ind-id 85 87 12
	
5 19 2 5 51
	
49 35 17
	
17 5 9
perp-org-id 52 52 12
	
0 7 1 0 33
	
33 72 23
	
23 6 3
perp-org-conf 52 52 4
	
2 13 0 2 33
	
33 72 10
	
10 63 5
phys-tgt-id 66 112 13
	
2 10 0 2 87
	
41 74 21
	
12 7 8
phys-tgt-type 66 112 10
	
4 11 0 3 87
	
41 74 18
	
11 78 4
phys-tgt-num 67 122 13
	
7 5 0 7 97
	
42 74 25
	
14 8 0
phys-tgt-nation 2 0 0
	
0 0 0 0 0
	
2 154 0
	
• • 0
phys-tgt-effect 39 112 6
	
6 2 0 5 98
	
25 82 23
	
8 88 1 0
phys-tgt-total-num 0 39 0
	
0 0 0 0 39
	
0 116 •
	
0 100
hum- tgt-name 57 173 22
	
5 9 1 5 137
	
21 68 43
	
14 7 9
hum-tgt-desc 132 222 29
	
24 17 1 24 152
	
62 35 31
	
18 6 8
hum-tgt-type 146 371 35
	
16 32 1 13 288
	
63 23 29
	
12 78 1 7
hum-tgt-num 146 389 35
	
32 16 1 26 306
	
63 23 35
	
13 7 9
hum-tip-nation 16 0 0
	
0 0 0 0 0
	
16 143 0
	
• • 0
hum-tgt-effect 124 386 35
	
20 11 1 18 320
	
58 26 36
	
12 83 2 0
hum-tgt-total-num 1 33 0
	
0 0 0 0 33
	
1 121 0
	
0 10 0
inc-total 530 428 156
	
63 34 23 33 175
	
277 252 35
	
44 4 1
perp-total 258 292 56
	
7 49 3 7 180
	
146 202 23
	
20 6 2
phys-tgt-total 240 497 42
	
19 28 0 17 408
	
151 574 21
	
10 8 2
hum-tat-total 622 1574 156
	
97 85 5 86 1236
	
284 439 33
	
13 7 8
MATCHED/MISSING 1650 1818 410
	
186 196 31 143 1026
	
858 1017 30
	
28 5 6
MATCHED/SPURIOUS 919 2791 410
	
186 196 31 143 1999
	
127 936 55
	
18 7 2
MATCHED ONLY 919 1818 410
	
186 196 31 143 1026
	
127 486 55
	
28 5 6
ALL TEMPLATES 1650 2791 410
	
186 196 31 143 1999
	
858 1467 30
	
18 7 2
SET FILLS ONLY 790 879 236
	
56 85 2 41 502
	
413 491 33
	
30 57 2
STRING FILLS ONLY 425 434 93
	
37 62 6 37 242
	
233 283 26
	
26 5 6
TEXT FILTERING 6 9
P&R
9 9
2P&R
68
	
•
P&2R
31
	
1 0 98
	
69 31 10 0
F-MEASURES 22 .5 19 .57
	
26.47
Table G4. TST3 Scores for HUGHE S
G-3
LOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON REC
	
PRE OVG FA L
template-id 112 312 62
	
0
	
0 0
	
0 250
	
50 12 55
	
20 8 0
inc-date 108 298 25
	
23
	
14 0
	
23 236
	
46 18 34
	
12 7 9
inc-loc 112 274 23
	
29
	
7 0
	
0 215
	
53 35 33
	
14 7 8
inc-type 112 312 54
	
8
	
0 0
	
0 250
	
50 0 52
	
18 80 1 0
inc-stage 112 246 54
	
0
	
3 0
	
0 189
	
55 61 48
	
22 77 2 0
inc-instr-id 33 10 2
	
0
	
0 0
	
0 8
	
31 329 6
	
20 8 0
inc-instr-type 52 49 2
	
3
	
0 0
	
0 44
	
47 275 7
	
7 90 0
perp-inc-cat 68 312 31
	
0
	
13 0
	
0 268
	
24 26 46
	
10 86 4 3
perp-ind-id 80 55 7
	
3
	
7 0
	
3 38
	
63 259 11
	
15 6 9
perp-org-id 47 8 1
	
1
	
0 0
	
1 6
	
45 311 3
	
19 7 5
perp-org-conf 54 312 0
	
15
	
21 0
	
0 276
	
18 35 14
	
2 88 1 4
phys-tgt-id 71 94 12
	
9
	
6 0
	
9 67
	
44 248 23
	
18 7 1
phys-tgt-type 71 90 3
	
0
	
20 0
	
0 67
	
48 248 4
	
3 74 2
phys-tgt-num 71 48 9
	
2
	
1 1
	
1 36
	
59 280 14
	
21 7 5
phys-tgt-nation 2 2 0
	
0
	
0 0
	
0 2
	
2 359 0
	
0 100 0
phys-tgt-effect 40 46 2
	
1
	
1 0
	
0 42
	
36 291 6
	
5 91 2
phys-tgt-total-num 0 0 0
	
0
	
0 0
	
0 0
	
0 362 a
hum-tgt-name 61 16 10
	
3
	
0 0
	
3 3
	
48 313 19
	
72 1 9
hum-tgt-desc 127 63 18
	
14
	
9 1
	
14 22
	
86 252 20
	
40 3 5
hum-tgt-type 137 59 16
	
5
	
16 0
	
5 22
	
100 247 14
	
31 37 1
hum-tgt-num 138 65 23
	
17
	
3 0
	
14 22
	
95 247 23
	
48 3 4
hum-tgt-nation 17 2 0
	
0
	
0 0
	
0 2
	
17 347 0
	
0 100 0
hum-tgt-effect 113 31 15
	
3
	
0 0
	
2 13
	
95 271 1S
	
53 42 0
hum-tgt-total-num 1 0 0
	
0
	
0 0
	
0 0
	
1 361 0
inc-total 529 1189 160
	
63
	
24 0
	
23 942
	
282 718 36
	
16 7 9
perp-total 249 687 39
	
19
	
41 0
	
4 588
	
150 631 19
	
7 8 6
phys-tgt-total 255 280 26
	
12
	
28 1
	
10 214
	
189 1788 12
	
11 7 6
hum-tat-total 594 236 82
	
42
	
28 1
	
38 84
	
442 2038 17
	
44 3 6
MATCHED/MISSING 1627 614 307
	
136
	
121 2
	
75 50
	
1063 1203 23
	
61 8
MATCHED/SPURIOUS 971 2392 307
	
136
	
121 2
	
75 1828
	
407 4601 39
	
16 7 6
MATCHED ONLY 971 614 307
	
136
	
121 2
	
75 50
	
407 629 39
	
61 8
ALL TEMPLATES 1627 2392 307
	
136
	
121 2
	
75 1828
	
1063 5175 23
	
16 7 6
SET FILLS ONLY 778 333 177
	
35
	
74 0
	
7 47
	
492 538 25
	
58 14 0
STRING FILLS ONLY 419 105 50
	
30
	
22 1
	
30 3
	
317 353 16
	
62 3
TEXT FILTERING 69 84 63
	
•
	
• •
	
• 21
	
6 12 91
	
75 25 6 4
P&R 2P&R
	
P&2 R
F-MEASURES 18 .87 17 .04 2L1 5
Table G5 . TST3 Scores for LS I
SLAT POS ACT COR
	
PAR
	
INC ICR
	
IPA S PU
	
MIS N ON REC
	
PRE OVG FAL
template-id 107 111 47
	
0
	
0 0
	
0 64
	
60 21 44
	
42 5 8
inc-date 104 104 24
	
12
	
6 0
	
12 62
	
62 5 29
	
29 6 0
lac-toe 107 91 13
	
26
	
2 0
	
6 50
	
66 14 24
	
28 5 5
inc-type 107 111 42
	
5
	
0 0
	
0 64
	
60 0 42
	
40 58 6
inc-stage 107 111 44
	
0
	
3 0
	
0 64
	
60 0 41
	
40 58 1 7
inc-instr-id 33 7 2
	
1
	
0 0
	
1 4
	
30 142 8
	
36 5 7
inc-instr-type 52 7 2
	
1
	
0 0
	
0 4
	
49 124 5
	
36 57 0
perp-inc-cat 67 111 30
	
0
	
3 0
	
0 78
	
34 26 45
	
27 70 3 0
perp-ind-id 80 49 8
	
2
	
6 0
	
2 33
	
64 78 11
	
18 6 7
perp-org-id 48 17 6
	
1
	
1 0
	
1 9
	
40 116 14
	
38 5 3
perp-org-conf 48 17 4
	
1
	
3 0
	
1 9
	
40 116 9
	
26 53 1
phys-tgt-id 63 38 10
	
2
	
4 0
	
2 22
	
47 113 17
	
29 5 8
phys-tgt-type 63 38 9
	
3
	
4 0
	
1 22
	
47 113 17
	
28 58 1
phys-tgt-num 64 37 9
	
3
	
4 0
	
3 21
	
48 113 16
	
28 5 7
phys-tgt-nation 2 0 0
	
0
	
0 0
	
0 0
	
2 170 0
	
• • 0
phys-tgt-effect 3 9 7 2
	
1
	
0 0
	
1 4
	
36 140 6
	
36 57 0
phys-tgt-total-num 0 0 0
	
0
	
0 0
	
0 0
	
0 17 1
hum-tgt-name 55 29 4
	
2
	
6 0
	
2 17
	
43 112 9
	
17 5 9
hum-tgt-desc 123 71 10
	
18
	
13 1
	
17 30
	
82 62 15
	
27 4 2
hum-tgt-type 133 82 12
	
24
	
4 0
	
20 42
	
93 49 18
	
29 51 2
hum-tgt-num 134 81 10
	
19
	
11 0
	
17 41
	
94 50 14
	
24 5 1
hum-tgt-nation 15 4 0
	
1
	
0 0
	
1 3
	
14 157 3
	
12 75 0
hum -tgt-effect 116 49 9
	
16
	
1 0
	
14 23
	
90 76 15
	
35 47 1
hum-tgt-total-num 1 0 0
	
0
	
0 0
	
0 0
	
1 170 0
	
• •
inc-total 510 431 127
	
45
	
11 0
	
19 248
	
327 285 29
	
35 5 8
perp-total 243 194 48
	
4
	
13 0
	
4 129
	
178 336 20
	
26 6 6
phys-tgt-total 231 120 30
	
9
	
12 0
	
7 69
	
180 820 15
	
29 5 8
hum-tat-total 577 316 45
	
80
	
35 1
	
71 156
	
417 676 15
	
27 4 9
MATCHED/MISSING 1561 530 250
	
138
	
71 101 71
	
1102 1157 20
	
60 1 3
MATCHED/SPURIOUS 780 1061 250
	
138
	
71 101 602
	
321 1416 41
	
30 5 7
MATCHED ONLY 780 530 250
	
138
	
71 101 71
	
321 456 41
	
60 1 3
ALL TEMPLATES 1561 1061 250
	
138
	
71 101 602
	
1102 2117 20
	
30 5 7
SET FILLS ONLY 749 260 154
	
52
	
18 0
	
38 36
	
525 536 24
	
69 14 0
STRING FILLS ONLY 402 115 40
	
26
	
30 25 19
	
306 329 13
	
46 1 6
TEXT FILTERING 68 58 47 • 11
	
21 21 69
	
81 19 3 4
P&R 2P&R
	
P&2R
F-MEASURES 24 .0 27 .27
	
2L43
Table G6. TST3 Scores for MDES C
G— 4
SLOT POS ACT COR
	
PAR
	
INC KR
	
IPA SPU
	
MIS
	
NON REC
	
PRE
	
OVG FAL
template-id 109 373 37
	
0
	
0 0
	
0 336
	
72
	
13 34
	
10
	
9 0
inc-date 105 359 20
	
9
	
7 3
	
9 323
	
69
	
17 23
	
7
	
9 0
inc-loc 109 195 6
	
11
	
5 0
	
2 173
	
87
	
163 10
	
6
	
8 9
inc-type 109 373 29
	
8
	
0 0
	
0 336
	
72
	
0 30
	
9
	
90 1 1
inc-stage 109 373 34
	
0
	
3 0
	
0 336
	
72
	
0 31
	
9
	
90 2 8
inc-instr-id 33 121 2
	
1
	
0 0
	
1 118
	
30
	
339 8
	
2
	
9 8
inc-instr-type 52 83 1
	
1
	
0 0
	
0 81
	
50
	
321 3
	
2
	
98 1
perp-inc-cat 67 373 31
	
0
	
3 0
	
0 339
	
33
	
39 46
	
8
	
91 4 2
perp-ind-id 82 147 15
	
6
	
4 2
	
6 122
	
57
	
259 22
	
12
	
8 3
perp-org- id 53 145 20
	
2
	
1 0
	
2 122
	
30
	
274 40
	
14
	
8 4
perp-org-conf 53 145 14
	
1
	
8 0
	
1 122
	
30
	
274 27
	
10
	
84 5
phys-tgt-id 65 0 0
	
0
	
0 0
	
0 0
	
65
	
400 0
	
•
	
•
phys-tgt-type 65 0 0
	
0
	
0 0
	
0 0
	
65
	
400 0
	
• 0
phys-tgt-num 66 0 0
	
0
	
0 0
	
0 0
	
66
	
400 0
	
•
	
•
phys-tgt-nation 2 0 0
	
0
	
0 0
	
0 0
	
2
	
444 0
	
'
	
• 0
phys-tgt-effect 39 0 0
	
0
	
0 0
	
0 0
	
39
	
417 0
	
•
	
. 0
phys-tgt-total-num 0 0 0
	
0
	
0 0
	
0 0
	
0
	
44 5
hum-tgt-name 54 0 0
	
0
	
0 0
	
0 0
	
54
	
404 0
	
.
	
.
hum-tgt-desc 119 0 0
	
0
	
0 0
	
0 0
	
119
	
363 0
	
•
	
.
hum-tgt-type 129 0 0
	
0
	
0 0
	
0 0
	
129
	
358 0
	
•
	
• 0
hum-tgt-num 130 0 0
	
0
	
0 0
	
0 0
	
130
	
358 0
	
•
	
*
hum-tgt-nation 15 0 0
	
0
	
0 0
	
0 0
	
15
	
434 0
	
•
	
. 0
hum-tgt-effect 109 0 0
	
0
	
0 0
	
0 0
	
109
	
370 0
	
•
	
• 0
hum-tgt-total-num 1 0 0
	
0
	
0 0
	
0 0
	
1
	
444 0
inc-total 517 1504 92
	
30
	
15 3
	
12 1367
	
380
	
840 21
	
7
	
9 1
perp-total 255 810 80
	
9
	
16 2
	
9 705
	
150
	
846 33
	
10
	
8 7
phys-tgt-total 237 0 0
	
0
	
0 0
	
0 0
	
237
	
2506 0
hum-tat-total 557 0 0
	
0
	
0 0
	
0 0
	
557
	
2731 0
MATCHED/MISSING 1566 266 172
	
39
	
31 5
	
21 24
	
1324
	
1208 12
	
72
	
9
MATCHED/SPURIOUS 592 2314 172
	
39
	
31 5
	
21 2072
	
350
	
6079 32
	
8
	
9 0
MATCHED ONLY 592 266 172
	
39
	
31 5
	
21 24
	
350
	
364 32
	
72
	
9
ALL TEMPLATES 1566 2314 172
	
39
	
31 5
	
21 2072
	
1324
	
6923 12
	
8
	
9 0
SET FILLS ONLY 749 145 109
	
10
	
14 0
	
1 12
	
616
	
563 15
	
79
	
8 0
STRING FILLS ONLY 406 63 37
	
9
	
5 2
	
9 12
	
355
	
338 10
	
66
	
1 9
TEXT FILTERING 68 80 61
	
•
	
' 19
	
7
	
13 90
	
76
	
24 5 9
P&R 2P&R
	
P&2R
F-MEASURES 9 .6 8 .57 10 .9 1
Table G7. TST3 Scores for MITRE
SLOT POS ACT COR
	
PAR
	
INC KR
	
IPA SPU
	
MIS
	
NON REC
	
PRE
	
OVG FA L
template-id 110 135 58
	
0
	
0 0
	
0 77
	
52
	
20 53
	
43
	
5 7
inc-date 105 131 42
	
9
	
6 0
	
9 74
	
48
	
8 44
	
35
	
5 6
inc-loc 110 128 10
	
37
	
8 0
	
9 73
	
55
	
4 26
	
22
	
5 7
inc-type 110 135 48
	
10
	
0 0
	
0 77
	
52
	
0 48
	
39
	
57 6
inc-stage 110 135 57
	
0 0
	
0 77
	
52
	
0 52
	
42
	
57 1 7
inc-instr-id 33 47 14 0
	
1 31
	
17
	
131 44
	
31
	
6 6
inc-instr-type 54 47 13
	
3 0
	
0 30
	
37
	
112 27
	
31
	
64 1
perp-inc-cat 70 135 34
	
0
	
7 0
	
0 94
	
29
	
23 48
	
25
	
70 3 3
perp-ind-id 83 14 0
	
2 0
	
1 11
	
80
	
113 1
	
4
	
7 8
perp-org-id 53 65 13
	
0
	
9 1
	
0 43
	
31
	
94 24
	
20
	
6 6
perp-org-conf 51 0 0
	
0
	
0 0
	
0 0
	
51
	
138 0
	
•
	
• 0
phys-tgt-id 71 76 12
	
5
	
10 1
	
5 49
	
44
	
91 20
	
19
	
6 4
phys-tgt-type 71 76 10
	
9
	
8 0
	
5 49
	
44
	
91 20
	
19
	
64 2
phys-tgt-num 70 0 0
	
0
	
0 0
	
0 0
	
70
	
142 0
	
.
phys-tgt-nation 2 0 0
	
0
	
0 0
	
0 0
	
2
	
186 0
	
•
	
• 0
phys-tgt-effect 40 63 0
	
9 0
	
0 53
	
30
	
105 1
	
1
	
84 5
phys-tgt-total-num 0 0 0
	
0
	
0 0
	
0 0
	
0
	
18 7
hum-tgt-name 56 52 14
	
4
	
2 2
	
4 32
	
36
	
112 28
	
31
	
6 2
hum-tgt-desc 126 90 7
	
9
	
22 1
	
9 52
	
88
	
50 9
	
13
	
5 8
hum-tgt-type 136 88 14
	
7
	
17 1
	
6 50
	
98
	
47 13
	
20
	
57 3
hum-tgt-num 135 0 0
	
0
	
0 0
	
0 0
	
135
	
98 0
hum-tgt-nation 18 0 0
	
0
	
0 0
	
0 0
	
18
	
175 0
	
•
	
• 0
hum-tgt-effect 113 140 6
	
26
	
13 0
	
5 95
	
68
	
36 17
	
14
	
68 6
hum-tgt-total-num 1 0 0
	
0
	
0 0
	
0 0
	
1
	
186 0
inc-total 522 623 184
	
60
	
17 0
	
19 362
	
261
	
255 41
	
34
	
5 8
perp-total 257 214 47
	
18 1
	
1 148
	
191
	
368 18
	
22
	
6 9
phys-tgt-total 254 215 22
	
15
	
27 1
	
10 151
	
190
	
802 12
	
14
	
7 0
hum-tat-total 585 370 41
	
46
	
54 4
	
24 229
	
444
	
704 II
	
17
	
6 2
MATCHED/MISSING 1618 666 294
	
122
	
116 6
	
54 134
	
1086
	
1108 22
	
53
	
2 0
MATCHED/SPURIOUS 927 1422 294
	
122
	
116 6
	
54 890
	
395
	
1513 38
	
25
	
6 2
MATCHED ONLY 927 666 294
	
122
	
116 6
	
54 134
	
395
	
492 38
	
53
	
2 0
ALL TEMPLATES 1618 1422 294
	
122
	
116 6
	
54 890
	
1086
	
2129 22
	
25
	
6 2
SET FILLS ONLY 775 380 182
	
56
	
56 1
	
16 86
	
481
	
499 27
	
55
	
23 0
STRING FILLS ONLY 422 174 60
	
20
	
46 5
	
20 48
	
296
	
299 16
	
40
	
2 8
TEXT FILTERING 6 6
P&R
6 9
2P&R
5 5
P&2R
14
	
I1
	
20 83
	
80
	
20 4 1
F-MEASURES 23 .4 24 .34
	
22 .54
Table G8. TST3 Scores for NMSU-BRANDEI S
G- 5
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS
	
NON REC PRE OVG FA L
template-id 107 114 71
	
0
	
0 0
	
0 43
	
36
	
24 66 62 3 8inc-date 103 110 37
	
22
	
10 0
	
22 41
	
34
	
6 47 44 3 7inc-loc
107 113 32
	
29
	
9 0
	
0 43
	
37
	
0 43 41 3 8inc-type 107
114 61
	
10
	
0 0
	
0 43
	
36
	
0 62 58 38 4inc-stage
107 114 69
	
0
	
2 0
	
0 43
	
36
	
0 64 60 38 1 3inc-instr-id 32
13 8
	
1
	
0 0 4
	
23
	
122 26 65 3 1
inc-instr-type 51 17 10
	
1
	
0 0
	
0 6
	
40
	
103 20 62 35 0perp-inc-cat 68
70 39
	
0
	
4 0
	
0 27
	
25
	
55 57 56 38 1 3
perp-ind-id 85 73 25
	
8
	
9 10
	
8 31
	
43
	
51 34 40 4 2perp-org-id
53 65 26
	
3
	
6 4
	
3 30
	
18
	
70 52 42 4 6perp-org-conf 51 31
11
	
4
	
8 0
	
0 8
	
28
	
94 25 42 26 2phya-tgt-id
65 38 12
	
3
	
4 4
	
3 19
	
46
	
90 21 36 5 0phys-tgt-type 66
38 11
	
1
	
7 2
	
0 19
	
47
	
90 17 30 50 1phys-tgt-num 66 38 14
	
2
	
3 2
	
2 19
	
47
	
90 23 39 5 0phys-tgt-nation
2 0 0
	
0
	
0 0
	
0 0
	
2
	
149 0 • • 0phys-tgt-effect
39 17 6
	
1
	
0 10
	
32
	
115 17 38 59 1
phys-tgt-total-num 0 0 0
	
0
	
0 0
	
0 0
	
0
	
15 0hum-tgt-name 57 45 20
	
3
	
2 0
	
3 20
	
32
	
90 38 48 4 4hum-tgt-desc
124 104 44
	
12
	
12 10
	
11 36
	
56
	
42 40 48 3 5hum-tgt-type 135 126 51
	
16
	
12 11 47
	
56
	
29 44 47 37 3hum-tgt-num
137 133 49
	
23
	
14 0
	
20 47
	
51
	
29 44 45 3 5hum-tgt-nation 15 9 2
	
0
	
1 0
	
0 6
	
12
	
133 13 22 67 0hum-tgt-effect
113 112 46
	
15
	
3 12 48
	
49
	
50 47 48 43 3hum-tgt-total-num 1 0 0
	
0
	
0 0
	
0 0
	
1
	
149 0 • •inc-total
507 481 217
	
63
	
21 0
	
23 180
	
206
	
231 49 52 3 7
perp-total 257 239 101
	
15
	
27 14
	
11 96
	
114
	
270 42 45 4 0phya-tgt-total
238 131 43
	
7
	
14 9
	
6 67
	
174
	
684 20 35 5 1hum-tat-total 582 529 212
	
69
	
44 12
	
57 204
	
257
	
522 42 46 3 8
MATCHED/MISSING 1584 926 573
	
154
	
106 35
	
97 93
	
751
	
1142 41 70 1 0
MATCHED/SPURIOUS 1150 1380 573
	
154
	
106 35
	
97 547
	
317
	
1273 56 47 4 0MATCHED ONLY 1150
926 573
	
154
	
106 35
	
97 93
	
317
	
708 56 70 1 0
ALL TEMPLATES 1584 1380 573
	
154
	
106 35
	
97 547
	
751
	
1707 41 47 4 0SET FILLS ONLY
754 436 306
	
48
	
37 5
	
24 45
	
363
	
541 44 76 10 0STRING FILLS ONLY 416
233 135
	
30
	
33 28
	
29 35
	
218
	
305 36 64 1 5
TEXT FILTERING 67 64 55 9
	
12
	
24 82 86 14 2 7P&R
2P&R
	
P&2R
F-MEASURES 43 .8 45 .66 42 .0 7
Table G9. TST3 Scores for NYU
SLAT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS
	
NON REC PRE OVG FA L
template-id 115 189 86
	
0
	
0 0
	
0 103
	
29
	
14 75 46 5 4inc-date 111 177
38
	
23
	
23 0
	
23 93
	
27
	
14 44 28 5 2
inc-loc 115 181 22
	
46
	
14 0
	
9 99
	
33
	
4 39 25 5 5inc-type 115 189
79
	
7
	
0 0
	
0 103
	
29
	
0 72 44 54 7inc-stage
115 189 81
	
0
	
5 0
	
0 103
	
29
	
0 70 43 54 2 0inc-instr-id
33 96 18
	
1
	
2 0 75
	
12
	
151 56 19 7 8
inc-instr-type 52 111 18
	
4
	
4 0 85
	
26
	
134 38 18 76 2perp-inc-cat
75 186 42
	
0
	
13 0
	
0 131
	
20
	
18 56 22 70 4 0
perp-ind-id 87 134 18
	
7
	
15 3
	
7 94
	
47
	
78 25 16 7 0perp-org-id
60 188 29
	
3
	
10 5
	
3 146
	
18
	
41 51 16 7 8
perp-org-conf 60 177 7
	
3
	
32 0
	
2 135
	
18
	
41 14 5 76 1 3
phya-tgt-id 67 98 17
	
3
	
10 0
	
3 68
	
37
	
124 28 19 6 9phys-tgt-type 67
98 17
	
3
	
10 0
	
1 68
	
37
	
124 28 19 69 2
phys-tgt-num 68 98 17
	
8
	
5 0
	
7 68
	
38
	
124 31 21 6 9phya-tgt-nation 2 0 0
	
0
	
0 0
	
0 0
	
2
	
217 0 • • 0
phys-tgt-effect 41 95 9
	
8
	
3 0
	
3 75
	
21
	
132 32 14 79 6phys-tgt-total-num 0 0
0
	
0
	
0 0
	
0 0
	
0
	
21 8
bum-tgt-name 56 70 22
	
4
	
3 0
	
4 41
	
27
	
145 43 34 5 8
hum-tgt-desc 135 325 45
	
18
	
30 6
	
17 232
	
42
	
47 40 17 7 1
hum-tgt-type 146 322 46
	
29
	
18 0
	
23 229
	
53
	
47 41 19 71 1 0
hum-tgt-num 147 328 43
	
36
	
20 0
	
32 229
	
48
	
47 41 18 7 0
hum-tgt-nation 16 0 0
	
0
	
0 0
	
0 0
	
16
	
206 0 • • 0
hum-tgt-effect 124 202 39
	
22
	
8 0
	
18 133
	
55
	
95 40 25 66 6
hum-tgt-total-num 1 0 0
	
0
	
0 0
	
0 0
	
I
	
217 0 •
inc-total 541 943 256
	
81
	
48 0
	
34 558
	
156
	
303 55 31 5 9perp-total 282
685 96
	
13
	
70 8
	
12 506
	
103
	
178 36 15 7 4
phys-tgt-total 245 389 60
	
22
	
28 0
	
14 279
	
135
	
939 29 18 7 2hum-tat-total
625 1247 195
	
109
	
79 6
	
94 864
	
242
	
804 40 20 6 9
MATCHED/MISSING 1693 1732 607
	
225
	
225 14
	
154 675
	
636
	
1025 42 42 3 9
MATCHED/SPURIOUS 1274 3264 607
	
225
	
225 14
	
154 2207
	
217
	
1907 56 22 6 8
MATCHED ONLY 1274 1732 607
	
225
	
225 14
	
154 675
	
217
	
708 56 42 3 9
ALL TEMPLATES 1693 3264 607
	
225
	
225 14
	
154 2207
	
636
	
2224 42 22 6 8
SET FILLS ONLY 813 827 338
	
76
	
93 0
	
48 320
	
306
	
466 46 45 39 2
STRING FILLS ONLY 438 489 149
	
36
	
70 14
	
35 234
	
183
	
264 38 34 4 8
TEXT FILTERING 6 8
P&R
7 8
2P&R
60
	
•
	
•
P&2R
18
	
8
	
14 88 77 23 5 6
F-MEASURES 28 .88 24 .32
	
35 .54
Table G10 . TST3 Scores for PARAMA X
G- 6
SLot POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON REC PRE OVG FA L
template-id 107 104 62
	
0
	
0 0
	
0 42
	
45 26 58 60 4 0inc-date
103 101 34
	
17
	
9 0
	
17 41
	
43 5 41 42 4 0inc-loc 107 104 26
	
35
	
1 0
	
3 42
	
45 0 41 42 4 0inc-type
107 104 53
	
9
	
0 0
	
0 42
	
45 0 54 55 40 4
inc-stage 107 104 60
	
0
	
2 0
	
0 42
	
45 0 56 58 40 1 3inc-instr-id
33 5 0
	
0
	
0 0
	
0 5
	
33 119 0 0 10 0
inc-instr-type 51 8 2
	
2
	
0 0
	
2 4
	
47 103 6 38 50 0
perp-inc-cat 64 40 24
	
0
	
3 0
	
0 13
	
37 72 38 60 32 7
perp-ind-id 80 39 15
	
4
	
2 2
	
4 18
	
59 76 21 44 4 6
perp-org-id 47 16 6
	
0
	
2 3
	
0 8
	
39 97 13 38 5 0perp-org-conf 47 16 3
	
2
	
3 0
	
2 8
	
39 97 8 25 50 1phys-tgt-id
64 40 19
	
0
	
2 1
	
0 19
	
43 88 30 48 4 8
phys-tgt-type 64 40 9
	
1
	
12 0
	
1 18
	
42 89 15 24 45 1
phys-tgt-num 65 43 19
	
1
	
2 0
	
1 21
	
43 86 30 45 4 9phys-tgt-nation
2 1 0
	
0
	
0 0
	
0 1
	
2 147 0 0 100 0
phys-tgt-effect 39 20 4
	
6
	
1 0
	
2 9
	
28 112 18 35 45 1
phys-tgt-total-num 0 0 0
	
0
	
0 0
	
0 0
	
0 14 9hum-tgt-name 55 6
1
	
3
	
1 0
	
3 1
	
50 106 4 42 1 7
hum-tgt-desc 122 97 21
	
12
	
17 1
	
12 47
	
72 30 22 28 4 8
hum-tgt-type 132 89 20
	
9
	
16 0
	
6 44
	
87 28 18 28 49 3
hum-tgt-num 133 97 24
	
17
	
10 1
	
13 46
	
82 27 24 34 4 7hum-tgt-nation
16 5 2
	
0
	
0 0
	
0 3
	
14 134 12 40 60 0
hum-tgt-effect 113 67 22
	
10
	
2 0
	
7 33
	
79 45 24 40 49 2
hum-tgt-total-num 1 0 0
	
0
	
0 0
	
0 0
	
1 148 0 • •
inc-total 508 426 175
	
63
	
12 0
	
22 176
	
258 227 41 48 4 1
perp-total 238 111 48
	
6
	
10 5
	
6 47
	
174 342 21 46 4 2phys-tgt-total 234 144
51
	
8
	
17 1
	
4 68
	
158 671 24 38 4 7
hum-tgt-total 572 361 90
	
51
	
46 2
	
41 174
	
385 518 20 32 4 8
MATCHED/MISSING 1552 641 364
	
128
	
85 8
	
73 64
	
975 1166 28 67 1 0MATCHED/SPURIOUS
1009 1042 364
	
128
	
85 8
	
73 465
	
432 1217 42 41 4 5
MATCHED ONLY 1009 641 364
	
128
	
85 8
	
73 64
	
432 625 42 67 1 0
ALL TEMPLATES 1552 1042 364
	
128
	
85 8
	
73 465
	
975 1758 28 41 4 5SET FILLS ONLY 742
303 199
	
39
	
39 0
	
20 26
	
465 546 29 72 8 0
STRING FILLS ONLY 401 129 62
	
19
	
24 7
	
19 24
	
296 328 18 55 1 9TEXT FILTERING 66
5 6 48 8
	
18 26 73 86 14 2 4
P&R 2P&R
	
P&2R
F-MEASURES 33 .28 37 .52 29 . 9
Table G11 . TST3 Scores for PR C
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON RFC PRE OVG FA L
template-id 107 132 58
	
0
	
0 0
	
0 74
	
49 27 54 44 5 6
inc-date 103 118 25
	
16
	
14 0
	
16 63
	
48 15 32 28 5 3inc-loc
107 131 21
	
31
	
5 0
	
1 74
	
50 0 34 28 5 6
inc-type 107 132 52
	
6
	
0 0
	
0 74
	
49 0 51 42 56 6inc-stage 107
132 52
	
0
	
6 0
	
0 74
	
49 0 48 39 56 1 8inc-instr-id 33 14 8
	
2
	
0 1
	
2 4
	
23 152 27 64 2 8inc-instr-type
52 25 10
	
5
	
0 0
	
1 10
	
37 128 24 50 40 0perp-inc-cat 63
90 33
	
0
	
2 0
	
0 55
	
28 63 52 37 61 1 9
perp-ind-id 78 38 10
	
8
	
3 1
	
8 17
	
57 102 18 37 4 5perp-org-id 49 74 17
	
0
	
10 0
	
0 47
	
22 89 35 23 6 4
perp-org-conf 49 74 12
	
7
	
8 0
	
7 47
	
22 89 32 21 64 5phys-tgt-id 65
50 17
	
1
	
5 4
	
1 27
	
42 109 27 35 5 4phys-tgt-type
66 50 15
	
2
	
6 2
	
1 27
	
43 109 24 32 54 1phys-tgt-num
66 45 16
	
4
	
0 2
	
3 25
	
46 111 27 40 5 6phys-tgt-nation 2
0 0
	
0
	
0 0
	
0 0
	
2 180 0 • • 0
phys-tgt-effect 40 15 3
	
3
	
0 0
	
1 9
	
34 143 11 30 60 1phys-tgt-total-aum 0 0
0
	
0
	
0 0
	
0 0
	
0 18 1
hum-tgt-name 55 12 7
	
1
	
1 0
	
1 3
	
46 136 14 62 2 5
hum-tgt-desc 120 72 14
	
4
	
10 2
	
4 44
	
92 54 13 22 6 1
hum-tgt-type 130 75 14
	
10
	
8 0
	
7 43
	
98 50 15 25 57 2hum-tgt-num 131 75
18
	
9
	
6 0
	
7 42
	
98 51 17 30 5 6
hum-tgt-nation 15 3 1
	
0
	
1 0
	
0 1
	
13 169 7 33 33 0hum-tgt-effect 110
66 13
	
8
	
0 0
	
7 45
	
89 61 15 26 68 2
hum-tgt-total-num 1 0 0
	
0
	
0 0
	
0 0
	
1 180 0inc-total 509 552
168
	
60
	
25 1
	
20 299
	
256 295 39 36 5 4
perp-total 239 276 72
	
15
	
23 1
	
15 166
	
129 343 33 29 6 0phys-tgt-total
239 160 51
	
10
	
11 8
	
6 88
	
167 833 23 35 5 5
hum-tat-total 562 303 67
	
32
	
26 2
	
26 178
	
437 701 15 27 5 9
MATCHED/MISSING 1549 609 358
	
117
	
85 12
	
67 49
	
989 1151 27 68 8
MATCHED/SPURIOUS 892 1291 358
	
117
	
85 12
	
67 731
	
332 1591 47 32 5 7MATCHED ONLY
892 609 358
	
117
	
85 12
	
67 49
	
332 570 47 68 8
ALL TEMPLATES 1549 1291 358
	
117
	
85 12
	
67 731
	
989 2172 27 32 5 7SET FILLS ONLY 741
304 205
	
41
	
31 2
	
24 27
	
464 536 30 74 9 0
STRING FILLS ONLY 400 136 73
	
16
	
29 8
	
16 18
	
282 321 20 60 1 3TEXT FILTERING 65 54
46
	
"
	
• •
	
" 8
	
19 27 71 85 15 2 3
P&R 2P&R
	
P&2R
F-MEASURES 29 .29 30 .86
	
27 .87
Table G12 . TST3 Scores for SRA
G- 7
SLOT POS ACT COR
	
PAR 1NC ICR
	
IPA SPU
	
MIS NON REC
	
PRE OVG FA L
template-id 110 105 80
	
0 0 0
	
0 25
	
30 24 73
	
76 2 4
inc-date 107 105 48
	
13 18 0
	
13 26
	
28 2 51
	
52 2 5
inc-loc 110 105 46
	
26 8 0
	
3 25
	
30 0 54
	
56 2 4
inc-type 110 105 77
	
3 0 0
	
0 25
	
30 0 71
	
75 24 3
inc-stage 110 103 78
	
0 1 0
	
0 24
	
31 1 71
	
76 23 8
inc-instr-id 33 22 12
	
2 0 0
	
2 8
	
19 104 39
	
59 3 6
inc-instr-type 52 38 15
	
9 2 0
	
1 '12
	
26 83 38
	
51 32 0
perp-inc-cat 65 47 23
	
0 8 0
	
0 16
	
34 54 3 .5
	
49 34 1 2
perp-ind-id 85 48 .23
	
2 5 1
	
2 18
	
55 60 28
	
50 3 8
perp-org-id 52 48 18
	
0 5 2
	
0 25
	
29 75 35
	
38 5 2
perp-org-conf 52 50 11
	
6 7 0
	
4 26
	
28 74 27
	
28 52 4
phys-tgt-id 67 49 23
	
1 6 '0
	
1 19
	
37 74 35
	
48 3 9
phys-tgt-type 67 49 19
	
2 9 0
	
2 19
	
37 74 30
	
41 39 1
phys-tgt-num 67 49 22
	
5 3 0
	
4 19
	
37 74 36
	
50 3 9
phys-tgt-nation 2 0 0
	
0 0 0
	
0 0
	
2 134 0
	
• • 0
phys-tgt-effect 41 23 10
	
2 2 0
	
1 9
	
27 98 27
	
48 39 1
phys-tgt-total-num 0 0 0
	
0 0 0
	
0 0
	
0 13 5
bum-tgt-name 64 40 22
	
5 2 0
	
5 11
	
35 81 38
	
61 2 8
hum-tgt-desc 133 111 50
	
21 13 6
	
20 ,27
	
49 30 45
	
54 2 4
hum-tgt-type 144 111 50
	
18 16 0
	
12 27
	
60 28 '41
	
53 24 2
hum-tgt-num 145 116 55
	
24 10 '2
	
19 27
	
56 28 46
	
58 2 3
hum-tgt-nation 20 4 2
	
0 0 0
	
0 2
	
18 121 10
	
50 50 0
hum-tgt-effect 121 85 42
	
14 1 1
	
10 28
	
64 40 '40
	
58 33 2
hum-tgt-total-num 1 0 0
	
0 0 0
	
0 0
	
1 134 0
	
• •
inc-total 522 478 276
	
53 29 0
	
19 120
	
164 190 58
	
63 2 5
perp-total 254 193 75
	
8 25 3
	
6 85
	
146 263 31
	
41 4 4
phys-tgt-total 244 170 74
	
10 20 0
	
8 66
	
140 589 32
	
46 3 9
hum-tgt-total 628 467 221
	
82 42 9
	
66 122
	
283 462 42
	
56 2 6
MATCHED/MISSING 1648 1055 646
	
153 1.16 12
	
99 140
	
733 1171 44
	
68 1 3
MATCHED/SPURIOUS 1230 1308 646
	
153 116 12
	
99 393
	
315 1165 59
	
55 3 0
MATCHED ONLY 1230 1055 646
	
153 116 12
	
99 140
	
315 832 59
	
68 1 3
ALL TEMPLATES 1648 1308 646
	
153 116 12
	
99 393
	
733 1504 44
	
55 3 0
SET FILLS ONLY 784 494 327
	
54 46 1
	
30 67
	
357 549 45
	
72 14 0
STRING FILLS ONLY 434 264 148
	
31 31 9
	
30 54
	
224 322 38
	
62 2 0
TEXT FILTERING 68 69 61
	
• • • 8
	
7 24 90
	
88 12 2 5
P&R 2P&R P&2R
F-MEASURES 48 .89 52 .38 45 .8 3
Table G13. TST3 Scores for SRI
SLOT POS ACT COR
	
PAR INC ICR
	
IPA SPU
	
MIS NON REC
	
PRE OVG FA L
template-id 103 41 10
	
0 0 0
	
0 31
	
93 30 10
	
24 7 6
inc-date 99 0 0
	
0 0 0
	
0 0
	
99 35 0
	
•
inc-loc 103 1 0
	
0 0 0
	
0 1
	
103 30 0
	
0 10 0
inc-type 103 41 8
	
2 0 0
	
0 31
	
93 0 9
	
22 76 4
inc-stage 103 41 10
	
0 0 0
	
0 31
	
93 0 10
	
24 76 1 0
inc-instr-id 33 8 0
	
0 0 0
	
0 8
	
33 101 0
	
0 10 0
inc-instr-type 51 8 0
	
0 0 0
	
0 8
	
51 84 0
	
0 100 0
perp-inc-cat 60 20 7
	
0 0 0
	
0 13
	
53 61 12
	
35 65 6
perp-ind-id 75 20 2
	
1 2 0
	
1 1S
	
70 60 3
	
12 7 5
perp-org-id 45 20 4
	
0 3 0
	
0 13
	
38 79 9
	
20 6 5
perp-org-conf 44 0 0
	
0 0 0
	
0 0'
	
44 92 0
	
• • 0
phys-tgt-id 62 3 0 0 0
	
0 2
	
61 90 2
	
33 6 7
phys-tgt-type 62 6 0
	
0 3 0
	
0 3
	
59 89 0
	
0 5 0
phys-tgt-num
phys-tgt-nation
6 3
2
0
0
0
	
0
0
	
0
0
0
0
	
0
0
	
0
0
	
6 3
0
	
2
9 2
133
0
	
•
0
	
•
•
• 0
phys-tgt-effect 38 0 0
	
0 0 0
	
0 0
	
38 107 0
	
• • 0
phys-tgt-total-num 0 0 0
	
0 0 0
	
0 0
	
0 13 4
hum-tgt-name 54 0 0
	
0 0 0
	
0 0
	
54 93 0
	
• `
hum-tgt-desc 118 6 4 1 1
	
4 0
	
112 53 2
	
50 0
hum-tgt-type 128 6 0
	
2 3 0
	
0 1
	
123 47 1
	
17 17 0
hum-tgt-num 129 0 0
	
0 0 0
	
0 0
	
129 48 0
	
• •
hum-tgt-nation 15 0 0
	
0 0 0
	
0 0
	
15 123 0
	
• • 0
hum-tgt-effect 109 0 0
	
0 0 0
	
0 0
	
109 59 0
	
. • 0
hum-tgt-total-num 1 0 0
	
0 0 0
	
0 0
	
1 133 0
inc-total 492 99 18
	
2 0 0
	
0 79
	
472 250 4
	
19 8 0
perp-total 224 60 13
	
1 5 0
	
1 41
	
205 292 6
	
22 6 8
phys-tgt-total 227 9 0 3 0
	
0 5
	
223 645 0
	
11 5 6
hum-t t-total 554 12 6 4 1
	
4 1
	
543 556 1
	
33 8
MATCHED/MISSING 1497 57 33
	
9 12 1
	
5 3
	
1443 1153 2
	
66 5
MATCHED/SPURIOUS 177 180 33
	
9 12 I
	
5 126
	
123 686 21
	
21 7 0
MATCHED ONLY 177 57 33
	
9 12 1
	
5 3
	
123 96 21
	
66 5
ALL TEMPLATES 1497 180 33
	
9 12 1
	
5 126
	
1443 1743 2
	
21 7 0
SET FILLS ONLY 715 36 25
	
4 6 0
	
0 1
	
680 540 4
	
75 3 0
STRING FILLS ONLY 387 21
	
. 8
	
5 6 1
	
5 2
	
368 326 3
	
50 1 0
TEXT FILTERING 6 5
P&R
2 6
2P&R
2 1
P&2R
5
	
44 30 32
	
81 19 1 4
F-MEASURES 3 .65 7 .24 2 .44
Table G14. TST3 Scores for UMD-CONQUES T
G-8
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON REC PRE OVG FA L
template-id 109 95 79
	
0
	
0 0
	
0 16
	
30 29 72 83 1 7
inc-date 106 93 53
	
11
	
13 1
	
11 16
	
29 3 55 63 1 7
inc-loc 109 95 39
	
37
	
3 0
	
3 16
	
30 0 53 60 1 7
inc-type 109 95 75
	
4
	
0 0
	
0 16
	
30 0 71 81 17 2
inc-stage 109 95 78
	
0
	
1 0
	
0 16
	
30 0 72 82 17 6
inc-instr-id 33 27 17
	
1
	
1 0
	
1 8
	
14 96 53 65 3 0
inc-instr-type 51 38 22
	
3
	
3 D
	
0 10
	
23 77 46 62 26 0
perp-inc-cat 66 45 30
	
0
	
4 D
	
0 11
	
32 48 45 67 24 8
perp-ind-id 80 45 21
	
2
	
11 1
	
2 11
	
46 50 28 49 2 4
perp-org-id 53 26 16
	
0
	
5 1
	
0 5
	
32 72 30 62 1 9
perp-org-conf 53 26 9
	
3
	
9 D
	
2 5
	
32 72 20 40 19 2
phys-tgt-id 67 49 22
	
1
	
9 D
	
1 17
	
35 67 34 46 3 5
phys-tgt-type 67 49 19
	
2
	
11 o
	
2 17
	
35 67 30 41 35 1
phys-tgt-num 68 49 22
	
7
	
3 D
	
6 17
	
36 67 38 52 3 5
phys-tgt-nation 2 0 0
	
0
	
0 0 0
	
2 124 0 • • 0
phys-tgt-effect 40 25 11
	
6
	
0 0
	
5 8
	
23 92 35 56 32 1
phys-tgt-total-num 0 0 0
	
0
	
0 0
	
0 0
	
0 125 • • r
hum-tgt-name 57 40 17
	
3
	
4 1
	
3 16
	
33 70 32 46 4 0
hum-tgt-desc 126 125 52
	
12
	
22 1
	
11 39
	
40 26 46 46 3 1
hum-tgt-type 138 133 56
	
19
	
21 1
	
17 37
	
42 22 47 49 28 4
hum-tgt-num 138 137 63
	
23
	
14 6
	
20 37
	
38 22 54 54 2 7
hum-tgt-nation 15 6 2
	
0
	
0 0
	
0 4
	
13 110 13 33 67 0
hum-tgt-effect 114 112 54
	
13
	
7 1
	
11 38
	
40 30 53 54 34 3
hum-tgt-total-num 1 0 0
	
0
	
0 0
	
0 0
	
1 124 0 *
inc-total 517 443 284
	
56
	
21 1
	
15 82
	
156 176 60 70 1 8
perp-total 252 142 76
	
5
	
29 2
	
4 32
	
142 242 31 55 2 2
phys-tgt-total 244 172 74
	
16
	
23 0
	
14 59
	
131 542 34 48 3 4
hum-tgt-total 589 553 244
	
70
	
68 10
	
62 171
	
207 404 47 50 3 1
MATCHED/MISSING 1602 1121 678
	
147
	
141 13
	
95 155
	
636 1155 47 67 1 4
MATCHED/SPURIOUS 1246 1310 678
	
147
	
141 13
	
95 344
	
280 1007 60 57 2 6
MATCHED ONLY 1246 1121 678
	
147
	
141 13
	
95 155
	
280 798 60 67 1 4
ALL TEMPLATES 1602 1310 678
	
147
	
141 13
	
95 344
	
636 1364 47 57 2 6
SET FILLS ONLY 764 533 356
	
50
	
56 2
	
37 71
	
302 543 50 71 13 0
STRING FILLS ONLY 416 270 145
	
19
	
52 4
	
18 54
	
200 317 37 57 2 0
TEXT FILTERING 6 7
P&R
6 5
2P&R
6 1
P&2R
4
	
6 29 91 94 6 1 2
F-MEASURES 51 .52 54 .67
	
48 .7 1
Table G15 .
	
TST3 Scores for UMASS
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON REC PRE OVG FAL
template-id 104 109 70
	
0
	
0 0
	
0 39
	
34 20 67 64 3 6
inc-date 100 102 42
	
18
	
8 0
	
18 34
	
32 9 51 50 3 3
inc-loc 104 102 27
	
30
	
9 0
	
2 36
	
38 3 40 41 3 5
inc-type 104 109 66
	
4
	
0 0
	
0 39
	
34 0 65 62 36 4
inc-stage 104 109 66
	
0
	
4 0
	
0 39
	
34 0 63 60 36 1 3
inc-instr-id 33 8 4
	
0
	
0 0
	
0 4
	
29 114 12 50 5 0
inc-instr-type 51 51 11
	
15
	
1 0
	
4 24
	
24 77 36 36 47 1
perp-inc-cat 67 109 42
	
0
	
7 0
	
0 60
	
18 16 63 38 55 3 1
perp-ind-id 76 35 11
	
1
	
7 0
	
1 16
	
57 66 15 33 4 6
perp-org-id 49 62 20
	
I
	
6 0
	
1 35
	
22 63 42 33 5 6
perp-org-conf 49 62 13
	
4
	
10 0
	
4 35
	
22 63 31 24 56 5
phys-tgt-id 63 45 22
	
2
	
4 1
	
2 17
	
35 89 36 51 3 8
phys-tgt-type 63 45 19
	
2
	
7 0
	
0 17
	
35 89 32 44 38 1
phys-tgt-num 63 45 21
	
4
	
3 0
	
2 17
	
35 89 36 51 3 8
phys-tgt-nation 2 0 0
	
0
	
0 0
	
0 0
	
2 142 0 * • 0
phys-tgt-effect 40 14 10
	
1
	
0 0
	
1 3
	
29 113 26 75 21 0
phys-tgt-total-num 0 0 0
	
0
	
0 0
	
0 0
	
0 143 * •
hum-tgt-name 55 29 20
	
1
	
0 1
	
1 8
	
34 94 37 71 2 8
hum-tgt-desc 122 174 41
	
15
	
23 2
	
15 95
	
43 20 40 28 5 4
hum-tgt-type 132 183 45
	
21
	
19 0
	
18 98
	
47 16 42 30 54 7
hum-tgt-num 133 190 46
	
25
	
18 2
	
23 101
	
44 16 44 31 5 3
hum-tgt-nation 15 8 0
	
0
	
3 0
	
0 5
	
12 128 0 0 62 0
hum-tgt-effect 114 106 31
	
11
	
12 0
	
9 52
	
60 46 32 34 49 4
hum-tgt-total-num 1 0 0
	
0
	
0 0
	
0 0
	
1 142 0
inc-total 496 481 216
	
67
	
22 0
	
24 176
	
191 203 50 52 3 6
perp-total 241 268 86
	
6
	
30 0
	
6 146
	
119 208 37 33 5 4
phys-tgt-total 231 149 72
	
9
	
14 1
	
5 54
	
136 665 33 51 3 6
hum-tat-total 572 690 183
	
73
	
75 5
	
66 359
	
241 462 38 32 5 2
MATCHED/MISSING 1540 1103
	
557
	
155
	
141
	
6
	
101
	
250
	
687 1071
	
41 58 2 3
MATCHED/SPURIOUS 1117 1588 557
	
155
	
141 6
	
101 735
	
264 1135 57 40 4 6
MATCHED ONLY 1117 1103 557
	
155
	
141 6
	
101 250
	
264 668 57 58 2 3
ALL TEMPLATES 1540 1588 557
	
155
	
141 6
	
101 735
	
687 1538 41 40 4 6
SET FILLS ONLY 741 549 303
	
58
	
63 0
	
36 125
	
317 488 45 60 23 1
STRING FILLS ONLY 398 249 118
	
20
	
40 4
	
20 71
	
220 299 32 51 2 8
TEXT FILTERING 68 74 61 13
	
7 19 90 82 18 4 1
P&R 2P&R
	
P&2 R
F-MEASURES 40 .49 40 .2
	
40 .8
Table G16. TST3 Scores for UMIC H
G-9
SLOT POS ACT COR PAR INC ICR IPA SPU MIS NON REC PRE OVG FAL
template-id 102 77 16 0 0 0 0 61 86 20 16 21 7 9
inc-date 98 72 11 3 1 3 3 57 83 8 13 17 7 9
inc-loc 102 14 0 4 2 0 0 8 96 53 2 14 5 7
inc-type 102 77 15 1 0 0 0 61 86 0 15 20 79 6
inc-stage 102 77 14 0 2 0 0 61 86 0 14 18 79 1 6
inc-instr-id 33 30 2 0 0 0 0 28 31 123 6 7 9 3
inc-instr-type 52 37 3 3 0 0 0 31 46 102 9 12 84 1
perp-inc-cat 58 70 8 0 4 0 0 58 46 47 14 11 83 2 3
perp-ind-id 76 29 3 1 3 0 1 22 69 80 5 12 7 6
perp-org-id 43 62 7 1 3 1 1 51 32 71 17 12 8 2
perp-org-conf 43 62 0 8 3 0 0 51 32 71 9 6 82 6
phys-tgt-id 61 19 6 0 0 0 0 13 55 113 10 32 6 8
phys-tgt-type 61 19 4 0 2 0 0 13 55 113 6 21 68 0
phys-tgt-num 62 19 6 0 0 0 0 13 56 113 10 32 6 8
phys-tgt-nation 2 0 0 0 0 0 0 0 2 162 0 • • 0
phys-tgt-effect 38 2 0 1 1 0 0 0 36 136 1 25 0 0
phys-tgt-total-num 0 0 0 0 0 0 0 0 0 16 3
hum-tgt-name 54 3 1 1 1 0 1 0 51 122 3 50 0
hum-tgt-desc 118 13 1 0 3 0 0 9 114 74 1 8 6 9
hum-tgt-type 128 .16 1 3 3 0 2 9 121 69 2 16 56 1
bum-tgt-num 129 16 2 3 2 0 3 9 122 69 3 22 5 6
hum-tgt-nation 15 0 0 0 0 0 0 0 15 152 0 • ' 0
hum-tgt-effect 109 0 0 0 0 0 0 0 109 88 0 • • 0
hum-tgt-total-num 1 0 0 0 0 0 0 0 1 162 0 • *
inc-total 489 307 45 11 5 3 3 246 428 286 10 16 8 0
perp-total 220 223 18 10 13 1 2 182 179 269 10 10 8 2
phys-tgt-total 224 59 16 1 3 0 0 39 204 800 7 28 6 6
hum-tat-total 554 48 5 7 9 0 6 27 533 736 2 18 5 6
MATCHED/MISSING 1487 174 84 29 30 4 11 31 1344 1129 7 57 1 8
MATCHED/SPURIOUS 322 637 84 29 30 4 11 494 179 1109 30 15 7 8
MATCHED ONLY 322 174 84 29 30 4 11 31 179 147 30 57 1 8
ALL TEMPLATES 1487 637 84 29 30 4 11 494 1344 2091 7 15 7 8
SET FILLS ONLY 710 89 45 16 15 0 2 13 634 531 7 60 15 0
STRING FILLS ONLY 385 48 20 3 10 1 3 15 352 316 6 45 3 1
TEXT FILTERING 6 7 43 30 ' • • 13 37 20 45 70 30 3 9
P&R 2P&R
	
P&2R
F-MEASURES 9 .55 12 .21
	
7 .84
Table G17. TST3 Scores for USC
SECTION 2: TST4
Note : The text filtering row has been inserted from the NRaD-scored file name d
scores.tst4-pass2 .nrad .
passl .nrad.
All other rows are from the NRaD-scored file named scores .tst4 -
SLOT POS ACT COR
	
PAR INC ICR
	
IPA SPU
	
MIS
	
NON REC PRE OVG FAL
template-id 74 91 53
	
0 0 0
	
0 38
	
21
	
33 72 58 4 2
inc-date 74 91 35
	
9 9 0
	
9 38
	
21
	
0 53 43 4 2
inc-loc 74 89 17
	
34 2 0
	
9 36
	
21
	
2 46 38 4 0
inc-type 74 91 50
	
3 0 0
	
0 38
	
21
	
0 70 56 42 5
inc-stage 74 91 51
	
0 2 0
	
0 38
	
21
	
0 69 56 42 1 5
inc-instr-id 33 30 13
	
1 1 0 15
	
18
	
72 41 45 5 0
inc-instr-type 52 46 17
	
2 5 0 22
	
28
	
47 35 39 48 1
perp-inc-cat 56 78 36
	
0 3 0
	
0 39
	
17
	
17 64 46 50 2 5
perp-ind-id 58 72 25
	
2 7 2 38
	
24
	
21 45 36 5 3
perp-org-id 50 35 24
	
0 1 0
	
0 10
	
25
	
60 48 68 2 8
perp-org-conf 50 35 16
	
1 8 0 10
	
25
	
60 33 47 28 3
phys-tgt-id 60 43 20
	
5 2 5 16
	
33
	
53 38 52 3 7
phys-tgt-type 60 43 16
	
3 8 0 16
	
33
	
53 29 41 37 1
phys-tgt-num 60 43 18
	
6 3 0 16
	
33
	
53 35 49 3 7
phys-tgt-nation 2 0 0
	
0 0 0
	
0 0
	
2
	
110 0 • • 0
phys-tgt-effect 43 23 9
	
3 4 0
	
0 7
	
27
	
72 24 46 30 2
phys-tgt-total-num 0 0 0
	
0 0 0
	
0 0
	
0
	
11 2
hum-tgt-name 31 19 4
	
2 1 0
	
2 12
	
24
	
78 16 26 6 3
hum-tgt-desc 70 49 14
	
3 7 6
	
2 25
	
46
	
42 22 32 5 1
hum-tgt-type 74 60 14
	
8 4 3
	
6 34
	
48
	
33 24 30 57 3
hum-tgt-num 76 61 16
	
5 6 4
	
3 34
	
49
	
33 24 30 5 6
hum-tgt-nation 5 0 0
	
0 0 0
	
0 0
	
5
	
108 0 • • 0
hum-tgt-effect 71 60 18
	
8 5 4
	
6 29
	
40
	
38 31 37 48 3
hum-tgt-total-num 6 0 0
	
0 0 0
	
0 0
	
6
	
106 0 •
inc-total 381 438 183
	
49 19 0
	
20 187
	
130
	
121 54 47 4 3
perp-total 214 220 101
	
3 19 3 97
	
91
	
158 48 46 4 4
phys-tgt-total 225 152 63
	
17 17 7 55
	
128
	
453 32 47 3 6
hum-tat-total 333 249 66
	
26 23 17
	
19 134
	
218
	
438 24 32 5 4
MATCHED/MISSING 1153 639 413
	
95 78 19
	
49 53
	
567
	
716 40 72 8
MATCHED/SPURIOUS 871 1059 413
	
95 78 19
	
49 473
	
285
	
938 53 43 4 5
MATCHED ONLY 871 639 413
	
95 78 19
	
49 53
	
285
	
484 53 72 8
ALL TEMPLATES 1153 1059 413
	
95 78 19
	
49 473
	
567
	
1170 40 43 4 5
SET FILLS ONLY 561 322 227
	
28 39 7
	
15 28
	
267
	
325 43 75 9 0
STRING FILLS ONLY 302 151 100
	
13 19 8
	
12 19
	
170
	
195 35 70 1 2
TEXT FILTERING 52 58 43 15
	
9
	
33 83 74 26 3 1
P&R 2P&R
	
P&2 R
F-MEASURES 41 .45 42 .36
	
40 .57
Table G18 . TST4 Scores for BB N
SLOT POS ACT COR
	
PAR INC ICR
	
IPA SPU
	
MIS
	
NON REC PRE OVG FA L
template-id 81 98 69
	
0 0 0
	
0 29
	
12
	
35 85 70 3 0
inc-date 81 98 46
	
15 8 15 29
	
12
	
0 66 54 3 0
inc-loc 81 98 37
	
31 1 0 29
	
12
	
0 65 54 3 0
inc-type 81 98 66
	
3 0 0
	
0 29
	
12
	
0 83 69 30 4
inc-stage 81 98 65
	
0 4 0
	
0 29
	
12
	
0 80 66 30 1 3
inc-instr-id 34 29 21
	
0 1 0
	
0 7
	
12
	
79 62 72 2 4
inc-instr-type 54 43 27
	
2 1 0
	
0 13
	
24
	
55 52 65 30 0
perp-inc-cat 61 73 40
	
0 3 0
	
0 30
	
18
	
19 66 55 41 2 1
perp-ind-id 56 70 27
	
6 3 2
	
6 34
	
20
	
39 54 43 4 8
perp-org-id 55 38 28
	
2 0 4
	
2 8
	
25
	
55 53 76 2 1
perp-org-conf 55 39 16
	
3 11 0
	
0 9
	
25
	
55 32 45 23 3
phys-tgt-id 67 100 45
	
3 3 2
	
3 49
	
16
	
49 69 46 4 9
phys-tgt-type 67 99 35
	
3 13 2 48
	
16
	
49 54 37 48 3
phys-tgt-num 67 99 46
	
2 3 2 48
	
16
	
49 70 47 4 8
phys-tgt-nation 2 1 1
	
0 0 0
	
0 0
	
1
	
108 50 100 0 0
phys-tgt-effect 48 61 27
	
4 4 2 26
	
13
	
67 60 48 43 4
phys-tgt-total-num 0 0 0
	
0 0 0
	
0 0
	
0
	
11 0
hum-tgt-name 35 26 21
	
1 0 2 4
	
13
	
80 61 83 1 5
hum-tgt-desc 77 101 47
	
4 5 4
	
4 45
	
21
	
37 64 48 4 4
hum-tgt-type 80 107 48
	
4 5 4 50
	
23
	
37 62 47 47 4
hum-tgt-num 83 107 48
	
4 5 0
	
3 50
	
26
	
37 60 47 4 7
hum-tgt-nation 6 3 2
	
0 0 0
	
0 1
	
4
	
104 33 67 33 0
hum-tgt-effect 74 92 39
	
7 4 6 42
	
24
	
42 57 46 46 4
hum-tgt-total-num 6 0 0
	
0 0 0
	
0 0
	
6
	
104 0
inc-total 412 464 262
	
51 15 16 136
	
84
	
134 70 62 2 9
perp-total 227 220 111
	
11 17 6
	
8 81
	
88
	
168 51 53 3 7
phys-tgt-total 251 360 154
	
12 23 6
	
8 171
	
62
	
432 64 44 4 8
hum-tat-total 361 436 205
	
20 19 8
	
18 192
	
117
	
441 60 49 4 4
MATCHED/MISSING 1251 1114 732
	
94 74 21
	
50 214
	
351
	
810 62 70 1 9
MATCHED/SPURIOUS 1092 1480 732
	
94 74 21
	
50 580
	
192
	
1043 71 53 3 9
MATCHED ONLY 1092 1114 732
	
94 74 21
	
50 214
	
192
	
678 71 70 1 9
ALL TEMPLATES 1251 1480 732
	
94 74 21
	
50 580
	
351
	
1175 62 53 3 9
SET FILLS ONLY 609 537 366
	
26 45 4
	
14 100
	
172
	
369 62 70 19 1
STRING FILLS ONLY 324 280 189
	
16 12 14
	
16 63
	
107
	
226 61 70 2 2
TEXT FILTERING 55 6 3
	
53
	
• 10
	
2
	
35
	
96 84 16 2 2
P&R 2P&R
	
P&2 R
F-MEASURES 57 .15 54 .58
	
59 .96
Table G19 . TST4 Scores for G E
G-1 1
SLOT' POS ACT COR PAR INC ICR
	
IPA SPU MIS NON REC
	
PRE
	
OVG FA L
template-id 77 80 58 0 0 0
	
0 22 19 39 75
	
72
	
2 8
inc-date 77 80 36 14 8 1
	
14 22 19 0 56
	
54
	
2 8
inc-loc 77 80 26 31 0
	
1 22 19 0 54
	
52
	
2 8
inc-type 77 80 53 5 0 0
	
0 22 19 0 72
	
69
	
28 4
inc-stage 77 80 56 0 2 0
	
0 22 19 0 73
	
70
	
28 1 1
inc-instr-id 32 25 17 0 2 0
	
0 6 13 70 53
	
68
	
2 4
inc-instr-type 51 36 21 3 2 4
	
2 10 25 47 44
	
62
	
28 0
perp-inc-cat 57 50 31 0 2 0
	
0 17 24 25 54
	
62
	
34 1 3
perp-ind-id 54 49 19 5 6 0
	
5 19 24 37 40
	
44
	
3 9
perp-org-id 51 26 17 3
	
1 7 32 51 34
	
67
	
2 7
perp-org-conf 51 27 13 2 4 0
	
0 8 32 50 27
	
52
	
30 2
phys-tgt-id 61 78 37 4 3 2
	
4 34 17 43 64
	
50
	
4 4
phys-tgt-type 61 77 30 3 11 1
	
2 33 17 43 52
	
41
	
43 3
phys-tgt-num 61 77 38 3 3 2
	
1 33 17 43 65
	
51
	
4 3
phys-tgt-nation 2 1 1 0 0 0
	
0 0 1 97 50
	
100
	
0 0
phys-tgt-effect 45 51 24 6 0 0
	
3 21 15 58 60
	
53
	
41 3
phys-tgt-total-num 0 0 0 0 0 0
	
0 0 0 9 9
hum-tgt-name 32 19 15 0 0
	
1 3 16 72 48
	
82
	
1 6
hum-tgt-desc 71 83 33 7 7 3
	
7 36 24 36 51
	
44
	
4 3
hum-tgt-type 74 84 34 7 4 1
	
7 39 29 36 51
	
45
	
46 4
hum-tgt-num 77 84 34 5 6 0
	
4 39 32 36 47
	
43
	
4 6
hum-tgt-nation 6 2 2 0 0 0
	
0 0 4 94 33
	
100
	
0 0
hum-tgt-effect 71 82 34 8 3 1
	
7 37 26 36 54
	
46
	
45 4
hum-tgt-total-num 5 0 0 0 0 0
	
0 0 5 94 0
inc-total 391 381 209 53 15 5
	
17 104 114 117 60
	
62
	
2 7
perp-total 213 152 80 8 13 3
	
6 51 112 163 39
	
55
	
3 4
phys-tgt-total 230 284 130 16 17 5
	
10 121 67 383 60
	
48
	
4 3
hum-tat-total 336 354 152 28 20 5
	
26 154 136 404 49
	
47
	
4 4
MATCHED/MISSING 1170 910 571 105 65 18
	
59 169 429 783 53
	
68
	
1 8
MATCHED/SPURIOUS 907 1171 571 105 65 18
	
59 430 166 866 69
	
53
	
3 7
MATCHED ONLY 907 910 571 105 65 18
	
59 169 166 582 69
	
68
	
1 8
ALL TEMPLATES 1170 1171 571 105 65 18
	
59 430 429 1067 53
	
53
	
3 7
SET FILLS ONLY 572 441 299 34 28 7
	
21 80 211 356 55
	
72
	
18 0
STRING FILLS ONLY 301 225 138 18 19 8
	
18 50 126 219 49
	
65
	
2 2
TEXT FILTERING 5 4
P&R
5 4
2P&R
4 7
P&2R
• 7 7 39 87
	
87
	
13 1 5
F-MEASURES 53 .0 53 .0
	
53 . 0
Table G20 . TST4 Scores for GE-CMU
SLOT POS ACT COR PAR INC ICR
	
IPA SPU MIS NON REC
	
PRE
	
OVG FA L
template-id 76 102 53 0 0 0
	
0 49 23 2 70
	
52
	
4 8
inc-date 76 102 30 8 15 30
	
8 49 23 0 45
	
33
	
4 8
inc-loc 76 77 1 40 2 0
	
8 34 33 15 28
	
27
	
4 4
inc-type 76 102 45 8 0 0
	
0 49 23 0 64
	
48
	
48 6
inc-stage 76 102 50 0 3 0
	
0 49 23 0 66
	
49
	
48 1710 0
inc-instr-id 31 22 11 0 0
	
0 10 19 96 35
	
50
	
4 5
inc-instr-type 50 22 8 0 7 0
	
0 7 35 80 16
	
36
	
32 0
perp-inc-cat 58 94 35 0 5 0
	
0 54 18 13 60
	
37
	
57 3 1
perp-ind-id 59 82 12 6 16 0
	
6 48 25 23 25
	
18
	
5 8
perp-org-id 49 33 12 2 3 1
	
2 16 32 68 26
	
39
	
4 8
perp-org-conf 49 33 3 13 0
	
0 16 32 68 7
	
11
	
48 4
phys-tgt-id 55 89 15 3 10 1
	
3 61 27 56 30
	
18
	
6 8
phys-tgt-type 55 89 5 6 17 0
	
5 61 27 56 14
	
9
	
68 4
phys-tgt-num 56 102 12 13 4 0
	
10 73 27 56 33
	
18
	
7 2
pbys-tgt-nation 2 0 0 0 0 0
	
0 0 2 123 0
	
•
	
• 0
phys-tgt-effect 38 89 6 8 7 0
	
7 68 17 62 26
	
11
	
76 1 0
phys-tgt-total-num 0 33 0 0 0 0
	
0 33 0 92 •
	
0
	
100
hum-tgt-name 33 175 10 4 6 0
	
4 155 13 51 36
	
7
	
8 8
hum-tgt-desc 75 120 20 9 9 0
	
9 82 37 47 33
	
20
	
6 8
hum-tgt-type 79 286 24 13 12 0
	
10 237 30 30 39
	
11
	
83 1 7
hum-tgt-num 80 292 22 17 10 2
	
12 243 31 30 38
	
10
	
8 3
hum-tgt-nation 5 0 0 0 0 0
	
0 0 5 121 0
	
•
	
• 0
hum-tgt-effect 73 289 19 12 13 0
	
9 245 29 31 34
	
9
	
85 2 0
hum-tgt-total-num 6 41 0 0 0 0
	
0 41 6 78 0
	
0
	
10 0
inc-total 385 427 145 56 28 30
	
16 198 156 191 45
	
40
	
4 6
perp-total 215 242 62 9 37 1
	
8 134 107 172 31
	
27
	
5 5
phys-tgt-total 206 402 38 30 38 1
	
25 296 100 445 26
	
13
	
7 4
hum-tat-total 351 1203 95 55 50 2
	
44 1003 151 388 35
	
10
	
8 3
MATCHED/MISSING 1157 1261 340 150 153 34
	
93 618 514 645 36
	
33
	
4 9
MATCHED/SPURIOUS 803 2274 340 150 153 34
	
93 1631 160 955 52
	
18
	
7 2
MATCHED ONLY 803 1261 340 150 153 34
	
93 618 160 404 52
	
33
	
4 9
ALL TEMPLATES 1157 2274 340 150 153 34
	
93 1631 514 1196 36
	
18
	
7 2
SET FILLS ONLY 561 612 195 48 77 0
	
31 292 241 314 39
	
36
	
48 2
STRING FILLS ONLY 302 294 80 24 45 2
	
24 145 153 179 30
	
31
	
4 9
TEXT FILTERING 5 6
P&R
9 8
2P&R
5 6
P&2R
42 0 2 100
	
57
	
43 9 5
F-MEASURES 24 .0 20 .0
	
30 .0
Table G21 . TST4 Scores for HUGHE S
G—1 2
SLOT POS ACT COR
	
PAR INC KR
	
IPA SPU
	
MIS NON REC PRE OVG FAL
template-id 74 278 40
	
0 0 0
	
0 238
	
34 18 54 14 8 6
inc-date 74 278 12
	
16 12 0
	
16 238
	
34 0 27 7 8 6
inc-loc 74 265 14
	
24 1 0
	
0 226
	
35 12 35 10 8 5
inc-type 74 278 36
	
4 0 0
	
0 238
	
34 0 51 14 86 1 1
inc-stage 74 216 32
	
0 2 0
	
0 182
	
40 56 43 15 84 2 1
inc-instr-id 30 13 4
	
0 0 0
	
0 9
	
26 280 13 31 6 9
inc-instr-type 49 44 4
	
5 4 0
	
0 31
	
36 239 13 15 70 0
perp-inc-cat 58 278 28
	
0 8 0
	
0 242
	
22 12 48 10 87 4 4
perp-ind-id 56 58 6
	
0 9 0
	
0 43
	
41 218 11 10 7 4
perp-org-id 49 18 0
	
2 3 0
	
2 13
	
44 258 2 6 7 2
perp-org-conf 50 278 0
	
15 13 0
	
0 250
	
22 20 15 3 90 1 4
phys-tgt-id 58 81 10
	
7 3 0
	
7 61
	
38 209 23 17 7 5
phys-tgt-type 58 74 3
	
1 14 0
	
0 56
	
40 214 6 5 76 1
phys-tgt-num 58 42 7
	
3 4 1
	
0 28
	
44 242 15 20 6 7
phys-tgt-nation 2 1 0
	
0 0 0
	
0 1
	
2 309 0 0 100 0
phys-tgt-effect 41 45 4
	
1 5 0
	
0 35
	
31 245 11 10 78 2
phys-tgt-total-num 0 0 0
	
0 0 0
	
0 0
	
0 312 • r r
hum-tgt-name 29 6 2
	
1 0 0
	
1 3
	
26 288 9 42 5 0
hum-tgt-desc 68 42 8
	
6 7 1
	
6 21
	
47 246 16 26 5 0
hum-tgt-type 71 40 9
	
7 3 0
	
5 21
	
52 246 18 31 52 1
hum-tgt-oum 76 40 8
	
6 5 0
	
4 21
	
57 246 14 28 5 2
hum-tgt-nation 5 2 0
	
0 0 0
	
0 2
	
5 306 0 0 100 0
hum-tgt-effect 71 28 8
	
8 0 0
	
5 12
	
55 254 17 43 43 0
hum -tgt-total-num 5 0 0
	
0 0 0
	
0 0
	
5 307 0 • r
inc-total 375 1094 102
	
49 19 0
	
16 924
	
205 587 34 12 8 4
perp-total 213 632 34
	
17 33 0
	
2 548
	
129 508 20 7 8 7
phys-tgt-total 217 243 24
	
12 26 1
	
7 181
	
155 1531 14 12 7 4
hum-tgt-total 325 158 35
	
28 15 1
	
21 80
	
247 1893 15 31 5 1
MATCHED/MISSING 1130 417 195
	
106 93 2
	
46 23
	
736 755 22 59 6
MATCHED/SPURIOUS 658 2127 195
	
106 93 2
	
46 1733
	
264 4142 38 12 8 1
MATCHED ONLY 658 417 195
	
106 93 2
	
46 23
	
264 378 38 59 6
ALL TEMPLATES 1130 2127 195
	
106 93 2
	
46 1733
	
736 4519 22 12 8 1
SET FILLS ONLY 553 234 124
	
41 49 0
	
10 20
	
339 333 26 62 8 0
STRING FILLS ONLY 290 71 30
	
16 22 1
	
16 3
	
222 218 13 54 4
TEXT FILTERING 54 76 48
	
' • •
	
' 28
	
6 18 89 63 37 6 1
P&R 2P&R
	
P&2R
F-MEASURES 15 .53 13 .2
	
18 .8 6
Table G22. TST4 Scores for LS I
SLOT POS ACT COR
	
PAR INC ER
	
IPA SPU
	
MIS NON REC PRE OVG FA L
template-id 73 82 35
	
0 0 0
	
0 47
	
38 36 48 43 5 7
inc-date 73 73 17
	
8 4 0
	
8 44
	
44 3 29 29 6 0
inc-loc 73 68 8
	
23 1 0
	
5 36
	
41 11 27 29 5 3
inc-type 73 82 30
	
5 0 0
	
0 47
	
38 0 44 40 57 6
inc-stage 73 82 35
	
0 0 0
	
0 47
	
38 0 48 43 57 1 6
inc-instr-id 31 7 2
	
0 0 1
	
0 5
	
29 91 6 28 7 1
inc-instr-type 51 7 2
	
0 0 0
	
0 5
	
49 72 4 28 71 0
perp-inc-cat 56 82 26
	
0 3 0
	
0 53
	
27 11 46 32 65 3 0
perp-ind-id 52 49 11
	
1 3 1
	
1 34
	
37 43 22 23 6 9
perp-org-id 48 16 5
	
0 1 0
	
0 10
	
42 71 10 31 6 2
perp-org-conf 48 15 3
	
1 2 0
	
1 9
	
42 71 7 23 60 2
phys-tgt-id 56 34 11
	
8 4 2
	
8 11
	
33 70 27 44 3 2
phys-tgt-typo 56 34 9
	
6 8 0
	
0 11
	
33 70 21 35 32 1
phys-tgt-num 56 34 10
	
7 6 0
	
1 11
	
33 70 24 40 3 2
phys-tgt-nation 2 0 0
	
0 0 0
	
0 0
	
2 118 0 • ~ 0
phys-tgt-effect 38 10 4
	
3 1 0
	
1 2
	
30 87 14 55 20 0
phys-tgt-total-num 0 0 0
	
0 0 0
	
0 0
	
0 12 0
hum-tgt-name 32 23 4
	
3 1 1
	
3 1S
	
24 86 17 24 6 5
hum-tgt-desc 69 40 7
	
9 3 1
	
9 21
	
50 57 17 29 5 2
hum-tgt-type 73 53 8
	
8 6 1
	
S 31
	
51 51 16 23 58 3
hum-tgt-num 75 53 7
	
8 7 0
	
5 31
	
53 51 15 21 5 8
hum-tgt-nation 5 6 0
	
0 0 0
	
0 6
	
5 110 0 0 100 0
hum-tgt-effect 70 33 8
	
5 3 1
	
3 17
	
54 62 15 32 52 2
hum-tgt-total-num 6 0 0
	
0 0 0
	
0 0
	
6 114 0 • •
inc-total 374 319 94
	
36 5 1
	
13 184
	
239 177 30 35 5 8
perp-total 204 162 45
	
2 9 1
	
2 106
	
148 196 22 28 6 5
phys-tgt-total 208 112 34
	
24 19 2
	
10 35
	
131 535 22 41 3 1
hum-tat-total 330 20 8 34
	
33 20 4
	
25 121
	
243 531 15 24 5 8
MATCHED/MISSING 1116 401 207
	
95 53 8
	
50 46
	
761 747 23 63 1 1
MATCHED/SPURIOUS 552 801 207
	
95 53 8
	
50 446
	
197 1040 46 32 5 6
MATCHED ONLY 552 401 207
	
95 53 8
	
50 46
	
197 348 46 63 1 1
ALL TEMPLATES 1116 801 207
	
95 53 8
	
50 446
	
761 1439 23 32 5 6
SET FILLS ONLY 545 200 125
	
28 23 2
	
10 24
	
369 336 26 70 12 0
STRING FILLS ONLY 288 86 40
	
21 12 6
	
21 13
	
215 213 18 59 1 5
TEXT FILTERING 54 47 37
	
• • •
	
• 10
	
17 36 68 79 21 2 2
P&R 2P&R
	
P&2 R
F-MEASURES 26 .76 29 .68
	
24 .37
Table G23 . TST4 Scores for MD C
G-13
SLOT POS ACT COR PAR INC ICR
	
IPA SPU
	
MIS NON REC PRE OVG FA L
template-id 73 366 45 0 0 0
	
0 321
	
28 13 62 12 8 8
inc-date 73 366 20 11 14 6
	
11 321
	
28 0 35 7 8 8inc-loc
73 241 1 22 4 0
	
1 214
	
46 107 16 5 8 9
inc-type 73 366 32 13 0 0
	
0 321
	
28 0 53 10 88 1 2iac-stage
73 366 43 0 2 0
	
0 321
	
28 0 59 12 88 2 9
inc-instr-id 31 120 8 1 2 0
	
1 109
	
20 294 27 7 9 1inc-instr-type
51 86 8 0 3 0
	
0 75
	
40 276 16 9 87 1
perp-inc-cat 57 366 43 0 1 0
	
0 322
	
13 15 75 12 88 4 4perp-ind-id
58 170 27 2 3 1
	
2 138
	
26 203 48 16 8 1
perp-org-id 50 135 19 2 6 3
	
2 108
	
23 244 40 15 8 0perp-org-conf
50 135 16 4 7 0
	
4 108
	
23 244 36 13 80 5
phys-tgt-id 59 0 0 0 0 0
	
0 0
	
59 351 0 ' *phys-tgt-type
59 0 0 0 0 0
	
0 0
	
59 351 0 • ' 0
phys-tgt-num 59 0 0 0 0 0
	
0 0
	
59 351 0
phys-tgt-nation 2 0 0 0 0 0
	
0 0
	
2 392 0 * ' 0
phys-tgt-effect 42 0 0 0 0 0
	
0 0
	
42 361 0 • • 0
phys-tgt-total-num 0 0 0 0 0 0
	
0 0
	
0 39 4
hum-tgt-name 31 0 0 0 0 0
	
0 0
	
31 372 0
hum-tgt-desc 68 0 0 0 0 0
	
0 0
	
68 350 0
hum-tgt-type 71 0 0 0 0 0
	
0 0
	
71 350 0 * • 0
hum-tgt-num 75 0 0 0 0 0
	
0 0
	
75 350 0 * .
hum-tgt-nation 5 0 0 0 0 0
	
0 0
	
5 390 0 ' • 0
hum-tgt-effect 68 0 0 0 0 0
	
0 0
	
68 351 0 • • 0
hum-tgt-total-num 6 0 0 0 0 0
	
0 0
	
6 388 0
inc-total 374 1545 112 47 25 6
	
13 1361
	
190 677 36 9 8 8
perp-total 215 806 105 8 17 4
	
8 676
	
85 706 51 14 8 4
phys-tgt-total 221 0 0 0 0 0
	
0 0
	
221 2200 0
hum-tat-total 324 0 0 0 0 0
	
0 0
	
324 2551 0 r .
MATCHED/MISSING 1134 325 217 55 42 10
	
21 11
	
820 745 22 75 3
MATCHED/SPURIOUS 750 2351 217 55 42 10
	
21 2037
	
436 5805 33 10 8 7
MATCHED ONLY 750 325 217 55 42 10
	
21 11
	
436 416 33 75 3
ALL TEMPLATES 1134 2351 217 55 42 10
	
21 2037
	
820 6134 22 10 8 7
SET FILLS ONLY 551 176 142 17 13 0
	
4 4
	
379 342 27 86 2 0
STRING FILLS ONLY 297 77 54 5 11 4
	
S 7
	
227 204 19 73 9
TEXT FILTERING 5 6
P&R
8 7
2P&R
5 6
P&2R
31
	
0 13 100 64 36 7 0
F-MEASURES 13 .75 11 .22
	
17 .7 4
Table G24 . TST4 Scores for MITRE
SLOT POS ACT COR PAR INC ICR
	
IPA SPU
	
MIS NON REC PRE OVG FA L
template-id 76 144 47 0 0 0
	
0 97
	
29 17 62 33 6 7
inc-date 76 144 26 11 10 1
	
11 97
	
29 0 41 22 6 7
inc-loc 76 140 1 41 5 0
	
9 93
	
29 4 28 15 6 6
inc-type 76 144 37 10 0 0
	
0 97
	
29 0 55 29 67 8
inc-stage 76 144 45 0 2 0
	
0 97
	
29 0 59 31 67 2 2
inc-instr-id 32 54 15 0 1 0
	
0 38
	
16 110 47 28 7 0
inc-instr-type 52 54 IS 1 1 0
	
0 37
	
35 92 30 29 68 1
perp-inc-cat 61 144 34 0 4 0
	
0 106
	
23 6 56 24 74 3 8
perp-ind-id 55 10 0 0 3 0
	
0 7
	
52 118 0 0 7 0
perp-org-id 53 71 7 1 10 0
	
1 53
	
35 75 14 10 7 5
perp-org-conf 52 0 0 0 0 0
	
0 0
	
52 129 0 • • 0
phys-tgt-id 60 72 10 8 12 0
	
8 42
	
30 88 23 19 5 8
ph ya-t gt-type 60 69 11 6 11 1
	
4 41
	
32 88 23 20 59 2
phys-tgt-num 60 0 0 0 0 0
	
0 0
	
60 129 0
phys-tgt-nation 2 0 0 0 0 0
	
0 0
	
2 171 0 • * 0
phys-tgt-effect 44 48 0 0 11 0
	
0 37
	
33 102 0 0 77 4
phys-tgt-total-num 0 0 0 0 0 0
	
0 0
	
0 173 • • .
hum-tgt-name 33 36 6 2 4 I
	
2 24
	
21 125 21 19 6 7
hum-tgt-desc 71 81 8 3 13 0
	
3 57
	
47 70 13 12 7 0
hum-tgt-type 74 81 6 8 10 0
	
6 57
	
50 70 14 12 70 3
hum-tgt-num 78 0 0 0 0 0
	
0 0
	
78 127 0 • .
hum-tgt-nation 6 0 0 0 0 0
	
0 0
	
6 168 0 • . 0
hum-tgt-effect 72 131 1 24 6 0
	
3 100
	
41 42 18 10 76 6
hum-tgt-total-num 6 0 0 0 0 0
	
0 0
	
6 167 0
inc-total 388 680 139 63 19 1
	
20 459
	
167 206 44 25 6 8
perp-total 221 225 41 1 17 0
	
1 166
	
162 328 19 18 7 4
phys-tgt-total 226 189 21 14 34 1
	
12 120
	
157 751 12 15 6 3
hum-tat-total 340 329 21 37 33 1
	
14 238
	
249 769 12 12 7 2
MATCHED/MISSING 1175 502 222 115 103 3
	
47 62
	
735 734 24 56 1 2
MATCHED/SPURIOUS 792 1423 222 115 103 3
	
47 983
	
352 1728 35 20 6 9
MATCHED ONLY 792 502 222 115 103 3
	
47 62
	
352 408 35 56 1 2
ALL TEMPLATES 1175 1423 222 115 103 3
	
47 983
	
735 2054 24 20 6 9
SET FILLS ONLY 575 285 149 49 45 1
	
13 42
	
332 322 30 61 1S 0
STRING FILLS ONLY 304 123 46 14 43 1
	
14 20
	
201 204 17 43 1 6
TEXT FILTERING 5 6
P&R
8 0
2P&R
5 3
P&2R
• •
	
* 27
	
3 17 95 66 34 6 1
F-MEASURES 21 .82 20 .69
	
23 .08
Table G25. TST4 Scores for NMSU/BRANDEIS
G-1 4
SLOT POS ACT COR
	
PAR INC ICR
	
IPA SPU
	
MIS NON REC
	
PRE
	
OVG FAL
template-id 76 96 59
	
0 0 0
	
0 37
	
17 36 78
	
61
	
3 8
inc-date 76 96 32
	
18 9 0
	
18 37
	
17 0 54
	
43
	
3 8
inc-loc 76 95 21
	
34 3 0
	
0 37
	
18 0 50
	
40
	
3 9
inc-type 76 96 48
	
11 0 0
	
0 37
	
17 0 70
	
56
	
38 5
inc-stage 76 96 59
	
0 0 0
	
0 37
	
17 0 78
	
61
	
38 1 4
inc-instr-id 31 10 6
	
0 0 0
	
0 4
	
25 85 19
	
60
	
4 0
inc-instr-type 51 17 8
	
2 0 0
	
1 7
	
41 64 18
	
53
	
41 0
perp-inc-cat 60 61 37
	
0 4 0
	
0 20
	
19 33 62
	
61
	
33 1 4
perp-ind-id 65 81 24
	
5 14 6
	
5 38
	
22 27 41
	
33
	
4 7
perp-org-id 50 48 29
	
0 4 1
	
0 15
	
17 57 58
	
60
	
3 1
perp-org-conf 50 35 20
	
1 4 0
	
0 10
	
25 61 41
	
58
	
28 2
phys-tgt-id 62 55 28
	
5 3 9
	
5 19
	
26 57 49
	
55
	
3 4
ph ys-t gt- t yp e 62 55 19
	
5 12 0
	
3 19
	
26 57 35
	
39
	
34 2
phys-tgt-num 62 55 27
	
5 4 0
	
1 19
	
26 57 48
	
54
	
3 4
phys-tgt-nation 2 1 0
	
0 0 0
	
0 1
	
2 110 0
	
0
	
100 0
phys-tgt-effect 48 33 14
	
6 1 0
	
1 12
	
27 72 35
	
52
	
36 2
phys-tgt-total-num 0 0 0
	
0 0 0
	
0 0
	
0 113 .
	
.
	
.
hum-tgt-name 32 23 12
	
3 2 1
	
3 6
	
15 86 42
	
59
	
2 6
hum-tgt-desc 75 79 25
	
6 6 8
	
6 42
	
38 36 37
	
35
	
5 3
hum-tgt-type 79 89 29
	
5 12 2
	
3 43
	
33 36 40
	
35
	
48 4
hum- tgt-n urn 81 92 30
	
10 7 4
	
7 45
	
34 36 43
	
38
	
4 9
hum-tgt-nation 5 5 0
	
0 0 0
	
0 5
	
5 104 0
	
0
	
100 0
hum-tgt-effect 75 76 21
	
13 2 2
	
8 40
	
39 42 37
	
36
	
53 4
hum-tgt-total-num 6 0 0
	
0 0 0
	
0 0
	
6 107 0
	
•
inc-total 386 410 174
	
65 12 0
	
19 159
	
135 149 53
	
50
	
3 9
perp-total 225 225 110
	
6 26 7
	
5 83
	
83 178 50
	
50
	
3 7
phys-tgt-total 236 199 88
	
21 20 9
	
10 70
	
107 466 42
	
49
	
3 5
hum-tat-total 353 364 117
	
37 29 17
	
27 181
	
170 447 38
	
37
	
5 0
MATCHED/MISSING 1200 756 489
	
129 87 33
	
61 51
	
495 770 46
	
73
	
7
MATCHED/SPURIOUS 973 1198 489
	
129 87 33
	
61 493
	
268 1042 57
	
46
	
4 1
MATCHED ONLY 973 756 489
	
129 87 33
	
61 51
	
268 572 57
	
73
	
7
ALL TEMPLATES 1200 1198 489
	
129 87 33
	
61 493
	
495 1240 46
	
46
	
4 1
SET FILLS ONLY 584 357 255
	
43 35 4
	
16 24
	
251 353 47
	
77
	
7 0
STRING FILLS ONLY 315 190 124
	
19 29 25
	
19 18
	
143 211 42
	
70
	
9
TEXT FILTERING 55 59 50
	
• • • 9
	
5 36 91
	
85
	
15 2 0
P&R 2P&R
	
P&2 R
F-MEASURES 46 .0 46 .0 46 .0
Table G26. TST4 Scores for NYU
SLOT POS ACT COR
	
PAR INC ICR
	
IPA SPU
	
MIS NON REC
	
PRE
	
OVG FA L
template-id 79 188 67
	
0 0 0
	
0 121
	
12 20 85
	
36
	
6 4
inc-data 79 188 44
	
9 14 0
	
9 121
	
12 0 61
	
26
	
6 4
inc-loc 79 188 11
	
49 7 0
	
7 121
	
12 0 45
	
19
	
6 4
inc-type 79 188 59
	
8 0 0
	
0 121
	
12 0 80
	
34
	
64 9
inc-stage 79 188 63
	
0 4 0
	
0 121
	
12 0 80
	
34
	
64 2 4
inc-instr-id 33 99 20
	
0 0 1
	
0 79
	
13 124 61
	
20
	
8 0
inc-instr-type 53 111 21
	
2 S 0
	
2 83
	
25 106 42
	
20
	
75 2
perp-inc-cat 62 181 48
	
0 6 0
	
0 127
	
8 16 77
	
26
	
70 3 9
perp-ind-id 63 201 26
	
9 15 0
	
9 151
	
13 51 48
	
15
	
7 5
perp-org-id 54 152 17
	
14 7 0
	
14 114
	
16 62 44
	
16
	
7 5
perp-org-conf 54 140 5
	
6 26 0
	
2 103
	
17 62 15
	
6
	
74 1 1
phys-tgt-id 67 129 30
	
5 12 0
	
5 82
	
20 107 48
	
25
	
6 4
ph ys-t gt- t yp e 67 128 29
	
4 14 1
	
3 81
	
20 107 46
	
24
	
63 3
phys-tgt-num 67 128 27
	
16 4 0
	
14 81
	
20 107 52
	
27
	
6 3
phys-tgt-nation 2 0 0
	
0 0 0
	
0 0
	
2 198 0
	
•
	
• 0
phys-tgt-effect 51 125 16
	
11 10 0
	
5 88
	
14 111 42
	
17
	
70 8
phys-tgt-total-num 0 0 0
	
0 0 0
	
0 0
	
0 20 0
hum-tgt-name 34 28 6
	
1 1 0
	
1 20
	
26 163 19
	
23
	
7 1
hum-tgt-desc 79 263 31
	
8 19 0
	
8 205
	
21 52 44
	
13
	
7 8
hum-tgt-type 82 263 30
	
15 13 0
	
12 205
	
24 52 46
	
14
	
78 1 0
hum-tgt-num 84 263 22
	
15 21 0
	
13 205
	
26 52 35
	
11
	
7 8
hum-tgt-nation 5 0 0
	
0 0 0
	
0 0
	
5 196 0
	
•
	
• 0
hum-tgt-effect 74 185 24
	
14 15 0
	
13 132
	
21 90 42
	
17
	
71 8
hum-tgt-total-num 6 0 0
	
0 0 0
	
0 0
	
6 194 0
	
a
	
•
inc-total 402 962 218
	
68 30 1
	
18 646
	
86 230 63
	
26
	
6 7
perp-total 233 674 96
	
29 54 0
	
25 495
	
54 191 47
	
16
	
7 3
phys-tgt-total 254 510 102
	
36 40 1
	
27 332
	
76 830 47
	
24
	
6 5
hum-tat-total 364 1002 113
	
53 69 0
	
47 767
	
129 799 38
	
14
	
7 6
MATCHED/MISSING 1253 1387 529
	
186 193 2
	
117 479
	
345 662 50
	
45
	
3 4
MATCHED/SPURIOUS 1088 3148 529
	
186 193 2
	
117 2240
	
180 1915 57
	
20
	
7 1
MATCHED ONLY 1088 1387 529
	
186 193 2
	
117 479
	
180 527 57
	
45
	
3 4
ALL TEMPLATES 1253 3148 529
	
186 193 2
	
117 2240
	
345 2050 50
	
20
	
7 1
SET FILLS ONLY 608 668 295
	
60 93 1
	
37 220
	
160 302 53
	
49
	
33 2
STRING FILLS ONLY 330 389 130
	
37 54 1
	
37 168
	
109 168 45
	
38
	
4 3
TEXT FILTERING 5 5
P&R
78 .
2P&R
53
	
•
P&2R
• 25
	
2 20 96
	
68
	
32 5 6
F-MEASURES 28 .57 22 .73
	
38 .46
Table G27 . TST4 Scores for PARAMA X
G-15
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON REC PRE OVG FA L
template-id 75 78 43
	
0
	
0 0
	
0 35
	
32 34 57 55 4 5
inc-date 75 78 25
	
9
	
9 1
	
9 35
	
32 0 39 38 4 5
inc-Ioc 75 78 15
	
28
	
0 0
	
1 35
	
32 0 39 37 4 5
inc-type 75 78 37
	
6
	
0 0
	
0 35
	
32 0 53 51 45 5
inc-stage 75 78 40
	
0
	
3 0
	
0 35
	
32 0 53 51 45 1 5
inc-instr-id 31 7 4
	
0
	
0 0
	
0 3
	
27 83 13 57 4 3
inc-instr-type 51 14 10
	
0
	
0 0
	
0 4
	
41 63 20 71 28 0
perp-inc-cat 59 42 19
	
0
	
6 0
	
0 17
	
34 34 32 45 40 1 4
perp-ind-id 54 31 15
	
1
	
1 2
	
1 14
	
37 49 29 50 4 5
perp-org-id 51 20 10
	
0
	
2 2
	
0 8
	
39 60 20 50 4 0
perp-org-conf 51 20 5
	
0
	
7 0
	
0 8
	
39 60 10 25 40 2
phys-tgt-id 60 30 14
	
5
	
3 1
	
5 8
	
38 59 28 55 2 7
phys-tgt-type 59 32 6
	
0
	
17 0
	
0 9
	
36 58 10 19 28 1
phys-tgt-num 60 32 17
	
3
	
3 2
	
2 9
	
37 58 31 58 2 8
phys-tgt-nation 2 3 1
	
0
	
0 0
	
0 2
	
1 106 50 33 67 0
phys-tgt-effect 43 23 6
	
10
	
0 0
	
1 7
	
27 70 26 48 30 1
phys-tgt-total-num 0 0 0
	
0
	
0 0
	
0 0
	
0 110 .
hum-tgt-name 32 3 2
	
0
	
0 0
	
0 1
	
30 85 6 67 3 3
hum-tgt-desc 70 64 16
	
3
	
8 8
	
3 37
	
43 31 25 27 5 8
hum-tgt-type 73 62 10
	
9
	
8 2
	
8 35
	
46 32 20 23 56 3
hum-tgt-num 76 64 15
	
6
	
6 6
	
4 37
	
49 31 24 28 5 8
hum-tgt-nation 6 2 1
	
0
	
0 1
	
0 1
	
5 104 17 50 50 0
bum-tgt-effect 72 42 10
	
7
	
1 4
	
5 24
	
54 40 19 32 57 2
hum-tgt-total-num 5 0 0
	
0
	
0 0
	
0 0
	
5 105 0 •
inc-total 382 333 131
	
43
	
12 1
	
10 147
	
196 146 40 46 4 4
perp-total 215 113 49
	
1
	
16 4
	
1 47
	
149 203 23 44 4 2
phys-tgt-total 224 120 44
	
18
	
23 3
	
8 35
	
139 461 24 44 2 9
hum-tat-total 334 237 54
	
25
	
23 21
	
20 135
	
232 428 20 28 5 7
MATCHED/MISSING 1155 478 278
	
87
	
74 29
	
39 39
	
716 754 28 67 8
MATCHED/SPURIOUS 703 803 278
	
87
	
74 29
	
39 364
	
264 888 46 40 4 5
MATCHED ONLY 703 478 278
	
87
	
74 29
	
39 39
	
264 404 46 67 8
ALL TEMPLATES 1155 803 278
	
87
	
74 29
	
39 364
	
716 1238 28 40 4 5
SET FILLS ONLY 566 239 145
	
32
	
42 7
	
14 20
	
347 338 28 67 8 0
STRING FILLS ONLY 298 96 61
	
9
	
14 13
	
9 12
	
214 214 22 68 1 2
TEXT FILTERING 54 51 39
	
•
	
• • 12
	
15 34 72 76 24 2 6
P&R 2P&R
	
P&2R
F-MEASURES 32 .94 36 .84
	
29 .7 9
Table G28.
	
TST4 Scores for PR C
SLOT POS ACT COR
	
PAR
	
INC KR
	
IPA SPU
	
MIS NON REC PRE OVG FA L
template-id 76 142 56
	
0
	
0 0
	
0 86
	
20 30 74 39 6 0
inc-date 76 142 24
	
18
	
14 1
	
18 86
	
20 0 43 23 6 0
inc-loc 76 143 17
	
36
	
3 0
	
0 87
	
20 0 46 24 6 1
inc-type 76 142 46
	
10
	
0 0
	
0 86
	
20 0 67 36 60 8
inc-stage 76 142 52
	
0
	
4 0
	
0 86
	
20 0 68 37 60 2 2
inc-instr-id 31 20 6 0
	
1 12
	
23 127 21 32 6 0
inc-instr-type 52 36 15 1
	
1 19
	
35 101 30 43 53 0
perp-inc-cat 61 107 43
	
0
	
2 0
	
0 62
	
16 39 70 40 58 2 4
perp-ind-id 55 54 11
	
5
	
6 1
	
5 32
	
33 87 24 25 5 9
perp-org-id 54 88 33
	
0 0
	
0 54
	
20 62 61 38 6 1
perp-org-conf 54 88 23
	
10 0
	
1 54
	
20 62 44 27 61 7
phys-tgt-id 63 60 19
	
3
	
6 5
	
3 32
	
35 90 32 34 5 3
phys-tgt-type 63 60 16
	
2
	
10 0
	
1 32
	
35 90 27 28 53 2
phys-tgt-num 63 59 16
	
7
	
5 0
	
4 31
	
35 91 31 33 5 2
phys-tgt•nation 2 0 0
	
0
	
0 0
	
0 0
	
2 160 0 • • 0
phys-tgt-effect 43 20 9
	
1 0
	
0 9
	
32 119 22 48 45 1
phys-tgt-total-num 0 0 0
	
0
	
0 0
	
0 0
	
0 16 2
hum-tgt-name 32 10 5
	
0
	
1 0
	
0 4
	
26 135 16 50 4 0
hum-tgt-deac 70 68 13
	
5
	
10 5
	
5 40
	
42 78 22 23 5 9
hum-tgt-type 74 70 17
	
8
	
4 5
	
7 41
	
45 77 28 30 58 2
hum-tgt-num 77 66 15
	
8
	
6 2
	
7 37
	
48 81 25 29 5 6
hum-tgt-nation 6 1 1
	
0
	
0 0
	
0 0
	
5 157 17 100 0 0
hum-tgt-effect 71 66 14
	
8
	
4 3
	
7 40
	
45 81 25 27 61 3
hum-tgt-total-num 6 0 0
	
0
	
0 0
	
0 0
	
6 156 0 • .
inc-total 387 625 160
	
66
	
23 2
	
20 376
	
138 228 50 31 6 0
perp-total 224 337 110
	
6
	
19 1
	
6 202
	
89 250 SO 34 6 0
phys-tgt-total 234 199 60
	
13
	
22 5
	
8 104
	
139 712 28 33 5 2
hum-tat-total 336 281 65
	
29
	
25 15
	
26 162
	
217 765 24 28 5 8
MATCHED/MISSING 1181 623 395
	
114
	
89 23
	
60 25
	
583 773 38 72 4
MATCHED/SPURIOUS 930 1442 395
	
114
	
89 23
	
60 844
	
332 1726 49 31 5 8
MATCHED ONLY 930 623 395
	
114
	
89 23
	
60 25
	
332 544 49 72 4
ALL TEMPLATES 1181 1442 395
	
114
	
89 23
	
60 844
	
583 1955 38 31 5 8
SET FILLS ONLY 578 314 236
	
31
	
36 9
	
17 11
	
275 350 44 80 4 0
STRING FILLS ONLY 305 136 87
	
14
	
25 11
	
14 10
	
179 217 31 69 7
TEXT FILTERING 5 4
P&R
6 5
2P&R
4 9
P&2R
16
	
5 30 91 75 25 3 5
F-MEASURES 34 .14 32 .19
	
36 .36
Table G29. TST4 Scores for SR A
G-1 6
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS
	
NON REC PRE OVG FA L
template-id 80 78 59
	
0
	
0 0
	
0 19
	
21
	
31 74 76 2 4
inc-date 80 78 31
	
10
	
18 0
	
10 19
	
21
	
0 45 46 2 4
inc-loc 80 78 23
	
31
	
5 0
	
3 19
	
21
	
0 48 49 2 4
inc-type 80 78 53
	
6
	
0 0
	
0 19
	
21
	
0 70 72 24 3
inc-stage 80 77 57
	
0
	
2 0
	
0 18
	
21
	
1 71 74 23 9
inc-instr-id 31 16 10
	
0
	
2 0
	
0 4
	
19
	
71 32 62 2 5
inc-instr-type 50 31 16
	
4
	
3 0
	
1 8
	
27
	
50 36 58 26 0
perp-inc-cat 60 48 30
	
0
	
6 0
	
0 12
	
24
	
27 50 62 25 1 3
perp-ind-id 60 50 23
	
2
	
1 4
	
2 24
	
34
	
33 40 48 4 8
perp-org-id 51 44 20
	
0
	
4 1
	
0 20
	
27
	
47 39 45 4 5
perp-org-conf 51 45 13
	
2
	
9 0
	
2 21
	
27
	
46 27 31 47 5
phys-tgt-id 62 58 32
	
1
	
2 6
	
1 23
	
27
	
46 52 56 4 0
phys-tgt-type 62 58 23
	
2
	
10 0
	
2 23
	
27
	
46 39 41 40 2
phys-tgt-num 62 58 29
	
5
	
1 1
	
4 23
	
27
	
46 51 54 4 0
phys-tgt-nation 2 3 0
	
0
	
0 0
	
0 3
	
2
	
94 0 0 100 0
phys-tgt-effect 44 33 13
	
7
	
2 0
	
3 11
	
22
	
60 38 50 33 2
phys-tgt-total-num 0 0 0
	
0
	
0 0
	
0 0
	
0
	
9 9
hum-tgt-name 34 22 11
	
5
	
1 0
	
5 5
	
17
	
69 40 61 2 3
hum-tgt-desc 77 60 21
	
7
	
10 2
	
7 22
	
39
	
34 32 41 3 7
hum-tgt-type 80 70 26
	
9
	
9 1
	
5 26
	
36
	
33 38 44 37 3
hum-tgt-num 82 71 27
	
12
	
6 2
	
6 26
	
37
	
33 40 46 3 7
hum-tgt-nation 5 1 0
	
0
	
0 0
	
0 1
	
5
	
94 0 0 100 0
hum-tgt-effect 75 64 26
	
9
	
4 1
	
4 25
	
36
	
34 41 48 39 3
hum-tgt-total-num 6 0 0
	
0
	
0 0
	
0 0
	
6
	
93 0 • .
inc-total 401 358 190
	
51
	
30 0
	
14 87
	
130
	
122 54 60 2 4
perp-total 222 187 86
	
4
	
20 5
	
4 77
	
112
	
153 40 47 4 1
phys-tgt-total 232 210 97
	
15
	
15 7
	
10 83
	
105
	
391 45 50 4 0
hum-tat-total 359 288 111
	
42
	
30 6
	
27 105
	
176
	
390 37 46 3 6
MATCHED/MISSING 1214 813 484
	
112
	
95 18
	
55 122
	
523
	
830 44 66 1 5
MATCHED/SPURIOUS 917 1043 484
	
112
	
95 18
	
55 352
	
226
	
823 59 52 3 4
MATCHED ONLY 917 813 484
	
112
	
95 18
	
55 122
	
226
	
597 59 66 1 5
ALL TEMPLATES 1214 1043 484
	
112
	
95 18
	
55 352
	
523
	
1056 44 52 3 4
SET FILLS ONLY 589 399 257
	
39
	
45 2
	
17 58
	
248
	
379 47 69 14 0
STRING FILLS ONLY 315 194 117
	
15
	
20 13
	
15 42
	
163
	
232 40 64 2 2
TEXT FILTERING 57 65 53 12
	
4
	
31 93 82 18 2 8
P&R 2P&R
	
P&2R
F-MEASURES 47 .67 50 .18
	
45 . 4
Table G30 .
	
TST4 Scores for SR I
SLOT POS ACT COR
	
PAR
	
INC IC.R
	
IPA SPU
	
MIS
	
NON REC PRE OVG FA L
template-id 72 42 10
	
0
	
0 0
	
0 32
	
62
	
39 14 24 7 6
ine-date 72 0 0
	
0
	
0 0
	
0 0
	
72
	
32 0
inc-loc 72 1 0
	
0
	
0 0
	
0 1
	
72
	
31 0 0 100
inc-type 72 42 7
	
3
	
0 0
	
0 32
	
62
	
0 12 20 76 5
inc-stage 72 42 10
	
0
	
0 0
	
0 32
	
62
	
0 14 24 76 1 3
inc-instr-id 30 6 1
	
0
	
0 0
	
0 5
	
29
	
76 3 17 8 3
inc-instr-type 49 6 0
	
0 0
	
0 5
	
48
	
57 1 8 83 0
perp-inc-cat 54 18 4
	
0
	
1 0
	
0 13
	
49
	
37 7 22 72 9
perp-ind-id 51 18 7
	
0
	
0 0
	
0 11
	
44
	
47 14 39 6 1
perp-org-id 49 18 0
	
0
	
4 0
	
0 14
	
45
	
49 0 0 7 8
perp-org-conf 49 0 0
	
0
	
0 0
	
0 0
	
49
	
63 0 • • 0
phys-tgt-id 53 2 0
	
0
	
1 0
	
0 1
	
52
	
61 0 0 5 0
phys-tgt-type 53 8 0
	
2 0
	
0 5
	
50
	
57 1 6 62 0
phys-tgt-num 53 0 0
	
0
	
0 0
	
0 0
	
53
	
62 0 . .
phys-tgt-nation 2 0 0
	
0
	
0 0
	
0 0
	
2
	
102 0 • • 0
phys-tgt-effect 36 0 0
	
0
	
0 0
	
0 0
	
36
	
73 0 • • 0
phys-tgt-total-num 0 0 0
	
0
	
0 0
	
0 0
	
0
	
104 .
hum-tgt-name 30 0 0
	
0
	
0 0
	
0 0
	
30
	
83 0 •
hum-tgt-desc 65 9 2
	
2 0
	
1 4
	
60
	
58 4 28 4 4
hum-tgt-type 68 12 0
	
5
	
0 0
	
0 7
	
63
	
55 4 21 58 0
hum-tgt-num 72 0 0
	
0
	
0 0
	
0 0
	
72
	
62 0 .
hum-tgt-nation 5 0 0
	
0
	
0 0
	
0 0
	
5
	
100 0 • • 0
hum-tgt-effect 66 0 0
	
0
	
0 0
	
0 0
	
66
	
62 0 • • 0
hum-tgt-total-num 6 0 0
	
0
	
0 0
	
0 0
	
6
	
98 0 .
inc-total 367 97 18
	
4
	
0 0
	
0 75
	
345
	
196 5 21 7 7
perp-total 203 54 11
	
0
	
5 0
	
0 38
	
187
	
196 5 20 7 0
phys-tgt-total 197 10 0
	
3 0
	
0 6
	
193
	
459 0 5 6 0
hum-tat-total 312 21 2
	
6
	
2 0
	
1 11
	
302
	
518 2 24 5 2
MATCHED/MISSING 1079 62 31
	
11
	
10 0
	
1 10
	
1027
	
753 3 59 1 6
MATCHED/SPURIOUS 153 182 31
	
11
	
10 0
	
1 130
	
101
	
715 24 20 7 1
MATCHED ONLY 153 62 31
	
11
	
10 0
	
1 10
	
101
	
99 24 59 1 6
ALL TEMPLATES 1079 182 31
	
11
	
10 0
	
1 130
	
1027
	
1369 3 20 7 1
SET FILLS ONLY 526 40 21
	
10
	
3 0
	
0 6
	
492
	
342 5 65 15 0
STRING FILLS ONLY 278 22 10
	
1
	
7 0
	
1 4
	
260
	
213 4 48 1 8
TEXT FILTERING 5 2
P&R
3 4
2P&R
2 5
P&2R
9
	
27
	
39 48 74 26 1 9
F-MEASURES 5 .22 9 .38
	
3 .61
Table G31. TST4 Scores for UMD-CONQUEST
G-17
SLOT POS ACT COR PAR INC ICR IPA SPU
	
MIS NON REC
	
PRE OVG FA L
template-id 75 85 54 0 0 0 0 31
	
21 35 72
	
64 3 6
inc-date 75 85 33 10 11 0 10 31
	
21 0 51
	
45 3 6
inc-loc 75 85 26 24 4 0 4 31
	
21 0 51
	
45 3 6
inc-type 75 85 53 1 0 0 0 31
	
21 0 71
	
63 36 5
inc-stage 75 85 52 0 2 0 0 31
	
21 0 69
	
61 36 1 4
inc-instr-id 31 35 16 0 0 0 0 19
	
15 68 52
	
46 5 4
inc-instr-type 51 43 18 1 0 0 1 24
	
32 46 36
	
43 56 1
perp-inc-cat 58 53 35 0 1 0 0 17
	
22 31 60
	
66 32 1 2
perp-ind-id 59 45 20 4 9 2 4 12
	
26 43 37
	
49 2 7
perp-org-id 49 30 20 0 3 1 0 7
	
26 60 41
	
67 2 3
perp-org-conf 49 30 13 1 9 0 0 7
	
26 60 28
	
45 23 2
phys-tgt-id 57 59 19 11 4 1 11 25
	
23 46 43
	
42 4 2
phys -t g t-typ e 57 56 18 6 8 3 1 24
	
25 46 37
	
38 43 2
phys-tgt-num 57 56 20 7 5 2 2 24
	
25 46 41
	
42 4 3
phys-tgt-nation 2 2 1 0 0 0 0 1
	
1 103 50
	
50 50 0
phys-tgt-effect 39 30 13 5 3 1 0 9
	
18 68 40
	
52 30 2
phys-tgt-total-num 0 0 0 0 0 0 0 0
	
0 10 6
hum-tgt-name 33 40 13 3 3 1 3 21
	
14 65 44
	
36 5 2
hum-tgt-desc 73 94 28 9 8 0 8 49
	
28 40 44
	
34 5 2
hum-tgt-type 76 105 32 11 5 2 6 57
	
28 38 49
	
36 54 5
hum-tgt-num 78 107 28 14 8 2 11 57
	
28 38 45
	
33 5 3
hum-tgt-nation 5 1 0 1 0 0 1 0
	
4 102 10
	
50 0 0
hum-tgt-effect 71 92 32 10 4 2 9 46
	
25 42 52
	
40 50 4
hum-tgt-total-num 6 0 0 0 0 0 0 0
	
6 100 0
	
• •
inc-total 382 418 198 36 17 0 15 167
	
131 114 56
	
52 4 0
perp-total 215 158 88 5 22 3 4 43
	
100 194 42
	
57 2 7
phys-tgt-total 212 203 71 29 20 7 14 83
	
92 415 40
	
42 4 1
hum-tgt-total 342 439 133 48 28 7 38 230
	
133 425 46
	
36 5 2
MATCHED/MISSING 1151 800 490 118 87 17 71 105
	
456 768 48
	
69 1 3
MATCHED/SPURIOUS 871 1218 490 118 87 17 71 523
	
176 907 63
	
45 4 3
MATCHED ONLY 871 800 490 118 87 17 71 105
	
176 527 63
	
69 1 3
ALL TEMPLATES 1151 1218 490 118 87 17 71 523
	
456 1148 48
	
45 4 3
SET FILLS ONLY 558 385 267 36 32 8 18 50
	
223 353 51
	
74 13 0
STRING FILLS ONLY 302 201 116 27 27 5 26 31
	
132 211 43
	
64 1 5
TEXT FILTERING 5 4
P&R
6 0
2P&R
4 9
P&2R
11
	
5 35 91
	
82 18 2 4
F-MEASURES 46 .45 45 .57
	
47 .3 7
Table G32. TST4 Scores for UMASS
SLOT POS ACT COR PAR INC I(.R IPA SPU
	
MIS NON REC
	
PRE OVG FA L
template-id 79 91 49 0 0 0 0 42
	
30 25 62
	
54 4 6
inc-date 79 91 19 16 14 1 16 42
	
30 0 34
	
30 4 6
inc-loc 79 89 20 27 1 0 2 41
	
31 1 42
	
38 4 6
inc-type 79 91 43 6 0 0 0 42
	
30 0 58
	
50 46 5
inc-stage 79 91 45 0 4 0 0 42
	
30 0 57
	
49 46 1 6
inc-instr-id 31 7 3 0 0 0 0 4
	
28 93 10
	
43 5 7
inc-instr-type 51 35 12 4 1 0 1 18
	
34 60 27
	
40 51 1
perp-inc-cat 62 91 35 0 4 0 0 52
	
23 7 56
	
38 57 3 1
perp-ind-id 56 30 14 1 3 0 1 12
	
38 59 26
	
48 4 0
perp-org-id 53 54 23 0 4 0 0 27
	
26 49 43
	
42 5 0
perp-org-conf 53 54 14 1 12 0 1 27
	
26 49 27
	
27 50 6
phys-tgt-id 55 47 12 11 5 1 11 19
	
27 70 32
	
37 4 0
phys-tgt-type 55 46 12 4 11 1 1 19
	
28 70 25
	
30 41 2
phys-tgt-num 55 46 12 13 2 2 5 19
	
28 70 34
	
40 4 1
phys-tgt-nation 2 1 0 0 0 0 0 1
	
2 118 0
	
0 100 0
phys-tgt-effect 39 21 5 4 I 0 2 11
	
29 85 18
	
33 52 2
phys-tgt-total-num 0 0 0 0 0 0 0 0
	
0 12 1
hum-tgt-name 35 28 14 2 0 0 2 12
	
19 85 43
	
54 4 3
hum-tgt-desc 77 120 22 5 13 0 4 80
	
37 28 32
	
20 6 7
hum-tgt-type 80 127 24 8 11 1 5 84
	
37 28 35
	
22 66 7
hum-tgt-num 82 128 24 11 8 1 8 85
	
39 28 36
	
23 6 6
hum-tgt-nation 6 2 2 0 0 1 0 0
	
4 116 33
	
100 0 0
hum-tgt-effect 74 65 19 8 5 0 4 33
	
42 53 31
	
35 51 3
hum-tgt-total-num 6 0 0 0 0 0 0 0
	
6 115 0
	
• *
inc-total 398 404 142 53 20 1 19 189
	
183 154 42
	
42 4 7
perp-total 224 229 86 2 23 0 2 118
	
113 164 39
	
38 5 2
phys-tgt-total 206 161 41 32 19 4 19 69
	
114 534 28
	
35 4 3
hum-tgt-total 360 470 105 34 37 3 23 294
	
184 453 34
	
26 6 2
MATCHED/MISSING 1188 730 374 121 99 8 63 136
	
594 802 36
	
60 1 9
MATCHED/SPURIOUS 764 1264 374 121 99 8 63 670
	
170 976 57
	
34 5 3
MATCHED ONLY 764 730 374 121 99 8 63 136
	
170 473 57
	
60 1 9
ALL TEMPLATES 1188 1264 374 121 99 8 63 670
	
594 1305 36
	
34 5 3
SET FILLS ONLY 580 357 211 35 49 3 14 62
	
285 362 39
	
64 17 0
STRING FILLS ONLY 307 171 88 19 25 1 18 39
	
175 227 32
	
57 2 3
TEXT FILTERING 5 5
MR
6 5
2P&R
4 5
P&2R
• • • • 20
	
10 25 82
	
69 31 4 4
F-MEASURES 34 .97 34 .38
	
35 .58
Table G33 . TST4 Scores for UMIC H
G-1 8
SLOT POS ACT COR PAR INC ICR IPA SPU MIS NON REC PRE OVG FAL
template-id 72 65 26 0 0 0 0 39 46 33 36 40 6 0
inc-date 72 61 14 6 3 8 6 38 49 1 24 28 6 2
inc-loc 72 20 1 4 1 0 1 14 66 25 4 15 7 0
inc-type 72 65 25 1 0 0 0 39 46 0 35 39 60 6
inc-stage 72 65 26 0 0 0 0 39 46 0 36 40 60 1 5
inc-instr-id 32 16 3 0 3 0 0 10 26 81 9 19 6 2
inc-instr-type 52 22 5 5 1 0 0 11 41 61 14 34 50 0
perp-inc-cat 54 61 16 0 4 0 0 41 34 16 30 26 67 2 7
perp-ind-id 54 34 6 3 4 0 3 21 41 41 14 22 6 2
perp-org-id 50 37 10 0 2 0 0 25 38 44 20 27 6 8
perp-org-conf 50 37 0 8 4 0 0 25 38 44 8 11 68 4
phys-tgt-id 56 11 6 0 1 0 0 4 49 65 11 54 3 6
phys-tgt-type 56 11 6 0 1 0 0 4 49 65 11 54 36 0
phys-tgt-num 56 11 6 0 1 0 0 4 49 65 11 54 3 6
phys-tgt-nation 2 0 0 0 0 0 0 0 2 109 0 * * 0
phys-tgt-effect 38 0 0 0 0 0 0 0 38 78 0 * * 0
phys-tgt-total-num 0 0 0 0 0 0 0 0 0 11 1
hum- tgt-name 30 0 0 0 0 0 0 0 30 90 0 * *
hum-tgt-desc 66 19 0 3 3 0 3 13 60 59 2 8 6 8
hum-tgt-type 69 19 0 6 0 0 6 13 63 59 4 16 68 1
hum-tgt-num 73 19 0 4 2 0 4 13 67 59 3 10 6 8
hum-tgt-nation 5 0 0 0 0 0 0 0 5 107 0 * * 0
hum-tgt-effect 68 0 0 0 0 0 0 0 68 68 0 * * 0
hum-tgt-total-num 6 0 0 0 0 0 0 0 6 105 0
inc-total 372 249 74 16 8 8 7 151 274 168 22 33 6 1
perp-total 208 169 32 11 14 0 3 112 151 145 18 22 6 6
phys-tgt-total 208 33 18 0 3 0 0 12 187 493 9 54 3 6
hum-tat-total 317 57 0 13 5 0 13 39 299 547 2 11 6 8
MATCHED/MISSING 1105 208 124 40 30 8 23 14 911 745 13 69 7
MATCHED/SPURIOUS 456 508 124 40 30 8 23 314 262 844 32 28 6 2
MATCHED ONLY 456 208 124 40 30 8 23 14 262 236 32 69 7
ALL TEMPLATES 1105 508 124 40 30 8 23 314 911 1353 13 28 6 2
SET FILLS ONLY 538 115 78 20 10 0 6 7 430 339 16 76 6 0
STRING FILLS ONLY 288 50 25 6 13 0 6 6 244 209 10 56 1 2
TEXT FILTERING 36 15 16 33 69 70 29 3 1
P&R 2P&R
	
P&2R
F-MEASURES 17 .76 22 .75
	
14 .56
Table G34 . TST4 Scores for US C
SECTION 3 : OPTIONAL RUNS FOR TST3 and TST 4
Note: The tables include only the summary rows from the summary score report .
BB N
There are several independent parameters that we would like to vary :
1. A threshold on whether a paragraph is relevant or irrelevant . As a threshold this is a
continuous variable . We would like to try several settings . For our required run th e
threshold will be set to 1 .
2. Several parameters that turn off/on code to more aggressively try to merg e
descriptions into a single view to search farther for possible slot fillers to mor e
aggressively/conservatively produce templates . These clearly are discrete options . For
the required run merging aggressiveness will be set at 3 .
3. Thresholds on the projected quality of a slot . These are continuous variables
independently settable on each slot and on the template as a whole . For our required ru n
this will be turned off.
TST 3
	SLOT	 POS	 ACT	 COR	 PAR	 INC	 ICR	 IPA	 SPU	 MIS	 NON
MATCHED/MISSING
	
1585
	
1051
	
566
	
154
	
133
	
9
	
114
	
198
	
732
	
1124
MATCHED/SPURIOUS
	
1169
	
1760
	
566
	
154
	
133
	
9
	
114 907
	
316
	
158 3
MATCHED ONLY
	
1169
	
1051
	
566
	
154
	
133 9
	
114
	
198
	
316
	
76 7
ALL TEMPLATES
	
1585
	
1760
	
566
	
154
	
133
	
9
	
114
	
907
	
732
	
1940
SET FILLS ONLY
	
757
	
512
	
298
	
54
	
58
	
0
	
41
	
102
	
347
	
51 6
STRING FILLS ONLY
	
407
	
245
	
124
	
16
	
47
	
9
	
15
	
58
	
220
	
31 3
TEXT FILTERING
	
6 9
	
8 5
	
68
	
•
	
•
	
•
	
•
	
17
	
1
	
1 4
R&P
	
2P&R P&2R
F-MEASURES
	
37.89 36 .73 39 .1 3
Table G35 . BBN : scores-l-objects .tst3-pass 2
SLOT	 POS	 ACT	 COR	 PAR	 NC	 ICR	 IPA	 SPU	 MIS	 NON	 REC	 PRE	 OVG FA L
MATCHED/MISSING
	
1594
	
867
	
533
	
137
	
106 9
	
97
	
91
	
818
	
1145 38
	
69
	
1 0
MATCHED/SPURIOUS
	
1152
	
1564
	
533
	
137
	
106 9
	
97
	
788
	
376
	
1609 52
	
38
	
5 0
MATCHED ONLY
	
1152
	
867
	
533
	
137
	
106 9
	
97
	
91
	
376
	
765
	
52
	
69
	
1 0
ALL TEMPLATES
	
1594
	
1564
	
533
	
137
	
106 9
	
97
	
788
	
818
	
1989 38
	
38
	
5 0
SET FILLS ONLY
	
762
	
437
	
285
	
46
	
51
	
0
	
34
	
55
	
380
	
523
	
40
	
70
	
12
	
0
STRING FILLS ONLY
	
409
	
190
	
115
	
15
	
31
	
9
	
14
	
29
	
248
	
323
	
30
	
64
	
1 5
TEXT FILTERING
	
69
	
85
	
68
	
•
	
•
	
•
	
•
	
17
	
1
	
14
	
98
	
80
	
20
	
5 5
R&P
	
2P&R P&2R
F-MEASURES
	
38 .0
	
38 .0
	
38 . 0
Table G36 . BBN : scores-add-criminal .tst3-pass2
	REC	 PRE	 OVG	 FAL
40
	
61
	
1 9
55
	
36
	
5 2
55
	
61
	
1 9
40
	
36
	
5 2
43
	
63
	
20
	
1
32
	
54
	
2 4
98
	
80
	
20
	
5 5
	SLOT	 POS	 ACT
MATCHED/MISSING
	
1639
	
105 6
MATCHED/SPURIOUS
	
1423 243 5
MATCHED ONLY
	
1423 105 6
ALL TEMPLATES
	
1639 243 5
SET FILLS ONLY
	
780
	
52 7
STRING FILLS ONLY
	
425
	
22 8
TEXT FILTERING
	
6 9
	
8 5
R&P 2P& R
F-MEASURES
	
36 .0
	
32 .14
	COR	 PAR	 INC
638
	
189
	
12 6
638
	
189
	
12 6
638
	
189
	
12 6
638
	
189
	
12 6
341
	
62
	
6 7
135
	
22
	
3 4
6 8
P&2R
40 .91
	 ICR	 IPA
13
	
13 3
13
	
13 3
13
	
13 3
13
	
13 3
1
	
4 3
11
	
21
	SPU	 MIS	 NON
103
	
686
	
116 3
1482 470
	
272 3
103
	
470
	
97 4
1482
	
686
	
291 2
57
	
310
	
53 5
37
	
234
	
32 1
17
	
1
	
14
	REC	 PRE	 OVG	 FAL
45
	
69
	
1 0
51
	
30
	
6 1
51
	
69
	
1 0
45
	
30
	
6 1
48
	
70
	
11
	
0
34
	
64
	
1 6
98
	
80
	
20
	
5 5
Table G37 . BBN: scores-limit-event-merge-distance-O .tst3-pass2
	ICR	 IPA
9
	
8 7
9
	
8 7
9
	
8 7
9
	
8 7
0
	
3 1
9
	
14
	 SPU	 MIS	 NON
87
	
873
	
113 8
725
	
338
	
143 4
87
	
338
	
690
725
	
873
	
188 2
54
	
407
	
51 9
26
	
259
	
32 3
11
	
5
	
22
	 REC	 PRE	 OVG	 FAL
35
	
70
	
1 1
53
	
38
	
5 0
53
	
70
	
1 1
35
	
38
	
5 0
37
	
70
	
13
	
0
27
	
64
	
1 5
92
	
85
	
15
	
3 3
	SLOT	 POS	 ACT
MATCHED/MISSING
	
1582 796
MATCHED/SPURIOUS
	
1047
	
143 4
MATCHED ONLY
	
1047 796
ALL TEMPLATES
	
1582
	
143 4
SET FILLS ONLY
	
756
	
403
STRING FILLS ONLY
	
406
	
17 3
TEXT FILTERING
	
6 7
	
7 3
R&P
	
2P&R
F-MEASURES
	
36 .44 37 .36
	COR	 PAR	 INC
491
	
125
	
9 3
491
	
125
	
9 3
491
	
125
	
9 3
491
	
125
	
9 3
262
	
42
	
4 5
104
	
15
	
2 8
6 2
P&2 R
35 .5 6
Table G38 . BBN : tst3/scores-thresh--l .tst3-pass2
G-20
	 SLOT
MATCHED/MISSING
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATES
SET FILLS ONLY
STRING FILLS ONLY
TEXT FILTERING
F-MEASURES
	POS	 ACT
1547
	
72 0
967
	
128 0
967
	
72 0
1547
	
128 0
742
	
36 4
397
	
15 3
69
	
8 0
R&P
	
2P& R
37 .17
	
39 .38
	COR	 PAR	 INC
470
	
108
	
7 4
470
	
108
	
7 4
470
	
108
	
7 4
470
	
108
	
7 4
252
	
36
	
3 4
101
	
10
	
2 2
65
	
•
	
•
P&2R
35 .2
	ICR	 IPA
9
	
7 6
9
	
7 6
9
	
7 6
9
	
7 6
q 2 7
9
	
9
	SPU	 MIS	 NON
68
	
895
	
1137
628
	
315
	
137 2
68
	
315
	
69 2
628
	
895
	
181 7
42
	
420
	
52 0
20
	
264
	
32 3
15
	
4
	
16
	REC	 PRE	 OVG	 FAL
34
	
73
	
9
54
	
41
	
4 9
54
	
73
	
9
34
	
41
	
4 9
36
	
74
	
12
	
0
27
	
69
	
1 3
94
	
81
	
19
	
4 8
Table G39. BBN: scores-mergeaggr3-graft-blockcomps .tst3-pass2
	SLOT
MATCHED/MISSING
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATES
SET FILLS ONLY
STRING FILLS ONLY
TEXT FILTERING
F-MEASURES
	POS	 ACT
1538
	
67 2
901
	
115 9
901
	
67 2
1538
	
115 9
737
	
34 3
396
	
14 2
67
	
6 9
R&P 2P& R
36 .32
	
39 .5 3
Table G40.
	COR	 PAR	 INC	 ICR	 IPA	 SPU	 MIS	 NON	 REC	 PRE	 OVG	 FAL
434
	
101
	
70
	
10
	
69
	
67
	
933
	
1121
	
32
	
72
	
1 0
434
	
101
	
70
	
10
	
69
	
554
	
296
	
1224
	
54
	
42
	
4 8
434
	
101
	
70
	
10
	
69
	
67
	
296
	
628
	
54
	
72
	
1 0
434
	
101
	
70
	
10
	
69
	
554
	
933
	
1717
	
32
	
42
	
4 8
232
	
34
	
33
	
0
	
25
	
44
	
438
	
511
	
34
	
72
	
13
	
0
92
	
10
	
22
	
10
	
9
	
18
	
272
	
319
	
24
	
68
	
1 3
59
	
•
	
•
	
•
	
•
	
10
	
8
	
23
	
88
	
86
	
14
	
3 0
P&2R
33 . 6
BBN: scores-thresh--l-mergeaggr3 .tst3-pass2
	 SLOT
MATCHED/MISSING
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATES
SET FILLS ONLY
STRING FILLS ONLY
TEXT FILTERING
F-MEASURES
	SLOT
MATCHED/MISSING
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATES
SET FILLS ONLY
STRING FILLS ONLY
TEXT FILTERING
F-MEASURE S
	SLOT
MATCHED/MISSING
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATES
SET FILLS ONLY
STRING FILLS ONLY
TEXT FILTERING
F-MEASURE S
	SLOT
MATCHED/MISSING
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATES
SET FILLS ONLY
STRING FILLS ONLY
TEXT FILTERING
F-MEASURE S
	SLOT
MATCHED/MISSING
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATES
SET FILLS ONLY
STRING FILLS ONLY
TEXT FILTERING
F-MEASURES
	POS	 ACT	 COR	 PAR	 INC
1562
	
763
	
465
	
120
	
9 1
1006
	
1358
	
465
	
120
	
9 1
1006
	
763
	
465
	
120
	
9 1
1562
	
1358
	
465
	
120
	
9 1
745
	
386
	
246
	
42
	
4 4
403
	
168
	
100
	
15
	
2 7
66
	
68
	
5 8
R&P
	
2P&R P&2R
36 .33
	
37 .89
	
34 .89
Table G41 . BBN:
	POS	 ACT	 COR	 PAR	 INC
1559
	
749
	
450
	
124
	
8 9
992
	
1289
	
450
	
124
	
8 9
992
	
749
	
450
	
124
	
8 9
1559
	
1289
	
450
	
124
	
8 9
745
	
379
	
242
	
44
	
4 0
401
	
164
	
96
	
14
	
2 8
65
	
66
	
5 7
R&P 2P&R P&2R
36 .16
	
38.37
	
34 . 2
Table G42 . BBN:
	POS	 ACT	 COR	 PAR	 INC
1558
	
707
	
420
	
121
	
8 6
949
	
1211
	
420
	
121
	
8 6
949
	
707
	
420
	
121
	
8 6
1558
	
1211
	
420
	
121
	
8 6
744
	
357
	
227
	
43
	
3 8
401
	
155
	
89
	
14
	
2 8
65
	
59
	
5 3
R&P
	
2R&P R&2P
34 .93
	
37 .8
	
32 .4 6
Table G43. BBN:
	POS	 ACT	 COR	 PAR	 INC
1535
	
674
	
402
	
116
	
8 1
891
	
1090
	
402
	
116
	
8 1
891
	
674
	
402
	
116
	
8 1
1535
	
1090
	
402
	
116
	
8 1
734
	
340
	
217
	
40
	
3 6
395
	
148
	
86
	
14
	
2 6
65
	
54
	
5 1
R&P
	
2P&R P&2R
35 .0
	
38 .89
	
31 .8 2
Table G44. BBN:
	POS	 ACT	 COR	 PAR	 INC
1592
	
852
	
526
	
133
	
10 2
1126
	
1565
	
526
	
133
	
10 2
1126
	
852
	
526
	
133
	
10 2
1592
	
1565
	
526
	
133
	
10 2
761
	
429
	
281
	
45
	
4 8
408
	
186
	
112
	
15
	
3 0
69
	
85
	
6 8
R&P
	
2P&R P&2R
37 .49
	
37 .8
	
37 . 2
Table G45. BBN :
scores-thresh-O .tst3-pass 2
scores-thresh-l .tst3-pass 2
scores-thresh-2.tst3-pass 2
scores-thresh-3 .tst3-pass 2
	ICR	 IPA
9
	
9 4
9
	
9 4
9
	
9 4
9
	
9 4
q 3 3
9
	
14
	SPU	 MIS	 NON
91
	
831
	
114 7
804
	
365
	
161 1
91
	
365
	
75 8
804
	
831
	
200 0
55
	
387
	
52 4
29
	
251
	
32 4
17
	
1
	
1 4
scores-vanilla .tst3-pass 2
	ICR	 IPA
9
	
8 2
9
	
8 2
9
	
8 2
9
	
8 2
q 3 1
9
	
14
	 SPU	 MIS	 NON
87
	
886
	
111 2
682
	
330
	
131 1
87
	
330
	
63 9
682
	
886
	
1784
54
	
413
	
50 8
26
	
261
	
31 4
10
	
8
	
2 4
	 ICR	 IPA
9
	
8 6
9
	
8 6
9
	
8 6
9
	
8 6
q 3 3
9
	
13
	SPU	 MIS	 NON
86
	
896
	
111 4
626
	
329
	
124 1
86
	
329
	
62 9
626
	
896
	
172 6
53
	
419
	
50 9
26
	
263
	
31 5
9
	
8
	
2 6
	 ICR	 IPA
8
	
8 4
8
	
8 4
8
	
8 4
8
	
8 4
q 3 2
8
	
13
	SPU	 MIS	 NON
80
	
931
	
112 1
584
	
322
	
114 2
80
	
322
	
58 6
584
	
931
	
167 7
49
	
436
	
51 4
24
	
270
	
31 7
6
	
12
	
2 9
	 ICR	 IPA
8
	
8 0
8
	
8 0
8
	
8 0
8
	
8 0
q 2 9
8
	
13
	SPU	 MIS	 NON
75
	
936
	
111 4
491
	
292
	
101 3
75
	
292
	
55 3
491
	
936
	
157 4
47
	
441
	
51 1
22
	
269
	
31 5
3
	
14
	
32
	 REC	 PRE	 OVG	 FAL
34
	
69
	
1 1
52
	
39
	
5 0
52
	
69
	
1 1
34
	
39
	
5 0
36
	
69
	
14
	
0
27
	
64
	
1 5
88
	
85
	
15
	
2 9
	 REC	 PRE	 OVG	 FAL
33
	
68
	
1 1
52
	
40
	
4 8
52
	
68
	
1 1
33
	
40
	
4 8
35
	
70
	
14
	
0
26
	
63
	
1 6
88
	
86
	
14
	
2 6
	 REC	 PRE	 OVG	 FA L
31
	
68
	
1 1
51
	
40
	
4 8
51
	
68
	
1 1
31
	
40
	
4 8
33
	
70
	
14
	
0
24
	
62
	
1 5
82
	
90
	
10
	
1 7
	 EEC	 PRE	 OVG	 FAL
30
	
68
	
1 1
52
	
42
	
4 5
52
	
68
	
1 1
30
	
42
	
4 5
32
	
70
	
14
	
0
24
	
63
	
1 S
78
	
94
	
6
	
8
	REC	 PRE	 OVG	 FAL
37
	
70
	
1 1
53
	
38
	
5 1
53
	
70
	
1 1
37
	
38
	
5 1
40
	
71
	
13
	
0
29
	
64
	
1 6
98
	
80
	
20
	
5 5
G-2 1
TST 4
	 SLOT
MATCHED/MISSIN G
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATE S
SET FILLS ONLY
STRING FILLS ONLY
TEXT FILTERING
	 POS	 ACT
1187
	
81 1
1058
	
2059
1058
	
81 1
1187
	
2059
580
	
40 4
306
	
18 7
53
	
7 2
R&P 2P&R
	 COR	 PAR	 INC
515
	
134
	
8 7
515
	
134
	
8 7
515
	
134
	
8 7
515
	
134
	
8 7
285
	
39
	
4 3
123
	
16
	
2 0
48
	
•
P&2R
	 ICR	 IPA
20
	
8 4
20
	
8 4
20
	
8 4
20
	
8 4
7
	
2 1
11
	
16
	SPU	 MIS	 NON
75
	
451
	
74 8
1323
	
322
	
207 8
75
	
322
	
65 3
1323
	
451
	
217 3
37
	
213
	
34 0
28
	
147
	
20 3
24
	
5
	
23
	 REC	 PRE	 OVG	 FAL
49
	
72
	
9
55
	
28
	
6 4
55
	
72
	
9
49
	
28
	
6 4
52
	
75
	
9
	
0
43
	
70
	
1 5
90
	
67
	
33
	
5 1
F-MEASURES 35 .64
	
30 .62
	
42 .6 1
Table G46. BBN : scores-limit-event-merge-distance-O .tst4-pass2
	 SLOT
MATCHED/MISSING
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATES
SET FILLS ONLY
STRING FILLS ONLY
TEXT FILTERING
F-MEASURES
	POS	 ACT
1156
	
64 7
861
	
119 0
861
	
64 7
1156
	
119 0
565
	
32 5
298
	
15 3
53
	
6 4
R&P 2P& R
40 .49 40 .2
	 COR	 PAR	 INC
429
	
90
	
7 5
429
	
90
	
7 5
429
	
90
	
7 5
429
	
90
	
7 5
235
	
23
	
3 9
106
	
11
	
1 7
4 3
P&2R
40 .8
	ICR	 IPA
20
	
5 4
20
	
5 4
20
	
5 4
20
	
5 4
7
	
1 2
11
	
11
	 SPU	 MIS	 NON
53
	
562
	
73 6
596
	
267
	
112 4
53
	
267
	
49 4
596
	
562
	
136 6
28
	
268
	
33 3
19
	
164
	
20 3
21
	
10
	
26
	 REC	 PRE	 OVG	 FAL
41
	
73
	
8
55
	
40
	
5 0
55
	
73
	
8
41
	
40
	
5 0
44
	
76
	
9
	
0
37
	
73
	
1 2
81
	
67
	
33
	
4 5
Table G47. BBN: scores-mergeaggr3-graft-blockcomps .tst4-pass2
	 SLOT
MATCHED/MISSING
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATE S
SET FILLS ONLY
STRING FILLS ONLY
TEXT FILTERING
F-MEASURES
	POS	 ACT
1129
	
73 7
951
	
141 4
951
	
73 7
1129
	
141 4
552
	
36 6
292
	
17 5
52
	
6 8
R&P 2P& R
41 .01
	
38 .5 1
Table
	COR	 PAR	 INC	 ICR	 IPA	 SPU	 MIS	 NON
457
	
127
	
89
	
19
	
80
	
64
	
456
	
69 8
457
	
127
	
89
	
19
	
80
	
741
	
278
	
1235
457
	
127
	
89
	
19
	
80
	
64
	
278
	
55 3
457
	
127
	
89
	
19
	
80
	
741
	
456
	
1380
252
	
40
	
44
	
7
	
24
	
30
	
216
	
31 8
115
	
14
	
21
	
10
	
14
	
25
	
142
	
18 9
46
	
•
	
•
	
•
	
•
	
22
	
6
	
2 6
P&2R
43 .8 7
G48. BBN: scores-thresh--l .tst4-pass2
	REC	 PRE	 OVG	 FAL
46
	
71
	
9
55
	
37
	
5 2
55
	
71
	
9
46
	
37
	
5 2
49
	
74
	
8
	
0
42
	
70
	
1 4
88
	
68
	
32
	
4 6
	 SLOT
MATCHED/MISSING
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATE S
SET FILLS ONLY
STRING FILLS ONLY
TEXT FILTERING
F-MEASURES
	POS	 ACT
1128
	
62 9
846
	
110 3
846
	
62 9
1128
	
110 3
551
	
31 5
292
	
14 8
51
	
6 1
R&P 2P& R
40 .49 40 .8
	 COR	 PAR	 INC
409
	
90
	
7 9
409
	
90
	
7 9
409
	
90
	
7 9
409
	
90
	
7 9
225
	
22
	
4 1
101
	
11
	
1 8
4 2
P&2R
40 .2
	 ICR	 IPA
20
	
5 4
20
	
5 4
20
	
5 4
20
	
5 4
7
	
1 2
11
	
11
	SPU	 MIS	 NON
51
	
550
	
71 2
525
	
268
	
99 5
S1
	
268
	
48 0
525
	
550
	
1227
27
	
263
	
32 2
18
	
162
	
19 6
19
	
9
	
30
	REC	 PRE	 OVG	 FAL
40
	
72
	
8
54
	
41
	
4 8
54
	
72
	
8
40
	
41
	
4 8
43
	
75
	
8
	
0
36
	
72
	
1 2
82
	
69
	
31
	
3 9
Table G49. BBN : scores-thresh-0-mergeaggr3 .tst4-pass2
	 POS	 ACT	 COR	 PAR	 INC
1129
	
715
	
449
	
118
	
8 8
922
	
1382
	
449
	
118
	
8 8
922
	
715
	
449
	
118
	
8 8
1129
	
1382
	
449
	
118
	
8 8
552
	
354
	
247
	
37
	
4 3
292
	
171
	
113
	
13
	
2 1
51
	
66
	
44
	
•
R&P 2P&R P&2R
40 .61
	
38 .36
	
43 .1 3
Table G50. BBN: scores-thresh-O .tst4-pass 2
	 SLAT
MATCHED/MISSIN G
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATE S
SET FILLS ONLY
STRING FILLS ONLY
TEXT FILTERING
F-MEASURES
	 SLOT
MATCHED/MISSING
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATES
SET FILLS ONLY
STRING FILLS ONLY
TEXT FILTERING
F-MEASURES
	 POS
111 8
89 8
89 8
111 8
54 7
28 9
5 0
R&P
40 .2
	ACT
69 3
131 5
69 3
131 5
34 4
16 5
6 1
2P&R
	COR	 PAR	 INC
433
	
115
	
8 6
433
	
115
	
8 6
433
	
115
	
8 6
433
	
115
	
8 6
239
	
37
	
42
108
	
13
	
20
4 2
P&2R
	ICR	 IPA
19
	
7 3
19
	
7 3
19
	
7 3
19
	
7 3
7
	
2 2
10
	
1 3
	JCR	 IPA
19
	
7 0
19
	
7 0
19
	
7 0
19
	
7 0
7
	
2 2
10
	
13
	SPU	 MIS	 NON
60
	
474
	
70 2
727
	
267
	
120 1
60
	
267
	
53 2
727
	
474
	
137 1
27
	
225
	
32 1
24
	
145
	
19 0
22
	
7
	
2 7
	SPU	 MIS	 NON
59
	
484
	
69 1
681
	
264
	
110 9
59
	
264
	
51 0
681
	
484
	
129 0
26
	
229
	
31 6
24
	
148
	
18 7
19
	
8
	
31
	 REC	 PRE	 OVG	 FAL
45
	
71
	
8
5S
	
37
	
5 3
55
	
71
	
8
45
	
37
	
5 3
48
	
75
	
8
	
0
41
	
70
	
1 4
86
	
67
	
33
	
4 5
	REC	 PRE	 OVG	 FAL
44
	
71
	
8
55
	
37
	
5 2
55
	
71
	
8
44
	
37
	
5 2
47
	
75
	
8
	
0
40
	
69
	
1 4
84
	
69
	
31
	
3 8
	 REC	 PRE	 OVG	 FAL
43
	
71
	
8
55
	
38
	
5 1
55
	
71
	
8
43
	
38
	
5 1
46
	
75
	
7
	
0
39
	
70
	
1 4
82
	
71
	
29
	
3 4
38 .22 42 . 4
Table G51 . BBN: scores-thresh- l .tst4-pass 2
	NON
69 5
104 1
50 0
123 6
31 8
18 9
3 3
F-MEASURES
SLOT	 POS	 ACT	 COR	 PAR	 INC	 ICR	 IPA SPU	 MIS
MATCHED/MISSING
	
1118
	
681
	
426
	
114
	
86
	
19
	
69
	
55
	
492
MATCHED/SPURIOUS
	
884
	
1269 426
	
114
	
86
	
19
	
69
	
643
	
25 8
MATCHED ONLY
	
884
	
681
	
426
	
114
	
86
	
19
	
69
	
55
	
25 8
ALL TEMPLATES
	
1118
	
1269
	
426
	
114
	
86
	
19
	
69
	
643
	
492
SET FILLS ONLY
	
547
	
338
	
235
	
37
	
42
	
7
	
22
	
24
	
23 3
STRING FILLS ONLY
	
289
	
162
	
107
	
13
	
20
	
10
	
13
	
22
	
14 9
TEXT FILTERING
	
50
	
5 8
	
41
	
•
	
•
	
•
	
•
	
17
	
9
R&P 2P&R P&2R
40.35 38 .9
	
41 . 9
Table G52 . BBN: scores-thresh-2 .tst4-pass2
G-22
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON REC
	
PRE
	
OVG FA L
MATCHED/MISSING 1117 663 407
	
108
	
90 19
	
62 58
	
512 693 41
	
70
	
9
MATCHED/SPURIOUS 860 1225 407
	
108
	
90 19
	
62 620
	
255 996 54
	
38
	
5 1
MATCHED ONLY 860 663 407
	
108
	
90 19
	
62 58
	
255 475 54
	
70
	
9
ALL TEMPLATES 1117 1225 407
	
108
	
90 19
	
62 620
	
512 1214 41
	
38
	
5 1
SET FILLS ONLY 547 331 225
	
37
	
44 7
	
21 25
	
241 317 44
	
74
	
8 0
STRING FILLS ONLY 288 158 102
	
13
	
20 10
	
13 23
	
153 189 38
	
69
	
1 4
TEXT FILTERING 5 0
R&P
5 5
2P&R
39
	
•
	
'
P&2R
'
	
• 16
	
11 34 78
	
71
	
29 3 2
F-MEASURES 39 .44 38 .56
	
40 .3 6
Table G53. BBN:
	
scores-thresh-3 .tst4-pass2
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON REC
	
PRE
	
OVG FAL
MATCHED/MISSING 1117 611 363
	
101
	
90 19
	
59 57
	
563 694 37
	
68
	
9
MATCHED/SPURIOUS 802 1084 363
	
101
	
90 19
	
59 530
	
248 845 52
	
38
	
4 9
MATCHED ONLY 802 611 363
	
101
	
90 19
	
59 57
	
248 419 52
	
68
	
9
ALL TEMPLATES 1117 1084 363
	
101
	
90 19
	
59 530
	
563 1120 37
	
38
	
4 9
SET FILLS ONLY 547 305 202
	
34
	
44 7
	
18 25
	
267 317 40
	
72
	
8 0
STRING FILLS ONLY 288 146 91
	
13
	
20 10
	
13 22
	
164 190 34
	
67
	
1 5
TEXT FILTERING 5 0
R&P
4 8
2P&R
37
	
•
	
•
P&2R
•
	
• 11
	
13 39 74
	
77
	
23 2 2
F-MEASURES 37 .49 37 .8
	
37 . 2
Table G54. BBN:
	
scores-thresh-4 .tst4-pass2
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON REC
	
PRE
	
OVG FAL
MATCHED/MISSING 1116 619 363
	
95
	
103 16
	
55 58
	
555 693 37
	
66
	
9
MATCHED/SPURIOUS 801 998 363
	
95
	
103 16
	
55 437
	
240 752 51
	
41
	
4 4
MATCHED ONLY 801 619 363
	
95
	
103 16
	
55 58
	
240 418 51
	
66
	
9
ALL TEMPLATES 1116 998 363
	
95
	
103 16
	
55 437
	
555 1027 37
	
41
	
4 4
SET FILLS ONLY 546 309 201
	
32
	
51 5
	
18 25
	
262 317 40
	
70
	
8 0
STRING FILLS ONLY 288 149 92
	
10
	
24 9
	
10 23
	
162 189 34
	
65
	
1 5
TEXT FILTERING 5 0
R&P
4 4
2P&R
37
	
•
	
•
P&2R
•
	
• 7
	
13 43 74
	
84
	
16 1 4
F-MEASURES 38 .9 40 .13
	
37 .7 4
Table G55. BBN:
	
scores-thresh-5 .tst4-pass2
SLOT POS ACT COR
	
PAR
	
INC KR
	
IPA SPU
	
MIS NON REC
	
PRE
	
OVG FAL
MATCHED/MISSING 1153 755 467
	
133
	
89 19
	
85 66
	
464 722 46
	
71
	
9
MATCHED/SPURIOUS 975 1487 467
	
133
	
89 19
	
85 798
	
286 1342 55
	
36
	
5 4
MATCHED ONLY 975 755 467
	
133
	
89 19
	
85 66
	
286 577 55
	
71
	
9
ALL TEMPLATES 1153 1487 467
	
133
	
89 19
	
85 798
	
464 1487 46
	
36
	
5 4
SET FILLS ONLY 564 376 259
	
43
	
43 7
	
26 31
	
219 329 50
	
75
	
8 0
STRING FILLS ONLY 297 178 116
	
14
	
22 10
	
14 26
	
145 196 41
	
69
	
1 5
TEXT FILTERING 5 3
R&P
7 1
2P&R
4 7
P&2R
24
	
6 23 89
	
66
	
34 5 1
F-MEASURES 40 .39 37 .64
	
43 .58
Table G56. BBN: scores-vanilla.tst4-pass2
GE
In this system configuration we generate those templates that contain the most conten t
and are the most accurate . Our advance tests show this to improve precision by 20% and
reduce reduce recall about the same amount .
TST 3
SLOT POS ACT COR
	
PAR INC ICR IPA SPU
	
MIS NON REC
	
PRE OVG FAL
MATCHED/MISSING 1618 896 614
	
109 101 18 71 72
	
794 1218 41
	
75 8
MATCHED/SPURIOUS 1117 1074 614
	
109 101 18 71 250
	
293 1035 60
	
62 2 3
MATCHED ONLY 1117 896 614
	
109 101 18 71 72
	
293 789 60
	
75 8
ALL TEMPLATES 1618 1074 614
	
109 101 18 71 250
	
794 1464 41
	
62 2 3
SET FILLS ONLY 773 421 309
	
36 41 6 26 35
	
387 571 42
	
78 8 0
STRING FILLS ONLY 41 8
R&P
20 9
2P&R
137
	
2 5
P&2R
23 11 25 24
	
233 342 36
	
72 1 1
F-MEASURES 49 .36 56 .24
	
43 .9 8
TST4
Table G57. GE: scores-opt.tst3-pass 1
SLOT POS ACT COR PAR INC KR IPA SPU MIS NON REC PRE OVG FA L
MATCHED/MISSING 1165 698 502 77 57 10 38 62 529 814 46 77 9
MATCHED/SPURIOUS 851 886 502 77 57 10 38 250 215 773 64 61 2 8
MATCHED ONLY 851 698 502 77 57 10 38 62 215 567 64 77 9
ALL TEMPLATES 1165 886 502 77 57 10 38 250 529 1020 46 61 2 8
SET FILLS ONLY 569 335 253 28 25 2 16 29 263 370 47 80 9 0
STRING FILLS ONLY 298 159 128 5 9 7 5
	
17 156 232
	
44 82 1 1
F-MEASURES 52 .45 57 .27
	
48 .38
Table G58. GE: scores-opt.tst4-pass 1
G-2 3
NYU
If we cannot get a full sentence parse we normally take the longest parse starting at wor d
1 . We have an option to search for and process additional clauses or NPs in th e
remaining portion of the sentence .
TST3
SLOT POS ACT COR PAR INC KR IPA SPU MIS NON REC PRE OVG FAL
MATCHED/MISSING 1586 944 599 149 95 41 96 101 743 1147 42 71 1 1
MATCHED/SPURIOUS 1154 1457 599 149 95 41 96 614 311 1332 58 46 4 2
MATCHED ONLY 1154 944 599 149 95 41 96 101 311 728 58 71 1 1
ALL TEMPLATES 1586 1457 599 149 95 41 96 614 743 1751 42 46 4 2
SET FILLS ONLY 758 442 317 47 34 5 25 44 360 543 45 77 10 0
STRING FILLS ONLY 41 3
R&P
24 0
2P&R
14 2
P&2R
18 37 35 18 43 216 305 36 63 1 8
F-MEASURES 43 .91 45 .14
	
42 .74
Table G59. NYU: scores-debris .tst3-pass l
SRI
We will change two parameters to turn off some filtering mechanisms we don't quite trust . One of
the parameters (*allow-templates-with-empty-and-military-targets*) will refrain from filterin g
sparse templates and templates with military targets on the theory that they'll contain a lot of
correct stuff. The other (*allow-templates-with-stale-dates*) will allow templates with stale dates o n
the same theory. We expect to gain a point or two in recall at the expense of 4 or 5 points i n
precision in this run.
TST3
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS
	
NON REC
	
PRE OVG FA L
MATCHED/MISSING 1596 1064 661
	
133
	
127 18
	
83 143
	
675
	
1161 46
	
68 1 3
MATCHED/SPURIOUS 1220 1340 661
	
133
	
127 18
	
83 419
	
299
	
1212 60
	
54 3 1
MATCHED ONLY 1220 1064 661
	
133
	
127 18
	
83 143
	
299
	
855 60
	
68 1 3
ALL TEMPLATES 1596 1340 661
	
133
	
127 18
	
83 419
	
675
	
1518 46
	
54 3 1
SET FILLS ONLY 760 500 335
	
48
	
48 0
	
27 69
	
329
	
544 47
	
72 14 0
STRING FILLS ONLY 41 6
R&P
26 1
2P&R
150
	
23
	
3 4
P&2R
15
	
23 54
	
209
	
320 39
	
62 2 1
F-MEASURES 49 .68 52 .18
	
47 . 4
Table G60. SRI :
	
scores-max -recall .tst3-pass 1
SLOT POS ACT COR
	
PAR
	
INC 1CR
	
IPA SPU
	
MIS
	
NON RFC
	
PRE OVG FA L
MATCHED/MISSING 1596 1064 661
	
133
	
127 18
	
83 143
	
675
	
1161 46
	
68 1 3
MATCHED/SPURIOUS 1220 1340 661
	
133
	
127 18
	
83 419
	
299
	
1212 60
	
54 3 1
MATCHED ONLY 1220 1064 661
	
133
	
127 18
	
83 143
	
299
	
855 60
	
68 1 3
ALL TEMPLATES 1596 1340 661
	
133
	
127 18
	
83 419
	
675
	
1518 46
	
54 3 1
SET FILLS ONLY 760 500 335
	
48
	
48 0
	
27 69
	
329
	
544 47
	
72 14 0
STRING FILLS ONLY 416 261 150
	
23
	
34 15
	
23 54
	
209
	
320 39
	
62 2 1
TEXT FILTERING 6 8
R&P
7 2
2P&R
64
	
•
	
•
P&2R
•
	
• 8
	
4
	
24 94
	
89 11 2 5
F-MEASURES
TST4
49 .68 52 .18
	
47 . 4
Table G61. SRI :
	
scores-max-recall .tst3-pass2
SLOT POS ACT COR
	
PAR
	
INC JCR
	
IPA SPU
	
MIS
	
NON REC
	
PRE OVG FA L
MATCHED/MISSING 1169 777 476
	
109
	
88 22
	
54 104
	
496
	
801 45
	
68 1 3
MATCHED/SPURIOUS 897 1077 476
	
109
	
88 22
	
54 404
	
224
	
864 59
	
49 3 8
MATCHED ONLY 897 777 476
	
109
	
88 22
	
54 104
	
224
	
578 59
	
68 1 3
ALL TEMPLATES 1169 1077 476
	
109
	
88 22
	
54 404
	
496
	
1087 45
	
49 3 8
SET FILLS ONLY 570 381 254
	
34
	
43 3
	
13 50
	
239
	
363 48
	
71 13 0
STRING FILLS ONLY 299 185 116
	
17
	
15 16
	
17 37
	
151
	
226 42
	
67 2 0
F-MEASURES 46 .91 48 .14
	
45 .7 5
Table G62. SRI : scores -max-recal1 .tst4-pass 1
G—2 4
SLOT POS ACT COR PAR INC ICR IPA SPU MIS NON REC PRE OVG FA L
MATCHED/MISSING 1169 777 476 109 88 22 54 104 496 801 45 68 1 3
MATCHED/SPURIOUS 8 97 1077 476 109 88 22 54 404 224 864 59 49 3 8
MATCHED ONLY 897 777 476 109 88 22 54 104 224 578 59 68 1 3
ALL TEMPLATES 1169 1077 476 109 88 22 54 404 496 1087 45 49 3 8
SET FILLS ONLY 570 381 254 34 43 3 13 50 239 363 48 71 13 0
STRING FILLS ONLY 299 185 116 17 15 16 17 37 151 226 42 67 2 0
TEXT FILTERING 5 5 6 5 51 • • ' • 14 4 31 93 78 22 3 1
F-MEASURES 46 .91 48 .14
	
45 .7 5
Table G63. SRI : scores-max-recall .tst4-pass2
UMAS S
Our optional run will provide us with a comparison of two dictionaries . The dictionary
used in our official run is based to a large extent on last-year's UMass/MUC-3 dictionary .
The dictionary used in our optional run is new and was derived from the corpus in a n
automated fashion requiring a minimal amount of manual programming effort . We
predict that the overall precision scores under all templates will be very close or identica l
for these two systems but the recall scores should differ by about 10 points (with the
optional system trailing) .
TST3
SLOT POS ACT COR
	
PAR INC ICR IPA SPU MIS NON REC
	
PRE
	
OVG
	
FA L
MATCHED/MISSING 1589 912 562
	
137 104 4 84 109 786 1167 40
	
69
	
1 2
MATCHED/SPURIOUS 1122 1063 562
	
137 104 4 84 260 319 943 56
	
59
	
2 4MATCHED ONLY
1122 912 562
	
137 104 4 84 109 319 722 56
	
69
	
1 2
ALL TEMPLATES 1589 1063 562
	
137 104 4 84 260 786 1388 40
	
59
	
2 4SET
FILLS ONLY 756 400 291
	
35 32 0 24 42 398 555 41
	
77
	
10
	
0STRING FILLS ONLY 41 2
R&P
23 1
2P&R
124
	
1 8
P&2R
42 4 17 47 228 318 32
	
58
	
2 0
F-MEASURES 47 .68 53 .88
	
42 .7 5
Table G64. UMASS :
	
scores-autoslog .tst3-pass 1
SLOT POS ACT COR
	
PAR INC ICR IPA SPU MIS NON REC
	
PRE
	
OVO
	
FAL
MATCHED/MISSING 1589 912 562
	
138 103 4 85 109 786 1167 40
	
69
	
1 2MATCHED/SPURIOUS
1122 1063 562
	
138 103 4 85 260 319 943 56
	
59
	
2 4MATCHED ONLY 1122
912 562
	
138 103 4 85 109 319 722 56
	
69
	
1 2
ALL TEMPLATES 1589 1063 562
	
138 103 4 85 260 786 1388 40
	
59
	
2 4SET FILLS ONLY
756 400 291
	
35 32 0 24 42 398 555 41
	
77
	
10
	
0
STRING FILLS ONLY 412 231 124
	
19 41 4 18 47 228 318 32
	
58
	
2 0TEXT FILTERING 6 7
R&P
6 3
2P&R
56
	
'
P&2R
• 7 I1 26 84
	
89
	
11
	
2 1
F-MEASURES 47 .68 53 .88
	
42 .75
Table G65. UMASS : scores-autoslog .tst3-pass2
TST4
SLOT POS ACE COR
	
PAR INC ICR
	
IPA SPU
	
MIS
	
NON REC
	
PRE
	
OVG
	
FAL
MATCHED/MISSING 1119 579 361
	
95 62 3
	
53 61
	
601
	
737 36
	
70
	
1 0
MATCHED/SPURIOUS 688 879 361
	
95 62 3
	
53 361
	
170
	
770 59
	
46
	
4 1MATCHED ONLY 688 579 361
	
95 62 3
	
53 61
	
170
	
431 59
	
70
	
1 0ALL
TEMPLATES 1119 879 361
	
95 62 3
	
53 361
	
601
	
1076 36
	
46
	
4 1SET FILLS ONLY 548 262 187
	
26 25 1
	
12 24
	
310
	
337 36
	
76
	
9
	
0
STRING FILLS ONLY 288 151 91
	
21 18 2
	
20 21
	
158
	
206 35
	
67
	
1 4F-MEASURES 40 .39 43 .58 37 .6 4
Table G66. UMASS :
	
scores-autoslog.tst4-pass I
SLOT POS ACT COR
	
PAR INC KR
	
IPA SPU
	
MIS
	
NON REC
	
PRE
	
OVG
	
FA L
MATCHED/MISSING 1119 579 361
	
95 62 3
	
53 61
	
601
	
737 36
	
70
	
1 0
MATCHED/SPURIOUS 688 879 361
	
95 62 3
	
53 361
	
170
	
770 59
	
46
	
4 1
MATCHED ONLY 688 579 361
	
95 62 3
	
53 61
	
170
	
431 59
	
70
	
1 0
ALL TEMPLATES 1119 879 361
	
95 62 3
	
53 361
	
601
	
1076 36
	
46
	
4 1SET FILLS ONLY 548 262 187
	
26 25 1
	
12 24
	
310
	
337 36
	
76
	
9
	
0
STRING FILLS ONLY 288 151 91
	
21 18 2
	
20 21
	
158
	
206 35
	
67
	
1 4TEXT FILTERING 52 56 45 •
11
	
7
	
37 86
	
80
	
20
	
2 3
F-MEASURES 40 .39 43 .58 37 .64
Table G67. UMASS : scores-autoslog .tst4-pass2
G-2 5
SECTION 4: PROGRESS TEST ON TST2 AND TST3
NOTE: TST2 data is included only for MUC-3 veteran sites . All tables show summary
rows only.
BBN
SLOT POS ACT COR
	
PAR
	
INC ICR IPA SPU
	
MIS
	
NON REC
	
PRE OVG FA L
MATCHED/MISSING 1242 687 402
	
117
	
135 12 92 33
	
588
	
657 37
	
67 5
MATCHED/SPURIOUS 1070 2264 402
	
117
	
135 12 92 1610
	
416
	
2593 43
	
20 7 1
MATCHED ONLY 1070 687 402
	
117
	
135 12 92 33
	
416
	
515 43
	
67 5
ALL TEMPLATES 1242 2264 402
	
117
	
135 12 92 1610
	
588
	
2735 37
	
20 7 1
SET FILLS ONLY 672 385 249
	
57
	
59 3 52 20
	
307
	
431 41
	
72 5 0
STRING FILLS ONLY 376 151 82
	
4
	
52 9 4 13
	
238
	
222 22
	
56 9
TEXT FILTERING 66 95 66
	
•
	
• • • 29
	
0
	
5 100
	
69 30 8 5
P&R 2P&R
	
R&2P
F-MEASURES 25 .96 22 .02 31 .62
Table G68. TST2 Progress
	
Scores
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS
	
NON REC
	
PRE OVG FAL
MATCHED/MISSING 1239 529 341
	
81
	
64 10
	
52 43
	
753
	
691 31
	
72 8
MATCHED/SPURIOUS 704 855 341
	
81
	
64 10
	
52 369
	
218
	
680 54
	
45 4 3
MATCHED ONLY 704 529 341
	
81
	
64 10
	
52 43
	
218
	
360 54
	
72 8
ALL TEMPLATES 1239 855 341
	
81
	
64 10
	
52 369
	
753
	
1011 31
	
45 4 3
SET FILLS ONLY 675 293 199
	
33
	
32 0
	
26 29
	
411
	
452 32
	
74 10 0
STRING FILLS ONLY 360 127 82
	
9
	
22 10
	
8 14
	
247
	
235 24
	
68 1 1
P&R 2P&R
	
R&2P
F-MEASURES
GE
36 .71 41 .27 33 .0 6
Table G69. TST3
	
Progress
	
Scores
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS
	
NON REC
	
PRE OVG FA L
MATCHED/MISSING 1169 944 538
	
93
	
113 15
	
55 200
	
425
	
584 50
	
62 2 1
MATCHED/SPURIOUS 971 1372 538
	
93
	
113 15
	
55 628
	
227
	
723 60
	
43 4 6
MATCHED ONLY 971 944 538
	
93
	
113 15
	
55 200
	
227
	
450 60
	
62 2 1
ALL TEMPLATES 1169 1372 538
	
93
	
113 15
	
55 628
	
425
	
857 50
	
43 4 6
SET FILLS ONLY 635 520 301
	
43
	
63 0
	
30 113
	
228
	
382 51
	
62 22 1
STRING FILLS ONLY 354 279 138
	
14
	
40 15
	
12 87
	
162
	
198 41
	
52 3 1
TEXT FILTERING 64 75 61
	
•
	
• •
	
• 14
	
3
	
23 95
	
81 19 3 8
P&R 2P&R
	
R&2 P
F-MEASURES 46 .24 44 .24 48 .4 2
Table G70. TST2 Progress Scores
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS
	
NON REC
	
PRE OVG FA L
MATCHED/MISSING 1356 1076 714
	
113
	
97 20
	
73 152
	
432
	
753 57
	
72 1 4
MATCHED/SPURIOUS 1101 1431 714
	
113
	
97 20
	
73 507
	
177
	
847 70
	
54 3 5
MATCHED ONLY 1101 1076 714
	
113
	
97 20
	
73 152
	
177
	
564 70
	
72 1 4
ALL TEMPLATES 1356 1431 714
	
113
	
97 20
	
73 507
	
432
	
1036 57
	
54 3 5
SET FILLS ONLY 737 586 406
	
38
	
54 6
	
27 88
	
239
	
500 58
	
72 15 0
STRING FILLS ONLY 394 314 194
	
32
	
24 14
	
32 64
	
144
	
250 53
	
67 2 0
P&R 2P&R
	
R&2 P
F-MEASURES
GE-CMU
55 .46 54 .57 56 .3 7
Table G71. TST3
	
Progress
	
Scores
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS
	
NON REC
	
PRE OVG FA L
MATCHED/MISSING 1357 901 617
	
73
	
99 40
	
31 112
	
568
	
784 48
	
72 1 2
MATCHED/SPURIOUS 976 1192 617
	
73
	
99 40
	
31 403
	
187
	
713 67
	
55 3 4
MATCHED ONLY 976 901 617
	
73
	
99 40
	
31 112
	
187
	
504 67
	
72 1 2
ALL TEMPLATES 1357 1192 617
	
73
	
99 40
	
31 403
	
568
	
993 48
	
55 3 4
SET FILLS ONLY 739 500 362
	
13
	
56 10
	
6 69
	
308
	
514 50
	
74 14 0
STRING FILLS ONLY 391 244 154
	
20
	
27 18
	
20 43
	
190
	
267 42
	
67 1 8
P&R 2P&R
	
R&2P
F-MEASURES 51 .26 53 .44
	
49 .25
Table G72. TST3 Progress Score s
G-26
HUGHES
	SLOT
MATCHED/MISSING
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATES
SET FILLS ONLY
STRING FILLS ONLY
TEXT FILTERING
F-MEASURE S
	SLOT
MATCHED/MISSING
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATES
SET FILLS ONLY
STRING FILLS ONLY
F-MEASURES
LS I
	SLOT
MATCHED/MISSING
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATE S
SET FILLS ONLY
STRING FILLS ONLY
TEXT FILTERING
F-MEASURES
	 POS	 ACT
1118
	
92 3
720
	
205 5
720
	
92 3
1118
	
205 5
604
	
53 7
340
	
27 9
66
	
9 9
P&R 2P& R
20 .62
	
17 .5 8
	 POS	 ACT
1357
	
141 3
788
	
217 8
788
	
141 3
1357
	
217 8
742
	
87 2
394
	
42 5
P&R
	
2P& R
24 .62
	
21 .62
	 POS	 ACT
1023
	
11 2
175
	
62 1
175
	
11 2
1023
	
62 1
553
	
6 7
312
	
2 0
59
	
4 1
P&R 2P&R
7 .76
	
9 .43
	 REC	 PRE	 OVG	 FAL
29
	
35
	
3 8
45
	
16
	
7 2
45
	
35
	
3 8
29
	
16
	
7 2
32
	
36
	
42
	
2
22
	
26
	
4 6
98
	
66
	
34
	
10 0
	 COR	 PAR	 INC	 ICR	 TPA	 SPU	 MIS	 NON
268
	
107
	
193
	
21
	
75
	
355
	
550
	
54 2
268
	
107
	
193
	
21
	
75
	
1487
	
152
	
118 6
268
	
107
	
193
	
21
	
75
	
355
	
152
	
29 3
268
	
107
	
193
	
21
	
75
	
1487
	
550
	
143 5
173
	
47
	
91
	
1
	
41
	
226
	
293
	
36 7
69
	
10
	
71
	
2
	
10
	
129
	
190
	
17 1
65
	
•
	
•
	
'
	
•
	
34
	
1
	
0
R&2 P
24 .9 5
Table G73. TST2 Progress Scores
	 COR	 PAR	 INC	 ICR	 IPA	 SPU	 MIS	 NON
374
	
131
	
172
	
44
	
91
	
736
	
680
	
61 8
374
	
131
	
172
	
44
	
91
	
1501
	
111
	
56 1
374
	
131
	
172
	
44
	
91
	
736
	
111
	
28 2
374
	
131
	
172
	
44
	
91
	
1501
	
680
	
89 7
241
	
36
	
99
	
5
	
17
	
496
	
366
	
42 1
101
	
35
	
50
	
18
	
35
	
239
	
208
	
19 5
R&2P
28 .5 7
Table G74. TST3 Progress Scores
	 COR	 PAR	 INC	 ICR	 IPA	 SPU	 MIS	 NON
55
	
22
	
22
	
5
	
11
	
13
	
924
	
54 3
55
	
22
	
22
	
5
	
11
	
522
	
76
	
71 8
55
	
22
	
22
	
5
	
11
	
13
	
76
	
7 1
55
	
22
	
22
	
5
	
11
	
522
	
924
	
119 0
36
	
8
	
11
	
0
	
5
	
12
	
498
	
35 0
11
	
2
	
6
	
4
	
2
	
1
	
293
	
18 9
34
	
•
	
•
	
~
	
*
	
7
	
25
	
3 4
R&2 P
6 . 6
Table G75. TST2 Progress Scores
	 REC	 PRE	 OVG	 FAL
32
	
31
	
5 2
56
	
20
	
6 9
56
	
31
	
5 2
32
	
20
	
6 9
35
	
30
	
57
	
2
30
	
28
	
5 6
	REC	 PRE	 OVG	 FAL
6
	
59
	
1 2
38
	
11
	
8 4
38
	
59
	
1 2
6
	
11
	
8 4
7
	
60
	
18
	
0
4
	
60
	
5
58
	
83
	
17
	
1 7
	IPA
5 4
5 4
5 4
5 4
6
2 5
	SLOT
MATCHED/MISSING
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATE S
SET FILLS ONLY
STRING FILLS ONLY
F-MEASURES
	 POS	 ACT
1309
	
51 4
750
	
223 3
750
	
51 4
'1309
	
223 3
715
	
307
378
	
9 4
P&R
	
2P&R
17 .68
	
15 .27
	 COR	 PAR	 INC	 ICR
259
	
108
	
100
	
4
259
	
108
	
100
	
4
259
	
108
	
100
	
4
259
	
108
	
100
	
4
165
	
32
	
66
	
0
48
	
25
	
18
	
4
R&2 P
2L 0
Table G76. TST3
	 SPU	 MIS	 NON
47
	
842
	
72 7
1766
	
283
	
292 5
47
	
283
	
34 3
1766
	
842
	
330 9
44
	
452
	
46 1
3
	
287
	
262
	REC	 PRE	 OVG	 FAL
24
	
61
	
9
42
	
14
	
7 9
42
	
61
	
9
24
	
14
	
7 9
25
	
59
	
14
	
0
16
	
64
	
3
Progress Scores
MDESC
	SLOT
MATCHED/MISSIN G
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATES
SET FILLS ONLY
STRING FILLS ONLY
TEXT FILTERING
F-MEASURES
	 SLAT
MATCHED/MISSING
MATCHED/SPURIOUS
MATCHED ONLY
ALL TEMPLATE S
SET FILLS ONLY
STRING FILLS ONLY
F-MEASURES
	 POS	 ACT
1114
	
52 8
760
	
97 1
760
	
52 8
1114
	
97 1
602
	
26 6
340
	
15 5
63
	
6 9
P&R
	
2P& R
29 .87
	
31 .1 1
	POS	 ACT
1280
	
42 2
622
	
92 4
622
	
42 2
1280
	
92 4
699
	
24 2
370
	
10 0
P&R
	
2P&R
24 .36
	
26 .95
	 REC	 PRE	 OVG	 FAL
28
	
59
	
1 9
41
	
32
	
5 6
41
	
59
	
1 9
28
	
32
	
5 6
27
	
62
	
18
	
0
20
	
44
	
3 3
82
	
75
	
25
	
4 6
	 COR	 PAR	 INC	 ICR	 IPA	 SPU	 MLS	 NON
277
	
72
	
79
	
6
	
52
	
100
	
686
	
59 9
277
	
72
	
79
	
6
	
52
	
543
	
332
	
102 4
277
	
72
	
79
	
6
	
52
	
100
	
332
	
37 5
277
	
72
	
79
	
6
	
52
	
543
	
686
	
124 8
148
	
33
	
36
	
1
	
29
	
49
	
385
	
39 4
62
	
11
	
31
	
5
	
11
	
51
	
236
	
20 1
52
	
•
	
•
	
•
	
•
	
17
	
11
	
2 0
R&2P
28 .7 2
Table G77. TST2 Progress Scores
	 COR	 PAR	 INC	 ICR	 IPA	 SPU	 MIS	 NON
228
	
79
	
70
	
6
	
44
	
45
	
903
	
72 4
228
	
79
	
70
	
6
	
44
	
547
	
245
	
90 1
228
	
79
	
70
	
6
	
44
	
45
	
245
	
26 1
228
	
79
	
70
	
6
	
44
	
547
	
903
	
136 4
149
	
27
	
37
	
2
	
13
	
29
	
486
	
47 4
42
	
16
	
27
	
4
	
15
	
15
	
285
	
24 8
R&2P
22 .23
Table G78. TST3 Progress Scores
	 REC	 PRE	 OVG	 FAL
21
	
63
	
1 1
43
	
29
	
5 9
43
	
63
	
1 1
21
	
29
	
5 9
23
	
67
	
12
	
0
14
	
50
	
1 5
G-27
MITRE
SLOT
	
POS
	
ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON REC
	
PRE OVG FA L
MATCHED/MISSING
	
1280
	
227 155
	
33
	
27 4
	
15 12
	
1065 766 13
	
76 5
MATCHED/SPURIOUS
	
401
	
2110 155
	
33
	
27 4
	
15 1895
	
186 4095 43
	
8 9 0
MATCHED ONLY
	
401
	
227 155
	
33
	
27 4
	
15 12
	
186 198 43
	
76 5
ALL TEMPLATES
	
1280
	
2110 155
	
33
	
27 4
	
15 1895
	
1065 4663 13
	
8 9 0
SET FILLS ONLY
	
695
	
126 101
	
7
	
11 2
	
0 7
	
576 503 15
	
83 6 0
STRING FILLS ONLY
	
371
	
4 9
P&R
	
2P&R
32
	
5
	
7
R&2P
0
	
5 5
	
327 259 9
	
70 1 0
F-MEASURES
	
9 .9
	
8 .67 11 .5 6
Table G79. TST3
	
Progress
	
Scores
NMSU/BRANDEI S
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON REC
	
PRE OVG FA L
MATCHED/MISSING 1326 625 289
	
95
	
114 27
	
32 127
	
828 659 25
	
54 2 0
MATCHED/SPURIOUS 750 1326 289
	
95
	
114 27
	
32 828
	
252 886 45
	
25 6 2
MATCHED ONLY 750 625 289
	
95
	
114 27
	
32 127
	
252 272 45
	
54 2 0
ALL TEMPLATES 1326 1326 289
	
95
	
114 27
	
32 828
	
828 1273 25
	
25 6 2
SET FILLS ONLY 721 360 187
	
35
	
54 18
	
0 84
	
445 436 28
	
57 23 0
STRING FILLS ONLY 390 152 52
	
7
	
50 9
	
7 43
	
281 218 14
	
36 2 8
P&R 2P&R
	
R&2 P
F-MEASURE S
NYU
25 .0 25 .0 25 . 0
Table G80. TST3 Progress
	
Scores
SLAT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON REC
	
PRE OVG FA L
MATCHED/MISSING 1151 712 401
	
111
	
114 26
	
75 86
	
525 566 40
	
64 1 2
MATCHED/SPURIOUS 921 1467 401
	
111
	
114 26
	
75 841
	
295 1316 50
	
31 5 7
MATCHED ONLY 921 712 401
	
111
	
114 26
	
75 86
	
295 413 50
	
64 1 2
ALL TEMPLATES 1151 1467 401
	
111
	
114 26
	
75 841
	
525 1469 40
	
31 5 7
SET FILLS ONLY 625 385 246
	
44
	
50 6
	
37 45
	
285 379 43
	
70 12 0
STRING FILLS ONLY 350 193 93
	
15
	
44 20
	
15 41
	
198 183 29
	
52 2 1
TEXT FILTERING 66 88 63
	
•
	
• '
	
• 25
	
3 9 95
	
72 28 7 4
P&R 2P&R
	
R&2 P
F-MEASURES 34 .93 32 .46 37 . 8
Table G81. TST2 Progress Scores
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON REC
	
PRE OVG FA L
MATCHED/MISSING 1303 788 496
	
118
	
94 37
	
66 80
	
595 709 42
	
70 1 0
MATCHED/SPURIOUS 927 1187 496
	
118
	
94 37
	
66 479
	
219 792 60
	
47 4 0
MATCHED ONLY 927 788 496
	
118
	
94 37
	
66 80
	
219 424 60
	
70 1 0
ALL TEMPLATES 1303 1187 496
	
118
	
94 37
	
66 479
	
595 1077 42
	
47 4 0
SET FILLS ONLY 708 426 296
	
45
	
39 5
	
22 46
	
328 478 45
	
75 11 0
STRING FILLS ONLY 383 223 130
	
17
	
42 32
	
16 34
	
194 227 36
	
62 1 5
P&R 2P&R
	
R&2P
F-MEASURE S
PARAMAX
44 .36 45 .91 42 .9 1
Table G82. TST3
	
Progress Scores
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON RFC
	
PRE OVG FA L
MATCHED/MISSING 1053 411 193
	
52
	
65 3
	
27 101
	
743 549 21
	
53 2 4
MATCHED/SPURIOUS 428 528 193
	
52
	
65 3
	
27 218
	
118 299 51
	
41 4 1
MATCHED ONLY 428 411 193
	
52
	
65 3
	
27 101
	
118 200 51
	
53 2 4
ALL TEMPLATES 1053 528 193
	
52
	
65 3
	
27 218
	
743 648 21
	
41 4 1
SET FILLS ONLY 569 228 118
	
23
	
34 0
	
15 53
	
394 360 23
	
57 23 0
STRING FILLS ONLY 318 120 47
	
4
	
21 3
	
4 48
	
246 185 15
	
41 4 0
TEXT FILTERING 60 40 36
	
•
	
• •
	
• 4
	
24 36 60
	
90 10 1 0
P&R 2P&R
	
R&2P
F-MEASURES 27 .77 34 .44
	
23 .27
Table G83. TST2 Progress Scores
G-2 8
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS
	
NON REC
	
PRE OVG FA L
MATCHED/MISSING 1388 1389 519
	
146
	
210 23
	
92 514
	
513
	
593 43
	
43 3 7
MATCHED/SPURIOUS 1043 2627 519
	
146
	
210 23
	
92 1752
	
168
	
1133 57
	
22 6 7MATCHED ONLY
1043 1389 519
	
146
	
210 23
	
92 514
	
168
	
394 57
	
43 3 7
ALL TEMPLATES 1388 2627 519
	
146
	
210 23
	
92 1752
	
513
	
1332 43
	
22 6 7SET FILLS ONLY
759 780 321
	
68
	
91 0
	
52 300
	
279
	
403 47
	
46 38 2
STRING FILLS ONLY 403 443 136
	
13
	
80 21
	
12 214
	
174
	
186 35
	
32 4 8
P&R 2P&R
	
R&2P
F-MEASURES
PRC
29 .11 24 .38 36 .1 1
Table G84. TST3
	
Progress
	
Scores
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS
	
NON REC
	
PRE OVG FA L
MATCHED/MISSING 1054 139 65
	
36
	
28 2
	
30 10
	
925
	
573 8
	
60 7MATCHED/SPURIOUS 207
1153 65
	
36
	
28 2
	
30 1024
	
78
	
1516 40
	
7 8 9
MATCHED ONLY 207 139 65
	
36
	
28 2
	
30 10
	
78
	
82 40
	
60 7ALL TEMPLATES
1054 1153 65
	
36
	
28 2
	
30 1024
	
925
	
2007 8
	
7 8 9
SET FILLS ONLY 570 88 46
	
23
	
9 0
	
22 10
	
492
	
369 10
	
65 11 0
STRING FILLS ONLY 320 22 10
	
2
	
10 2
	
2 0
	
298
	
200 3
	
50 0
TEXT FILTERING 66 83 65
	
•
	
• •
	
• 18
	
1
	
16 98
	
78 22 5 3
P&R 2P&R
	
R&2 P
F-MEASURES 7 .47 7 .18 7 .7 8
Table G85. TST2 Progress Scores
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS
	
NON REC
	
PRE OVG FA L
MATCHED/MISSING 1269 519 322
	
75
	
75 15
	
26 47
	
797
	
734 28
	
69 9
MATCHED/SPURIOUS 744 885 322
	
75
	
75 15
	
26 413
	
272
	
791 48
	
41 4 7MATCHED ONLY 744
519 322
	
75
	
75 15
	
26 47
	
272
	
369 48
	
69 9ALL TEMPLATES 1269 885 322
	
75
	
75 15
	
26 413
	
797
	
1156 28
	
41 4 7SET FILLS ONLY 691
284 198
	
23
	
39 4
	
5 24
	
431
	
483 30
	
74 8 0
STRING FILLS ONLY 368 119 66
	
13
	
17 11
	
13 23
	
272
	
247 20
	
61 1 9
P&R 2P&R
	
R&2P
F-MEASURES
SRA
33 .28 37 .52 29 . 9
Table G86. TST3
	
Progress
	
Scores
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS
	
NON REC
	
PRE OVG FAL
MATCHED/MISSING 1265 510 311
	
88
	
67 13
	
44 44
	
799
	
717 28
	
70 9MATCHED/SPURIOUS 688 1132
311
	
88
	
67 13
	
44 666
	
222
	
1012 52
	
31 5 9
MATCHED ONLY 688 510 311
	
88
	
67 13
	
44 44
	
222
	
341 52
	
70 9ALL TEMPLATES 1265 1132
311
	
88
	
67 13
	
44 666
	
799
	
1388 28
	
31 5 9
SET FILLS ONLY 689 280 196
	
32
	
25 1
	
16 27
	
436
	
472 31
	
76 10 0STRING FILLS ONLY 366
122 70
	
13
	
22 12
	
13 17
	
261
	
241 21
	
63 1 4
P&R 2P&R
	
R&2P
F-MEASURES
SRI
29 .42 30 .35 28 .5 5
Table G87. TST3
	
Progress
	
Scores
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS
	
NON REC
	
PRE OVG FA L
MATCHED/MISSING 1089 395 217
	
85
	
69 6
	
54 24
	
718
	
585 24
	
66 6MATCHED/SPURIOUS 558 650 217
	
85
	
69 6
	
54 279
	
187
	
664 46
	
40 4 3MATCHED ONLY
558 395 217
	
85
	
69 6
	
54 24
	
187
	
304 46
	
66 6ALL TEMPLATES 1089 650 217
	
85
	
69 6
	
54 279
	
718
	
945 24
	
40 4 3
SET FILLS ONLY 590 221 133
	
36
	
36 2
	
26 16
	
385
	
380 26
	
68 7 0STRING FILLS ONLY 331
87 41
	
14
	
24 4
	
14 8
	
252
	
201 14
	
55 9
TEXT FILTERING 63 53 44
	
•
	
• •
	
• 9
	
19
	
28 70
	
83 17 2 4P&R 2P&R
	
R&2P
F-MEASURES 30 .0 35.29 26 .0 9
Table G88. TST2 Progress Scores
SLOT POS ACT COR PAR INC LCR
	
IPA SPU
	
MIS
	
NON REC
	
PRE OVG FA L
MATCHED/MISSING 1306 850 539 96 101 13
	
62 114
	
570
	
719 45
	
69 1 3MATCHED/SPURIOUS 963 1063
539 96 101 13
	
62 327
	
227
	
726 61
	
55 3 1
MATCHED ONLY 963 850 539 96 101 13
	
62 114
	
227
	
504 61
	
69 1 3ALL TEMPLATES 1306 1063
539 96 101 13
	
62 327
	
570
	
941 45
	
S5 3 1
SET FILLS ONLY 708 455 310 37 45 0
	
25 63
	
316
	
479 46
	
72 14 0STRING FILLS ONLY 383 237
135 21 31 13
	
21 50
	
196
	
238 38
	
61 2 1
P&R 2P&R
	
R&2 P
F-MEASURES 49 .5 52 .66
	
46 .7
Table G89. TST3 Progress Scores
G-2 9
UMBC/ConQuest
SLOT POS ACT COR
	
PAR
	
INC ICR IPA SPU
	
MIS NON REC
	
PRE
	
OVG
	
FA L
MATCHED/MISSING 1018 13 7
	
1
	
5 0 1 0
	
1005 561 1
	
58
	
0MATCHED/SPURIOUS 23 296 7
	
1
	
5 0 1 283
	
10 766 33
	
2
	
9 6MATCHED ONLY
23 13 7
	
1
	
5 0 1 0
	
10 12 33
	
58
	
0
ALL TEMPLATES 1018 296 7
	
1
	
5 0 1 283
	
1005 1315 1
	
2
	
9 6SET FILLS ONLY
550 10 4
	
1
	
5 0 1 0
	
540 365 1
	
45
	
0
	
0
STRING FILLS ONLY 310 2 2
	
0
	
0 0 0 0
	
308 192 1
	
100
	
0TEXT FILTERING
60 28 22
	
•
	
• • • 6
	
38 34 37
	
78
	
21
	
1 5
P&R 2P&R
	
R&2 P
F-MEASURES 1 .33 1 .67 1 .1 1
Table G90. TST2 Progress
	
Scores
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON REC
	
PRE
	
OVG
	
FA L
MATCHED/MISSING 1220 57 33
	
9
	
12 1
	
5 3
	
1166 731 3
	
66
	
5
MATCHED/SPURIOUS 141 164 33
	
9
	
12 1
	
5 110
	
87 475 26
	
23
	
6 7
MATCHED ONLY 141 57 33
	
9
	
12 1
	
5 3
	
87 55 26
	
66
	
5
ALL TEMPLATES 1220 164 33
	
9
	
12 1
	
5 110
	
1166 1151 3
	
23
	
6 7SET FILLS ONLY 664
36 25
	
4
	
6 0
	
0 1
	
629 479 4
	
75
	
3
	
0
STRING FILLS ONLY 354 21 8
	
5
	
6 1
	
5 2
	
335 248 3
	
50
	
1 0
P&R 2P&R
	
R&2P
F-MEASURE S
UMASS
5 .31 9 .86 3 .6 3
Table G91. TST3
	
Progress
	
Scores
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON REC
	
PRE
	
OVG
	
FA L
MATCHED/MISSING 1110 680 451
	
74
	
90 10
	
48 65
	
495 581 44
	
72
	
1 0
MATCHED/SPURIOUS 825 1193 451
	
74
	
90 10
	
48 578
	
210 1019 59
	
41
	
4 8
MATCHED ONLY 825 680 451
	
74
	
90 10
	
48 65
	
210 414 59
	
72
	
1 0
ALL TEMPLATES 1110 1193 451
	
74
	
90 10
	
48 578
	
495 1186 44
	
41
	
4 8
SET FILLS ONLY 602 373 260
	
31
	
47 1
	
24 35
	
264 380 46
	
74
	
9
	
0
STRING FILLS ONLY 336 179 112
	
11
	
26 9
	
11 30
	
187 197 35
	
66
	
1 7
TEXT FILTERING 64 76 62
	
•
	
• 14
	
2 22 97
	
82
	
18
	
3 9
P&R 2P&R
	
R&2P
F-MEASURES 42 .45 41 .57 43 .3 7
Table G92. TST2 Progress Scores
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON REC
	
PRE
	
OVG
	
FAL
MATCHED/MISSING 1310 89S 545
	
118
	
108 5
	
71 124
	
539 710 46
	
67
	
1 4
MATCHED/SPURIOUS 994 1059 545
	
118
	
108 5
	
71 288
	
223 628 61
	
57
	
2 7
MATCHED ONLY 994 895 545
	
118
	
108 5
	
71 124
	
223 470 61
	
67
	
1 4
ALL TEMPLATES 1310 1059 545
	
118
	
108 5
	
71 288
	
539 868 46
	
57
	
2 7
SET FILLS ONLY 712 496 326
	
49
	
50 0
	
34 71
	
287 475 49
	
71
	
14
	
0
STRING FILLS ONLY 383 247 130
	
22
	
42 5
	
21 53
	
189 232 37
	
57
	
2 1
P&R 2P&R
	
R&2P
F-MEASURES
UMICH
50 .91 54 .4 47 .8 5
Table G93. TST3
	
Progress
	
Scores
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON REC
	
PRE
	
OVG
	
FAL
MATCHED/MISSING 1254 868 469
	
81
	
138 11
	
42 180
	
566 661 41
	
59
	
2 1
MATCHED/SPURIOUS 874 1289 469
	
81
	
138 11
	
42 601
	
186 709 58
	
40
	
4 7
MATCHED ONLY 874 868 469
	
81
	
138 11
	
42 180
	
186 392 58
	
59
	
2 1
ALL TEMPLATES 1254 1289 469
	
81
	
138 11
	
42 601
	
566 978 41
	
40
	
4 7
SET FILLS ONLY 687 506 287
	
14
	
88 4
	
1 117
	
298 433 43
	
58
	
23
	
1
STRING FILLS ONLY 363 232 115
	
13
	
41 7
	
13 63
	
194 224 33
	
52
	
2 7
P&R 2P&R
	
R&2 P
F-MEASURES
USC
40.49 40 .2 40 . 8
Table G94. TST3
	
Progress
	
Scores
SLOT POS ACT COR
	
PAR
	
INC ICR
	
IPA SPU
	
MIS NON REC
	
PRE
	
OVG
	
FA L
MATCHD/MISSING 1226 132 72
	
25
	
24 4
	
9 11
	
1105 725 7
	
64
	
8
MATCHED/SPURIOUS 266 535 72
	
25
	
24 4
	
9 414
	
145 731 32
	
16
	
7 7
MATCHED ONLY 266 132 72
	
25
	
24 4
	
9 11
	
145 89 32
	
64
	
8
ALL TEMPLATES 1226 535 72
	
25
	
24 4
	
9 414
	
1105 1367 7
	
16
	
7 7
SET FILLS ONLY 667 75 43
	
13
	
14 0
	
1 5
	
597 477 7
	
66
	
7
	
0
STRING FILLS ONLY 357 36 18
	
4
	
8 1
	
4 6
	
327 244 6
	
56
	
1 7
P&R 2P&R
	
R&2 P
F-MEASURES 9 .74 12 .73
	
7 .89
Table G95. TST3 Progress Scores
G-3 0
SECTION 5: F-MEASURES USED FOR STATISTICAL ANALYSI S
The F-measures used for statistical analysis are calculated using the floating point value s
of recall and precision rather than those derived by integer arithmetic in the scorin g
program. The F-measures in this section differ from those produced and reported by th e
scoring program because the F-measures in the scoring program are calculated based o n
the rounded values of recall and precision that are shown in the score reports .
Site F(P&R) F(2P&R) F(P&2R)
BBN 36.01 40.58 32.37
GE 56.01 54.97 57.08GE-CMU 51.98 53.92 50.17
HUGHES 22.65 19.63 26.7 8
LSI 18.66 16.75 21.07MDESC 24.33 27.48 21 .83
MITRE 9.87 8.85 11 .1 6
NMSU-BRANDEIS 23.36 24.30 22.49NYU 43.86 45.75 42.12
PARAMAX 29.03 24.39 35.85PRC 33.00 37.41 29.52
SRA 29.33 31 .02 27.8 1SRI 48 .88 52.51 45.73
UMBC-CONQUEST 4.47 8.46 3.04UMASS 51.61 54.92 48.68
UMICHIGAN 40.57 40.20 40.95USC 9.27 12.21 7.48
Table G96. F-Measures for TST3
SITE F(P&R) F(2P&R) F(P&2R)
BBN 41 .64 42.73 40.60
GE 57.05 54.32 60.07GE-CMU 53.27 53.25 53.28
HUGHES 24.19 20.24 30.06LSI 15.23 12.87 18.66
MDESC 26.55 29.46 24.17MITRE 14.03 11.60 17 .75
NMSU-BRANDEIS 21.52 20.35 22.82NYU 46.16 46.19 46.1 4
PARAMAX 28.27 22.46 38 .1 1
PRC 32.84 36.81 29.64SRA 34.46 32.52 36.65
SRI 47.85 50.13 45.77UMBC-CONQUEST 5.79 10.10 4.06
UMASS 46.35 45.58 47.1 5UMICHIGAN 35.44 34.79 36.1 1
USC 17.85 22.95 14.61
Table G97. F-Measures for TST4
