Inter- Reference Data Total 
preted 
data 1 2 3 4 other 
1 Xa Xp Xj Xu Xo T 
2 21 X» X X o T; 
3 31 X2 X4 X4 X o T, 
4 Xa Xo X4 Xa Xo Ty 
Missed Na Son a Xie - - 
Total R, R, R; R, - ER 
  
  
  
  
  
  
  
  
  
  
  
  
Table 3: Concept of a confusion matrix 
nant 
X,: quantity of features interpreted as feature "i 
data and as feature j in the reference data. 
X,.: quantity of features j in the reference data, which have no 
equivalent in the test data ("missed"). 
X,: quantity of features "i" in the test data, which have either 
no equivalent in the reference data, or an equivalent which 
is not one of the features considered here ("other"). 
in the test 
As quantity was taken: the number of features for point features, 
the line length for line features and the area (in m?) for area 
features . 
From these tables were derived: 
Producers accuracy: P, - (X; / R;) * 100% (also called "inter- 
pretation accuracy".) Gives for feature "i" the percentage of 
detected and correctly classified features. The opposite of it is 
Omissions (O, = 100% - P;), which gives for feature "i" the 
percentage of undetected or wrongly classified features. 
Users Accuracy: U, - (X, / Tj) * 10096 (also called "feature 
accuracy".) Gives for all features which were classified as 
feature "i" how many are really "i". The opposite of it is 
Commissions (C, = 100% - U;), which gives for all features 
classified as feature "i" the percentage of wrong classifications. 
Missed rate: M, - (X,, / Rj) (also called "undetection rate" or 
"total omissions".) Gives the percentage of features "i", which 
were not detected. 
Fault rate: F, — 10096 - P, -M, Percentage detected, but 
misclassified features "i". 
Overall Producers Accuracy: OPA = X(Xj) / ER the 
weighted mean (by sample size) of the Producers accuracy per 
feature type. 
To distinguish the different operators, the producers accuracy 
was calculated separately for the eastern and western half of the 
test area. After eliminating the most obvious gross errors the 
final calculations were made. For the different digitizations the 
Producers accuracy (P), the Users accuracy (U) and the Missed 
rate (M) were tabulated per feature, and the Overall Producers 
Accuracy (OPA) per feature group added. 
3. RESULTS OF EXPERIMENTS 
3.1 Stereo interpretations 
A comparison of the stereo interpretability of digitized images 
and their analogue ancestors is given in table 4. There were no 
monuments and no powerlines in the reference data and only a 
few towers, reservoirs and cemeteries. These features are not 
included in the table, as their sample size is too small to be 
considered separately. In the "Overall Producers Accuracy" they 
are however included. 
  
  
  
photoscale 1:60,000 
1:30,000 
  
pixel size 60um (3.6m) 30um (1.8m) 
15um (0.9m) 
60um (1.8m) 30um (0.9m) 15um (0.45m) 
  
Feature P U M P U M P U 
M P U M p U M P U|M 
  
Is. houses | 24 16 76 36 23 64 54 45 
46 (38 E33 01621556 1.33 | 44 |] 68 | 40 1 22 
  
bridges 44 4 504.56]. 434,35} 57.46 |i67 
54 | ‘60 100} 404-67 | 67 | 331 (822750 18 
  
OPA-point 26 35 47 
36 55 70 
  
footpaths 0 - 100. 35.1 81.:|: 48:4 66.5] 
3° 776771 19 | $6 19471714 ]^85 1782 1 IS 
  
sing.road 1 57 | (66.1 43 | 64: |.:53 |: 35 ] 78. | 74 
22.1 66.190 |. 32.1279 1. 951 21.1. 85,1. 92.015 
  
sing.railw. | 100 | 100| 0 |100|100| O | 100 | 100 
0 | 100|100| O | 100| 100) O | 100| 100| 0 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
ditches 0 - 100 | 73 |.100. 27 | 79 | 73 | 21° 58.190 | 42 | 31°} 100-419 | 36 1,100 | 14 
OPA-line 47 66 78 67 80 86 
vineyards | O - oro 31-4063 4762 | 58171 | 424];:/36:)5999:1 62: | 0111. 91 4 95 | 90 | 4 
orchards. | - 17 | 29-|. 83 |. 46. | 54-152. J55..| 180 | 40 | 63^, 63 1° 36 | 90 | 47 6 87 ] 75 | 1i 
OPA-area 6 42 57 46 91 95 
  
  
  
  
  
  
Table 4: Global results of the stereo interpretations 
The comparison between different operators showed consider- 
able inconsistencies. Some of them are easily explainable by the 
limited experience of the operators, which resulted in inconsis- 
tent application of the interpretation key. This is considered as 
unavoidable noise on the data. There were also gross errors (e.g. 
in one test an operator had simply forgotten to digitize bridges). 
308 
International Archives of Photogrammetry and Remote Sensing. Vol. XXXI, Part B4. Vienna 1996 
  
Such d 
had to 
size fo 
5096, v 
Fortun: 
ples, s 
change 
The ov 
with a 
lookin; 
the res 
for the 
the sm: 
feature 
worse . 
This in 
18m! 
from 1: 
of phot 
of ima; 
The mi 
sureme 
most c: 
fusion 
ted but 
in char 
identifi 
The us 
digitize 
for the 
ficatior 
  
  
Table : 
For line 
the ster 
photogr 
output | 
3.6 m 
contrad: