International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol XXXV, Part B3. Istanbul 2004 
  
Application 
  
Data | Software 
  
  
  
  
  
; 7 = Requirements 
: Performance: : B on the output 
2 “Characterization R(c, ) 
E : | | eium 
= PSS : cost 
= | Input | Performance : te functions 
Characterization | Charcieristics 5S 7 
kn Gey 5 ERO S. 
  
  
  
  
   
  
  
  
Evaluation: | 
    
  
   
  
  
Instanciation of : à P DH : 
Sr new — cInstanciation of | rus iY ame 
S 2 input Input performance Output | Instanciation of costs due to 
= pu Input i characteristics [=| characteristics |—»— characteristics. ^" | characteristics |-æ| —— costfunctions . |- deficiencies 
$$ ET à & c = > & 
GR C ( i ) € = Ge) © = ER(c, ) Co ) SS : 
= o i = : 
  
  
mL 
  
  
Figure 1: Work-flow for characterization and performance evaluation of algorithms. 
fm nn 
iin alent ret Model I 
UE Intention im D 
level 
; Intention 
  
level 
| 
I I 
| I 
| nd 
fo : 
Function F1 ) ze «uq 
ia ge emendi 
| a 
I 
| 
| I 
Function : 
  
  
level 
> 
UE 
= 
= 
> 
Algorithm 
Level 
Implementation 
CE ; 
Spr il > Impl. o 
ENIM uere Reality! 
I 
umm um mem 7 ce ma am me en fs ne Em MA ee 
Figure 2: Abstraction hierarchy of algorithms 
implementation. 
In the model area, functions, algorithms or implementations are 
analyzed theoretically based on models that describe the data and 
models that describe the function, the algorithm or the implemen- 
tation. In the reality domain, empirical tests on synthetic data are 
needed to validate the implementation or to simulate the behavior 
of the implementation on data that follows a certain model. 
Neither theoretical analysis nor empirical testing on synthetic data 
gives full information about how well real world input data fits 
the data model that is inherent in an algorithm or an implementa- 
tion. To validate model assumptions in reality, tests in real envi- 
ronments have to be carried out using reference data to measure 
deviations of system-immanent model assumptions from reality. 
3 METHODS FOR REFERENCE DATA DEFINITION IN 
REAL ENVIRONMENTS 
There are several ways to define reference data for characterizing 
implementations of algorithms in real environments. Reference 
data for the output of an implementation may be specified ex- 
plicitely by a human or be defined as the result of a reference 
implementation of a reference algorithm on given data. Further- 
more, reference data for the output of an arbitrary implementation 
I may be defined as the output of / on reference input data. 
Our first approach to generate reference data matches best the 
concept of reference data from a reference algorithm. In our case, 
the reference algorithm employs an algorithm A on many hetero- 
geneous data sets D; and combines the results o; in a robust es- 
timation process to provide the reference output of the algorithm 
1062 
A on each data set. The second approach supports defining refer- 
ence output based on reference input. It provides noiseless refer- 
ence input data by exploiting the noise characteristics of Gaussian 
image pyramids. 
3.1 Reference data from multiple views. 
The character and the amount of deficiencies in the output of 
computer vision algorithms may depend on the perspective under 
which an object is observed. For characterizing view-dependent 
properties of an algorithm, test scenarios are needed based on 
multiple views, with reference data defining the true result for 
each view. In the following, we propose a method for automati- 
cally generating reference data from multiple views. 
Procedure. Assuming that the outputs an of algorithm on multi- 
perspective images can be combined in such a way that the com- 
bination of all results yields an error-free or nearly error free re- 
sult, reference data for the output on each data set may be esti- 
mated as follows: 
I. A large number of images is taken from the same (parts of) 
an object. To minimize viewpoint-dependent errors in step 
2, the exposure setup chosen is such that effects of errors in 
analyzing individual views are widely compensated for over 
the whole set of views. 
t2 
A reference algorithm or the algorithm to be characterized 
itself is applied to each image 7;, yielding the individual 
outputs o;. 
3. The individual outputs o; are fused to obtain a common error 
free result a. 
4. Reference outputs for the individual views are derived from 
the error free result, e.g. by projection. 
3.2 Reference data from multiple resolutions 
Our approach to reference data acquisition from multiple resolu- 
tions provides nearly noise-free input images with natural image 
structures by exploiting the noise characteristics of Gaussian im- 
age pyramids. It is applicable to single images and serves refer- 
ence data for at least two questions: 
I. Investigations concerning the noise sensitivity of an algo- 
rithm may compare its results on noise-free input with the 
results on noisy data with known noise characteristics. For 
this purpose, noiseless test data and test data with known 
noise characteristics is required. 
    
Internat 
to 
z 
C 
Gaussiz 
] at mul 
up a Ga 
Ji eta 
We start 
2 
[pel] a 
image 7 
with an 
of filter 
sampled 
for all (: 
ing the 
Noise cl 
Smooth; 
noise. 
inal ima 
given by 
Thus, if 
image p 
by 
Obvious 
creasing 
Procedi 
images 
the sam 
qualifie: 
data for 
ture is v 
as the de 
compen 
To denx 
acquisiti 
feature «