bul 2004 
PERFORMANCE EVALUATION IN NATURAL AND CONTROLLED ENVIRONMENTS 
APPLIED TO FEATURE EXTRACTION PROCEDURES 
Marc Luxen 
University of Bonn 
Institute for Photogrammetry, NuBallee 15, 53115 Bonn, Germany 
http://www.ipb.uni-bonn.de/~marc 
luxen@ipb.uni-bonn.de 
Commission III/8 
KEY WORDS: Algorithms, Performance, Environment, Reference Data, Feature, Extraction, Comparison 
ABSTRACT: 
The paper highlights approaches to reference data acquisition in real environments for the purpose of performance evaluation of image 
analysis procedures. Reference data for the input and for the output of an algorithm is obtained by a) exploiting the noise characteristics 
of Gaussian image pyramids and b) exploiting multiple views. The approaches are employed exemplarily in the context of evaluating 
low level feature extraction algorithms. 
1 INTRODUCTION 
As most image analysis procedures are addressed to real world 
applications, performance evaluation in natural environments is 
needed for the design, optimization and selection of algorithms 
(Canny, J.E., 1983; Fórstner, 1996; Maimone and Shafer, 1996). 
Considering the fact that many approaches to performance eval- 
uation rely on reference data, we propose two methods for refer- 
ence data acquisition in natural environments. 
The first approach provides reference output data based on a large 
number of multi-perspective images. It assumes that individ- 
ual outputs of an algorithm on images of different views can be 
fused in an estimation process that yields practically error-free 
estimates for the outputs on each individual image. 
The second approach provides almost noise-free images with nat- 
ural image structures by exploiting the noise characteristics of 
Gaussian image pyramids. It is applicable to single images and 
provides reference input at least for investigating the noise sensi- 
tivity of algorithms. 
Both methods are applied exemplarily for characterizing low level 
feature extraction algorithms. 
2 REFERENCE DATA IN PERFORMANCE 
CHARACTERIZATION 
To sketch the impact of reference data for performance character- 
ization purposes, this section discusses the role of reference data 
in characterizing and evaluating algorithms. 
2.1 Characterizing and evaluating algorithms 
Algorithms generally fulfill the requirements of specific tasks only 
to a limited extent. As an example, a point detection algorithm 
may only partly fulfill the requirements of object reconstruction, 
as it erroneously may leave out relevant points. 
We refer to evaluating an algorithm as the process and the re- 
sult of deriving statements on the usefulness of an algorithm with 
respect to a specific task, resulting e. g. in a score or expected 
1061 
costs. For example, an algorithm detecting screws on an assem- 
bly line may be evaluated by means of the expected costs which 
are associated with misdetection. 
Evaluation may be based on the results of performance character- 
ization. Performance characterization wants to provide applica- 
tion independent models C',(-) describing relevant properties c. 
of algorithms f and of their output dependent on properties c; of 
the input. As an example, performance characterization of corner 
extraction modules may provide models describing the precision 
of extracted points dependent on the image noise (cf. section 4). 
A general scheme for evaluating an algorithm may follow fig. I. 
Firstly, the input is characterized using methods for input data 
characterization, resulting in input characteristics c;. These input 
characteristics are used to instantiate the models C, (+) which re- 
sult from the characterization and describe the behavior of the al- 
gorithm dependent on the input characteristics, yielding estimates 
Co = Co(c;) for characteristics co of the algorithm and its output 
on the given input. Based on user-specified output requirements 
R(co) and cost functions E(co), the estimated characteristics 0; 
may be used to estimate the costs & = E(6,) which are to be 
expected in case the algorithm is applied to the given data. 
Abstraction hierarchy. Algorithms are implemented in pro- 
grams to fulfill a function with a certain intention. In design- 
ing a computer vision system, several alternative functions may 
be considered to follow a certain intention. For example, feature 
based matching and intensity based matching may be considered 
as functions following the intention of image matching. For each 
function there may exist multiple algorithms, e. g. cross correla- 
tion and least squares matching as algorithms for intensity based 
matching. Again, several different implementations of the same 
algorithm may be available. 
Characterization and evaluation may therefore take place on the 
level of the intention, the level of the function, the level of the al- 
gorithm or the level of the implementation, with the levels build- 
ing a characterization hierarchy of decreasing abstraction (cf. fig. 
2). 
2.2 The role of reference data 
Reference data serves for investigations on the lowest level of 
abstraction, i. e. for empirical investigations on the level of the