ISPRS Commission III, Vol.34, Part 3A ,,Photogrammetric Computer Vision", Graz, 2002
image using a least-squares adjustment (see also (Wallack and
Manocha, 1998)). Approximate transformation parameters are
assumed to be known, which can be obtained by any preceding
object recognition method that uses the edge position and orien-
tation as features, e.g., the shape-based matching or the modified
Hough transform. The minimization is realized using a single
step algorithm (Press et al., 1992). This approach is described
more extensively in (Ulrich and Steger, 2002). We implemented
the least-squares adjustment as an extension of the shape-based
matching, which returns the requested approximate values accu-
rately enough.
3 EVALUATION
3.1 Evaluation Criteria
We use three main criteria to evaluate the performance of the six
object recognition methods and to build a common basis that fa-
cilitates an objective comparison.
The first criterion to be considered is the robustness of the ap-
proach. This includes the robustness against occlusions, which
often occur in industrial applications, e.g., caused by overlapping
objects on the assembly line or defects of the objects to be in-
spected. Non-linear as well as local illumination changes are also
crucial situations, which cannot be avoided in many applications
over the entire field of view. Therefore, the robustness against
arbitrary illumination changes is also examined. A multitude of
images were taken to simulate different overlapping and illumi-
nation situations (see Section 3.2). We measure the robustness
using the recognition rate, which is defined as the number of im-
ages in which the object was correctly recognized divided by the
total number of images.
The second criterion is the accuracy of the methods. Most appli-
cations need the exact transformation parameters of the object as
input for further investigations like precise metric measurements.
In the area of quality control, in addition, the object in the search
image must be precisely aligned with the transformed reference
image to ensure a reliable recognition of defects or other varia-
tions that influence certain quality criteria, e.g., by subtracting the
gray values of both images. We determine the subpixel accuracy
by comparing the exact (known) position and orientation of the
object with returned parameters of the different candidates.
The computation time represents the third evaluation criterion.
Despite the increasing computation power of modern micropro-
cessors, efficient and fast algorithms are more important than
ever. This is particularly true in the field of object recognition,
where a multitude of applications enforce real time computation.
Indeed, it is very hard to compare different recognition methods
using this criterion because the computation time strongly de-
pends on the individual implementation of the recognition meth-
ods. Nevertheless, we tried to find parameter constellations (see
Section 3.2) for each of the investigated approaches that at least
allow a qualitative comparison.
Since the Hausdorff distance does not return the object position
in subpixel accuracy and in addition does not use image pyra-
mids resulting in unreasonably long recognition times, the cri-
teria of accuracy and computation time are only applied to the
five remaining candidates. The least-squares adjustment is im-
plemented as a subsequent refinement step in combination with
the shape-based matching. Therefore, only the accuracy and the
recognition time of the least-squares adjustment are analyzed,
since the robustness is not affected and hence is the same as the
robustness of the underlying recognition approach.
A - 370
Tunt Fa ad
NN
»
LS
-
Li
=
-
-
-
ra
-
=
i
-
-
-
-
-
-
a
p
LLLI
A
LLLA ul
14] LL LA
PL ed
Figure 1: An IC is used as the object to be recognized.
3.2 Experimental Set-Up
In this section the experimental set-up for the evaluation is ex-
plained in detail. We chose an IC, which is shown in Figure 1,
as the object to be found in the subsequent experiments. Only
the part within the bounding box of the print on the IC formes
the ROI, from which the models of the different recognition ap-
proaches are created. For the recognition methods that segment
edges during model creation (Hausdorff distance, shape-based
matching, modified Hough transform, least-squares adjustment)
the threshold for the minimum edge amplitude in the reference
image was set to 30 during all our experiments. The images we
used for the evaluation are 8 bit gray scale of size 652 x 494
pixels. For all recognition methods using image pyramids, four
pyramid levels were used to speed up the search, which we found
to be the optimum number for our specific object. When using
PatMax®, there is no parameter that allows to explicitly spec-
ify the number of image pyramids to use. Instead, the parameter
coarse grain limit can be used to control the depth of the hier-
archical search, which has a similar meaning as, but can not be
equated with, the number of pyramid levels. Since this parame-
ter can be set automatically, we assumed the automatically deter-
mined value as the optimum one and did not use a manual setting.
3.2.1 Robustness To apply the first criterion of robustness and
determine the recognition rate two image sequences were taken,
one for testing the robustness against occlusions the other for test-
ing the sensibility to illumination changes. We defined the recog-
nition rate as the number of images, in which the object was rec-
ognized at the correct position divided by the total number of
images.
The first sequence contains 500 images of the IC, which was oc-
cluded to various degrees with various objects, so that in addition
to occlusion, clutter of various degrees was created in the im-
age. Figure 2 shows two of the 500 images that we used to test
the robustness against occlusion. For the approaches that seg-
ment edges in the search image (modified Hough transform and
Hausdorff distance) the minimum edge amplitude in the search
image was set to 30, i.e., to the same value as in the reference
image. The size of the bounding box is 180 x 120 pixels at the
lowest pyramid level, i.e., at original image resolution, contain-
ing 2127 edge points extracted by the Sobel filter. In addition to
the recognition rate, the correlation between the actual occlusion
and the returned score values are examined, because the corre-
lation between the visibility of the object and the returned score
value is also an indicator for robustness. If, for example, only
half of the object is visible in the image then, intuitively, also the
score should be 50%, i.e., we expect a very high correlation in
the ideal case. For this purpose, an effort was made to keep the
IC in exactly the same position in the image in order to be able
to measure the degree of occlusion. Unfortunately, the IC moved
very slightly (by less than one pixel) during the acquisition of
the images. The true amount of occlusion was determined by ex-
tracting edges from the images and intersecting the edge region
with the edges within the ROI in the reference image. Since the
