CMRT09: Object Extraction for 3D City Models, Road Databases and Traffic Monitoring - Concepts, Algorithms, and Evaluation 
Within the last years, the matching of multiple views of 
an object enabled the reconstruction of 3D object points 
with high accuracy and high density. Previous approaches 
as (Kanade and Okutomi, 1994) are based on a low-level 
preprocessing of the image to extract points of interest. 
Then, the correspondences of such points are used to es 
timate the 3D position of the object points. In many ap 
plications, Forstner-features (Forstner and Gulch, 1987) 
or SIFT-features (Lowe, 2004) are used, but the derived 
point clouds are either sparse or have been extracted from 
many images or video, e. g. (Mayer and Reznik, 2005) and 
(Gallup et al., 2007). In (Tuytelaars and Van Gool, 2000), 
the correspondences are determined over local affinely in 
variant regions, which were extracted from local extrema 
in intensity images. This procedure is liable to make match 
ing mistakes if the image noise is relatively high. 
Dense point clouds from only a few images are obtained 
by adjusting the correspondence between pixels by correla 
tion based on (semi-) global methods, e. g. (Hirschmuller, 
2005). Assuming the observed objects have a smooth sur 
face, the accuracy of the obtained point clouds gets in 
creased by including information on the relations between 
the pixels by a Markov random field, e. g. (Yang et al., 
2009), or from image segmentation, e. g. (Tao and Sawh- 
ney, 2000). 
In our approach, we take up the idea of (Khoshelham, 2005) 
to improve an initial image segmentation using additional 
3D information. From multi-view analysis, we derive a 
point cloud, which is used for deriving additional features 
for the segmented image regions. We focus on building 
scenes, whose objects mostly consist of planar surfaces. 
So, it is reasonable to look for dominant planes in the point 
cloud, where the search is guided by the image segmenta 
tion. 
For us, it is important to realize an approach, which has 
the potential to get automatized since there are many ap 
plications with thousands of images. There is a need for a 
completely automatic procedure if additional features are 
derived from a reconstructed point cloud to improve the 
segmentation or interpretation of the images. Our input are 
only two or more images from the object, which were taken 
by a calibrated camera. An example is shown in fig. 1. 
3 RECONSTRUCTION OF THE 3D SCENE 
In this section, we describe the generation of the point 
cloud C from the given images. For this generation, there 
are two conditions, which should be fulfilled: (a) the ob 
served objects should be textured sufficiently and (b) the 
views must overlap, otherwise we have problems to deter 
mine the relative orientation between the images. So far, 
the implemented algorithms need some human interaction 
for setting the point cloud scale and the disparity range pa 
rameters, but under certain conditions, the whole approach 
could get designed to perform completely automatically. 
We describe the procedure with two or three given images 
Ji, 12 and /3. Two views are necessary to reconstruct the 
Figure 1: Three aerial views of a building scene consisting 
of a flat roofed part and a gable roofed part. The initial 
segmentation of the upper view is shown on its right side. 
The ground consists of several weirdly shaped regions, and 
the flat roof is also not well segmented. 
Figure 2: Reconstructed 3D-points are projected back into 
2D-image (white). Left: all pairs of matches are shown. 
The point cloud is very dense with approximately 75% of 
pixels having a 3D point, but these points are very impre 
cise. Right: only matches in all three images are shown. 
The point cloud is still dense with approximately 30% of 
pixels having a 3D point with higher precision. 
observed 3D data, but if the matching is performed over 
three images, the point cloud is still dense, see fig. 2, and it 
contains more reliable points, thus less outliers. The recon 
struction process can get improved if even more images are 
considered. If all used images were taken by a calibrated 
camera, we are able to reconstruct the 3D scene by per 
forming the following steps. 
In the first step we determine the relative orientations be 
tween the given images. Of course, it can be skipped if the 
projection matrices have been estimated during image ac 
quisition. Otherwise, due to the calibration of the camera 
we eliminate automatically the non-linear distortions using 
the approach of (Abraham and Hau, 1997). The matching 
of extracted key-points using the approach of (Lowe, 2004) 
leads to the determination of the relative orientations of all 
images, i. e. their projection matrices P n , cf. (Labe and 
Forstner, 2006). The success of the relative orientation can 
212