640 
4. Computation of super resolution images: With the applied 
strategy for matching and linking the pixels in the images 
which participated in the matching process were observed 
multiple times. This fact is exploited to compute images 
preserving the same geometry as the original ones, but with 
an enhanced image quality regarding the noise and the ef 
fective resolution. 
5. Forward intersection: The correspondence information as 
retrieved from the matching and the subsequent linking are 
used for multi-view forward intersection to obtain 3D coor 
dinates for the matched points including colour information. 
Input video frames 
For the implementation of the workflow at hand, the commercial 
software Boujou (2d3, 2008) is currently being used. Besides the 
fully automatic reconstruction up to scale, it is possible to define 
constraints on the actual scene geometry, like known distances 
in object space between feature points. Further, the coordinate 
frame can be fixed through the definition of plane constraints. 
As an additional unknown the radial distortion coefficient is es 
timated and the possibility to compute undistorted images is of 
fered to the user. Refer to (Dobbert, 2005) for detailled informa 
tion on the approach as implemented in Boujou. 
In the subsequent steps the undistorted images, 3D feature coor 
dinates, the corresponding image points and the individual pro 
jection matrices are used. 
2.2 Matching strategy 
The aim of the data processing described in this paper is to derive 
two final datasets, namely so-called super resolution images and 
a 3D representation of the scene which can be used e.g. for vi 
sualisation tasks. Both products require to establish dense image 
correspondences. Apart from some special cases (Heinrichs et 
al., 2007), matching is normally done in stereo image pairs, thus 
it is required to link stereo correspondences across the sequence. 
In this paper it is proposed to increase the reliability of match 
ing by applying two kinds of matches: long baseline matches and 
short baseline matches, refer also to Figure 2. The basic idea is 
long baseline matches 
1 
2 
3 
4 
5 
6 
7 
8 
9 
Super resolution images 
Figure 1: Workflow 
Forward intersection: 
coloured 3D point cloud 
2.1 Structure and motion recovery 
In most cases where a procedure as described in this paper is ap 
plied, uncalibrated, non-metric cameras are used, and in contrast 
to conventional (airborne) remote sensing, precise navigation in 
formation through GPS/IMU is normally not available. Thus, the 
full information on the individual camera poses throughout the 
sequence including intrinsic camera parameters need to be re 
covered from the images. The initial step consists in retrieving 
image-to-image correspondences by feature tracking. Through 
a subsequent bundle adjustment including self-calibration, the 
scene can be reconstructed up to scale if no additional knowledge 
on the scene geometry is available. Further information on the 
structure and motion recovery can be found in several sources, 
e.g. (Hartley and Zisserman, 2004, Pollefeys et al., 2004). 
short baseline matches 
Figure 2: Matching Strategy 
that through the short baseline matches correspondences between 
consecutive matches are established and linked, i.e. a matching 
pair (rrii, TOj+i) in image i is linked with (rnj+i, rm + 2) and thus 
establishing the additional match (mi,rrn + 2) if the respective 
pixel rrii+i refers to an identical location in image i + 1. Besides 
this linking chain, the long baseline matches establish a direct 
match between the pairs which are already connected through the 
linked short baseline matches. This procedure results in a higher 
redundancy of matches and thus helps to increase the reliability: 
If for instance a correspondence (mi, rrn+2) as derived through 
short baseline matches does not fit to the direct match (mi, m' +2 ) 
from the long baseline match, the correspondences are regarded 
as wrong and skipped in the subsequent processing. 
2.3 Dense stereo matching 
The approach to dense stereo matching as applied in the current 
implementation is the Semi-Global Matching algorithm (Hirsch- 
miiller et al., 2005, Hirschmiiller, 2008). The basic idea behind 
this technique is to aggregate local matching costs by a global 
energy function, which is approximated by an efficient pathwise 
1-dimensional optimisation. 
The local matching costs can be derived by several methods, like 
cross-correlation or intensity differences; in the present case they 
are computed using an hierarchical Mutual Information approach 
(Viola and Wells, 1997). During cost aggregation not only the 
local matching cost is considered, but additional penalties are 
defined by considering disparities in the vicinity of a particular 
pixel p with the aim to preserve smoothness and height jumps: