VIEW SYNTHESIS WITH THE TRIFOCAL TENSOR FROM UNCALIBRATED IMAGERY
Helmut Mayer“, Wolfram Büchner“, Thomas Riegel"
“Institute for Photogrammetry and Cartography, Bundeswehr University Munich, D-85577 Neubiberg, Germany
Helmut.Mayer @UniBw-Muenchen.de
bCorporate Technology, Siemens AG, Otto-Hahn-Ring 6, 81730 Munich, Germany
Thomas.Riegel @mchp.siemens.de
Working Group V/2
KEY WORDS: Visualization, Orientation, Hierarchical, Matching, Geometry
ABSTRACT
This paper describes a prototypical production flow ranging from (possib
orientation is based on hierarchical matching, the robust estimation o
ly) uncalibrated images to view synthesis based visualization. Sensor
f the fundamental matrix or the trifocal tensor, and (possibly) the
calibration of the camera. From epipolar resampled images we estimate a disparity map based on an approach which takes advantage of
the local neighborhood in three dimensional (3D) disparity space and which can deal well with large depth ranges and occlusions. This
information is then used for an efficient view synthesis based on the trifocal tensor.
1 INTRODUCTION
This paper presents a fully automatic prototypical production flow to
synthesize novel views with general view point and orientation from
given, (possibly) uncalibrated images. Our major achievements are
as follows:
e By means of a hierarchical approach based on image pyra-
mids we significantly reduce the search space for image match-
ing. Efficiency but also robustness are improved consider-
ably. Highly precise conjugate point are obtained from a least-
squares matching of points obtained from the Fórstner operator
(Fórstner and Gülch, 1987)
e We use the RANSAC (Fischler and Bolles, 1981) algorithm to
deal with the remaining mismatches based on the seven point
solution for the fundamental matrix or the six point solution for
the trifocal tensor. We take calibration information into account
and optimize the parameters via the Levenberg-Marquardt non-
linear algorithm.
e For view interpolation homologous points in two images are
needed. We utilize the approach proposed in (Zitnick and
Kanade, 2000) which smoothes locally in a 3D disparity array
and assumes that the surface is opaque, i.e, one ray of view will
hit the surface only once. Our modifications comprise an auto-
matic determination of the disparity range, determination of a
stop condition, sub-pixel improvement, and a fast computation
by separation of the 3D box-filter.
e We have optimized the view synthesis scheme proposed in
(Avidan and Shashua, 1998) by linearly projecting the points
as proposed in (Hartley and Zisserman, 2000) and, particularly,
by parameterizing the points according to the epipolar lines.
In Section 2 we introduce notations and basic concepts for stereo
pairs and image triplets comprising the fundamental matrix, the es-
sential matrix, and the trifocal tensor. Section 3 shows how the trifo-
cal tensor can be used for point transfer. The estimation of the image
orientations based on the RANSAC algorithm and the reduction of
the search space by image pyramids and geometric constraints, such
as the-epipolar constraint, is presented in Section 4. Section 5 details
the algorithm which we use for disparity estimation from epipolar
resampled imagery. This is the basis for view synthesis presented in
Section 6. We end up with conclusions.
2 BASICS OF LINEAR ORIENTATION
In our notation we distinguish homogeneous 2D and 3D vectors x
and X, respectively, as well as matrices P, which represent the same
object also after a change of the scaling factor A (bold), from Eu-
clidean vectors x and X as well as matrices R (bold italics).
A perspective transformation given the point in object space X is de-
scribed by equation (1). The exterior orientation is defined by pro-
jection center O(Xo) and rotation matrix R. The interior orientation
is modeled by principal distance c, principal point (25, y/,). scale
difference m of the coordinate axes, and skew of the axes s.
The 5 parameters chosen for the interior orientation are collected
into the calibration matrix
e cs T
K= 0 c(1+m) yn
0 0 1
With the projection matrix (K is the matrix K multiplied by an arbi-
trary scalar # 0) we finally end up with
P — KR(I| - Xo) and x = PX.
The fundamental matrix describes the (projective) relative orienta-
tion of the image pair. We assume that we have homologous points
x’ in the first image and x” in the second image. With the 3 x 3
fundamental matrix the coplanarity equation as condition for the ho-
mology of image points can be represented in a simple and elegant
way:
x'Fx' — 0.
This relation is linear in the image coordinates of both images, i.e.,
bilinear, and has some important properties:
e Because they refer to the original measured data, there is no
need for a reduction of the image coordinates. The reduction is
contained in the fundamental matrix.
—192—
If cal
esser
redu
essel
and |
geon
spac
For t
com]
(Fór:
the t
base
base
