MODEL-BASED 3D SCENE ANALYSIS FROM STEREOSCOPIC IMAGE SEQUENCES
Reinhard Koch
Institut für Theoretische Nachrichtentechnik und Informationsverarbeitung
Universität Hannover, Appelstrasse 9A, 3000 Hannover 1, Germany
ISPRS Commission V
An approach for the modeling of complex 3D scenes like outdoor street views from a sequence of stereoscopic
image pairs is presented. Starting with conventional stereoscopic correspondence analysis a 3D model scene
with 3D surface geometry is generated. Not only the scene geometry but also surface texture is stored within
the model. The 3D model permits to detect and correct geometric errors by comparison of synthesized images
with real input images through analysis by synthesis techniques. 3D camera motion can be estimated directly
fromthe image sequence to track camera motion and to fuse measurements from different viewpoints through-
out the sequence into a common 3D model scene. From the textured 3D model realistic looking image se-
quences from arbitrary view points can be synthesized using computer graphics methods.
Key Words: 3D Scene Analysis, Stereoscopic Image Sequence Analysis, Robot Vision, Scene Reconstruction,
Close Range Photogrammetry.
INTRODUCTION
Conventional stereoscopic image analysis tries to re- Storeoscopic image sequence
construct a 3D scene from pairs of camera images L R
through triangulation of corresponding 2D image k k
points while the relative orientation of the cameras to Image Rectification &
each other is known. By triangulation a depth map Correspondence Analysis
can be constructed where the distance of each corre-
sponding 3D scene point to the camera focal point is Scene Knowledge
Lk | Rk| Dk| Cx
v
measured. This approach is sufficient for simple & Interaction
scene geometry without occluded areas but will fail | Scene Segmentation &
when analyzing complex scenes like street views. Depth Interpolation €
The reconstruction of complex 3D scenes requires
that a series of problems needto be solved, especially | Lk | "| xo si
the problems of 2D correspondence and 3D registra-
tion. To be able to triangulate the scene from a particu-
lar pair of images a calibration of the projection geom-
etry and the relative orientation of the cameras to
eachotheris needed. When using a binocular camera
setup, the calibration can be performed once before
the measurement and will remain constant through-
out the measurement phase. Measurement of scene
geometry relative to such a binocular camera system
can then be obtained through triangulation of corre-
sponding image points. During correspondence anal-
ysis one tries to uniquely identify the projections of a
scene element onto the camera targets. In a complex from 3D model scene
natural scene parts of the scene may be occluded to
the camera system so that the camera has to be Ly | R'k Dy Sk
moved throughout the scene. The scene is then ana-
lyzed from a sequence of image pairs. To register all e
measurements into a common scene coordinate sys- 3D Scene Synthesized
tem the 3D motion of the camera system must be Model image sequence
tracked and measurements from multiple view points
must be integrated to build a 3D model of the scene
[Aloimonos, 1989]. Ly : left image
Rx : right image
3D Motion Estimation &
1 3D Model Building from
image sequences
3D Model
parameter
Image Synthesis &
Sequence Prediction
The presented approach addresses the problems
h ;
i ENG stated above for building a 3D model of a complex D. JPA Ian a
| rock: scene from a sequence of stereoscopic image pairs. S. : segmantation N ap
ratory Fig. 1 displays the structure of a 3D scene analysis «: Frame k of sequence
systemthat automatically extracts 3D shape, motion,
and surface texture of a 3D scene viewed by a stereo- ;
scopic camera. Input to the system is a stereoscopic Fig. 1: Structure of 3D scene analysis.
427
