ystems are able
to the required
le, and further
acial structures,
8.
, D., Hallbauer,
measurement of
to the spine
and Remote
for small format
1 5(86), pp. 325-
n for automated
| Archives of
p. 265-272.
1995. An
CD for digital
5(86), pp. 301-
onal Computers.
sing, 55(11), pp.
A VIDEO-RATE STEREO MACHINE AND ITS APPLICATION TO VIRTUAL REALITY
Kazuo Oda, Masaya Tanaka, Atsushi Yoshida, Hiroshi Kano and Takeo Kanade
The Robotics Institute, Carnegie Mellon University
5000 Forbes Avenue Pittsburgh, PA 15213 USA
Commission V, Working Group 1
KEY WORDS: Close Range, Real-Time Vision, Stereo, Virtual Reality, Z keying, Hardware, Computer Vision.
ABSTRACT
We have developed a video-rate stereo machine which has the capability of generating a dense range map at video rate.
The CMU video-rate stereo machine has the following performance: 1) multi image input of up to 6 cameras; 2) throughput
of 30 million point x disparity measurements per second; 3) frame rate of 30 frames/sec; 4) a dense depth map of 256 x
240 pixels; 5) disparity search range of up to 60 pixels; and 6) high precision of up to 8 bits (with interpolation). The capa-
bility of producing such a high resolution depth map (3D representation) at video rate opens up a new class of applications
for 3D vision. We report one such application: z keying, which merges the real and virtual worlds in real time.
1. INTRODUCTION
Stereo range imaging uses correspondence between sets
of two or more images for depth measurement. Despite a
great deal of research during the past two decades, no
stereo systems developed so far have achieved adequate
throughput and precision to enable video-rate dense depth
mapping. The throughput of a stereo machine can be most
effectively measured by the product of the number of
depth measurements per second (pixels/sec) and the
range of disparity search (pixels); the former determines
the density and speed of depth measurement and the lat-
ter the dynamic range of distance measurement. There
are several advanced real-time stereo systems (Nishihara,
1990, Webb, 1993, Matthies, 1992, Faugeras et al., 1993);
yet none of them is able to provide complete video-rate
output of range as dense as the input image with low
latency.
We have developed a video-rate stereo machine which
has the throughput of 30 million pixel?/sec. This through-
put translates to a 200 x 200 x 5bit depth image at the
speed of 30 frames per second - the speed, density and
depth resolution high enough to be called a video-rate 3D
depth measurement camera. Our video-rate stereo
machine is based on a new stereo algorithm, the multi-
baseline stereo theory (Okutomi et al., 1992, Nakahara
and Kanade, 1992, Okutomi and Kanade, 1993). It uses
multiple images obtained by multiple cameras to produce
different baselines in length and direction.
Video-rate stereo range mapping has many advantages. It
is passive and it does not emit any radio or light energy.
With appropriate imaging geometry, optics, and high-reso-
lution cameras, stereo can produce a dense, precise
range image. Stereo performs sensor fusion inherently;
range information is aligned with visual information in the
common image coordinates. Stereo depth mapping is
scanless; thus it does not have the problem of apparent
shape distortion from which a scanning-based range sen-
sor suffers due to motion during a scan. These features of
video-rate dense depth mapping open up a new class of
applications of 3D vision. We report one such application:
2 keying which merges the real and virtual worlds in real
ime.
2. CMU VIDEO-RATE STEREO MACHINE AND ITS
PERFORMANCE
The CMU video-rate stereo machine comprises special-
purpose high-performance hardware. Table 1 summarizes
its current performance.
Table 1: Performance of CMU stereo machine
Number of cameras 2106
Processing time/pixel 33ns x (disparity range + 2)
Frame rate up to 30 frames/sec
Depth image size up to 256 x 240
Disparity search range up to 60 pixels
(a) Processor (b) Five-eye camera head
Figure 1: The CMU video-rate stereo machine
(b) range image
(a) intensity image
Figure 2: An example scene and its range image
International Archives of Photogrammetry and Remote Sensing. Vol. XXXI, Part B5. Vienna 1996
