International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol XXXV, Part B3. Istanbul 2004
regions and disturb the viewer. In our approach, we propose
initially painting the occluded regions of the second view using
the same painting algorithm as for the reference view, and then
superimposing the projected pixel values of the non-occluded
regions onto the paint applied to the occlusion gaps. When
painting the occluded regions, we render those strokes whose
starting points lie within the occluded regions. We allow the
strokes to extend beyond the occlusion boundaries, since
confining them to the occlusion gaps can change the stroke
characteristics perceived by the user. We also found that a
morphological dilation of the occlusion map prior to painting
could improve the results.
In Hertzmann's original algorithm, the strokes are rendered in a
random order, to prevent an undesirable appearance of
regularity on the final painting. In our implementation, we use
randomization within individual depth layers, and order the
layers according to decreasing depth. This would support a
future implementation of different levels of detail, rendered
with coarser and finer stroke sizes, depending on the distance of
an object from the viewer. Generally, regions rendered with
many small brush strokes tend to attract the attention of the
viewer. Whereas our current implementation focuses attention
on image regions containing high-frequency information (i.e.,
fine image details), the depth layer implementation might be
used to emphasize foreground elements.
3. TESTS AND RESULTS
We performed tests on both benchmark images obtained from
the Middlebury Stereo Vision website (Middlebury, 2004) and
self-recorded data. We captured our video frames in 24 bit RGB
format using two Dragonfly IEEE-1394 video cameras
(Pointgrey, 2004) in a stereo configuration. The images were
calibrated according to (Zhang, 2000) and projected into
epipolar geometry.
Figures 2 through 5 demonstrate the application of the proposed
stereoscopic painting algorithm to the Sawroorh test set. The
initial stereo image pair is given in figure 2. We started our
stereoscopic rendering tests by first applying an implementation
of Hertzmann's original algorithm to the left and right image
individually. We chose a painting style that imitates work by
impressionist artists. Figures 3 (a) and (b) show the results
obtained after painting two consecutive layers with a circular
brush of 8 and 4 pixels radius, respectively. Since the depth
discontinuities are not taken into account in Hertzmann's
original implementation, the object contours are not well
preserved. Figures 3 (c) and (d) show the same original images
rendered with finer brush strokes. In this case, the effect of
paint spilling is less pronounced, but still present. In figure 3,
one can also recognize the loss of coherence between the brush
strokes computed for the left and right stereo view. This effect
becomes more pronounced with larger brush sizes and
increasing geometric dissimilarity between the images of the
original stereo pair.
We employed our segmentation-based stereo matching
algorithm described in section 2.1 to compute the corresponding
disparity map, which is shown in figure 4 (a) in the geometry of
the left image. For comparison, the ground truth for this data set
can be seen in figure 4 (c). Visual comparison of (a) and (c)
indicates a very good quality of the stereo-derived disparities in
(a). This observation was confirmed by a quantitative analysis.
The evaluation testbed provided by (Scharstein and Szeliski,
2002) found for our disparity map a percentage of 0.2 % “bad”
pixels (i.e., unoccluded pixels whose absolute disparity error is
greater than 1). At the time of writing this paper, this error rate
resulted in a first rank of our matching algorithm for the
Sawtooth image pair among 30 algorithms listed on the stereo
evaluation website (Middlebury, 2004). The high percentage of
correctly matched pixels is reflected by the almost perfect
reconstruction of the depth layer boundaries in figure 4 (a).
We projected the computed depth map into the geometry of the
right view and determined those pixels in the right view that are
not visible in the left view. Figure 4 (b) shows the location of
the occluded regions marked in white. These areas are painted
in a separate step in our modified version of Hertzmann’s
algorithm, since paint cannot be propagated from the left view.
The painting of the occlusion gaps from figure 4 (b) is
illustrated in figure 4 (d).
(a) left image (b) right image
Figure 2. Sawtooth stereo pair (size 436 x 380 pixels) from the
Middlebury Stereo Vision website.
(c) left image (fine strokes)
(d) right image (fine strokes).
Figure 3. Results obtained by painting the stereo images from
figure 2 separately using Hertzmann's original algorithm. The
images in the top row were rendered using two layers of paint
with stroke radii of 8 and 4 pixels. An additional layer of paint
with a brush stroke radius of 2 pixels was applied to generate
the refined version shown in the bottom row.
The results of the proposed stereo painting algorithm can be
seen in figure 5. In contrast to figure 3, depth discontinuities
and coherence between the two images are now well preserved.
One can recognize that the painting of the occluded regions in
Inte
e Ro FT 0.
re
es
— 0 [y EEE PN rS 0 (In
—
