ONS
; with a
ge pairs
sign of
zorithm
ted as a
ated at
pression
1.
ed from
aintings
18 René
or Dali
dea of
1e New
Art of
. When
nize the
shift of
in the
(warm
e into a
intings,
he term
tensive
> same
painters
others
that the
'd with
why the
tial of
itations
rate on
d to be
nerated
International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol XXXV, Part B3. Istanbul 2004
stereo painting would be to apply the same single image
rendering technique to both images individually. However, this
will usually not produce satisfactory results. Depending on the
paint style, individual painting of the two images may result in
non-coherent stroke patterns that do not fuse well
stereoscopically. Furthermore, brush strokes that cross the
borders of an object can cause the color to spill to adjacent
objects located at different depths. While intersurface paint
spilling does not necessarily degrade the quality of a single
painting, in stereo painting it becomes a more noticeable and
mostly unpleasant effect. These and other related effects are
still largely unexplored.
(a) left tb) right image
Figure 1. Example of a stereoscopic painting by Salvador Dali
(“The sleeping smoker”, 1972/73).
In this study, we propose a stereoscopic painterly rendering
algorithm that uses a stereo-derived depth map to prevent paint
spilling and to preserve coherence between the brush strokes of
the two stereo views. We also tackle the related problem of
occlusions. The algorithm is described in section 2. In section 3,
we show experimental results obtained from a stereo benchmark
pair and self-recorded video frames. Suggestions for future
work are given in section 4.
2. ALGORITHM
2.1 Stereo Analysis
We developed a stereo matching algorithm that uses a color-
based image segmentation for the precise location of depth
discontinuities and the superior handling of large untextured
regions. Whereas the majority of existing stereo matching
algorithms were originally tailored to grey-value images, color
images play an important role in many multimedia applications,
especially computer-generated art. The need to deal with
untextured regions of significant extent is illustrated by figure
I. Precise localization of depth discontinuities is required to
prevent paint spilling, as described in the introduction.
The matching algorithm can be described as follows. We first
apply color segmentation to the reference image and use a
planar model to represent the disparity inside the derived
segments. This approach is based on the assumption that for
regions of homogeneous color the disparity varies smoothly and
depth discontinuities coincide with the boundaries of those
regions (Zhang and Kambhamettu, 2002), which holds true for
most natural scenes. We compute an initial disparity map using
window-based correlation with additional consistency checks to
determine the reliability of the depth values computed for each
segment. The initial disparity map is used to derive a planar
model of each segment. The segments are grouped into depth
layers by a mean-shift clustering algorithm. The layer
assignment is refined iteratively using a global cost function.
The quality of the current assignment is measured by projecting
the reference image into the geometry of the second view and
evaluating the difference between the projected and original
pixel values in the second image. Besides pixel dissimilarity,
the cost function also penalizes discontinuities and occlusions.
Since the problem of finding an exact solution to the
minimization problem can be shown to be np-complete, we
employ an efficient greedy search strategy to find a local
optimal solution. More details on the algorithm along with
results obtained in benchmark tests are given in (Bleyer and
Gelautz, 2004).
2.2 Stereoscopic Painterly Rendering
The proposed stereo painting algorithm is an extension of
Hertzmann's single image painting algorithm (Hertzmann,
1998) that was designed for the image-based generation of
hand-painted effects. The authors use a multiresolution
approach, in which layers of successively finer strokes are
placed on top of each other. The brush strokes are computed as
B-splines that follow the isocontours of the original image.
Each stroke is rendered by dragging a circular brush mask along
the spline. The color of the stroke is derived from a blurred
source image, which also serves as underpaint for the
hierarchical refinement of the computer-generated strokes.
Finer strokes are applied to regions of high-frequency content,
whereas in relatively uniform areas the coarser strokes of
previous layers of paint are preserved. For a more detailed
description of the spline planning and rendering procedure, the
reader is referred to the original paper by (Hertzmann, 1998).
Our modifications to the algorithm make use of the stereo-
derived depth map to handle depth discontinuities and
occlusions during the painting procedure as follows.
Depth discontinuities: When painting the reference image we
use the depth map to prevent brush strokes from painting across
depth discontinuities in order to avoid paint spilling between
different layers of depth. As we calculate the control points for
the B-splines, we check for each control point whether a
significant depth discontinuity, according to a pre-defined
threshold, is located in its vicinity, which is defined by the
current size of the circular brush. If such a discontinuity is
encountered, the stroke is terminated. This adds a fourth
constraint to the three termination criteria (brush length, color,
and magnitude of image gradient) that are already part of the
original algorithm. Our tests in section 4 will demonstrate that
the object boundaries are well preserved using this depth
constraint.
Occlusions: We employ the depth map to project the rendered
strokes from the reference view into the geometry of the second
view in order to preserve coherence between the two images of
the stereo pair. Those areas in the second view which are not
visible in the reference view require special consideration. For
the sake of simplicity, we refer to these regions as "occlusions",
although the gaps can also be caused by other conditions (e.g..
missing information along image borders due to the spatial
displacement between the two cameras) A straightforward
solution to filling the occlusion gaps would be to interpolate the
missing information from the projected neighboring pixels or to
employ the color values of the underpaint (i.e. the blurred
original image) at those locations. While this may work for
minor gaps, the resulting artifacts are clearly visible on larger