ONS 
; with a 
ge pairs 
sign of 
zorithm 
ted as a 
ated at 
pression 
1. 
ed from 
aintings 
18 René 
or Dali 
dea of 
1e New 
Art of 
. When 
nize the 
shift of 
in the 
(warm 
e into a 
intings, 
he term 
tensive 
> same 
painters 
others 
that the 
'd with 
why the 
tial of 
itations 
rate on 
d to be 
nerated 
International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol XXXV, Part B3. Istanbul 2004 
  
stereo painting would be to apply the same single image 
rendering technique to both images individually. However, this 
will usually not produce satisfactory results. Depending on the 
paint style, individual painting of the two images may result in 
non-coherent stroke patterns that do not fuse well 
stereoscopically. Furthermore, brush strokes that cross the 
borders of an object can cause the color to spill to adjacent 
objects located at different depths. While intersurface paint 
spilling does not necessarily degrade the quality of a single 
painting, in stereo painting it becomes a more noticeable and 
mostly unpleasant effect. These and other related effects are 
still largely unexplored. 
(a) left tb) right image 
   
Figure 1. Example of a stereoscopic painting by Salvador Dali 
(“The sleeping smoker”, 1972/73). 
In this study, we propose a stereoscopic painterly rendering 
algorithm that uses a stereo-derived depth map to prevent paint 
spilling and to preserve coherence between the brush strokes of 
the two stereo views. We also tackle the related problem of 
occlusions. The algorithm is described in section 2. In section 3, 
we show experimental results obtained from a stereo benchmark 
pair and self-recorded video frames. Suggestions for future 
work are given in section 4. 
2. ALGORITHM 
2.1 Stereo Analysis 
We developed a stereo matching algorithm that uses a color- 
based image segmentation for the precise location of depth 
discontinuities and the superior handling of large untextured 
regions. Whereas the majority of existing stereo matching 
algorithms were originally tailored to grey-value images, color 
images play an important role in many multimedia applications, 
especially computer-generated art. The need to deal with 
untextured regions of significant extent is illustrated by figure 
I. Precise localization of depth discontinuities is required to 
prevent paint spilling, as described in the introduction. 
The matching algorithm can be described as follows. We first 
apply color segmentation to the reference image and use a 
planar model to represent the disparity inside the derived 
segments. This approach is based on the assumption that for 
regions of homogeneous color the disparity varies smoothly and 
depth discontinuities coincide with the boundaries of those 
regions (Zhang and Kambhamettu, 2002), which holds true for 
most natural scenes. We compute an initial disparity map using 
window-based correlation with additional consistency checks to 
determine the reliability of the depth values computed for each 
segment. The initial disparity map is used to derive a planar 
model of each segment. The segments are grouped into depth 
layers by a mean-shift clustering algorithm. The layer 
assignment is refined iteratively using a global cost function. 
The quality of the current assignment is measured by projecting 
the reference image into the geometry of the second view and 
evaluating the difference between the projected and original 
pixel values in the second image. Besides pixel dissimilarity, 
the cost function also penalizes discontinuities and occlusions. 
Since the problem of finding an exact solution to the 
minimization problem can be shown to be np-complete, we 
employ an efficient greedy search strategy to find a local 
optimal solution. More details on the algorithm along with 
results obtained in benchmark tests are given in (Bleyer and 
Gelautz, 2004). 
2.2 Stereoscopic Painterly Rendering 
The proposed stereo painting algorithm is an extension of 
Hertzmann's single image painting algorithm  (Hertzmann, 
1998) that was designed for the image-based generation of 
hand-painted effects. The authors use a multiresolution 
approach, in which layers of successively finer strokes are 
placed on top of each other. The brush strokes are computed as 
B-splines that follow the isocontours of the original image. 
Each stroke is rendered by dragging a circular brush mask along 
the spline. The color of the stroke is derived from a blurred 
source image, which also serves as underpaint for the 
hierarchical refinement of the computer-generated strokes. 
Finer strokes are applied to regions of high-frequency content, 
whereas in relatively uniform areas the coarser strokes of 
previous layers of paint are preserved. For a more detailed 
description of the spline planning and rendering procedure, the 
reader is referred to the original paper by (Hertzmann, 1998). 
Our modifications to the algorithm make use of the stereo- 
derived depth map to handle depth discontinuities and 
occlusions during the painting procedure as follows. 
Depth discontinuities: When painting the reference image we 
use the depth map to prevent brush strokes from painting across 
depth discontinuities in order to avoid paint spilling between 
different layers of depth. As we calculate the control points for 
the B-splines, we check for each control point whether a 
significant depth discontinuity, according to a pre-defined 
threshold, is located in its vicinity, which is defined by the 
current size of the circular brush. If such a discontinuity is 
encountered, the stroke is terminated. This adds a fourth 
constraint to the three termination criteria (brush length, color, 
and magnitude of image gradient) that are already part of the 
original algorithm. Our tests in section 4 will demonstrate that 
the object boundaries are well preserved using this depth 
constraint. 
Occlusions: We employ the depth map to project the rendered 
strokes from the reference view into the geometry of the second 
view in order to preserve coherence between the two images of 
the stereo pair. Those areas in the second view which are not 
visible in the reference view require special consideration. For 
the sake of simplicity, we refer to these regions as "occlusions", 
although the gaps can also be caused by other conditions (e.g.. 
missing information along image borders due to the spatial 
displacement between the two cameras) A straightforward 
solution to filling the occlusion gaps would be to interpolate the 
missing information from the projected neighboring pixels or to 
employ the color values of the underpaint (i.e. the blurred 
original image) at those locations. While this may work for 
minor gaps, the resulting artifacts are clearly visible on larger