STEREO-BASED IMAGE AND VIDEO ANALYSIS FOR MULTIMEDIA APPLICATIONS
M. Gelautz *, E. Stavrakis, M. Bleyer
Institute for Software Technology and Interactive Systems, Vienna University of Technology,
Favoritenstrasse 9-11/188/2, A-1040 Vienna, Austria - (gelautz, stavrakis, bleyer)@ims.tuwien.ac.at.
http://www.ims.tuwien.ac.at
KEY WORDS: Vision Sciences, Stereoscopic, Multimedia, Video, Reconstruction, Vision, Graphics, Rendering
ABSTRACT:
In this paper we utilize stereo images and video frames of real scenes to automatically generate artistic stereoscopic views with a
hand-painted appearance (“painterly rendering"). The approach is motivated by contemporary artists who painted stereo image pairs
on canvases. Although a variety of monoscopic painterly rendering algorithms have been proposed in the literature, the design of
paintérly rendering algorithms suited to stereo views of real scenes presents a largely unexplored field.
In our approach, we use stereo analysis to generate a disparity map which is then employed by the stereoscopic rendering algorithm
to preserve coherence between the brush strokes of the two stereo views. The painting of the occluded regions is incorporated as a
separate step into the rendering procedure. We utilize the disparity map to prevent paint spilling between surfaces located at
different depths. Stereoscopic viewing of the stereo paintings produced by our algorithm demonstrates that the aesthetic impression
resulting from the hand-painted appearance of the individual images has been enhanced with an additional perception of depth.
1. INTRODUCTION AND MOTIVATION
The 3D reconstruction of image and video scenes is an
important requirement in many multimedia applications,
including video editing, compositing, and the generation of
artificial images based on real image content (“image-based
rendering”). In this paper, we focus on a non-photorealistic
rendering (NPR) application which uses an input stereo pair of
images from a real scene to produce painterly rendering effects
on stereo views, similar to artwork painted by hand. The
computer-generated paintings can be viewed stereoscopically,
which combines the artistic rendering effects of the individual
images with the additional sense of depth.
1.1 Non-photorealistic Rendering
Over the past years, the development of techniques for non-
photorealistic rendering (NPR) - as opposed to traditional
photorealistic rendering - has received increasing attention in
computer graphics research (Gooch and Gooch, 2001). Studies
on computer-generated painterly effects include the work by
(Litwinowicz, 1997), which deals with the automated
generation of impressionist paintings. A comprehensive survey
on stroke-based rendering techniques is presented by
(Hertzmann, 2003). The combination of painterly rendering
methods with computer vision techniques has been addressed
by (Shiraishi and Yamaguchi, 2000) and (Gooch et al., 2002).
The authors of the latter study compute the medial axis of
segmented objects to guide the automated creation of brush
strokes. As an extension, they suggest the incorporation of
depth information into their stroke-based rendering system,
which can, for example, assist the segmentation.
Most non-photorealistic rendering work that utilizes depth maps
relies on synthetic depth maps computed from a 3D model.
However, our approach makes use of stereo analysis to process
images of real scenes, which usually provide richer and more
* Corresponding author
visually interesting content than synthetic images rendered from
3D scene models.
1.2 Artistic Rendering of Stereo Views
Although not widely known, traditional stereoscopic paintings
have been created by several well-known artists including René
Magritte (“Man with Newspaper”, 1928) and Salvador Dalí
(Maur, 1989). (Ferragallo, 1972) promoted the idea of
stereoscopic painting in his “Manifesto Directed to the New
Aesthetics of Stereo Space in the Visual Arts and the Art of
Painting”.
A stereo image pair painted by Dali is shown in figure 1. When
comparing the left and right view, one can clearly recognize the
stereo parallaxes between the two images (see, e.g., the shift of
the cigarette with respect to the head and window in the
background). Interestingly, Dalí used different colors (warm
and cool tones) for the left and right image, which fuse into a
new color impression on the stereo view. For his 3D paintings,
which combine real and surreal elements, he created the term
*Metaphysical Hyperrealism" (Maur, 1989).
The manual creation of stereo paintings is a labor intensive
task, since it requires the artist to reproduce the same
composition twice from different viewpoints. Some painters
used stereo photography to base their compositions on; others
restricted themselves to geometric forms. We believe that the
technical requirements and excessive effort associated with
creating stereoscopic paintings were principal reasons why the
technique has not become widespread. The potential of
computer vision techniques to overcome these limitations
provides a major motivation for our study.
When devising painterly rendering algorithms that operate on
stereo images of real scenes, several requirements need to be
taken into account. An obvious solution to computer-generated
Inter
—
-— —
— wu [0 M9 pea
rn
