MULTISENSOR DATA FUSION FOR AUTOMATIC SCENE INTERPRETATION
Bea Csathó
Byrd Polar Research Center
The Ohio State University
1090 Carmack Rd.
Columbus, OH 43210
csatho.1©osu.edu
Toni Schenk
Department of Civil Engineering
The Ohio State University
2070 Neil Ave.,
Columbus, OH 43210
schenk.2Qosu.edu
KEY WORDS: Object Recognition, Data Fusion, Multisensor/Multispectral Fusion
ABSTRACT
The ultimate goal of digital photogrammetry is to make maps automatically. This entails understanding aerial imagery and
recognizing objects—both hard problems. Despite of the increased research activities and the remarkable progress that has
been achieved, we are still far from operational systems and for many, the goal remains a dream. The utilization of multiple
sensory input data opens new avenues to approach the problem. By combining sensors that use different physical principles
and record different properties of the object space, complementary and redundant information becomes available. If merged
properly, multisensor data may lead to a more stable and consistent scene description. In this paper we address issues related -
to the question on how to merge multisensor data and how to combine different processes, such as classification and fusion
approaches in remote sensing with processes from object recognition. To illustrate the potential of multisensor integration
for urban mapping we describe experiments obtained from a data set comprising aerial imagery, multispectral scanner, and
scanning laser altimetry.
1 Introduction
The ultimate goal of digital photogrammetry is to generate
maps automatically. Map compilation is a very labor intensive
process. It may take several hours per model. Thus, there
is a great economical incentive to automate the map com-
pilation process. A more esoteric reason for the increasing
research activities in identifying and localizing objects from
aerial imagery is the scientific challenge the problem poses.
The two tasks are known as object recognition—a term we
use in this paper instead of automatic map compilation.
Before an object, say a building, can be measured it must first
be identified as such. This involves understanding an image,
at least to a certain extent and this is precisely what makes
object recognition a hard problem. Humans are remarkably
adept at object recognition. How the human visual system
solves the problem is not known well enough to mimic the so-
lution by computers. Instead of trying to unravel the image
understanding abilities of humans, many researchers attempt
to improve the current status of recognizing objects from
aerial scenes by increasing the sensory input sources. For ex-
ample, laser altimeter data are used to generate DEMs; color
imagery is preferred to take advantage of color information.
Expanding this idea leads to the inclusion of multispectral
or even hyperspectral sensor data into the object recogni-
tion process. This trend is facilitated by the fact that the
spatial, spectral and temporal resolution of different airborne
and spaceborne sensors rapidly increases while the cost asso-
ciated with data collection decreases.
Now, the cardinal question is how to exploit the potential
these different data sources offer to tackle the object recog-
nition more effectively. This is precisely the main item on the
research agenda of Working Group 5 of ISPRS Commission
I. Ideally, proven concepts and methods in remote sensing,
digital photogrammetry, and computer vision should be com-
bined in a synergistic fashion. We argue that such attempts
should first be launched on a conceptual level before specific
algorithms are devised, modified or merged. The purpose
of this paper is to stimulate discussions that hopefully will
broaden our understanding of object recognition with multi-
sensor and multispectral data sources.
Since we have the conceptual level in mind we start out with a
brief description of the object recognition paradigm, followed
by a summary of state-of-the-art data fusion in remote sens-
ing. We then present some examples that illustrate the po-
tential of using information from multisensor sources for the
purpose of interpreting aerial scenes automatically. A data
set of an urban area is processed by remote sensing tech-
niques. We conclude by identifying major issues that need to
be addressed by future research.
2 Object Recognition Paradigm
At the heart of the paradigm is the recognition that it is
impossible to bridge the gap between sensory input data and
the desired output in one step. Consider a gray level image
as input and a GIS as the result of object recognition. The
computer does not see an object, say a building. All it has
available at the outset is an array of numbers. On the output
side, however, we have an abstract description of the object,
for example, the coordinates of its boundary. There is no
direct mapping between the two sets of numbers.
A commonly used paradigm begins with preprocessing the raw
sensory input data, followed by feature extraction and seg-
mentation. Features and regions are perceptually organized
until an object emerges from the data. This data model is
then compared with a model of the physical object. If there
is sufficient agreement, the data model is labeled accordingly.
In a first step, the sensor data usually require some prepro-
cessing. For example, images may be radiometrically ad-
justed, oriented and perhaps normalized. Raw laser altimeter
data are processed to 3-D points in object space.
The motivation for feature extraction is to capture informa-
tion from the processed sensory data that is somehow related
336 Intemational Archives of Photogrammetry and Remote Sensing. Vol. XXXII, Part 7, Budapest, 1998
to the
Excep
by ev
clude
in refl
tures
way ©
Segm
tion a
gethei
quite
by the
quires
indep«
The o
than t
bols.
primit
subjec
organi
ceptio
symbo
ing pri
The n
compa
with a
called
model
proced
and th
object
erties |
color, í
matic
roof de
The se
called |
approa
cific kn
occur |
3
Remot:
multise
tempor
reflecta
such as
a GIS |
in remc
includir
pattern
more ce
shadow
Multise
formati
solution
gration
stantial
to Abid
tisensor
any staj
