6A-4-2 
achieved by using iteratively rectified images; it serves as 
a natural starting point in our investigation on integrating 
stereo imagery and LIDAR data for surface extraction. 
The Center for Mapping has been a pioneer in developing 
modern mapping technologies. The most recent mobile 
mapping system developed by the Center is the Airborne 
Integrated Mapping System (AIMS™) - a tightly coupled 
GSP/INS integrated positioning system supporting 
primarily digital sensor-based image data collection 
(Grejner-Brzezinska et. al. 1998). The prototype system 
currently employs a 4K by 4K imaging sensor (Toth 
1998), and recently, test flights were combined with a 
LIDAR sensor. Based on the initial experiences, the 
sample data from these flights illustrate the implications of 
integrating laser range data with stereo-imagery and 
provide an early insight into future multisensor fusion- 
based surface extraction techniques. 
2. SURFACE EXTRACTION METHODS 
Past surface extraction techniques have evolved around 
the use of predominantly analog film-based aerial 
photography. The usually large-format aerial camera- 
recorded films have been scanned at various resolutions, 
ranging from 30 to 5 microns, resulting in image sizes 
from 10K by 10K up to 30K by 30K. The radiometric 
representation is generally on 8 bits, or 3 x 8 bits for 
color; however, the effective resolution is typically not 
more than 6-7 bits. The large image sizes posed almost 
unmanageable difficulties for the early systems (which 
were unable to handle several hundreds of Mbytes of 
data). Apart from memory and disk limitations, the rather 
modest processing power was the main obstacle in 
developing surface extraction systems on general-purpose 
computers. Therefore, the first systems built by military 
mapping specialists (Helava 1989) were based on 
hardware implementations and were ultimately limited to 
perform only simple image correlations. With rapidly 
advancing generic computer hardware and software 
technologies, the implications of the inadequate computer 
processing power have completely disappeared by now. 
From an algorithmic point of view, existing surface 
extraction techniques have gone far beyond simple image 
correlation, although that function is still an integral part 
of the methods. Current systems can usually handle only 
fully oriented stereo pairs with monochrome image data. 
To minimize the number of operations, the massive 
amount of image data is handled at various resolutions. 
Forming an image pyramid, the matching procedure 
usually starts at a coarse resolution, typically at 512 by 
512, and then by establishing an approximate registration, 
the search space is gradually narrowed down as the 
processing moves to the next higher resolution level of the 
pyramid. By tracing down the conjugate image primitives 
from the coarsest to the highest resolution, not only are 
the computational savings enormous, but also this scale- 
space approach makes the whole procedure reasonably 
robust. There is diversity among the leading techniques 
concerning the choice of the image primitives used to find 
conjugate elements in the images. Typical image features 
(primitives) are points, edges and regions, and a variety of 
interest points and edge extraction operators are also 
available with numerous segmentation techniques. Once 
the image features are matched at the highest level of 
resolution, an area correlation or a least squares matching 
is performed to refine the conjugate image locations. 
The performance of existing surface extraction techniques 
can be adequately measured since these systems have been 
widely used in production. Although different systems 
may deliver very different results for identical image data 
sets, some generic conclusions can be drawn (Gruen 
1998). For smooth, rolling terrain at small and medium 
scale performance is usually good. However, it decreases 
rapidly for more complex scenes and with a larger scale, 
such as steep terrain, lush vegetation and dense urban 
areas with a variety of man-made objects. A 
comprehensive review of the large variety of surface 
extraction techniques is beyond the scope of this paper. 
Instead, recent relevant trends impacting the surface 
extraction process are listed here. 
Direct GPS/INS Orientation Data. The availability of 
exterior orientation parameters has always been 
assumed, since such parameters are the primary tool to 
provide the basic object space constraining for the 
matching process. However, in many cases, the 
orientation data determination process itself provides a 
large number of surface points, for example, 
automated relative orientation or automated aerial- 
triangulation. Then these object points can be used as 
seed points for the consecutive surface extraction 
procedure. With the recent development and widening 
use of GPS/INS-based direct orientation systems 
(Schwarz 1995; Grejner-Brzezinska 1997), points are 
no longer available for such purposes. 
Epipolar Geometry. The use of epipolar geometry 
provides a computationally efficient technique to 
reduce the otherwise two-dimensional search space 
into a more manageable one-dimensional matching 
problem. Obviously, confining the search space to a 
line may severely impact the robustness of the 
matching process. Inspired by the somewhat modest 
accuracy of the early experimental GPS/INS 
positioning systems, techniques have been developed 
to perform 1.5-D matching along epipolar bands. 
Direct Digital Imagery. Recent rapid technological 
developments have finally reached the imaging sensor 
- the last bastion of analog mapping techniques. CCD- 
based sensors have long been used in space and on the 
ground, but now they have started to penetrate the 
airborne surveying market. Since the size of the focal- 
plane CCD arrays currently falls short of the large- 
format aerial film dimensions, CCD frame sensors 
produce rather small footprints to maintain the