International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXIX-B3, 2012 
XXII ISPRS Congress, 25 August — 01 September 2012, Melbourne, Australia 
    
MANHOLE COVER DETECTION USING VEHICLE-BASED MULTI-SENSOR DATA 
Ji Shunping*, Shi Yun", Shi Zhongchao © 
? School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430049, China 
jishunping2000(2)163.com 
? CSIS, the University of Tokyo, Tokyo, Japan 
shiyun@jiis.u-tokyo.ac.jp 
* Dept. of Environmental and Information Studies, Tokyo City University, Yokohama, Japan 
shizc@tcu.ac.jp 
III/4: Complex Scene Analysis and 3D Reconstruction 
KEY WORDS: cover detection; matching; edge detection; manhole cover; multi-sensor 
ABSTRACT: 
A new method combined with multi-view matching and feature extraction technique is developed to detect manhole covers on the 
streets using close-range images combined with GPS/IMU and LIDAR data. The covers are an important target on the road traffic as 
same as transport signs, traffic lights and zebra crossing but with more unified shapes. However, the different shoot angle and 
distance, ground material, complex street scene especially its shadow, and cars in the road have a great impact on the cover detection 
rate. The paper introduces a new method in edge detection and feature extraction in order to overcome these difficulties and greatly 
improve the detection rate. The LIDAR data are used to do scene segmentation and the street scene and cars are excluded from the 
roads. And edge detection method base on canny which sensitive to arcs and ellipses is applied on the segmented road scene and the 
interesting areas contain arcs are extracted and fitted to ellipse. The ellipses are then resampled for invariance to shooting angle and 
distance and then are matched to adjacent images for further checking if covers and . More than 1000 images with different scenes 
are used in our tests and the detection rate is analyzed. The results verified our method have its advantages in correct covers 
detection in the complex street scene. 
1. INTRODUCTION 
Manhole cover is a necessary facility in urban design and traffic 
management. The missing of manhole may cause great security 
risks and economic losses, so the near real-time report of cover 
missing seems very important. There are many ways to prevent 
loses of covers. One way is to ensure the cover and road that are 
built as a whole entity with physical methods. Another way is to 
install a sensor with a transmitter in covers, which can report 
cover’s position in real-time (Tang ef al, 2003). However, the 
first way may cause some difficulties to get down to the 
underground for pipeline repair or draining. The second way 
may cause some additional needs of operations and economy. 
Cover detection and management with close-range images is a 
more popular way due to the development of computer vision 
science and photogrammetry. 
Manhole covers have a more uniform appearance than other 
traffic features, such as traffic signs, traffic lights and zebra 
crossing It hasa relatively standard shape of circle or rectangle. 
The basic idea of cover detection is to find ellipses or rectangles 
in images and then determine if they are covers. Chen presents 
an ellipse detection method based on improved Hough 
transform (Chen and Wang, 2006). Liu utilises least squares 
fitting to find ellipses, and then the false targets are eliminated 
by parameters such as max and minor axis ratio and rotation 
angles (Liu, 2010). 
However, the appearance of covers in images is not always a 
single and complete ellipse due to the complex street 
background, such as shadows of trees or buildings, the 
occlusion from cars or road railings. So in many cases only an 
  
: Corresponding author. 
arc of part ellipse can be found. In addition, some ellipse targets 
may not be covers such as wheels, traffic signs, etc. 
In this paper we present an arc detection and multi-view 
matching method for cover detection, location and false target 
eliminating using close-range images, GPS/IMU and LIDAR 
data. In 2.1 the system workflow is briefly introduced and in 2.2 
and 2.3 the arc detection, ellipse fitting and filter checking are 
presented. In 2.4 the multi-view matching is done to further 
check and locate the 3D position of manhole covers. In para.3 
we use 1100 images from vehicle-based camera with GPS/IMU, 
and LIDAR data to test our system and the results are discussed 
and in 4 the conclusion are drown. 
2. METHODOLOGY 
2.1 Data and system workflow 
The vehicle based close-range imaging system Ladybug 
(Ladybug, 2012) is used as data acquisition equipment. There 
are six separate fisheye cameras in the car. No.0-4 cameras are 
around the car with 72 degree between each other and No.5 
camera points to the sky and all camera can form a panoramic 
image in geometry (see Fig. 1). We use three front cameras 
No.0, 1 and 4 to detect manholes. GPS/IMU system with high 
accuracy is installed in the car to supply external orientation 
parameters (EOs) at the exposure time. The LIDAR data is also 
provided to get the road height. 
The workflow in Fig. 2 includes two main steps, edge detection 
and texture detection. In image pre-process, the fisheye images