are resampled to ideal plane images using the camera 
calibration parameter supported by PGR Ladybug systems, and 
the interesting area is extracted for efficiency. The canny based 
on EDISON (Meer and Georgescu, 2001) edge detection 
method is used to detect arcs in the interesting area. Then each 
arc is fitted to the most possible ellipse using least square 
method. The direction, area and height information of extracted 
ellipse are then checked to exclude false detection such as 
shadows, wheels and traffic signs. In the texture verification 
step, the texture complexity is firstly estimated by variance of 
intensity in ellipse to exclude the area lack of texture. Then the 
multiview images are matched to further validation and at the 
same time the location of covers are obtained with GPS/IMU. 
aS : 
Figure 1. Separate fisheye images 
  
Edge 
EET verification 
steps 
FishEye views 
ideal plane 
views 
y t 
Arc Detection 
|^ 
um A or o. 
Ellipse fitting 
r7Directio n.area E] CUNT 
| andheight «| GPS*LIDAR | 
_Threthods 
i Texture 
Texture steps 
| verification | 
: | 
| p Mdilew 
| | image e n dun | 
| — matching + GPS/IMU+ | | 
|: Ld LIDAR | 
Georeferencin |: ERE ROMA. uA 
Lt see = ar ] 
Figure 2. System workflow 
2.2 Arc detection 
The shape of manhole is mainly ellipse or rectangle. However, 
those two shapes are symmetric, and same extraction method 
International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXIX-B3, 2012 
XXII ISPRS Congress, 25 August — 01 September 2012, Melbourne, Australia 
   
could be adapted to those two features. This paper mainly deals 
with manholes with ellipse shapes. For the reasons such as 
occlusion, shadows, illumination, blurring, the ellipse cover 
does not show whole and perfect ellipse, and in many cases 
only one part of ellipse can be discovered. Our method is to 
detect arcs firstly, and then finds the most possible ellipse after 
multi-step checking. 
For efficiency, the area of 1024*527 in pixels is extracted for 
the next processing. We use EDISON method to detect arcs and 
lines. The gradient window radius is set to 3 and the minimum 
length of arc is set to 40. The type of non-maximum 
suppression is set as vertical line and the type of hysteresis High 
threshold is set as horizontal line and low threshold as arc. All 
the values of rank and confidence take the default. 
2.3 Ellipse fitting and thresholds 
Most of the arcs been extracted do not belong to covers for the 
complex street scenes. The ellipse fitting is then used to fine the 
possible ellipses that agree to the proper shape and size of 
covers. The elliptic equation with unknown parameters as long 
axle a, minor axle b and rotation angle was solved with 
RANSAC (Fischler and Bolles, 1981) method according to the 
points in the arcs. 
After the ellipses are obtained some thresholds are then used to 
eliminate those ones which not fitting a cover. Because only the 
covers in front of the car are considered, long axle is likely 
rectangular to the road direction. So is limited to 45 degree. 
For the area restriction, long axle a is set to between 0.5m and 
Im and P is set to 0.3m and 0.9m and a > b. 
There are still some objects with elliptic shape in the interesting 
area, such as wheels and some traffic signs. The LIDAR 
observation values are used to eliminate those false arcs by 
height information. If the height of ellipse center is 0.2m higher 
than the ground height, the ellipse will be excluded as false 
detection. 
2.4 Multi-view matching and locating 
Except shape, texture is another important feature for cover 
detection and false checking. In the paper matching method is 
utilised to confirm the candidate ellipses. Due to the deferent 
sight angle, deferent material and shadow, lamination etc, it is 
difficult to find a universal template to match all candidates 
correctly. In this paper, the multi-view matching for adjacent 
images are used instead, so as to confirm the detection results 
and to obtain the accrue geo-position of covers combined with 
GPS/IMU data. 
Before matching the ellipse is verified to have enough texture 
information. The variance in pixel intensity within the ellipse 
should be more than 50. However, there are some covers have 
less variance in intensity for the light situation (see Fig. 3d). In 
such situation the cover will be missed. 
The covers have been taken at deferent distance to the car and 
there is identified deformation between those covers. The 
deformation formula can be described like eq.2 deducted from 
eq.1 when the cover is regarded as in a plane. In eq.1, x; and X 
are the coordinate vector of reference image and ground 
coordinate respectively, and R; and T; are the rotation matrix 
and translation vector, which provided by GPS/IMU 
observations. R,, T; and x are the correspondent observations 
of an arbitrary matching image. Combined with LIDAR data, À 
can be calculated from the first formula in eq.| while Z 
coordinate in X is known.