AN ACCURACY ASSESSMENT MEASURE FOR OBJECT BASED IMAGE 
SEGMENTATION 
nent s and 2/3 of a 
' ni P accuracy 
reflected 
in 
» the 
are measured 
Prevement 
in the 
ges from different 
ger ) " S1( k lap h/h 
ction. 
ad analog images 
acies with small 
ma g e $ also allow 
)f 5 Pixels. The 
; al accuracy of up 
and buildings. In 
se (approximately 
lot well represent 
lings or other man 
elusion is that the 
lore information, 
ore comfort in the 
it loss of the relief 
ieight accuracy, it 
itions in the final 
n the horizontal 
al component can 
:ts. 
¡ages in the paper 
enerated from a 
ire known. The 
the LSF function 
lion of the system, 
tion, using USAF 
Iso be applied to 
The results then 
r of the DMC, in 
Vrbiol, R.. Boned 
¡z, M. À., Palma, 
fanon process of 
“High-Resolution 
nnover. 
seeker, S;* 
F ,f DMC. to 
I, Paris. 
ie selection 
»rial triangulati° n 
Nicholas Clinton 3 ’*, Ashley Holt 3 , Li Yan b , Peng Gong 3C . 
“Department of Environmental Science, Policy and Management. University of California, Berkeley CA 94720-3114 
b Intemational Institute for Earth System Science. Nanjing University, China. 210093 - nclinton@nature.berkeley.edu 
c State Key Laboratory of Remote Sensing Science, Jointly Sponsored by Institute of Remote Sensing Applications, Chinese 
Academy of Sciences, and Beijing Normal University, 3 Datun Road, Chaoyang District, Beijing 100101 
KEY WORDS: Segmentation, ASTRO, BerkeleylmageSeg, Ecognition, Definiens 
ABSTRACT: 
Traditional approaches to accuracy assessment are inadequate for object oriented image processing. We tested some measures to assess 
the accuracy of object based image segmentation in a supervised context. The measures quantify the extent to which objects in the 
segmentation match training objects in terms of over-segmentation, under-segmentation, and distance to a perfect match. Using high 
resolution digital aerial photographs over an urban setup, we obtained segmentation results for a variety of parameter combinations using 
two software packages: eCognition and ASTRO. We compute the accuracy measures using three types of objects: vehicles, trees and 
buildings. The measures were used to compare the software, identify ideal parameter combinations, and identify objects that each 
software is better at extracting from the images. The measures are shown to be an intuitive, useful technique for consistency checking 
different segmentation results and assessing segmentation accuracies among a large set of disparate segmentation results. 
INTRODUCTION 
In object based image processing, the first step is generally a 
segmentation of the image of interest. A wide variety of 
segmentation results may be obtained through different parameter 
combinations or different segmentation software. Prior to 
classification or even to training of a suitable classifier, one of the 
segmentation results must be chosen. In this paper, we describe 
well defined measures that can be used in the identification of a 
“best” segmentation and the “best” objects within that 
segmentation for training a classifier. These measures are 
applicable in the supervised setting only, and “bestness” is 
therefore relative to a set of pre-defined training objects (assumed 
polygons) over the image of interest. 
Assume first that the landscape of interest is a finite population of 
objects (Bian 2007). The spatial information about these objects 
is useful in the ultimate classification of the object (Gong and 
Howarth 1990). It is obvious that exact representation of the 
objects in the segmentation is important, since this shape 
information will eventually be presented to a classifier for the 
identification of a pattern. The accuracy of the classification is 
thus dependent on the accuracy of the shape information 
submitted to the classifier. Measures of the segmentation result 
are therefore relevant to the interpretation and optimization of 
ultimate classification accuracy. The measures we tested are not 
measures of classification accuracy, but are related. Assuming 
that accuracy assessment is conducted with statistical rigor, a 
probability sample will be obtained on the population of objects 
(Stehman and Czaplewski 1998, Stehman 1999). If the population 
is assumed to be represented by the segmentation result and a 
simple random sample is used to generate accuracy statistics, then 
the accuracy of the shapes has been completely ignored! On the 
other hand, if a sample is taken from the landscape directly (e.g. 
human delineated training polygons are used) and compared to the 
segments, then areas of intersection between mapped classes and 
reference classes affect the resultant accuracy. The accuracy of 
the segmentation will thus directly influence the classification 
accuracy, unless classification is performed on object primitives, a 
different problem discussed below. 
There are a large number of methods with which to judge 
segmentations (Zhang 1996). This study is focused on the 
scenario in which a set of training objects is available for a static 
image and segmentation results are to be compared to these pre 
defined training objects. Unlike unsupervised evaluation of 
segmentation results (Levine and Nazif 1985, Ng and Lee 1996, 
Borsotti et al. 1998, Chabrier 2006), spectral aspects (such as 
homogeneity within segment or within class) of the resultant 
segments are not considered and the quality of segments is 
evaluated solely in respect to the shape of training objects. In this 
context, a segmentation result should contain segments that match 
the training objects. For automatically checking this, a simple, 
intuitive measure of polygon matching can be computed. This 
measure relies on the observation that there should be a one-to- 
one correspondence (in area) between human identified objects 
(training objects) and segments. A measure of this 
correspondence was first proposed by Levine and Nazif (1982) 
and demonstrated by Yang et al. (1995). Moller et al. (2007) 
describe a similar measure called Relative Area (RA) which relies 
on the ratio of intersected area to segment and reference object 
area. We present the intuition behind this measure, a refinement 
of its computation, and a case study using high resolution urban 
imagery. 
SEGMENTATION GOODNESS MEASURES