CMRT09: Object Extraction for 3D City Models, Road Databases and Traffic Monitoring - Concepts, Algorithms, and Evaluation 
(a) Matikainen et al. (2007) (b) Rottensteiner (2008) (c) Olsen and Knudsen (2005) 
Figure 1. Evaluation of change detection in Lyngby, for (a), (b) and (c) Green: TP; red: FN; orange: FP; blue: TN. 
Unchanged 
Demolished 
New 
(Champion, 2007) 
Completeness [%] 
93.5 
88.9 
95.2 
Correctness [%] 
99.8 
18.4 
63.5 
(Matikainen et al., 2007) 
Completeness [%] 
94.7 
100 
97.6 
Correctness [%] 
100 
23.7 
75.6 
(Rottensteiner, 2008) 
Completeness [%] 
94.1 
100 
94.0 
Correctness [%] 
100 
22.0 
96.3 
Table 2. Completeness and correctness for the Marseille test 
area, depending on the update status. 
| Unchanged \ Demolished \ New 
(Matikainen et al., 2007) 
Completeness [%1 
81.7 
100 
91.8 
Correctness \%] 
100 
22.6 
100 
(Olsen and Knudsen, 2005) 
Completeness [%1 
87.8 
100 
93.9 
Correctness [%] 
100 
30.4 
82.1 
(Rottensteiner, 2008) 
Completeness [%] 
95.9 
100 
87.8 
Correctness [%) 
100 
56.8 
91.8 
Table 3. Completeness and correctness for the Lyngby test 
area, depending on the update status. 
Unchanged 
Demolished 
New 
(Champion, 2007) 
Completeness f%j 
82.8 
100 
75.0 
Correctness \%] 
100 
42.9 
65.2 
(Rottensteiner, 2008) 
Completeness [%] 
80.2 
86.7 
82.6 
Correctness [%] 
97.9 
36.1 
59.4 
Table 4. Completeness and correctness for the Toulouse test 
area, depending on the update status. 
ranges from 18.4% with (Champion, 2007) to 23.7% with 
(Matikainen et al., 2007). The situation is a bit better for new 
buildings, with a correctness rate larger than 63% for all the 
methods and even rising to 96.8% with (Rottensteiner, 2008). In 
spite of such limitations, all the methods presented here are very 
efficient in classifying unchanged buildings, for which the 
completeness rates are higher than 93%, which indicates that a 
considerable amount of manual work is saved and also 
demonstrates the economical efficiency of these approaches in 
the context of aerial imagery. 
Analyzing Table 3 leads to similar conclusions for the LIDAR 
context. The correctness rate for the reported demolished 
buildings are again poor and only (Rottensteiner, 2008) 
achieves less than 50% false positives. However, the methods 
are very effective in detecting demolished buildings and achieve 
a completeness rate of 100% for this class. Compared to the 
outcomes obtained in Marseille, the main difference concerns 
the new buildings, which appear to be more difficult to extract. 
Thus, between 6.1% (Olsen and Knudsen, 2006) and 12.2% 
(Rottensteiner, 2008) of the new buildings are missed. If these 
percentages of missed new buildings can be tolerated, our tests 
indicate that LIDAR offers a high economical effectiveness and 
thus may be a viable basis for a future application. If these error 
rates for new buildings are unacceptable, manual post-process is 
required to find the missed buildings, at the expense of a lower 
economical efficiency. 
The situation is not quite as good with the satellite context 
(Table 4). The method by (Champion, 2007) is very effective in 
detecting demolished buildings (100%), but this is achieved at 
the expense of a low correctness rate (42.9%). The same 
analysis can be carried out with (Rottensteiner, 2008), but this 
method even misses quite a few demolished buildings. It has to 
be noted that, even though the completeness rates for 
unchanged buildings achieved by both methods are relatively 
low compared to those obtained in the Marseille and Lyngby 
test areas, they also indicate that even under challenging 
circumstances, 80% of unchanged buildings need not be 
investigated by an operator. The main limitation appears to be 
the detection of new buildings. As illustrated for an example in 
Figures 3e and 3f, 17.4% and 25% of new buildings are missed 
with (Rottensteiner, 2008) and (Champion, 2007) respectively, 
which is clearly not sufficient to provide a full update of the 
database and requires a manual intervention in order to find the 
remaining new buildings. 
In order to obtain deeper insights into the reasons for failure, in 
the subsequent sections we will focus our analysis on some 
factors that affect the change detection performance. 
4.2 Impact of the Size of a Change 
To analyse the performance of change detection as a function of 
the change size, we compute the completeness and correctness 
rates depending on this factor. For that purpose, new and 
demolished buildings are placed into bins representing classes 
148