SEMI-SUPERVISED INCREMENTAL LEARNING OF HIERARCHICAL APPEARANCE 
MODELS 
Susanne Wenzel and Wolfgang Fôrstner 
Department of Photogrammetry 
Institute of Geodesy and Geo Information, University of Bonn 
susanne.wenzel@uni-bonn.de, wf@ipb.uni-bonn.de 
http://www.ipb.uni-bonn.de 
Commission III/IV 
KEY WORDS: Detection, Building, Structure, Interpretation, Classification, Incremental Learning, Recognition 
ABSTRACT: 
We propose an incremental learning scheme for learning a class hierarchy for objects typically occurring multiple in images. Given one 
example of an object that appears several times in the image, e.g. is part of a repetitive structure, we propose a method for identifying 
prototypes using an unsupervised clustering procedure. These prototypes are used for building a hierarchical appearance based model 
of the envisaged class in a supervised manner. For classification of new instances detected in new images we use linear subspace 
methods that combine discriminative and reconstructive properties. The used methods are chosen to be capable for an incremental 
update. We test our approach on facade images with repetitive windows and balconies. We use the learned object models to find new 
instances in other images, e. g. the neighbouring facade and update already learned models with the new instances. 
1 INTRODUCTION 
The interest in geo-applications like Google Earth and Microsoft 
Virtual Earth has increased tremendously lately. So far, most 3D 
city models are very simple and without semantic information. 
Furthermore, the creation of such models needs a lot of user in 
teractions. The need for semantic enrichments is seen in different 
applications of 3D city models of high level of detail. E. g., for 
accurate navigation to a certain address one would like to know 
the exact position of the door. Tasks of urban management or 
catastrophe management need to know the number of stories and 
the availability of balconies. Architects who plans a new building 
need to know the types and number of windows in the neighbour 
ing facades. Automatic procedures for deriving such information 
are of great use. The present paper focuses on simplifying the 
development tools for interpreting images of man-made scenes, 
especially of building scenes. 
As building parts often show some degree of symmetry, we ad 
dress the following problem: Given one example of an object and 
given the prior knowledge that it is repeatedly present in the same 
image we can learn its object class appearance. This provides a 
tool to detect other objects of the same type in other images with 
minimal need of user interaction. Additionally, we build up an 
object class hierarchy with a minimal amount of user interactions. 
As we want to increase our knowledge about class appearances 
with every new image, we want to propose models that are capa 
ble of incremental learning. 
The paper is structured as follows. First we give an overview 
over our concept for semi-supervised incremental learning. Sec. 3 
refers to related work and Sec. 4 describes our concept in more 
detail. Experiments are described in Sec. 5. We conclude with 
Sec. 6. 
2 OVERVIEW 
Our objective is to develop an incremental learning scheme which 
requires the least user interaction. To achieve this goal, we need 
four procedures: (1) a detector for finding new instances (2) a 
prototype generator for finding new class candidates unsuper 
vised, (3) a classifier for the envisaged classes and finally (4) an 
GUI for the interaction between the system and the supervisor. 
The internal representation of the classes needs to be updatable 
incrementally. 
In order to achieve a clear representation we change the order and 
start bottom up. So the first task of this work is to automatically 
identify prototypes, i. e. characteristic instances of their classes 
for supporting autonomous learning. In general, we cannot as 
sume single prototypes to be sufficient for complex object classes 
like windows, balconies or doors, but we need to assumethat mul 
tiple prototypes per class exist, which possibly are structurally 
related e. g. by a hierarchy. As an example Fig. 1 shows some in 
stances of windows. We can obviously find a hierarchy of proto 
types, e. g. rectangular and arclike windows, which further have 
subclasses depending on the number of crossbars. We propose a 
method for identifying prototypes using an unsupervised cluster 
ing procedure. For this purpose we use a similarity graph, which 
is built up recursively by detecting similar objects given only one 
or a few examples, and take its connected components as clus 
ters. Thus, given a single instance of a class we let the system 
find as many clusters as possible for that superclass. Based on 
user’s judgement they are specified, e. g. as an object of the same 
superclass but maybe a new subclass or as background. We learn 
the object class appearance and use their prototypes p c to auto 
matically detect probable new instances of class c G 1... C in 
new images. 
Secondly, for classification we represent identified classes as a 
kind of Fisher-images, see (Belhumeur et al., 1997) and combine 
reconstructive (PCA) and discriminative subspace (LDA) meth 
ods, see (Fidler, 2006) for classification of new instances. When 
detecting new instances in new images we incrementally update 
our class representations and the object hierarchy. 
The methods described in this paper are conceived very general. 
They can be used whenever one deals with objects which appear 
multiple times in an image, e.g. in repetitive structures and which 
occur with similar appearance in other images. As special appli 
cation this paper addresses the detection and classification of win-