324 
The multisensorial camera: a new approach in multi-sensory pattern recognition 
Robert Massen 
MASSEN machine vision systems GmbH 
Am Seerhein 8 , D-78467 Konstanz 
tel. +49-7531-57502 , fax +49-7531-53740 
Key-words: multi-sensoriai, 3D camera, colour camera, classification, segmentation 
Abstract 
Segmentation of a scene often suffers from the lack of information contained in every pixel. The multi-sensorial 
camera produces for every pixel not a scalar attribute , but a complete feature vector with many , preferrably 
uncorrelated components. We show howa ,,Colour&3D" line scan camera can be designed which generates a 
feature vector ( Intensity, Hue, Saturation, Z= height) for every pixel at resolutions of typically 2048 pixels 
along the line of scan and with scanning frequencies up to several kHz. The processing of such a vectorial i image 
starts with a LUT-based, trainable pixel classifier who transforms the vectorial image into a stack of binary class 
label images. This significant data reduction results in only little information loss and leads to further proces- 
sing based on well-established binary image processing techniques. 
1. Image segmentation with little robustness 
One of the most common problems in vision systems is their lack of robustness against small changes in 
illumination, object arrangement, shadows and specular reflections or changes in the optical characteristics of the 
surface of the objects to be segmented . The poor success of symbolic image processing technologies in 
industrial applications is mainly due to this lack of robustness: extraction of symbols like corners, polygones 
etc. tends to fail completely, if the iconic preprocesssing such as edge detection does not perform in a proper and 
reliable way. Traditionally, there have been two attempts to alleviate this uncomfortable situation: 
1. use more intelligent algorithms based on more a - priori knowledge. This works well in a well-known 
test scene but not so in real life where generally little reliable a - priori knowledge is available. 
2. restrict to very simple pixel-based algorithms like blob analysis, projections etc. This is the day-to-day 
approach for most actual vision systems. 
3. work on more than just one single image, preferrably on a whole sequence of images taken under 
different illuminations. This is a good approach but often fails due to the increased processing time. 
We prefer to remember the good old rule of pattern recognition : 
Do not try to squeeze out the last drop of information from a single feature but find additional, 
uncorrelated features. 
We like to translate this into : Get more information from a pixel than just grey-level intensity. 
2. À feature vector attached to every pixel 
With the introduction of colour, we jump from a scalar pixel to a vectorial one: every pixel carries now 
information on intensity and saturation and hue . Of course we should be careful about the RGB colour vector, 
whose components are highly correlated and which is not a good colour space for segmentation and 
classification. Our Colour Brain family of vision systems which uses colour matrix- or line cameras and 
trainable, look-up table based classifiers / 1, 2/ has shown how the segmentation of a scene is dramatically 
improved due to the increased information content in every IHS colour pixel compared to a dumb greylevel 
pixel. 
IAPRS, Vol. 30, Part 5W1, ISPRS Intercommission Workshop “From Pixels to Sequences”, Zurich, March 22-24 1995