The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Vol. XXXVII. Part B7. Beijing 2008
5. PROBLEMS IN THE IEEE GRSS 2006 DATA FUSION
CONTEST
5.1 Background
Giga bites of testing data were made available to the contest
participants for image fusion. The testing data contain two
types of images: (1) QuickBird and (2) simulated Pleiades Pan
and MS images. The Pleiades Pan images were simulated using
green and red channels, which did not cover the designed
spectral coverage of Pleiades Pan, 500-850 nm, analogously to
Ikonos and QuickBird Pan (Alparone et al.,2007). The data
volume of QuickBird images occupies over 80% of the total
data volume provided to the participants for the contest. The
fusion results generated by the participants were sent to one of
two official contest judges, Dr. L. Alparone.
5.2 Contest results
The contest evaluation concluded that the fusion results of the
Generalized Laplacian Pyramid Decomposition Featuring a
Modulation Transfer Function Reduction Filter and a Context
Based Decision Injection Rule (GLP-MTF-CBD), also called
GLP-CBD, outperformed the other competing algorithms for
most of the criteria [MB, VD, SDD, CC, SAM, ERGAS, and
Q4] (Gamba et al.,2006). An IEEE Certificate of Recognition
was granted to the GLP-CBD developers at the IEEE IGARSS
2006 conference in August 2006.
The paper on 2006 data fusion contest outcome (Alparone et al.,
2007), published in IEEE Transactions of Geoscience and
Remote Sensing, provided results of quantitative analysis and
visual evaluation. The visual analysis stated:
• “GLP-CBD: Image is nice as a whole. Colors
should be better synthesized. This would enhance the
legibility of the image. Details are there, except for the
most colored (blue, red). Errors in colors lead to
interpretation errors. Contours should be sharper. There
is no bias, except for Strasbourg outskirts. Unacceptable
for detailed visual analysis.”
• “UNB-Pansharp: Image is too noisy. There are
many artifacts. Colors are not well synthesized as a
whole and locally. Green trees are not green enough.
Red or blue cars are absent. Shapes are not well defined;
they are sometimes underlined by black lines. Too large
bias is observed. There is lack of variance as a whole.
At times, unacceptable. In best cases, unacceptable for
detailed visual analysis.”
5.3 Irregularity of the evaluation
After the contest award in August 2006, numerous requests
were sent to the contest committee for an opportunity to review
some fusion examples by the contest participants. The requests
were rejected and the participants were asked to wait for the
publication of the paper on the contest outcome. Finally, the
evaluation examples were provided to the participants for
review in late December 2006.
outcome paper in 2007 (Alparone et al.,2007) cannot be
displayed here. However, readers can still see some difference
by comparing the GLP-CBD QuickBird fusion result published
in the IEEE GRSS Newsletter (Gamba et al.,2006) and that in
the contest outcome paper (Alparone et al.,2007), even though
In the reviewing of the fusion results used in the contest
evaluation, it was found that the QuickBird fusion results
produced by UNB-Pansharp were not evaluated in the contest,
even though giga bytes of QuickBird fusion results of UNB-
Pansharp were sent to the judge, Dr. L. Alparone, together with
fusion results of the simulated Pleiades data.
Two subsets of the UNB QuickBird fusion results are given in
Figure 4. Readers can compare the original QuickBird Pan and
MS images with the fusion results to evaluate whether the
visual analysis of the IEEE fusion contest outcome by Alparone,
et al. 2007 (see above) is objective, or not. Internal evaluation
among the contest participants clearly agreed that the results
produced by UNB-Pansharp are superior to those of GLP-CBD.
Literature review after the fusion contest, especially after the
publication of the contest outcome (Alparone et al.,2007),
proves that the judge Dr. L. Alparone is also a co-developer/co-
author of the top winning GLP-CBD algorithm (Alparone et al.,
2003; Aiazzi & Alparone et al., 2002; and Aiazzi & Alparone et
al.,2006).
A request for permission to use the GLP-CBD fusion results
provided to the 2006 contest participants for publications was
denied. The original answer to the participants is quoted below
to avoid misinterpretation:
“In particular, on the Quickbird fused image I noted few
small areas where a proper spatial enhancement did not occur
because of statistical instabilities of the adaptive injection
model. Such fusion inaccuracies appear only in few small
areas and cannot change the global evaluation of my
algorithm. However, il [if] one extracts the misfused patches
and compares only them with those of other algorithms, he
might be erroneously lead to believe that GLP-CBD is not
the best algorithm among those compared in the Contest.
After the contest I realized of the inconvenience by watching
the fused images in the DFC [data fusion contest] web site
and I fixed it. On the other side, the DFC site should contain
the images that were evaluated for the Contest and cannot be
changed. Therefore, if you want use GLP-CBD fused data
for any publications, I will be pleased to provide fused
versions with the fixed algorithm, which performs identically
to the earlier one except on the above mentioned small areas.
So, I do not give you, or any other may request it, the
permission of using the GLP-CBD fused data found in the
DFC contest, because such data refer to the Contest only and
do not reflect the current progress of my activity, as it should
appear in an unbiased future publication.”
In the comparison between the GLP-CBD QuickBird fusion
results received by the contest participants in 2006 and that
published in the IEEE contest outcome paper (Alparone et
al.,2007), it is found that the GLP-CBD QuickBird fusion result
in Alparone et al.,(2007) is clearly better than the one received
by the participants—misfused patches and blurred areas are
clearly reduced.
Because the request for permission was rejected, the
comparison between the GLP-CBD QuickBird fusion result
used in the contest in 2006 and that published in the contest
the images displayed are very small and do not cover the same
area. To see the misfused patches, readers can see the GLP-
CBD QuickBird fusion result published in the IEEE GRSS
Newsletter (Gamba et al.,2006) and pay attention to the area
circled in Figure 4 of this paper. The difference leads to a