In: Wagner W., Szekely, B. (eds.): ISPRS TC VII Symposium - 100 Years ISPRS, Vienna, Austria, July 5-7, 2010, IAPRS, Vol. XXXVIII, Part 7B
650
of laboratory, in-situ, air and space-borne measurements. This
was primarily due to difficulties associated with the acquisition
of accurate and spatially detailed descriptions of 1) plant
architectural properties, like foliage orientation and density,
wood distribution and branching patterns, etc., 2) directional
scattering characteristics of plant and background constituents
suitable for inclusion into canopy RT models, and 3)
directionally resolved solar radiation fields, that are all needed
to guarantee a faithful reproduction of the actual 3D target (at
the time of observation) within the RT models. The evaluation
of models through comparison with observation requires also
access to information regarding the angular and spectral
resolution of the measuring devices, as well as, the uncertainties
associated with eventual up-scaling and correction techniques
(e.g., atmosphere, adjacency effect, point spread function).
To avoid these issues RAMI evaluates models under perfectly
controlled experimental conditions, i.e., all structural, spectral,
illumination and observation related characteristics are known
without ambiguity. Deviations between RT simulations can thus
only be due to - explicit or implicit - assumptions and shortcuts
entering model-specific implementations of the radiative
transfer equation. This mathematical foundation of physically-
based canopy RT models allows furthermore to verify model
predictions of arbitrary sub-components of the total (absorbed,
transmitted and reflected) radiation, i.e., quantities that could
not be measured in reality, and to check that the model
simulations remain consistent with physical reasoning even if
the environmental conditions deviate from those encountered in
nature. The latter two aspects are crucial since they allow - in a
few select cases - to assess RT model performance in absolute
terms, i.e., against analytical solutions of directionally-varying
or hemispherically-integrated radiative quantities and to
increase the confidence in model simulations relating to new
species/biomes and phenological conditions, respectively.
As a general rule, RT model comparison activities have to deal
with the fact that the true solution is not known. RAMI deals
with this issue trough a three-pronged evaluation approach
based on:
1. model consistency tests: that verify the internal
consistency of RT models, for example, with respect
to energy conservation, or, when radiative quantities
are modelled that vary in a pre-determined manner
across spectral bands, with background brightness, or,
with changing illumination conditions,
2. absolute performance tests: that compare the
magnitude of model simulated radiative quantities
against those predicted by analytical solutions (which
can be derived for some types of canopy targets
having certain well defined characteristics),
3. relative performance tests: that compare simulations
of different models in the light of knowledge obtained
from 1) the above model consistency and absolute
performance tests, and 2) an analysis of the shortcuts
and assumptions contained in their respective
implementations/formulations of the RT equation.
In order to obtain viable assessments of the trends, patterns and
perhaps also biases in the performance of canopy RT models it
is imperative to compare model simulations over an as large as
possible set of structural, spectral and illumination related
conditions. Such an approach is also conform with the paradigm
stating that computer simulation models can never be
completely validated and that efforts should focus instead on
the invalidation of such tools (Oreskes, 1994). In other words, a
model may yield the correct solution but for the wrong reasons,
and therefore nothing can be said with absolute certainty about
the reliability/accuracy of its predictions when applied to cases
that were not actually tested beforehand.
2.2 Outcome
As an open-access and community-driven activity RAMI
operates in successive phases each one aiming at re-assessing
the capability, performance and agreement of the latest
generation of RT models (http://rami-benchmark.ec.europa.eu/).
RAMI-1 involved a small yet somewhat abstract set of canopy
scenarios specifically designed to suit both 1-D and 3-D canopy
RT models. The results of RAMI-1 underlined the need for
model verification since many of the submitted simulations
differed quite substantially between the 8 participating models
(Pinty et al., 2001). In some cases, the cause of these
discrepancies may have been due to operator errors or software
bugs (some of which were identified during the data analysis
stage). RAMI-2, therefore proposed a rerun of all earlier
experiments together with two new test cases addressing issues
of topography and spatial resolution. This time 13 canopy RT
models participated and their agreement was much better
especially for the homogeneous canopies (Pinty et al., 2004a).
Expanding the set of experiments yet again, RAMI-3 concluded
with an unprecedented level of agreement amid its 18
participating RT models and this for both the homogeneous and
heterogeneous vegetation canopies (Widlowski et al., 2007).
candidate canopv RT model
credible’ canopy RT mode!
Figure 1. The selection process of‘credible’ canopy RT models.
Using the process outlined in Figure 1 it was possible to
identify six 3-D Monte Carlo models from among the RAMI-3
participants that differed by ~1% over several thousands of BRF
and flux simulations. A ‘surrogate truth’ reference data set was
then generated on the basis of the simulations of these ‘credible’
canopy RT models (Widlowski et al, 2008). This, in turn, lead
to the development of the RAMI Online Model Checker
(ROMC), a web-based benchmarking facility providing quasi-
real time statistics of the differences existing between the
simulations of a user's canopy RT model and the RAMI-3
"surrogate truth" data set (http://romc.jrc.-ec.europa.eu/).
