| 2004
orest -
Non-
rest or
: Non-
cess of
oducts
eneral,
lerived
o gi In
hat are
)). This
matrix
classes
quality
sample
tracted
rs and
on and
matrix
ovide a
al and
Iso that
ta, all
fusing a
> with a
image
sses the
ss using
e of the
and the
*hard
lologies
ropriate
This is
y at the
in situ
International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol XXXV, Part B7. Istanbul 2004
2.2 Continuous improvement protocol
The National Carbon Accounting System considered evaluating
the quality of the change maps critical to the monitoring
process. The goal of this evaluation was two-fold. Firstly, to
assess the quality of the change maps; and secondly, to identify
reasons why the change maps appeared to have problems so
that the classification procedure(s) could be continuously
improved.
The continuous improvement protocol was initiated in 2001
after completion of the first 10 change maps. Subsequently,
some of the map sheets were revised and the classification
methodology amended for the 11% change map production
(2000-2002). Continuous improvement responds to the ongoing
development and updating of the NCAS land cover program by
looking at the source and significance of potential errors. This
allows for a targeted and prioritised rectification of any
problems that can be assessed and, if necessary, further
amended on each round of updating and continuous
improvement. In contrast, verification only provides a one-off
statement of accuracy.
Initially, a traditional approach to accuracy assessment was
envisaged whereby area-based samples would be extracted from
a given change map, and concurrent areas extracted from higher
resolution reference data for dates "close to" those of the
satellite imagery used to produce the change map. As an
alternative, it was proposed that enough points — rather than
areas —be extracted from the change maps and verified against
the reference data to allow a statistically valid statement of the
accuracy of change maps to be produced. The area-based
approach (e.g., Tian et al, 2002) finds its strength in a
quantified statement of accuracy of areas of change, albeit
qualified by the unavoidable difficulties of geo-rectification and
the interpretation of differing image products. The suggested
point-based method is more targeted at determining how good
the current methodology for detecting land cover change from
multi-temporal satellite images is at discriminating between
Forest and Non-Forest conditions under a range of different
circumstances (forest types, soil types, relief changes, etc.) and
to testing the robustness of the methodology employed across
the diverse landforms of Australia.
Verification of state (Forest/Non-Forest) and change map
accuracy was accomplished by first verifying the existence of
suitable high-spatial resolution images for a given change map
area. Given the 30-year temporal scale, the best suited imagery
were usually aerial photographs, with a spatial scale of around
1:50000, whose acquisition date were co-incidental or close to
the satellite image acquisitions. These three factors — the region
covered, the image scale, and the acquisition date —vary with
each change map area. In total 384 aerial photo pairs were used
in the verification; the majority of photos had scales between
1:25000 and 1:80000. Both panchromatic (black & white) and
colour aerial photos were used.
Aerial photographs were the only data that could be employed
as the reference data against which change maps would be
assessed; no other information source was available nationwide
from 1972 to 2000. A nationwide inventory was made of aerial
photographs that were readily available from government
agencies and their location relative to the 1:1,000,000 map
sheets was noted. Where possible, 10 stereo pairs of aerial
photographs were selected for each map sheet. In the more
remote areas of Australia, it was sometimes not possible to
749
obtain this many; 24 of the 37 mapsheets evaluated used 10
stereo pairs and 13 used fewer than this number. Photographs
were selected based on their geographic and ecological
distribution within a map sheet, their temporal distribution over
the 30-year period, the scale and film type of the photograph,
and the quality of the photographs.
Each of the 37 map sheets was evaluated individually and all
map sheets were processed in an identical fashion. Each aerial
photo was then gridded and converted to digital format and co-
registered to one of the satellite images used in developing the
associated change map. In general, the rectified and calibrated
2000 TM image was employed for this purpose. However, in
some cases where cloud cover was too heavy for the photo
location on the 2000 image, or land cover changes between the
photo date and 2000 required that the 1991 TM image be
employed.
Forty randomly selected points were then located on each co-
registered photograph. For each point, a photo-interpreter
determined what was present using a fuzzy classification with
five classes:
e Definitely Forest; (see definition below)
* Probably Forest;
Unsure;
Probably Non-forest; and,
e Definitely Non-forest.
Fuzzy Logic Definitions
Definitely Forest:
Where the photo interpreter has no doubt that both the corresponding
points on the change map and the photo are forest. The confidence of
the interpreter relates to their knowledge of the local area, the
stereoscopic information available from the photography, the relatively
close alignment in time between the photo and the satellite images
used to compile the change maps.
Probably Forest:
Where the photo interpreter has expressed a good degree of
confidence in the matching relationship between the change map and
the aerial photo but some uncertainty exists. For example, i) making a
judgement call as to whether a forest was 1.9 m or 2.1 m tall or had a
19% crown cover or the required 20%; ii) unambiguously resolving the
exact neighbourhood of the point being checked due to
heterogeneities in the forest structure; iii) a slightly longer elapsed
time between aerial photo and the satellite image and subsequent
After interpreting all forty points for a photo, the interpreter
determined their image classification for the relevant change
map. For example, a 1998 photo would be evaluated against
the 1998 change map that covered the period 1995-1998. If the
date for a photo did not exactly match a time slice date, it was
evaluated against the closest date — e.g., a 1997 photograph was
still evaluated against the 1998 change map. In such cases, if
deforestation had clearly occurred between the date of a photo
and its satellite-image equivalent, the areas affected were not
sampled, or a different photo was employed.
In addition, (s)he recorded if the at least four of the eight
neighbours of the pixel in question were of the same type in
order to be able to subsequently identify isolated pixels.
Finally, the photo-interpreter examined the temporal profile of
the sampled pixel to determine if it was always forest (Forest
Throughout), álways non-forest (Non-forest Throughout),
deforested one time and remaining non-forest (Deforestation),
regencrated one time and remaining forest (Regrowth), or was
Forest/Non-forest more than once separated by a
