ISPRS, Vol.34, Part 2W2, “Dynamic and Multi-Dimensional GIS”, Bangkok, May 23-25, 200t
148
The primary objective of the script-based system is to evaluate
the effectiveness of alternative tutorial systems. It is foremost
important to formulate the learning objectives, which will serve
as the ultimate evaluation criteria. These objectives should be
defined in three areas: conceptual learning, critical thinking, and
operational skills. For example, the intended learning objectives
(ILO) for the session on “Plane Coordinate Systems" are defined
as following:
• Be able to describe the basic elements of plane
coordinate systems.
• Understand the characteristics and applications of
UTM and US State Plane Coordinate Systems.
• Be able to select or specify an appropriate plane
coordinate system for a given region.
Identifying learning topics
The topics will cover the normal scope of Introduction to GIS.
These are:
• Digital representations of spatial phenomena
o Object representation
o Database representation
o Visual representation
• Selection
o Attribute query, SQL
o Spatial query: simple, complex
• Join
o Relational join
o Spatial join
o Geometric overlay
Decomposing learning topics into tutorial topics
Each learning topic may contain several subtopics. For
instance, object representation needs to deal with how the
geometries as well as the attributes of point, line, and polygon
are represented. In such case, two tutorials may be required. In
addition, the prime reason to develop this system is to make a
comparison with existing menu-selection systems. That target
system is “Getting to Know ArcView GIS”. The system contains
some thirty chapters, each including several tutorials. It is
important to carefully define the tutorial topics in the script
system so that they correspond to the topics in the menu
selection system. Each topic should be defined with reference
to the concept and methods (functions) the students expected to
learn.
Design and implementation
Avenue objects and requests are identified for each tutorial
topic. Data sets identical to those used in the menu-driven
tutorials will be used. Similar procedures are implemented.
Testing
Scripts for each tutorial topic are tested using sample data to
identify bugs and errors.
Documentation
Comments will be added to the scripts. Descriptions of the
tutorial are placed at the beginning, including the purpose, data,
and the date of the script. Short comments will be inserted
where explanation may be needed.
Conduct
The two tutorial systems were subsequently evaluated through a
control experiment. Students were divided into two groups, A
and B. Group A used the menu-driven system and B used the
script-based tutorials. There were ten students in each of the
group. To minimize academic bias, the grouping was done so
that the mean GPA scores were approximately the same. With
the group formed, the following activities were conducted for
each module:
• Attend lecture (same for both groups).
• Complete tutorials (Group A used the ESRI tutorials,
Group B used the script-based one).
• Review (separately for each group).
• Quiz (same questions for both groups).
The lecture introduces the concepts and corresponding software
components. The software part includes both the graphic
interface elements and the Avenue objects/requests. When
working on the tutorials, Group A students would read the
instruction then point-and-click through the steps. Group B
students would have to type in the scripts, debug, and execute
them. During the tutorial process, the instructor was available to
provide feedbacks and answering questions.
The purpose of the review is to provide a summary on the
operational matters, highlight the key processes, and make
connections to the learning objectives for the session. This is
also the time for students to ask questions related to the lecture
and the tutorials.
After the review, a quiz follows. The quiz is the primary means
to evaluating the learning effectiveness of the tutorials. The
quizzes were designed based on a set of intended learning
objectives (ILO). Quiz questions fall into three types, multiple
choice, essay, and problem solving. Each quiz is graded on a
10-point scale.
Analysis of results
This study falls into the category of group comparison. There
are numerous statistical methods for this type of analysis. T-
test, for example, can be used to test for a statistically significant
difference in the means of the test sores between the two
groups. In addition to the comparison for average scores, we
also test the differences for different types of questions, namely
conceptual and problem solving. Furthermore, for a sample of
this size, direct observation and qualitative methods should also
play important roles. For instance, questions students asked
during the tutorial and the quiz sessions can be recorded and
summarized. (Students were allowed to ask questions even
during the quiz session. It is the instructor’s judgment on how
the question should be answered so that the quiz reflects the
objective measures of the student’s capability.)
PRELIMINARY RESULTS
An experiment was conducted through a one-semester
introductory GIS course. Twenty students were participated,
resulting in ten for each group. The reference textbook is
“Getting Started with Geographic information Systems”, third
edition, by Keith Clarke. Students have access to the digital
tutorial “Getting to Know ArcView GIS” and all related software
documentations for ArcView.
Preliminary analysis reveals two findings:
• Statistical analysis shows no significant difference
between the overall test scores of the two groups.
• Students using the script-based tutorials achieved
higher scores on conceptual questions but had similar
scores on problem solving.
We selected three topics as examples to illustrate the
experiment.
1. Plane coordinate systems
ILOs:
• Be able to describe the basic elements of plane
coordinate systems.
• Understand the characteristics and applications of
UTM and US State Plane Coordinate Systems.
• Be able to select or specify an appropriate plane
coordinate system for a given region.
