wing for 
All these 
y of 
pe 
) is 
ofa 
ific on- 
x 10'in 
An 
this 
to 
r that 
Boulanger, Pierre 
purpose, we developed a keyboardless, two-handed UI that combines a mouse and a three DOF joystick that are used in 
parallel. 
Two mutually exclusive operator modes (monitoring and control) are possible and can be selected by using the mouse. 
In the monitoring mode, the joystick is used to navigate in the environment while the mouse is used to display 
information about the instrumented machines by clicking on specific parts of them. The clickable parts are highlighted 
to indicate their function when they are brushed by the mouse, i.e. when the mouse cursor is over them, as illustrated in 
Figure 4 (virtual query system). The highlighting style indicates the function of the object, which can be either to query 
and display information about it (sensor information or others), or to toggle between the monitoring and the control 
mode. Also, in the monitoring mode, the pose of the trucks and the excavator are displayed by placing the avatars in the 
same pose as the real ones. 
In the control mode, the joystick is used for the rate control of the virtual excavator joints in either joint or coordinated 
control mode [Withney, 1969]. The operator chooses the type of control mode (joint or coordinated) by clicking on a 
different mouse button. The coordinated control mode allows the direct control of the bucket motion instead of having 
to coordinate the movement of each separate joint of the excavator. It has been implemented in order to help novice 
users for the control of the excavator since it has been shown that this control mode significantly improves their initial 
performance compared to the traditional joint control [Lapointe, 1999]. Once that the virtual excavator is positioned 
correctly, the operator push the joystick's trigger to transmit joints positions of the virtual excavator through the 
network to the real excavator. The real excavator then uses a local position controller to position itself accordingly to 
the virtual excavator pose, before sending back an acknowledgement signal to the operator station. It is important to 
note that both the monitoring and control mode require a calibration of the virtual world with the real one. 
23 Data communication controller 
In complex and distributed systems such as the one developed here, the management of the communication 
infrastructure becomes critical for initialization, maintenance, and failsafe performance of the system. Within the virtual 
private network, the current system prototyped here employs a network bulletin board implementation which all 
network nodes use to post their latest status. In this way, any node that requires data from other nodes can check their 
status using the bulletin board and thereby avoid the bandwidth overhead associated with the use of embedded heartbeat 
data and regular dedicated heartbeat packets. 
The processing of all the incoming data from the mine site also required the development of a central local 
communication controller that is exclusively used by the observer station. This controller is in charge of data 
synchronization, logging operation, dealing with obsolescence, data validity, formatting, firewall, and allow to separate 
display from communication issues. 
2.4 On-board remote monitoring sensing system 
The remote equipment system for the Hitachi EX200 excavator was retrofitted with remote-control capabilities using 
their CAN-networked Genie II controllers and a number of sensors installed to provide condition indicators for overall 
machine health monitoring. The sensors are used to measure hydraulic pressure, joint angles, motor speed, temperature, 
position from GPS, etc. One particular sensor monitors the acoustic characteristics of various portions of the machine 
using an acoustic anomaly detection function. This sensing function also samples audio segments for playback at the 
observer station. A digital camera returns up-to-date images of the workspace and a laser range is used to create local 
terrain model. These two imaging sensors are mounted on a remotely controlled pan-tilt unit. Using this REMCAM 
system (see Figure 4), video images can be texture-mapped onto polygonal surfaces measured by the time-of-flight 
range sensor. 
Local control of the machine is accomplished with a local version of a script executor called PACE from Dynacom. 
This script executor issues commands to the robot control computer (RCC). The RCC communicates via a CAN bus to 
drive the Genie II controllers and collect machine sensor data for networked transmission back to the observer stations. 
Communication between this mobile equipment and the internet-based data communication network can use in-house 
radio modems or digital cellular telephone modems. The RCC also gathers machine status and sensor data for trending 
and performance evaluation, by using wireless Ethernet modems for data transmission between the machine and local 
area network hardware. 
  
International Archives of Photogrammetry and Remote Sensing. Vol. XXXIII, Part B5. Amsterdam 2000. 97