each camera are recorded on the video tapes. The 
subjects are 31 persons who are unskilled to the personal 
computer and a word processor. It is easy to distinguish 
the perplex situations by observing facial expression, 
utterance, behavior and keyboard operation. We picked 
up the perplexed behaviors and counted the number of the 
subjects who took those actions by observing the replayed 
image sequence. 
ms 2 
Personal computer 
CRT Camera 1 
|. 
  
  
  
  
  
  
  
  
  
Lt 
  
  
  
  
Keyboard 
^ Camera 3 
d ] 
Operator 
Camera 4 
Figure 1 Experimental system for observing 
perplexed behaviors. 
2.2 Observation results 
The subjects took the perplexed behaviors when they lost 
the operational method. The representative cases were 
as the following. : 
(D The inputted Roman characters can't be transformed 
in the correct Chinese character. 
The double consonant in the Japanese syllabary can't 
be inputted with Roman characters. 
The sentences can't be edited. 
The operation mode once changed can't be returned 
to origin. 
eo © 
The representative perplexed behaviors were as the 
following. : 
(D Change ofthe facial expression without the key input : 
knit the eyebrows, open the eyes wide, open the 
mouth, smile, laugh while showing the teeth 
@ Motion of the upper half of the body without the key 
input : bend forward to gaze on the CRT display, 
bend backward 
Motion of the head without the key input : shake, tilt 
Change of the glance without the key input: gazeon 
the CRT display, gaze on the keyboard, move 
between the CRT display and the keyboard frequently 
@ Motion of the arms without the key input : support 
the jaw, hold the head 
(9 Keyboard operation : take the longer time interval 
for key input, hit a same key repeatedly, hit keys at 
random 
©@ 
The words such as “Why?” are uttered in the perplex 
situations. However, the voice signal is excluded here 
because it is easily disturbed by other person’s voice and 
noise in the practical scene. 
Table 1 shows the number of the subjects who take the 
above behaviors. Every subject took a few kinds of the 
above perplexed behaviors. The perplexed behaviors 
other than the item (6) "Keyboard operation" are attended 
with the break of the key input for morethan 3 sec. While 
the jaw or the head is supported by the arm, the head 
position and attitude keeps still. Accordingly, supporting 
the jaw and holding the head can be classified as motion 
of the glance “gaze”. It is clear that the detection of the 
perplex situations is possible 10096 by combining the 
motion of the head and the time interval of the key input, 
while the rate of the subjects who don't change the facial 
expression is 29% (9 persons). 
Table 1 Observation results of perplexed behaviors. 
  
  
  
  
  
  
  
  
  
number of 
object and behavior persons 
(overlapped) 
upper |bend backward | 8 | 
half body | bend forward | 13 
lance | comeandgo | 7 
9 gaze 26 | 31 
motion í tilt 15 31 
age shake 2 
key |_ samekey [12/44 
stroke random key 2 
knit the eyebrows 4 
facial |. opentheeyes wide — | 1| 
open the mouth 12 20 
expression|” 77777777 smile ^^^ 27702 We 
laugh while showing the teeth| 8 | 
  
  
  
3. RECOGNITION OF HEAD MOTIONS 
3.1 Chase of both pupils 
According to our observation, it became clear that the 
state of mind was shown in action patterns rather than the 
facial expression. We adopted the method of presuming 
the operator's perplex situations from his action. In our 
method, the movement of the head is captured by the 
image processing. 
Infrared illumination 
© © 
  
CCD 
camera 
  
oe 
0 0 
Q/ ss 
[e 
Image processor 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
D, À 
Personal computer 
Figure 2 Experimental system for detecting 
perplexed behaviors. 
436 
  
  
In gener: 
image p! 
the move 
because 
be easily 
recogniti 
camera 
display a 
word pro 
the oper: 
face turn 
by such 
pupils in 
in near ir 
infrared : 
a low-pa 
(IR-85: F 
(infrarec 
Calcu 
J [2 
Figui