International Archives of Photogrammetry and Remote Sensing. Vol. XXXII, Part 5. Hakodate 1998 
FACIAL EXPRESSION RECOGNITION FROM IMAGE SEQUENCES 
USING SELF-ORGANIZING MAPS 
Ayako KATOH*, Yasuhiro FUKUI** 
*Student, **Professor, Applied Systems Engineering, Science and Engineering 
Graduate School of Tokyo Denki University 
Ishizaka, Hatoyama, Hiki, Saitama, 355-0311 
E-mail: aya@bme f.dendai.ac.jp, fukui@f.dendai.ac.jp 
Japan 
Commission V, Working Group V/4 
KEYWORDS: facial expression, image sequences, self-organizing map, neural networks 
ABSTRACT 
Just as humans use body language or nonverbal language such as gestures and facial expression in communication, in 
the future computers will also be able to communicate with humans. In medical engineering, it is possible that 
recognition of facial expression can be applied to support communication with persons who have trouble communicating 
verbally such as infants, aged persons and mental patients. The purpose of this study is to enable recognition of human 
emotions by facial expressions using engineering methods. Our observations of facial expressions founds that the 
important facial segments for recognition are the eyebrows, the eyes and the mouth. We also found that it is important to 
recognize facial expressions by identifying changes in facial expressions by using sequences of images. Self-organizing 
maps, which are neural networks of two-layer structure, are used to extract features of image sequences. A self- 
organizing map is taught for each facial segment. The image sequences of six types of facial expressions are recorded 
on VTR and made into image sequences consisting of 30 images per second. Subjects are asked to make facial 
expressions ranging from expressionless to one of the six types of facial expressions under consideration. Gray levels of 
each segment are input into the self-organizing map corresponding to each segment. The neuron in the output layer, 
called the victory neuron, reacts to the feature nearest the input segment reacts. Our analysis of the changes in victory 
neurons demonstrates that they have characteristic features which correspond to each of the six facial expressions. 
1. INTRODUCTION emotions manifested in facial expressions using a instant 
image (Hitoshi et al, 1994. Katsuhiro et al., 1994). The 
Sensitivity information such as emotions plays an six types of facial expressions shown by Ekman et.al. are 
important role in communication. The means for famous (Paul et al., 1975). Many studied have also been 
conveying the sensitivity information are nonverbal made on facial expression recognition using an instant 
languages such as gestures, facial expressions and voice image. But, a number of studied have pointed out 
tone. Until now, research on sensitivity information has limitations in recognizing facial expressions using only an 
been carried out primarily in the field of psychology instant image (Hiroshi, K. et al., 1995. Hiroshi, K. et al, 
(Hiroshi et al., Y., 1994). 1993. Kenji, 1991. Tatsumi, 1995.). Thus, in this study, 
But in recent years, it has also been taken up in the fields an attempt was made to recognize facial expressions from 
of engineering. Focus has been shifted from artificial image sequences. 
intelligence based on knowledge provided by the Our observations demonstrate the important of 
researcher towards intelligence based on self-generated recognizing facial expressions from movements of several 
knowledge in computer science. In the field of medical facial segments. A method for extraction features using 
engineering the mental condition of patients is considered self-organizing maps is reported this paper. 
in addition to his or her physical condition (Noboru, 1994). ; 
Recently extraction of sensitive information shown by 
nonverbal language is given considerable attention in the 2. SELF-ORGANIZING MAP 
field of human interface (Shigeo, 1994). 
The purpose of this study is to recognize human emotions Self-organizing maps are neural networks designed to 
in facial expressions by focusing on facial expressions as input data into one or two dimensional space while 
a form of nonverbal language. maintaining the same distance found in the original 
Research on facial expressions has been carried out in the dimensional space (Teuvo, 1996). Two-layer of self- 
field of psychology. Most of the research discusses organizing maps consists of input and output layer. All 
443