Fua, Pascal 
2. We perform 3-D tracking from [f-1] into [f], thus identifying a certain number of markers in [f], i.e. attaching them 
to their legitimate joint. 
3. If all markers are still not found, we attempt to identify the 3-D markers that are still anonymous. We find all the 
skeleton's joints that are missing one or more markers. Assuming that displacement is minimal from one frame 
to another, we retrieve the coordinates of these joints in the previous frame, and calculate the distance from these 
joints to each remaining unidentified 3-D marker; the distance closest to the marker-to-joint distance specified by 
the marker model yields an association of the 3-D marker to that joint. 
4. If the distance from marker to joint is larger than the distance specified by the marker model, we "bind" the coordi- 
nates of the 3-D marker to the joint: We change its 3D coordinates so that the marker moves within an acceptable 
distance of the joint. We however leave all reliably reconstructed 3-D markers untouched. 
5. In the worst-case scenario, there may still be joints that are missing markers. We retrieve these markers in the three 
previous frame [f-3], [f-2] and [f-1], and calculate the acceleration; we apply this acceleration to the position in [f-1], 
thus obtaining an estimated position of the marker in the current frame [f]. As before, we calculate the distance from 
this inferred position to its associated joint. If it is out of range, we "bind" the coordinates. 
3.3 Fitting Results 
Percentage of markers identified by tracking without skeleton 
(over 200 frames) 
90% + 
80% + 
70% 
60% 
50% 
40% 4: 
Percentage 
30% + 
20% 
10% 
  
0% 
4 |41 |24 |31 |41 [51 |61 | 71 | 81 | 91 |102 112 |122 |132 |142 [152 |162 |172 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
[g Markers identified (83% |57% |42% |40% |33% |3096 33% (28% |26% | 26% | 22% | 23% [23% [22% |27% | 22% |15% 16% [11% | 0% 
  
Figure 6: Percentage of markers identified by simple tracking, for the karate motion of Figurefig:SkeletonResults. Without 
skeleton-based tracking, most of them are quickly lost. 
Figures 4 and 5 show the results obtained for a difficult karate motion that involves complex movements and sudden 
accelerations. Using the skeleton has enabled us to improve every step of the process, from 3-D reconstruction, to 
tracking and identification of the markers. It is robust with respect to noisy data: Out-of-bound and non-identified markers 
are rejected and occluded markers are properly handled. The effectiveness of skeleton-based tracking is illustrated by 
Figure 6: For the karate motion of Figure 4, our system does not loose any marker whereas, if we were to use only the 
simple tracking described above, we would quickly loose most of them. 
- 
4 VIDEO-BASED MODELING 
As video cameras become increasingly prevalent, for example as attachment to most computers, video-based approaches 
become increasingly attractive means of deriving models of people such as clones for video-conferencing purposes. Such 
approaches also allow the exploitation of ordinary movies to reconstruct the faces and bodies of actors or famous people 
that cannot easily be scanned using active techniques, for example because they are unavailable or long dead. In the 
remainder of this section, we first discuss our approach to face and then body modeling. 
4.1 Face Modeling 
Given a set of potentially uncalibrated images or a video sequence, our goal is to fit the animation mask of Figure 1(a). 
Our challenge, here, is to solve the structure from motion problem in a case where 
  
International Archives of Photogrammetry and Remote Sensing. Vol. XXXIII, Part B5. Amsterdam 2000. 259