Fua, Pascal 
approximate 3-D location of thirteen major joints, namely the joints of the arms and legs, as well as the location of the 
pelvic joint, at the base of the spine. 
3.1.3 Skeleton Initialization Given these thirteen joint locations in all frames, we take the median distances between 
them to be estimates of the length of the performer's limbs. We then use anthropometric tables to infer the length of the 
other skeleton segments. This gives us a skeleton model scaled to the size of the actor. This model, however, is a static 
one. It has the appropriate dimensions but does not yet capture the postures for the gym sequence or the relative position 
of markers and joints. 
To estimate those distances, we first need to roughly position the skeleton in each frame by minimizing the distance of 
the thirteen key joints to the corresponding centers of rotation. This is done by minimizing an objective function that is 
the sum of square distances from the centers of rotation to the joint it is attached to. Given the fact that we use a sampling 
rate of 100 Hertz and that the gym motion is slow, the displacement from one frame to another is very small. Fitting is 
performed one frame at a time, and the initial parameter values for frame [f] are the optimized parameters obtained from 
the fitting in the previous frame [f-1]. As we only have thirteen observations for each frame, we do not attempt to estimate 
all of the skeleton's degrees of freedom. Only ten joints (shoulders, elbows, hips, knees, pelvic joint and the fourth spine 
vertebra) are active while all the others remain frozen. This yields the postures of the skeleton in all frames of the gym 
motion. In other words, we now have values of the global positioning vectors and degrees of freedom in each frame, as 
well as a better approximation to the limb lengths of the skeleton. 
3.1.4 Global Fitting We now have a skeleton model that is scaled to the size of the performing actor, but we are still 
missing a complete marker model, that is one that specifies where the markers are positioned on the actor's body and their 
distance to the joints to which they are attached. This is computed by performing a second least-squares minimization 
where the actual 3—D marker locations become the data to which we intend to fit the skeleton. 
Markers are not located exactly on the joints and the marker-to-joint distances must be estimated. To this end, we super- 
impose the markers' 3-D coordinates with the previously computed skeleton postures. In each frame, we then compute 
the distance from the marker to the joint and we take the median value of these distances to be our initial estimate of 
the marker-to-joint distance. Taking the marker model to be the distance from marker to joint means that the marker is 
expected to always be located on a sphere centered at the joint. We now have all the information required to fit the skeleton 
model to the observation data. The initial state is given by the previously obtained skeleton postures. As we need to check 
that all markers are present and identified before fitting, we do it one frame at a time. 
For each frame and for each marker, once the fitting is complete, the distance between marker and joint is stored. At the 
end of the gym motion sequence, we have as many such distances per marker as there are frames. The median value of 
these distances is an improved approximation of the marker-to-joint distance and becomes the final marker model. 
3.2 Capturing Complex Motions 
The resulting skeleton-and-marker model can now be applied to motions that we actually wish to capture. The procedure is 
very similar to the one used in the global fitting step of the previous section. However, we are now dealing with potentially 
complex motions. Consequently, even though 2-D and 3-D tracking ensure the identification of a large number of markers 
from one frame to another, ambiguities, sudden acceleration or occlusions will often cause breaks in the tracking links or 
erroneous reconstructions. For this reason, it has proved to be necessary to increase our procedure's robustness by using 
the skeleton to drive the reconstruction process, as discussed below. 
The user is once again required to identify the markers in the first frame. However, he will no longer be associating 3-D 
markers to joints, but directly to 3—D markers located on the body model as computed during the calibration phase. 
3.21 Skeleton Based Tracking In order to improve the results of stereo matching, we use the skeleton for applying a 
visibility and occlusion test to each pair of 2-D markers used to construct a 3—D marker, thus verifying the validity of the 
reconstruction. 
Visibility Check A marker is expected to be visible in a given view if it is seen more or less face on as opposed to 
edge on, that is if the surface normal at the marker's location and the line of sight form an acute angle. Suppose that 
we have reconstructed a certain 3-D marker using the 2-D pair (marker il, view j1) and (marker i2, view j2); we check 
that these two markers il and i2 are indeed visible in views j1 and j2 respectively. Still assuming that displacement is 
minimal from one frame to the next, we use the skeleton's posture in the previous frame and calculate the normal at 
the 3-D marker’s location with respect to its underlying body part segment. We draw the line joining the 3-D marker 
coordinates to the position in space of the camera and if the angle between the normal and the line is acute, then the 
marker is visible. If this test shows that we have used the wrong 2-D coordinates for reconstruction, we must select other 
candidate 2—D coordinates: As discussed in Section 3.1.1, each 3-D marker is associated to two sets of 2-D coordinates 
  
International Archives of Photogrammetry and Remote Sensing. Vol. XXXIII, Part B5. Amsterdam 2000. 257