The following scoring procedures were developed to deal with the 
problems involved in scoring photointerpreter performance : 
Standard Response.  Photointerpreters traditionally provide & 
written report using words of their own choice. From the point of view 
of objective scoring, this constitutes & problem, since the judgment of 
the scorer must be a factor in the determination as to whether or not a 
response is correct. A structured coding system by which responses were 
elicited in & standard fashion was developed. By use of this code, & 
particular item was explicitly and uniquely identified. 
  
Scoring Key. Once the problem of obtaining responses had been 
solved, it was necessary to determine the correctness of response. For 
this purpose, a 'ground truth key' was established. This key was pro- 
vided by highly experienced photointerpreters who availed themselves of 
better photographs, references and maps. This scoring key was used in 
the determination of the accuracy of all responses. 
Information Level. Another problem dealt with the level of infor- 
mation to be scored--gross identifications or fine discriminations. For 
the present study, it was decided to obtain two scores, one for the 
grossest type of response, component only, and one for & slightly more 
refined level of detail, component and first modifier unit combination. 
  
Scoring Formula. A final problem dealt with the scoring formula. 
PI information provided is either correct or in error, but there are no 
adequate means of combining correct and error scores. Moreover, there 
is the problem of 'omits' of relevant militarily significant components 
(or component-modifier units). It was decided to run two separate 
analyses (where applicable), one for correct and one for error scores. 
Omits as such were not considered. An analysis incorporating the concept 
of omits was conducted separately in determining completeness of infor- 
mation extraction. 
  
RESULTS 
PERFORMANCE OF EXPERIENCED PI's 
Accuracy of Information Extraction. An index of the accuracy of 
the information provided by the photointerpreter was derived by deter- 
mining the ratio (in percent) of correct information produced by the 
interpreters to total information they provided. Table l presents aver- 
aged accuracy data obtained on the experienced PI's at two information 
levels, the grossest or component level and the somewhat more specific 
component-modifier level. For the component scoring method, the error 
rate was approximately 25%, for the component-modifier method, 50%. 
- LO— 
  
     
  
  
  
  
   
  
   
   
  
  
  
  
   
  
  
  
  
   
   
  
  
  
  
  
   
  
  
  
  
  
   
  
  
  
  
  
  
  
    
   
Archives 6