Optical Computer Recognition of Stress, Affect and Fatigue during Performance in Spaceflight

Astronauts must maintain high-level performance while experiencing demanding workload and work schedules, extreme environmental risks, and psychosocial stressors in space (e.g., isolation, confinement). Stress, negative emotions, and fatigue can jeopardize their cognitive performance and neurobehavioral status. The proposed research is developing and validating an objective, unobtrusive, computational model-based tracker of the human face that reliably identifies when astronauts are experiencing stress, emotion, and fatigue at levels that compromise performance in space. This optical computer recognition (OCR) system will provide feedback to them for autonomous selection of countermeasures for stress, depression, and fatigue. The project is being accomplished through collaborative efforts of Dr. David Dinges (Unit for Experimental Psychiatry) at the Perelman School of Medicine at the University of Pennsylvania, and Dr. Dimitris Metaxas (Computational Biomedicine Imaging and Modeling Center) at Rutgers University. The project has four specific aims: (1) Create an OCR system capable of monitoring facial displays of specific emotions (i.e., angry, happy, and sad). (2) Improve our current OCR system's ability to detect facial expressions of high versus low performance-induced stress. (3) Develop OCR algorithms to identify fatigue due to sleep loss based on slow eyelid closures (PERCLOS). (4) Test the technical feasibility of data acquisition and reliability of the advanced OCR system in spaceflight analogs that contain neurobehavioral stressors relevant to spaceflight. The project has primary relevance to strategic goals of the National Space Biomedical Research Institute (NSBRI) Neurobehavioral and Psychosocial Factors (NBPF) Team.

The two major laboratory experiments on the accuracy of OCR using a single camera were completed in the final funding period. One study involved measuring how well the single-camera OCR algorithm accurately tracked emotional expressions in N= 31 healthy subjects who underwent emotional induction techniques. Preliminary analyses of the overall extent to which the initial 1-camera OCR algorithm could identify specific emotional expressions in many individuals with limited training revealed that the algorithm often failed to discriminate among emotions. That is, the algorithm had modest sensitivity and low specificity (i.e., it selected negative emotions too often and failed to discriminate among them). Although the facial expression models being used by the tracker were appropriate, it became apparent that a great deal of OCR inaccuracy was due to problems in identifying facial expressions when the face was partially out of view, which occurs frequently as people move their heads in all dimensional planes as they move about, work, etc. Thus, although we trained the OCR facial expression models with frontal images of facial expressions of emotion, the videos of subjects experiencing emotions would many times show subjects in non-frontal poses. The OCR algorithm model would then fail to correctly recognize the facial expression. To correct for this problem, the Metaxas Lab developed a sufficiently approximate warping transformation to warp the tracked face to a frontal pose (which is what the OCR algorithm expects to evaluate), as well as enhancing the algorithm with other analytic techniques that improve single-camera face tracking and extrapolation of facial expressions when the face is moving and/or partially out of view. The second validation experiment was conducted on a separate group of N=33 healthy adults randomized to either sleep deprivation (N=18) or no sleep deprivation (N=15). The total number of subjects to be studied was decreased in order to focus on improved computational detection of emotion and fatigue. The goal of this experiment was to determine the extent to which the model-based OCR algorithm validly detected and tracked ocular changes in slow eyelid closures (PERCLOS) when sleep deprived and when not sleep deprived, and the extent to which the PERCLOS measure reliably tracked lapses of attention during psychomotor vigilance test (PVT) performance.

Experiment 2 was our first attempt to track PERCLOS with a novel 1-camera OCR algorithm. The results were very promising. Subjects completed a 20-min PVT every 2 h while awake. Images of the face were recorded during each performance test. PERCLOS was manually scored frame-by-frame by 4 human scorers blinded to sleep condition and time of testing, and it was scored by the OCR algorithm (programmers blind to these factors). Coherence was calculated as the extent to which PVT lapses of performance were tracked by OCR-scored PERCLOS while subjects were and were not sleep deprived. Although OCR PERCLOS did not predict the occurrence of every PVT performance lapse, it was remarkably accurate for the vast majority of them, showing excellent coherence across a 36-h period of sleep deprivation, and a marked increase after sleep deprivation relative to before it, but no change in subjects who were resistant to the effects of sleep deprivation on PVT performance, or subjects who were not sleep deprived. Thus, the sensitivity of the current 1-camera OCR algorithm for detecting PERCLOS as an index of fatigue-related performance failures was 73.3% and the specificity was 89.1%. This is excellent sensitivity-specificity. These data confirm that the OCR PERCLOS detector rarely yielded false positives, and that it was acceptably high in sensitivity to predicting performance lapses. OCR PERCLOS will never reach 100% sensitivity when validated against PVT lapses because some performance lapses can occur when the eyelids are open. The findings suggest a single-camera OCR algorithm could detect the presence of performance-impairing fatigue from sleep loss in spaceflight.

More »