S-PRINT: Development and Validation of a Tool to Predict, Evaluate, and Mitigate Excessive Workload Effects

The objective of this research was to develop tools and empirically-based guidelines that support human performance researchers, mission planners, automation designers, and astronauts in long-duration missions. Specifically, the products from this research will help users to (a) anticipate and avoid potential problems related to unexpected workload transitions by identifying the empirically established effects of operator fatigue, automation system design, and task factors on overload performance with particular emphasis on the fatigued operator’s response to unexpected emergencies; and (b) assure that systems can be designed in such a way as to minimize transient or longer-term impacts on performance in space exploration missions. The proposed work contributes to the Program Requirements Document (PRD) by helping to mitigate both 1) risk of errors due to poor task design, and 2) risk of performance errors due to sleep loss, circadian cycle, fatigue, and work overload, especially in instances when high workloads are imposed by off-nominal events. Alion Science and Technology, together with Dr. Christopher Wickens, Colorado State University, and Dr. Thomas Jones, proposed to develop and empirically validate the S-PRINT tool. S-PRINT is based on human-performance models that are accessed through a usable interface. S-PRINT allows users to evaluate the effects of automation system design, operator fatigue, and task factors on predicted performance in automation failure scenarios. The project consists of three main lines of work: 1) literature review and meta-analyses, 2) S-PRINT model and tool development, and 3) empirical data collection and validation studies. The literature review and meta-analyses were conducted to identify and evaluate factors that affect astronaut performance on long-duration space missions. In our literature review effort, we identified three primary areas of research: 1) fatigue and underload effects on performance, 2) human-automation interaction, including factors such as automation reliability and operator complacency, and 3) overload, multitasking, and operator strategies for performing tasks in these conditions. These three areas were researched in parallel to provide a qualitative understanding of the issues (goal of the literature review), and to provide empirically-based data to inform human performance model development (goal of the meta-analyses). The review provided sufficient data to develop analytic models for predicting the effects of sleep disruption fatigue on complex task performance, and for developing a preliminary model of task selection in overload conditions. It also revealed a need for targeted empirical research in the areas of human-automation interaction and in task selection in overload. The S-PRINT model and tool development area included four main subtasks: 1) S-PRINT tool development, 2) human performance model development, 3) implementation of analytic models and performance shaping factors developed from the meta-analyses and targeted experimentation, and 4) beta test evaluation and tool improvements. These were all completed in the project. The S-PRINT tool was developed and the models were implemented and tested. S-PRINT is included within the Improved Performance Research Integration Tool (IMPRINT), a tool that Alion has developed and maintains for the Army Research Laboratory. IMPRINT allows users to build task network models to predict human performance in complex scenarios. S-PRINT allows users to develop and evaluate scenarios using a particular model of operator performance. S-PRINT, as delivered to NASA in March, 2015, includes two library models, and it also provides the capability for users at NASA to build their own custom models using IMPRINT. S-PRINT provides an easy-to-use interface that allows users to create, run, and compare scenarios using already-existing library models. By changing input parameters regarding astronaut fatigue, automation system design, and task characteristics, S-PRINT users can create literally thousands of scenarios. The output from these scenarios can be compared to identify sleep mitigations, automation design changes, allocation of individuals to tasks, or task factor changes that can be adjusted to provide better performance. In addition, S-PRINT was evaluated at a beta test performed with potential system users at NASA. We identified two scenarios for the basis of the library models. The primary deliverable in S-PRINT was a long-duration mission scenario that would impose significant mental workload on an astronaut. It is of an astronaut working with a remotely-manipulated robotic arm and monitoring an environmental process control system. A fault occurs in the process control system, and rapidly becomes a high-workload off-nominal event. We developed the model of this scenario using data from NASA trainers, astronauts, and from our robotics and process control simulations. Another model, involving an operator using one of three different types of fire detection systems, was developed for testing the S-PRINT human-automation interaction capabilities and for the beta test. The third task in the tool and model development area – implementing the analytic models and performance shaping factors – has also been completed. The fatigue meta-analysis provided algorithms that specify performance degradations based on sleep deprivation (hours of continual wakefulness), restricted sleep, circadian cycle effects, and sleep inertia. These algorithms, considerably expanding on existing algorithms (e.g., SAFTE) have been added to IMPRINT. The human-automation interaction (HAI) literature review provided a robust HAI framework and relative importance measures of different factors such as automation reliability and automation failure salience. However, to parameterize the HAI model further, we conducted targeted research. The data collected from the research allowed us to develop a performance shaping factor that applies a benefit to performance when the scenario includes 1) automation that is implemented at a high degree (where most of the functions are allocated to the automation rather than the operator) and 2) is highly reliable, if the automation is functioning normally. This performance shaping factor applies a penalty (e.g., the time to perform tasks is longer, or the accuracy associated with task completion is degraded) in cases where highly automated, highly reliable systems fail. It also applies a penalty in automation failure situations when a salient failure indication is not provided. From the meta-analysis of task overload and multitasking that might apply to an unexpected emergency management situation, we developed a model of operator task selection and task shedding in overload. This is the Strategic Task Overload Management (STOM) model. The factors of task difficulty, salience (the presence of a reminder), priority, and engagement all affect the probability that an operator will select a given task and, by extension, neglect others. This model has also been implemented within S-PRINT. The data gathering and validation studies conducted in this effort included a set of ground-based human-in-the-loop (HITL) studies performed at Colorado State University (CSU), specifically designed to provide data for model development and validation. Experiments were conducted to investigate operator performance in working with automation, and in multitasking conditions. These experiments provided data regarding the effects of automation design on operator performance, and the interaction of automation design with fatigue and the interaction of multitasking with fatigue. Three experiments examined the effects of task factors on operator multi-task performance. In particular, within the long duration mission scenario of multi-tasking between robotic arm control and environmental control, the HITL experiment provided data used to validate the STOM model predictions. Over 95% of the variance in actual task switching behavior within this pair of complex, competing tasks was accounted for within the model. The experimental studies also provided data (e.g., times to complete tasks, probability of failure on a given task, performance distributions on the tasks) that was used to populate the human performance model of an astronaut controlling a robotic arm while also monitoring the environmental systems. The S-PRINT research project was successfully completed. We developed and tested a model-based tool that includes analytic models, performance shaping factors, and task network models of operator performance in complex human-automation interaction scenarios with workload transitions. The models have been developed and validated using empirical data.