Space Human Factors and Habitability MIDAS-FAST: Development and Validation of a Tool to Support Function Allocation

In the third year of the contract, a number of results were achieved. Briefly summarized, the major accomplishments were the completion of the second and third experimental studies, and model parameterization and validation efforts. This resulted in a number of minor modifications to the human performance model, to simulate more accurately actual operator performance during the robotics tasks. Final development efforts were implemented prior to delivering the software, and a user manual was written. In addition, the project was presented and the MIDAS-FAST tool was shown during a live demonstration as part of an Office of Management and Budget (OMB) deliverable (August 14, 2012).

A primary goal of year 3 was to verify and validate our model of the robotic arm operator, to be employed in the function allocation tool, and to collect data that would further validate the Function Allocation Support Tool tradeoff model. To accomplish these purposes, data from the Human in the loop (HITL) Experiments 2 and 3 were analyzed, and both models developed and refined. The nature of Experiment 2, which was the most critical in both validations, will first be described in some detail.

HITL simulation: In Experiment 2, 36 participants were first trained extensively to manually operate a simulated version of the robotic arm manipulation task, in which a payload was first raised, then moved across a horizontal “table top”, and then lowered to a target destination. While maneuvering, participants needed to avoid proximity with hazards (wall and the table top) and joint states that would freeze the arm motion. In carrying out the task, they were assisted by three forms of automation. (1) All participants were alerted when proximity to a hazard was violated. (2) All participants were advised, on half the trials, as to the appropriate selection of two of the 4 possible camera views that would provide them with optimal viewing perspective. Finally, (3) participants were divided into three groups receiving different levels of trajectory control automation: none, presence of a 3D guidance path for the correct trajectory (autoguidance), and full autopilot control (autocontrol). In terms of the FAST tradeoff model, the two automation conditions varied in the stage of automation supported (early versus late).

Automation of all types functioned correctly through most of the trials. However on the final trial (last few minutes of the 6 hour experiment), both automation systems “failed.” Either the line directed the arm too close to a hazard (autoguidance), or the autopilot actually moved the arm into this close proximity with the hazard (autocontrol). In both instances, the collision warning system also failed.

Consistent with our FAST tradeoff model, increasing degrees of automation (from none, to autoguidance, to autocontrol) produced progressively better routine performance and lower workload. However, also consistent with the tradeoff model, late stage automation produced significantly worse performance in automation failure management on this final trial, and was associated with a significantly different visual scan path. The camera advisory automation was not failed. The advice of this automation was complied with, and did improve participant’s view of the workspace.

Validation of robotic arm operator model. We developed a computational model of the robotic arm operator on MIDAS. Because it was MIDAS controlling Boris, we called the model MORIS. MORIS consisted of three primary sub models: (1) A decision model, based on utility theory, chose the best camera views and decided which 4D trajectories to take (XYZ and speed) to reach the ultimate goal and avoid hazards. (2) A spatial cognition model called FORT (Frame of reference transformation) continuously calculated the challenges to spatial cognition caused by different levels of motion ambiguity (portrayed by camera views), control incompatibility (created by misalignment between control motion and perceived display motion), and by visibility challenges within the workplace. (3) a visual scanning model, across the workplace, known as SEEV, which controls simulated eye movement particularly on the basis of the bandwidth of information source changes, their value to the task, and their location in the workplace. MORIS then generated outputs of performance time, trajectory error, workload, camera selection, and scan-based situation awareness. Predictions were different across the three degrees of trajectory automation (none, autoguidance, autocontrol). MORIS model parameters were adjusted so that close fits were obtained between MORIS predictions and the empirical data from the HITL.

While the proceeding was essentially “parameterization” of the model, two efforts were made to make true validation: that is, predictions of the model in which the model parameters were not adjusted to maximize the fit with what was predicted. First, we predicted reasonably well, performance of participants in the one condition of experiment 3 that corresponded to one of our conditions in experiment 2. Second, and more significantly, we used MORIS to predict performance of participants in all three conditions in responding to the unexpected automation failures (see above). We did this by modeling, with MORIS, a reduction in the scanning of critical displays, thereby using SEEV to produce an automation complacency prediction. It was this prediction that was validated, with a high correlation between predicted and obtained failure management performance, with the actual performance of participants in the three conditions. In fact, as predicted by the FAST (Function Allocation Simulation Tool) tradeoff model, our complacency measure precisely predicted the poorer performance in the auto-control, compared to the auto-guidance automation.

The third experimental study evaluated context-sensitive function allocation. Specifically, four conditions were used to compare operator performance: (1) adaptable automation, where changes in the allocation of functions are initiated by the user, (2) adaptive automation, where the automation triggers changes based on operator or system performance, (3) a hybrid approach, where the system and operator collaborate on selecting and activating automation levels, and (4) a fixed automation approach, in which moderate degrees of automation were consistently applied.

Twelve participants (university students with 6-9 hours of previous robotics experience) worked with the BORIS simulation. They were instructed to perform their tasks (executing 3-segment trajectories) in a safe, accurate, and efficient manner. All subjects performed tasks in all 4 automation conditions (within-subjects design). Performance measures included trajectory completion time, trajectory deviation, subjective workload, and subjective preferences regarding automation type. Results indicate that trajectory deviations were significantly smaller in the adaptive, adaptable, and hybrid automation conditions compared with the fixed automation condition. Further, deviations in the hybrid automation condition were smaller than deviations in the adaptive and adaptable conditions. While the difference in time to complete the trajectory was not significant, the trend showed the similar results (slowest for fixed automation, fastest for hybrid). Workload was highest in the fixed condition, and lowest in both the adaptable and hybrid conditions. Finally, participants preferred the adaptable and hybrid automation types for the control it allowed them and – in the hybrid condition – the knowledge that the system was monitoring their performance.

In summary, Experiment 3 identified empirical support for the adaptable and hybrid automation schemes over fixed and adaptive schemes. The experiment also provided data (from the fixed and hybrid conditions) for validation of model predictions, described previously.

Based on the parameterization and validation efforts, the robotics operator model was updated. Further refinements in the model and simulation integration were implemented. New capabilities were added to the MIDAS-FAST tool, to allow researchers to examine a variety of potential conditions. Further, a user manual was developed and provided as part of the project deliverables.

During the third year of research, long-term hospitalization of two key personnel (different issues) resulted in a no-cost extension of the contract until April 30, 2013.

On June 5, 2013, the MIDAS-FAST tool was delivered to NASA Johnson Space Center (JSC) and a final briefing was delivered. This briefing included a summary of the project (purpose, goals, approach) with a focus on the experimental results and model parameterization and validation efforts (methods, results). The final report and user manual were also delivered to NASA.