Evidence-based Metrics Toolkit for Measuring Safety and Efficiency in Human-Automation Systems--NNX15AR28G

Phase 1: The purpose of this phase was to synthesize and translate findings from the extant literature relevant to human automation/robotic integration in order to determine specific metrics that characterize the safety and efficiency of a human-automation interaction. The final outcome of Phase 1 was the development of the content for human automation interaction metrics (HAIM) toolkit. This content was then packaged inside an automated online tool that went a step beyond our original aim for this project by providing designers with a tool that not only contained content, but also provided information, tips, and advice for how to apply this content to their own measurement situations.

Phase 2: The original goal for this second phase was to complete in-depth preparation for scientifically sound experiments. This detailed preparation time in phase 2 was meant to ensure adequate time and methodology for meaningful outcomes for Phase 3. The intended outcome of Phase 2 was be the final development of the experimental testbed and experimental protocol. However, as more was learned throughout phase 1 about Human-Automation measurement and as the toolkit development was progressing, it was determined that Phase 2 should be adjusted to better meet the goals of the overall project. Therefore, Phase 2 was adjusted to be preparation for Subject Matter Expert (SME) interviews and iterative usability evaluations of the HAIM Toolkit instead of validation experiments to test the theoretical framework. This proved to be a very fruitful decision as the SME interviews provided valuable feedback and data that was able to be incorporated into the final design of the toolkit.

Phase 3: Preparation for and execution of, experiments. This final phase of the project was originally planned to include the design and execution of a set of multi-level empirical studies aimed at validating the theoretical framework and metrics toolkit. The focus of the validation studies was to be on testing different aspects of human automation interaction (e.g., levels of automation, task complexity, and the number and configuration of system operators). The intended outcome of the proposed effort was to provide NASA a set of evidence-based, empirically-validated guidelines and a measurement toolkit for mitigating the risk of inadequate design of human and automation/robotic integration as it pertains to the development of safety and efficiency metrics for human automation systems. Phase 3 was adjusted to focus on validation of the automated HAIM Toolkit through execution of usability evaluations by potential end users as this was deemed more valuable than the originally planned validation studies. The usability studies that were conducted were designed to evaluate both usability of the automated HAIM Toolkit as well as the value of the information that the toolkit provided. Multiple rounds of testing took place along with subsequent periods or toolkit redesign to incorporate feedback gleaned during the usability testing. The final outcome of this effort along with the research conducted in Phase 1 is a set of principles, guidelines, and tips for measurement that have been submitted to NASA in white papers and are in the process of being transformed into multiple manuscripts currently under preparation. Additionally, the final outcome of this phase is the development of the HAIM Toolkit, which is complete and will be delivered to NASA with the final report.

Manuscripts in various stages:

Hughes, A.M., Marlow, S.L., Hancock, G.M., Oglesby, J.M., Stowers, K. & Salas, E. (Under Revision). Physiological assessment of workload: A meta-analysis. Human Factors

Stowers, K., Iwig, C., Salas, E. (In preparation). Measurement in the Second Machine Age: Considerations for Future Human-Machine Systems. To be submitted to Human Factors.

Stowers, K., Sonesh, S., Iwig, C., Salas, E. (In preparation). Technology and Teams: The Impact of Technological Evolution on Team Performance. To be submitted to Group and Organization Management.

Stowers, K., Sonesh, S.C., Iwig, C., Salas, E. (In preparation.) Capturing Human Machine System Safety in Spaceflight. To be submitted to Aerospace Medicine and Human Performance.