Evidence-based Metrics Toolkit for Measuring Safety and Efficiency in Human-Automation Systems--NNX13AO51G

APRIL 2016 NOTE: Principal Investigator moved to Rice University in mid-2015. Project continues at Rice with the same title (Evidence-based Metrics Toolkit for Measuring Safety and Efficiency in Human-Automation /Systems--NNX15AR28G), period of performance 8/1/2015-7/31/2017, grant #NNX15AR28G. See that project for continued reports. JULY 2015 PROGRESS REPORT The second year of the project involved extending our qualitative literature review for identifying factors and metrics utilized for assessing human-automation system performance and safety. Additionally, a collaborative workshop was hosted by the University of Central Florida (UCF) research team to provide an opportunity for experts to share their insights regarding automation design and measurement, as well as to obtain feedback regarding our methods and findings from the project. Current efforts involve preparing to conduct subject matter expert interviews to obtain recommendations from potential end users to optimize the effectiveness and usability of the web-based metrics toolkit. Studies will be the focus of future efforts after the transition of team members to Rice University during the end of Year 2. AUGUST 2014 PROGRESS REPORT There is a heavy reliance of automation systems to complete tasks and sustain habitability in the spaceflight context. However, past research has shown that certain risks associated with human-automation interaction can have an impact on performance and safety. In order to design and integrate automation systems effectively, it is important for designers, evaluators, and operators to monitor and assess certain factors associated with human-automation performance and safety. In order for effective monitoring, access to valid metrics and resources to properly assess various conditions of the human-automation system can benefit in the design and application process for a variety of contexts, including spaceflight. The purpose of the research project is to conduct efforts in developing a metrics toolkit that can be practically utilized for evaluating design and applications of human-automation systems. In this case, a metrics toolkit is defined as a set of resources and metrics that can be used for determining what conditions in the human-automation system can impact performance and safety. Our project involves several phases in order to meet this objective: (1) theoretical framework development, (2) systematic literature review of metrics used to assess human-automation systems, (3) experimental validation of selected metrics. Year one efforts have been dedicated to the development of a theoretical framework to delineate how various factors can impact human-automation system performance, efficiency, and safety. The framework was guided by an initial literature review about critical factors that have been of interest regarding human-automation interaction. Based on our initial review, we have categorized and identified factors into three separate facets: (1) inputs, (2) processes, and (3) outcomes. Inputs involve the traits of the operator (e.g., personality, expertise), the automation system (e.g., reliability, level of automation), and context (e.g., environmental factors, nature of the task, task complexity). Inputs of each of these elements are not impacted by their interaction with one another, but can play a role in how the operators interact with the task environment and automation system during taskwork. Processes are variables that emerge from the interaction between the operator, automation agent, and the environment, and are considered the attitudinal, behavioral, and cognitive factors experienced by the user (e.g., trust, workload, situation awareness) that can impact the performance of the entire system. The outcomes are the dependent variables of interest, such as performance, safety, and efficiency. Based on our framework, it is theorized that the inputs and processes play a role in the outcomes, and that metrics that capture these critical factors can aid in the assessment of performance and safety in system integration. Based on our theoretical framework, the research team has also conducted a systematic literature in order to identify metrics utilized in assessing human-automation interaction, and extract key information that can be used to develop guidelines for measuring these systems. While the literature review is still currently ongoing, we have discovered a wide variety of types metrics used to assess human-automation system performance. Insights based on our literature review will play a role in the development of a metrics toolkit for further phases of the research project. Additionally, the research team has initiated efforts in hosting a collaborative workshop with measurement and automation experts to (1) gain feedback and insights into our initial work, and (2) provide recommendations and guidelines for applying metrics. Expert outreach is also involving the development of an expert panel that will review our work and provide guidance for metrics selection and validation testing during the future phases of our research project. After the work conducted in year one, efforts in year two will involve initializing the experimental study to investigate the validity and properties of selected metrics for the proposed toolkit. It is expected that this work will help in identifying which metrics are accurate in their assessment of identifying changes in factors associated with human-automation system performance and safety based on experimental manipulations set to impact critical factors outlined in our theoretical framework.