Evidence-based Metrics Toolkit for Measuring Safety and Efficiency in Human-Automation Systems--NNX15AR28G

N/A More »

The Rice University research team continues to work to provide a tool guiding the evaluation of human-automation systems. Research and design efforts throughout the life-cycle of this project have focused on developing best practices for human-automation system measurement and incorporating these best practices into a web-based decision aid tool for providing system designers with practical guidelines and recommendations for how to evaluate their systems. Recent progress has focused on development of this online tool, called HASMAT: Human Automation System Measurement and Assessment Toolkit. Development of HASMAT has followed an iterative design and development process including multiple phases of prototype design, testing, and revision. Major accomplishments during this year have included (1) transitioning the toolkit decision tree architecture, which was developed in the previous year, into the design of the first HASMAT prototype, (2) testing the prototype using validated usability evaluation techniques, (3) developing a second HASMAT prototype based on results from initial evaluations, (4) testing of the second prototype in a structured laboratory based usability study utilizing NASA SMEs (subject matter experts), (5) development of a vast database of measurement information, guidelines, and tips to incorporate into the toolkit (in-progress), and (6) development of the final, fully automated, HASMAT prototype (in-progress). Research efforts this year began with the refinement of the toolkit decision tree architecture, which was developed during the previous year. Once refinement of this architecture was complete, it was used as a basis of the first toolkit prototype. This initial prototype was created using Axure prototyping software. The main goal for this initial prototype was to incorporate all decision tree questions, determine the most appropriate visual delivery method for the questions, and develop and ideal organization and flow of the questions within the decision tree. Once this initial prototype was complete, initial evaluation techniques were conducted by the Rice research team to evaluate all aspects of the initial prototype's design, flow, and support features. The results of these evaluations informed revisions to the toolkit. The second HASMAT prototype incorporated all recommendations for improvement that resulted from evaluations of the initial prototype. The purpose of this prototype was to provide a semi-functional prototype that could be tested with potential end-users. This prototype was still lacking automated measure selection features, but the decision tree architecture had been refined in this prototype and other features were added. Therefore, usability testing of this prototype with NASA SMEs focused on obtaining input as to how the decision tree architecture could be improved as well as the content that had already been incorporated into this version of HASMAT. Results from the structured usability studies aimed at evaluating the second HASMAT prototype were very enlightening and led to an extensive overhaul of the entire toolkit design. Additionally, this extensive update of HASMAT, which is currently in progress, is also aimed at adding in full functionality including automation of the dynamic measure selection feature that allows the toolkit to provide customized measurement recommendations based on system and contextual information provided by the toolkit’s users during their completion of the decision tree questions. Once complete, this final HASMAT prototype will be tested in a final round of usability testing using NASA SMEs, followed by completion of necessary revisions, and resulting in delivery of the final HASMAT prototype to NASA. Overall, major progress on the development of the final HASMAT prototype has been made in year 4 and will continue to be made through the end of July. More »