Collaborative Augmented Reality with Hands‐Free Gesture Control for Remote Astronaut Training and Mentoring

The per-minute cost of operations for crew aboard the International Space Station (ISS) is orders of magnitude higher than that for ground crew. To illustrate, a recent letter from NASA Administrator Charles Bolden to Congress [1] disclosed that transportation of astronauts to and from the ISS would cost $490 million. Since a tour typically consists of six astronauts with a length of stay of about six months, this amounts to nearly $20,000 per hour aboard the ISS per astronaut just for transportation to and from the ISS. Thus, it should be clear that technologies and methods to increase astronaut efficiency are important. Currently, many tasks, such as scheduling, are handled by ground teams to increase the time astronauts can spend conducting research. Despite this effort, crew is still required to load and unload cargo from visiting cargo vehicles. This task, though simple, is extremely time consuming. It can take many days for crew to finish unloading from a vehicle, packing the cargo transfer bags (CTBs) to return to the vehicle, loading the vehicle, and checking the results. We created a prototype augmented reality (AR) application, called StowageApp, to aid crew and ground planners in executing this process. StowageApp aims to improve a crew member’s ability to understand instructions provided by ground planners, to collaborate with other crew members in executing cargo operations, and ultimately to complete cargo operations faster and with greater accuracy. StowageApp runs on a Microsoft HoloLens head-worn AR computer that blends a user’s view of the real world with virtual content. Figures 1 through 5 show a user following StowageApp instructions as he completes a stowage procedure in the full-scale ISS mockup at NASA Johnson Space Center (JSC). (These images were taken by a second user running StowageApp on a separate Microsoft HoloLens.) Figure 1. User wearing Microsoft HoloLens in JSC full-scale ISS mockup consults StowageApp as he retrieves items during a stowage procedure. (Image is actual screenshot from StowageApp.) Figure 2. User wearing Microsoft HoloLens consults StowageApp as he transits in full-scale ISS mockup at JSC. (Image is actual screenshot from StowageApp.) Figure 3. StowageApp navigation aid guides user wearing Microsoft HoloLens toward PMM Leonardo module in full-scale ISS mockup at JSC. (Image is actual screenshot from StowageApp.) Figure 4. User reviews StowageApp instructions as he enters PMM Leonardo module in full-scale ISS mockup at JSC. (Image is actual screenshot from StowageApp.) Figure 5. User follows StowageApp instructions as he stows CTB inside compartment in PMM Leonardo module in full-scale ISS mockup at JSC. (Image is actual screenshot from StowageApp.) StowageApp impacts processes including crew-side execution of instructions given by ground planners, authoring of these instructions by ground planners, and communication between ground planners and onboard crew regarding the results of or problems with the instructions. We developed a series of mock stowage tasks to be executed in the full-scale ISS mockup at JSC. We conducted two user studies (N=25, N=9) comparing two different iterations of StowageApp with an electronic procedure document displayed on a tablet device. We measured completion time of the stowage tasks, incidence of errors, and responses to a NASA TLX workload survey. We are still in the process of analyzing the data. This work represents one of the first attempts at developing a prototype application deployed to a head-worn display for use in tasks requiring crew to travel between multiple ISS modules. As a result of this work, we also developed a new method for expressing cargo operation procedure instructions suitable for AR environments. This new method will help inform the design of new authoring tools that will enable NASA Inventory and Stowage Officers to efficiently write procedures that could be displayed by many different types of devices, including AR headsets. Figure 6. User views instructions in virtual model of the ISS in VR version of StowageApp running on a Windows Immersive Mixed Reality head-worn display. I also developed a Virtual Reality (VR) version of StowageApp that immerses the user in a virtual model of the ISS while providing the same user interface as our AR prototype. The VR version of StowageApp runs on Windows Immersive Mixed Reality head-worn display. Figure 6 shows a screenshot of the VR version of StowageApp. I gave a live demo of the VR version of StowageApp at SIGGRAPH 2018 [2] and I delivered a technical presentation on the demonstration. An abstract describing our work was also accepted for technical paper publication and oral presentation at the 69th International Astronautical Congress. In addition, I was awarded a 2018 Future Space Leaders Foundation Fellowship in part based on the merit of my work performed during this fellowship. References [1] C. Bolden. Letter Notifying Congress about Space Station Contract Modification with Russia. Washington, DC, 2015. [2] SIGGRAPH 2018. Virtual, Augmented and Mixed Reality. https://s2018.siggraph.org/conference/conference-overview/virtual-augmented-mixed-reality/ (Follow link to: “Immersive Pavilion - Village” and search for: “Furuya”.)