Robotic Localization using Shadow Information in Imagery

My overall goal was to develop algorithms to effectively use shadow (occlusion) information in visual imagery as a vehicle navigational aid, be it absolute navigation with respect to static terrain (e.g. using landing imagery for rover localization), or with respect to a moving target (e.g. using visual imagery of a tumbling asteroid in an AR&D application). During my four years of NSTRF work, my research has touched on a few application areas. A significant portion of the research focused on uses of shadow information in sonar imagery for underwater robotic localization, and this underwater research was the subject of my thesis. Additionally, a portion of the research focused on the use of shadow information in visual descent imagery for lunar rover localization with respect to prior maps. Furthermore, in the first year of NSTRF work I developed an algorithm for the fusion of visual imagery and sparse pattern laser range data for tumbling target pose and shape estimation. Underwater Robotic Localization using Shadow Information in Sonar Imagery Robotic localization in the underwater environment is a challenging problem, as GPS signals do not penetrate the water column. There has been a wealth of research in methods for underwater localization, including those in the general area of dead reckoning, visual-aided navigation, and sonar-aided navigation. My research was the first to successfully demonstrate localization using sonar imagery with respect to prior topographical seafloor maps. Imaging sonars are commonly found on underwater vehicles, and high-resolution topographical maps are increasingly available. In my research, I demonstrated meter-level accuracy using sonar images collected from remotely-operated underwater vehicles (ROVs) and autonomous underwater vehicles (AUVs), in collaboration with the Monterey Bay Aquarium Research Institute (MBARI). The major conclusions of the research are as follows: • Meter-level localization accuracy can be achieved with sonar imagery with respect to prior topographical seafloor maps by using the acoustic shadow data in the imagery as the information source. • Acoustic shadow data in sonar imagery are primarily geometric in nature, and as such are predicted well from seafloor topography alone. This conclusion was not obvious from the outset of the investigation. • For ROVs, which are capable of holding a fixed position with respect to the seafloor, meter-level localization was achievable from a single sonar image (100m maximum range), provided there was a reasonable amount of shadow information in the image. • By accounting for shadow uncertainty based on the terrain map, localization was made far more robust. This research holds the potential to enable map-relative localization for vehicles that previously could not do so. Please refer to my thesis for details. Lunar Rover Localization using Shadow Information in Visual Descent Imagery Localization is a key capability for planetary rovers. Rovers carry onboard sensors (cameras, laser range finders, etc.) that measure terrain, where these measurements may be compared to prior maps (e.g. topography). Furthermore, there is an opportunity to improve the initial rover position estimate using descent imagery from the lander. In collaboration with Ara Nefian at NASA Ames IRG, we developed a new method to use shadow information from a prior planetary (lunar) digital elevation map (DEM) to localize a landing rover using descent imagery. • Position localization accuracy on the order of map resolution was obtained in a simulation study, along with sub-degree pointing accuracy. A real lunar DEM was used in the study. • By pre-computing the shadow map, given known sun position relative to the DEM for the landing window, the computational burden can be significantly decreased such that 6- DOF pose estimation is feasible. Fusion of Visual Imagery and Sparse-Pattern Laser Range Data for Tumbling Target Pose and Shape Estimation. In the first year of NSTRF work I developed an estimation framework for tumbling target pose and shape estimation through the fusion of vision and sparse-pattern range data. The method was rooted in a Rao-Blackwellized particle filter formulation. The high-level concept of this filter was to use the bearings-only vision measurements to track sparse features (e.g. SIFT, SURF) for pose estimation, with enforcement of vision-range consistency in order to recover absolute scale and a dense reconstruction (from projection of the LIDAR returns). • First development of topography-relative localization using imaging sonars, a class of sonar commonly found on operational underwater vehicles. • Demonstrated meter-level localization using imaging sonars and prior seafloor topography maps with real operational underwater vehicles (ROVs and AUVs). • Authored four peer-reviewed conference publications. • Thesis defended in the school of Aeronautics & Astronautics at Stanford University.