NASA Information And Data System for Earth Science Data Fusion and Analytics (NAIADS)

One of the key elements of advancing our understanding of Earth system via remote sensing is integration of diverse measurements into the observing system. As remote measurements capture larger amounts of data and higher quality data, the demand for advanced data applications and high-performance information processing systems becomes a greater challenge. These challenges are outlined in the OSTP Guidelines for Civil Space Observations (2013), recognized in the NASA Strategic Space Technology Investment Plan (2013), and addressed in the NASA Strategic Objective 2.2 (2014) and its implementation by "...developing new technologies and predictive capabilities, and demonstrating innovative and practical uses of the program's data and results for societal benefit". In response to these challenges, we propose to develop the NASA Information And Data System (NAIADS) - a prototype framework for the next generation Earth Science multi-sensor data fusion, processing, and analytics. The concept of maximizing information content by combining multi-sensor data and enabling advanced science algorithms, was successfully used by several past and on-going projects: CERES experiment (Earth radiation budget), fusion of the CERES, MODIS and MISR observations (for estimating instantaneous shortwave flux uncertainties, and multi-instrument calibration comparison), fusion of MODIS and PARASOL observations to enhance cloud and aerosol retrievals, fusion of data from CALIPSO, CloudSat, CERES, and MODIS (A-Train) for comprehensive aerosol and cloud information, as well as CERES-derived fluxes. Advanced science algorithms allowed to reduce uncertainty in weather and climate parameters. The future satellite constellations and NASA missions: RBI, TEMPO, CLARREO, ACE, and GEO-CAPE will require tools for efficient data fusion and massive process scaling. Objective of proposed effort is to develop a prototype of a conceptually new middleware framework to modernize and significantly improve efficiency of the Earth Science data fusion, big data processing and analytics. The key components of the NAIADS include: Service Oriented Architecture (SOA) framework, into-memory Data Staging, multi-sensor coincident Data Predictor, multi-sensor data-Event Builder, complete data-Event streaming (a workflow with minimized IO), on-line data processing control and analytics services. The NAIADS project will leverage existing CLARA SOA framework, developed in Jefferson Lab, and integrated with the ZeroMQ messaging library. The services will be prototyped and incorporated into a system. Data merging and follow-on aerosol retrieval from combined TEMPO, GOES-R ABI simulated, and real VIIRS observations will be used for NAIADS demonstration and performance tests in Compute Cloud and Cluster environments. The proposed NASA Information And Data System (NAIADS) provides a novel approach to signi?cantly improve e?ciency in the Earth Science multi-sensor big data processing and analysis by deploying conceptually new workflow and state of-the-art software technologies. Within the 2-year project, beginning May 2015, the NAIADS technology readiness will increase from TRL 3 to TRL 6. More »