Open-Source Pipeline for Large-Scale Data Processing, Analysis and Collaboration

NASA's observational products generate petabytes of scientific data, which are highly underutilized due to computational requirements; disjoint data access protocols; and task-specific, non‑reusable code development. Our overall objective is to accelerate NASA science through development of an open-source, Python-based Pipeline for Observational Data Processing, Analysis, and Collaboration (PODPAC). The PODPAC software framework will enable widespread exploitation of earth science data by enabling multi-scale and multi-windowed access, exploration, and integration of available earth science datasets to support both analysis and analytics; automatically accounting for geospatial data formats, projections, and resolutions; simplifying implementation and parallelization of geospatial data processing routines; unifying sharing of data and algorithms; and enabling seamless transition from local development to cloud processing. To achieve these objectives, we will work with NASA Science Team members involved with the SMAP (Soil Moisture Active Passive) and EOSDIS (Earth Observing System Data and Information System) programs and the wider scientific community to define technical specifications for the software, and plan a list of prioritized enhancements for each quarterly release cycle; further develop the core Python library based on user feedback and using agile development practices; develop integrations with cloud computing resources, specifically targeting Amazon Web Services; develop and demonstrate best-available remotely sensed soil moisture, high-resolution downscaled soil moisture, and flood/drought monitoring applications to promote infusion into NASA programs; and engage with scientific community through conferences, meetings, webcasts, and by providing support in order to promote adoption of the software. More »