Challenges and Successes of Implementing NOAA/NESDIS/NCEI's Ingest of Archival Data in a Cloud Environment

Since its formation in 2015, the National Centers for Environmental Information (NCEI) has used disparate, legacy systems spread across several IT networks of the National Environmental Satellite, Data, and Information Service (NESDIS) to fulfill its data-stewardship functions. As part of its modernization and consolidation of these functions, NCEI implemented Common Ingest as the functional component that ingests approximately 200 data streams every month into its enterprise archival information system. In parallel, NESDIS completed the Secure Ingest Gateway Project (SIGP), a pilot project to establish standard-enterprise secure methods for NESDIS and the rest of the National Oceanic and Atmospheric Administration (NOAA) to receive data in a cloud environment from their external partners. SIGP is now transitioning to operations as the Operational Secure Ingest Service (OSIS), which will be the on-ramp to NCEI's "Common Ingest" functionality when it too moves to the cloud. In addition, this ingest function will populate and use a cloud-based metadata catalog, which will be the beating heart of the NESDIS and NCEI information systems in the cloud environment. The vision is to scale their ingest of environmental data to keep pace with its ever increasing volume, veracity, variety, and velocity. In this presentation to the ocean data community, the authors describe NESDIS and NCEI's challenges and successes with the implementation of the ingest function of their archival information system in a cloud environment.