NOAA’s Big Data Partnership at the National Centers for Environmental Information

Abstract:

In April of 2015, the U.S. Department of Commerce announced NOAA’s Big Data Partnership (BDP) with Amazon Web Services, Google Cloud Platform, IBM, Microsoft Corp., and the Open Cloud Consortium through Cooperative Research and Development Agreements. Recent progress on the activities with these Partners at the National Centers for Environmental Information (NCEI) will be presented. These activities include the transfer of over 350 TB of NOAA’s archived data from NCEI’s tape-based archive system to BDP cloud providers; new opportunities for data mining and investigation; application of NOAA’s data maturity and stewardship concepts to the BDP; and integration of both archived and near-realtime data streams into a synchronized, distributed data system. Both lessons learned and future opportunities for the environmental data community will be presented.