Providing an Analysis Environment with Access to High-Volume Simulation and Observational Data for Climate Science

Abstract:

Environmental science requires the fusion of ever growing volumes of data from multiple simulation and observation platforms. In the UK, the Centre for Environmental Analysis (CEDA) provides infrastructure to support the analysis of such data. CEDA delivers both a curated archive, and an environment to exploit that data alongside other datasets. Over 3 petabytes (PB) of data are now available in the archive and this will rise considerably in the next 12 months as key datasets are acquired and made available via the JASMIN super data cluster. JASMIN incorporates over 17 PB of disk, co-located with tape and computing facilities for data analysis via batch, hosted and cloud computing

The first of the European Sentinel series of satellites, Sentinel-1A was launched in April 2014, followed by Sentinel-2A in July 2015. Synthetic Aperture Radar data from Sentinel-1A data is already flowing, with around one terabyte per day being archived at CEDA. Recent data are stored on-line for direct access to users; older data will be moved to near-line tape, reinstating for users on demand. It is expected that most UK science users will access, process and analyse the data in the JASMIN-CEMS hosted environment avoiding the need to download and store data on their local machines. Sentinel 2 and Sentinel 3 data will follow soon, with landcover mapping expected to be an important application for Sentinel 2 data.

The same approach will be adopted for CMIP6 data, where we expect to both host the largest possible CMIP cache and provide a specific resource for one of the constituent MIPS: HIRESMIP. We expect to provide the UK climate community (and related European collaborators, such as the PRIMAVERA consortium) a 2 PB disk cache alongside a complete copy of HIRESMIP (10-50 PB) on tape.

This paper describes these challenging use cases and presents recent infrastructure developments, including how the facility for Climate and Environmental Monitoring from Space, CEMS, exploits JASMIN.