Building the Petascale National Environmental Research Interoperability Data Platform (NERDIP): Minimizing the ‘Trough of Disillusionment’ and Accelerating Pathways to the ‘Plateau of Productivity’

Abstract:

The National Computational Infrastructure (NCI) at the Australian National University (ANU) has evolved to become Australia’s peak computing centre for national computational and Data-intensive Earth system science. More recently NCI collocated 10 Petabytes of 34 major national and international environmental, climate, earth system, geophysics and astronomy data collections to create the National Environmental Research Interoperability Data Platform (NERDIP). Spatial scales of the collections range from global to local ultra-high resolution, whilst sizes range from 3PB down to a few GB. The data is highly connected to both NCI HPC and cloud resources via low latency internal networks with massive bandwidth.

Now that the collections are collocated on a single data platform, the ‘Hype’ and expectations around potential use cases for the NERDIP are high. Not unexpected issues are emerging such as access, licensing issues, ownership, and incompatible data standards. Many communities are standardised within their domain, but achieving true interdisciplinary science will require all communities to move towards open interoperable data formats such as NetCDF4/HDF5. This transition will impact on software using proprietary or non-open standards.

But before we reach the ‘Plateau of Productivity’, there needs to be greater ‘Enlightenment’ of users to encourage them to realise that this unprecedented Earth system science platform provides a rich mine of opportunities for discovery and innovation for a diverse range of both domain-specific and interdisciplinary investigations including climate and weather research, impact analysis, environment, remote sensing and geophysics and develop new and innovative interdisciplinary use cases that will guide those architecting the system and help minimise the amplitude of the ‘Trough of Disillusionment’ and ensure greater productivity and uptake of the collections that make NERDIP unique in the next generation of Data-intensive Science.