Using GDAL to Convert NetCDF 4 CF 1.6 to GeoTIFF: Interoperability Problems and Solutions for Data Providers and Distributors

Abstract:

An increasing number of new Earth science datasets are being produced
by data providers in self-describing, machine-independent file formats
including Hierarchical Data Format version 5 (HDF5) and Network
Common Data Form version 4 (netCDF-4). Furthermore data providers may
be producing netCDF-4 files that follow the conventions for Climate
and Forecast metadata version 1.6 (CF 1.6) which, for datasets mapped
to a projected raster grid covering all or a portion of the earth,
includes the Coordinate Reference System (CRS) used to define how
latitude and longitude are mapped to grid coordinates, i.e. columns
and rows, and vice versa. One problem that users may encounter is that
their preferred visualization and analysis tool may not yet include
support for one of these newer formats. Moreover, data distributors
such as NASA's NSIDC DAAC may not yet include support for on-the-fly
conversion of data files for all data sets produced in a new format to
a preferred older distributed format.

There do exist open source solutions to this dilemma in the form of
software packages that can translate files in one of the new formats
to one of the preferred formats. However these software packages
require that the file to be translated conform to the specifications
of its respective format. Although an online CF-Convention compliance
checker is available from cfconventions.org, a recent NSIDC user
services incident described here in detail involved an NSIDC-supported
data set that passed the (then current) CF Checker Version 2.0.6, but
was in fact lacking two variables necessary for conformance. This
problem was not detected until GDAL, a software package which relied
on the missing variables, was employed by a user in an attempt to
translate the data into a different file format, namely GeoTIFF.

This incident indicates that testing a candidate data product with one
or more software products written to accept the advertised conventions
is proposed as a practice which improves interoperability. Differences
between data file contents and software package expectations are
exposed, affording an opportunity to improve conformance of software,
data or both. The incident can also serve as a demonstration that data
providers, distributors, and users can work together to improve data
product quality and interoperability.