NSIDC: Coping with a Big Data World

Abstract:

The National Snow and Ice Data Center (NSIDC) has been serving cryospheric and related data to the research community since 1976, though our actual holdings include data extending back into the earliest days of polar exploration and in other cases back to the earliest days of scientific data collection itself. Reflecting diversity in both funding sources and research themes, NSIDC’s data holdings vary dramatically in size and variety. A considerable amount of NSIDC's data is not yet even available digitally! Digital holdings range all the way from simple spreadsheets from an individual investigator to large and constantly updated NASA and NOAA satellite data sets to model output to photographs and visualized data. While most of our data holding relate to the physical sciences, we are increasingly handling data from the social sciences that may include oral histories, photographs, and surveys.

Over the years the audiences interested in these data have broadened dramatically, from investigators within the discipline that provided the data, to investigators in other disciplines, journalists, decision makers, and the motivated public. How is NSIDC addressing these challenges and evolving its data management practices? In this invited presentation NSIDC's approaches for tackling the Big Data challenges of volume, velocity, veracity, visibility, and variety are described.