Integrating a Collaborative Infrastructure with a Big Data Technology to Boost Data-analysis Productivity

Abstract:

Because the Earth is a “system of systems”, i.e., it is composed of many strongly coupled and complicated subsystems, Earth scientists must frequently collaborate beyond their individual focus areas. As a result, Earth Science data analysis typifies the four V’s of Big Data: 1) We need to analyze a great “variety” of data from all subsystems; 2) We need to analyze greater “volumes” of data because, to improve understanding, we are resorting to ever finer resolutions in both observations and model simulations; 3) We need to analyze these data with greater “velocity,” for climate change has imposed urgency and the current processes are taking too long to digest these data; 4) We need to analyze these data with “veracity” lest we contribute more to confusion than solution. However, current data-analysis practices exhibit numerous inefficiencies and are poorly suited to many interdisciplinary collaborations. We demonstrate, using a realistic science scenario, a prototype system that integrates a collaborative infrastructure with Big Data techniques to address these inefficiencies. This system has great potential to provide higher return on investment and boost significantly science data analysis productivity. We will discuss its implications for the (near) future of scientific data analysis.