A Methadology for Near-Real-Time Access to Environmental Data through Federation

Abstract:

The availability of near-real-time data can be critical for response to rapid changes including violent storms, tsunamis and earthquakes. While climate changes relatively slowly, compared to a tsunami, the increasing variance in weather over time and warming must also be considered in terms of civil impacts. A simple example is the decreasing resilience of coastal communities to severe weather as sea level increases. The integration of these data for modeling and response activities in near-real-time must be pursued to make data collection practical. We present an approach to data and metadata integration that has occurred over the past 10-20 years in Earth and Ocean sciences that provide a model for the future. The NSF Data Federation Consortium (DFC) is working to integrate data and metadata from a number of fields using iRODS (Integrated Rule-Oriented Data System). iRODS is open source software for building distributed data collections. In particular, the SCION (SCIence Observatory Network) funded by the NSF provides Python-based software for data and metadata access from a variety of near-real-time data sets relevant to climate studies including weather and hazards from other observational systems. As an example, we are working on the integration of data on shore and offshore in southern California using resources from the High Performance Wireless Research and Education Network (HPWREN) and the Southern California Coastal Ocean Observing System (SCCOOS). National and International integration of near-real-time earthquake data through the Incorporated Research Institutions for Seismology (IRIS) and the International Federation of Digital Seismic Networks (FDSN) provide a well-integrated data and metadata system for both research and civil uses. ObsPy, written in Python, has proved to be a highly successful methodology for accessing global data from thousands of stations with well-developed metadata. The persistence of the data and metadata, in turn, provides long-term provenance particularly important for climate data. We discuss these various tools and the current state of efforts in broader integration of Earth data.