Processing a Large Amount of Seismic (Geophysical?) Data Using Hadoop

English Abstract:

Processing large amounts of seismic data to perform complex operations such as sophisticated event detection, event location in complex 3D media, or seismic interferometric imaging in a reasonable amount of time often requires the use of computer clusters. Traditionally, researchers mainly used frameworks based on a message passing system (MPS) to harness the capability of high performance computing facilities. MPSs provide flexibility and scalability which allow for the processing to be distributed on several machines both across local and wide area networks. Most MPSs are, however, dependent on network architecture, and require that the communication and data distribution between a master and slaves, as well as network failure, be explicitly managed. Today the map-reduce framework implemented in Apache Hadoop provides an alternative to message passing systems for distributed computing. In contrast to message passing, Hadoop is conceptually simple, automatically manages the distribution and processing of data and seamlessly handles node failures. To use Hadoop, problems have to be cast as a series of map/reduce operations. So far, we have found that most computing problems in seismology are amenable to this. In addition, we have found that it is possible to integrate into the processing flow operations performed by existing programs, not necessarily designed for parallel processing, without any re-factoring of the code. We present a processing scheme that illustrates how Hadoop can be used to parallelize operations commonly performed on seismic data. We then apply the processing scheme to process 8 TB of continuous data performing complex operations in real-time using standard software widely used in seismology such as NonLinLoc and Obspy.