Optimizing Oceanographic Big Data Browse and Visualization Response Times by Implementing the Lambda Architecture

Visualizing large-scale data sets using standard web-based mapping tools can result in significant delays and response time issues for users. Load times for data sets comprised of millions of records can be in excess of thirty seconds when the data sets are served using traditional architectures and techniques. In this paper we demonstrate the efficiency gains created by utilizing the Lambda Architecture on a low velocity, high volume hypoxia-nutrient decision support system with 25M records. While traditionally employed on high velocity, high volume data we demonstrate significant improvements in data load times and the user browse experience on low velocity, high volume data. Optimizing query and visualization response times becomes increasingly important as data sets grow in size. Time series data from extended autonomous underwater vehicle deployments can exceed 500M records. Applying the Lambda Architecture to these data sets will allow users to browse, visualize and fuse data in a manner not possible using traditional methodologies.