Condensing Massive Satellite Datasets For Rapid Interactive Analysis

Abstract:

Our goal is to enable users to interactively analyze massive satellite datasets, identifying anomalous data or values that fall outside of thresholds. To achieve this, the project seeks to create a derived database containing only the most relevant information, accelerating the analysis process. The database is designed to be an ancillary tool for the researcher, not an archival database to replace the original data. This approach is aimed at improving performance by reducing the overall size by way of condensing the data. The primary challenges of the project include:

- The nature of the research question(s) may not be known ahead of time.

- The thresholds for determining anomalies may be uncertain.

- Problems associated with processing cloudy, missing, or noisy satellite imagery.

- The contents and method of creation of the condensed dataset must be easily explainable to users.

The architecture of the database will reorganize spatially-oriented satellite imagery into temporally-oriented columns of data (a.k.a., “data rods”) to facilitate time-series analysis. The database itself is an open-source parallel database, designed to make full use of clustered server technologies. A demonstration of the system capabilities will be shown.

Applications for this technology include quick-look views of the data, as well as the potential for on-board satellite processing of essential information, with the goal of reducing data latency.