Feature extraction from scientific datasets using Apache Spark

Abstract:

We present an example of feature extraction from scientific datasets such as global numerical models using Apache Spark. The algorithm uses a simple penalized linear regression technique and a training dataset to learn and extract a similar feature from the rest of the data. Thanks to Apache Spark, algorithm can scale to a large number of computing nodes.