Development of new metric evaluations for magnetic in situ solar wind model forecasts at L1

We present results from predictive models of interplanetary CMEs arriving at Earth. The architecture of the empirically based Bz4Cast model is compared to the MHD based SUSANOO model. We show that appropriate simplifications to the real-world, which are made by these models, still generate insightful skill above random noise. In particular, we demonstrate two key behaviors: 1) By investigating 78 CME events, the Bz4Cast model can statistically predict the NOAA geomagnetic storm scale above random noise and to an equal level of skill as a trained on-duty forecaster. 2) Injecting a magnetic spheromak into an inner boundary of an MHD simulation can produce an obstacle akin to a flux rope which is more observationally-appropriate at L1.

For Operations-to-Research (O2R) and Research-to-operations (O2R) activities, ideally, an internationally agreed metric is needed to fairly compare and contrast between current operational models and future model concepts. Using the facilities provided by NASA CCMC, we have been developing an online framework to test new models that forecast the magnetic field at L1. We present preliminary examples of the metric framework using two specific metric methods: 1) ROC curve, 2) Cost-Loss analysis. We will discuss the difficulties in their interpretations and their strengths. Finally, we will demonstrate the strategy at CCMC for helping the community to R2O transition their new models by providing openly available metric analysis using a new Bz scoreboard website that is under development.