Scientific assessment of accuracy, skill and reliability of ocean probabilistic forecast products.

Mozheng Wei, John C. Stennis Space Center, Stennis Space Center, MS, United States, Clark David Rowley, John C. Stennis Space Center, Oceanography Division, Stennis Space Center, MS, United States, Charlie N. Barron, Naval Research Laboratory SSC, Stennis Space Center, MS, United States and Patrick J Hogan, John C. Stennis Space Center, Oceanography, Stennis Space Center, MS, United States
Abstract:
As ocean operational centers are increasingly adopting and generating probabilistic forecast products for their customers with valuable forecast uncertainties, how to assess and measure these complicated probabilistic forecast products objectively is challenging. The first challenge is how to deal with the huge amount of the data from the ensemble forecasts. The second one is how to describe the scientific quality of probabilistic products. In fact, probabilistic forecast accuracy, skills, reliability, resolutions are different attributes of a forecast system.

We briefly introduce some of the fundamental metrics such as the Reliability Diagram, Reliability, Resolution, Brier Score (BS), Brier Skill Score (BSS), Ranked Probability Score (RPS), Ranked Probability Skill Score (RPSS), Continuous Ranked Probability Score (CRPS), and Continuous Ranked Probability Skill Score (CRPSS). The values and significance of these metrics are demonstrated for the forecasts from the US Navy’s regional ensemble system with different ensemble members. The advantages and differences of these metrics are studied and clarified.

Back to: Advances in Operational Oceanography: Modeling, Product Services, and Skill Assessment I