A Scenario Is A Projection Is A Forecast: And All Should Be Verified

Abstract:

Bray and von Storch (2009) showed that there is considerable confusion among scientists in differentiating forecasts, projections, and scenarios, and that this confusion is echoed by the IPCC.

These misunderstandings are not only found in making predictions, and in classifying which future-statements count as predictions, but also under which circumstances predictions must be verified. There is general recognition that good models produce good forecasts, but there is no escape from the implication that a bad forecast arises from a flawed model by claiming a forecast wasn't a forecast (or prediction) but a projection or scenario (see e.g. Risbey, 2014).

We show forecasts, projections, and scenarios (when used prospectively) are equivalent, and that all face the same burden of verification.

Any model M, statistical or physical, is a complex mixture of past data, time, and external evidence which specifies the model form. Forecasts f about some observable y are conditional on M and on a guess "x" about what the future holds. (Confusingly, sometimes x is called a "scenario" and sometimes the whole forecast is called a "scenario".)

The simplest x is that the future comes in discrete time points, e.g. f(y_t+1 |M, t+1...). Assuming its components are probative of y, x may be compound. E.g., suppose a climate model is built so that it is sensitive to oil price, thus x = "Price of oil > p and t+1".

If at t+1 the price of oil was not <= p, then the forecast is null: no forecast has been made, because the conditions x were not met. At this point, some might call this forecast a "projection" and ignore its verification. But since this x is probative of y, the model also implies the forecast f(y | Price <= p, t+1), which should be (after-the-fact) computed and then verified.

Examples where predictions which have "escaped" verification, as well as strategies of verification, will be given.