1、1Decision-Making from Probability Forecasts using Calculations of Forecast ValueKenneth R. MylneThe Met. Office, Bracknell, UK(To be submitted to Meteorological Applications)How do I make a decision based on a probability forecast?Abstract: A method of estimating the economic value of weather foreca
2、sts for decision-making is described. The method may be applied equally to either probability forecasts or deterministic forecasts, and provides a forecast user with a direct comparison of the value of each in terms of money saved, which is more relevant to users than most standard verification scor
3、es. For a user who wishes to use probability forecasts to decide when to take protective action against a weather event, the method identifies the optimum probability threshold for action, thus answering the question of how to use probability forecasts for decision-making. The system optimises decis
4、ion-making for any probability forecast system, whatever its quality, and therefore removes any need to calibrate the probability forecasts. The method is illustrated using site-specific probability forecasts generated from the ECMWF ensemble prediction system and deterministic forecasts from the EC
5、MWF high-resolution global model. It is found that for most forecast events and most users the probability forecasts have greater user value than the deterministic forecasts from a higher resolution model.1. IntroductionA weather forecast, however skilful, has no intrinsic value unless it can be use
6、d to make decisions which bring some benefit, financial or otherwise, to the end user. Conventionally in most weather forecast services the forecast provider supplies the user with their best estimate of whether a defined event will occur (e.g. wind speed will or will not exceed 15ms-1), or of a val
7、ue for a measurable parameter (e.g. maximum wind speed =18 ms-1). Decision-making is often based on whether a defined event is expected to occur or not. For example the owner of a small fishing boat may decide to seek shelter when the forecast wind speed exceeds 15 ms-1. The nature of atmospheric pr
8、edictability is such that there is frequently a significant uncertainty associated with such deterministic forecasts. Forecast uncertainty can be expressed in many ways, either qualitatively or quantitatively, and where such information is included in a forecast this can aid the decision-maker who u
9、nderstands the potential impact of a wrong decision. However, uncertainty is most commonly estimated subjectively by a forecaster; such estimates are often inconsistent, and may be affected by factors such as forecasters “erring on the safe side”, which may not lead to optimal decision-making. In re
10、cent years there has been considerable development of objective methods of estimating forecast uncertainty, notably ensemble prediction systems (EPS) such as those operated by the European Centre for Medium Range Weather Forecasts (ECMWF) (Molteni et al, 1996, Buizza and Palmer 1998) and the US Nati
11、onal Centers for Environmental 2Prediction (NCEP) (Toth and Kalnay, 1993). Output from an EPS is normally in the form of probability forecasts, and there is growing evidence (e.g. Molteni et al, 1996, Toth et al 1997) that these have greater skill than equivalent deterministic forecasts based on sin
12、gle high-resolution model forecasts, particularly on medium range time-scales. To make use of this additional skill, the decision-maker needs to know how to respond to a forecast such as There is a 30% probability that the wind speed will exceed 15 ms-1. This paper will describe a technique which es
13、timates the economic value of a probability forecast system for a particular user based on verification of past performance, and use it to determine the users optimal decision-making strategy. The value of deterministic forecasts can be calculated in the same way, and this allows a direct comparison
14、 of the utility of probability and equivalent deterministic forecasts in terms which are clear and relevant to the user. 2. Background to Ensemble Probability ForecastsUncertainty in weather forecasts derives from a number of sources, in particular uncertainty in the initial state of the atmosphere
15、and approximations in the model used to predict the atmospheric evolution. Errors in the analysis of the initial state result from observational errors, shortage of observations in some regions of the globe and limitations of the data assimilation system. Model errors are due to numerous approximati
16、ons which must be made in the formulation of a model, most notably the many small-scale processes which cannot be resolved explicitly, and whose effect must therefore be represented approximately by parametrization. The non-linear nature of atmospheric evolution means that even very small errors in
17、the model representation of the atmospheric state, whether due to the analysis or the model formulation, will be amplified through the course of a forecast and can result in large errors in the forecast. This sensitivity was first recognised by Lorenz (1963), and was influential in the development o
18、f chaos theory. Gross errors in the synoptic-scale evolution are common in medium range forecasts (over 3 days), but can occasionally occur even at less than 24 hours. Errors in the fine detail of a forecast, such as precipitation amounts or locations, are common even in short-range forecasts. Ensem
19、ble prediction systems have been developed in an attempt to estimate the probability density function (pdf) of forecast solutions by sampling the uncertainty in the analysis and running a number of forecasts from perturbed analyses (Molteni et al, 1996; Toth and Kalnay, 1993). In more recent develop
20、ments, Buizza et al (1999) have included some allowance for model errors in the ECMWF EPS, by adding stochastic perturbations to the effects of model physics. Houtekamer et al (1996) describe an ensemble which accounts for both model errors and analysis errors by using a range of perturbations in bo
21、th model formulation and analysis cycles. With any of these ensemble systems, probability forecasts may be generated by interpreting the proportion of ensemble members 3predicting an event to occur as giving a measure of the probability of that event. A range of standard verification diagnostics are
22、 used to assess the skill of such probability forecasts. For example the Brier Score (see Wilks, 1995), originally introduced by Brier (1950), is essentially the mean square error for probability forecasts of an event. Murphy (1973) showed how the Brier score could be decomposed into three terms, re
23、liability, resolution and uncertainty, which measure different aspects of probabilistic forecasting ability. Of these the reliability term measures how well forecast probabilities relate to the actual probability of occurrence of the event, and resolution measures how effectively the forecast system
24、 is able to distinguish between high and low probabilities on different occasions. ROC (Relative Operating Characteristics), described by Stanski et al (1989), measures the skill of a forecast in predicting an event in terms of Hit Rates and False Alarm Rates. Rank Histograms (Hamill and Colucci, 19
25、97 and 1998) specifically measure the extent to which the ensemble spread covers the forecast uncertainty, and can also reveal biases in the ensemble forecasts. However while all these diagnostics are of great value to scientists developing ensemble systems, they are of little interest or relevance
26、to most forecast users. In particular they do not tell users how useful or valuable the forecasts will be for their applications, nor do they answer the question of how to use probability forecasts for decision-making. 3. Calculation of Forecast ValueAn overview of techniques for estimating the econ
27、omic value of weather forecasts is given by Murphy (1994), and a comprehensive review by Katz and Murphy (1997). The method applied in this study is closely related to ROC verification (Stanski et al, 1989). It has recently been discussed by Richardson (2000), and is rapidly becoming accepted as a v
28、aluable tool for user-oriented verification of probability forecasts. The method has also been applied to seasonal forecasts by Graham et al (2000). The aim of this paper is to present the method in a way which is particularly suitable for aiding forecast users with decision making.The concept of fo
29、recast value is that forecasts only have value if a user takes action as a result, and the action saves the user money. Calculation of forecast value for predictions of a defined event therefore requires information on (a) the ability of the forecast system to predict the event, and (b) the users co
30、sts and losses associated with the various possible forecast outcomes. Consequently the value depends on the application as well as on the skill of the forecast. Forecast value will be defined first for a simple deterministic forecast, and the generalisation to probability forecasts will be consider
31、ed in more detail in section 3.6.3.1 Ability of the Forecast SystemThe basis of most estimates of forecast value is the cost-loss situation described by Murphy (1977). This is based on forecasts of a simple binary 4event, against which a user can take protective action when the event is expected to
32、occur. For such an event the ability of a forecast system is fully specified for a deterministic forecast by the 22 contingency table shown in Table 1, where h, m, f and r give the frequencies of occurrence of each possible forecast outcome. 3.2 User Costs and LossesFor any user making decisions bas
33、ed on forecasts, each of the four outcomes in table 1 has an associated cost, or loss, as given in table 2. For Event ForecastYes - User ProtectsNo - No Protective ActionEvent Yes Hit H(h)Miss M(m)Observed No False Alarm F(f)Correct Rejection R(r)Table 1: Contingency table of forecast performance. U
34、pper case letters H, M, F, R represent the total numbers of occurrences of each contingency, while the lower case versions in brackets represent the relative frequencies of occurrences.Event ForecastYes - User ProtectsNo - No Protective ActionYes Mitigated Loss LmLoss LEvent Observed No Cost C Norma
35、l Loss N=0Table 2: Contingency table of generalised user costs and losses. Note: in the simple cost/loss situation described by Murphy (1977), this is simplified such that Lm = C. convenience it is normal to measure all costs and losses relative to the users costs for a Correct Rejection, so the Nor
36、mal Loss N for this contingency is set to zero. (Note however that this assumption is not necessary, and the method readily accounts for non-zero values of N.) If the event occurs with no protective action being taken, the user incurs a loss L. If the event is forecast to occur, the user is assumed
37、to take protective action at cost C. In the simple 5cost/loss situation (Murphy, 1977), this action is assumed to give full protection if the event does occur, so the user incurs the cost C for both Hits and False Alarms. In reality protection will often not be fully effective in preventing all loss
38、es, and the losses may be generalised by specifying a Mitigated Loss Lm for Hits, as in table 2.For a forecast to have value it is necessary that LmL. In most circumstances it would be expected that C LmL, but it is possible that in some circumstances LmC. For example, protective action could involv
39、e using an alternative process which works effectively in the weather conditions specified by the event, but does not work in the non-event conditions - in this case the cost C of a False Alarm would be high compared to Lm. The above examples assume that costs, losses and forecast value are specifie
40、d in monetary terms. They could, instead, be calculated in terms of a users energy consumption, for example - the concept is the same. Note that one limitation of the system comes where a forecast is used to protect life, due to the difficulty in objectively placing a cost on lives lost. 3.3 Mean Ex
41、pense Following ForecastsGiven the information specified in tables 1 and 2, and assuming the user takes protective action whenever the event is forecast, it can be expected that over a number of forecast occasions the user will experience a mean expense Efx of (1)rNfCmLhEfxNote that the last term rN
42、 in equation (1) is normally zero, but this specifies the generalisation to any definition of the Normal Loss. 3.4 Climatological Reference ExpenseForecast value represents the saving the user gains by using the forecasts, and therefore requires a baseline reference expense for comparison with Efx.
43、If no forecast is available the user has two options, either always protect or never protect. In Murphys (1977) simple cost-loss situation where Lm=C these options will incur mean expenses Ecl over many occasions of C or L respectively, where is the climatological frequency of oooccurrence of the ev
44、ent. The users best choice is always to protect if C L, oor C/L , and never to protect otherwise. Assuming the user takes the best option, the mean climatological expense in the simple cost-loss situation is thus given by (2)L,Cmin(EclFor the generalised user loss matrix given in Table 2, the mean e
45、xpense of the always protect option is given by , and the mean mLo)(1expense of following climatology is given by (3),o)i(mcl 16Fully generalising this to allow for the possibility of N0 gives (4)Lo(,LoC)min(Emcl 11In this case the users best strategy is to always take protective action if )m(5where
46、 is the generalised cost/loss ratio introduced by Richardson (2000). In some circumstances one of the climatological options may not be viable for a user, since taking protective action may involve stopping doing their normal economic activity (e.g. a fishermans protective action against strong wind
47、s may be to leave his boat in port). The user cannot do that all the time or he would go out of business. In this case the forecast value should be calculated using the viable climate option. This will be considered further in section 4.3.3.5 Definition of Forecast ValueThe value of the forecast in
48、monetary terms, V, is the saving the user can expect to make from following the forecast, averaged over a number of occasions: (6)fxclEVThis basic definition of value is the most relevant for a user, except that it does not account for the cost of buying the forecast service. The true value to the u
49、ser is therefore (7)fxfclfxuCwhere Cfx is the purchase price of the forecast. However, although Vu is the correct definition to use when estimating the value of a forecast to a user, Cfx is specific to any forecast service and cannot be estimated in general terms. For the purposes of this paper this term will therefore be ignored and value will be defined as in equation (6).For general assessments of forecast value, it is convenient to scale V relative to the value of a perfect forecast, in a similar fashion to the normal definition of a skill sc
