Self-Adaptive Quantiles for Precipitation Forecasting

1. Introduction

While weather forecasting was approached as a deterministic initial value problem at the beginning of the 20^th century [1], ensemble prediction systems are nowadays common practice in meteorology and many other scientific fields [2]. Ensembles consist of a set of plausible weather outcomes that allow estimating the uncertainty about the future state of the atmosphere [3]. Accounting for forecast uncertainty in decision-making has proven more valuable than relying on a single best-guess forecast [[4,5], and others in between].

Forecast end-user’s ability to deal with uncertainty is however heterogeneous. For some, the uncertainty associated with a forecast is useful [6,7], while for others, this information is difficult to apprehend [8]. Strategies to account for forecast uncertainty can consist in extracting a summary forecast that would, in the long term, optimize one’s decision-making process.

Moreover, automation of forecasting tasks are developed by meteorological agencies to mitigate production overload and limited resources [9]. By nature, forecast automation relies on maximizing a measure of accuracy. In this context, the question arises of how the diversity of information generated by an ensemble, on the one hand, and the automation requirement, on the other hand, can fit together while acknowledging the diversity of users’ needs. One approach is to derive a single actionable forecast potentially suitable for a wide range of users.

So-called best-guess forecasts can be derived from an ensemble by using, for example, clustering. Different clustering techniques exist to create consistent scenarios, with one single model representing a cluster [10]. However, the use of a single model is questionable: the definition of the best member is a highly multidimensional task [11], and the concept of the dominance of one model among all realizations is sometimes considered as a flawed concept [12].

Alternatively, the whole ensemble can be used to create a pseudo-deterministic meteorological field, a so-called consensus forecast [13]. For example, optimal forecast with respect to square and/or absolute errors can be derived using genetic programming [14] or spatially-patched precipitation field from ensemble members closest to the mean [15]. More recently, optimal threshold optimization based on F2 and Equitable Threat Score has been empirically investigated by Bouttier and Marchal [16]. A common aspect of all these studies on consensus forecasts is that optimization relies on the definition of a measure of accuracy.

What is the ideal consensus forecast for a given measure of accuracy? The answer to this question is related to the concept of forecast elicitation or consistency with a scoring rule [17]. The development of a mathematical framework for a formal definition of elicitation is out of scope here, but we introduce this concept based on 2 simple examples: 1) the ensemble mean is consistent with the root mean squared error (RMSE), and 2) quantile forecasts are consistent with quantile scores (QS). A special case of the latter is the 50%-quantile (or median) which is consistent with the mean absolute error.

Figure 1 illustrates the concept of elicitation with 2 examples. While more details about the data and the different forecast types are provided below, we can already make the following remarks. The ensemble mean ($em$) is the best-performing forecast in terms of RMSE for all lead times in Figure 1(a). The concept of elicitation is powerful because one doesn’t need to test if the ensemble mean is the best forecast in terms of RMSE for a given application, but rather know beforehand that it is the case. Similarly, the quantile forecast ($mx$) is the best forecast in terms of the quantile score QSmax for all lead times in Figure 1(b), as expected. In this example, $em$ is also one of the worst performing forecasts in terms of QSmax and $mx$ the worst performing forecast in terms of RMSE. In other words, the best forecast for one application can be the worst for another one.

In this manuscript, we introduce a new type of point-forecast, the crossing-point quantile, and demonstrate that this forecast is optimal in terms of Peirce skill score [PSS, [18]] in the case of a reliable ensemble system, whatever the event of interest. The work presented here is based on the crossing-point forecast (CPF) developed and explored in previous studies. [19] showed that the CPF is consistent with the Diagonal score, a threshold/quantile-weighted continuous ranked probability score introduced in [20].

This paper is organized as follows: Section 2 introduces the point-forecasts used for comparison together with a description of the theoretical properties of the crossing-point quantile, Section 3 displays illustrations of forecasts and observations alongside an analysis of the consistency between them, Section 4 discusses verification results before concluding in Section 5.

2. From Ensemble to Point-Forecasts

2.1. Benchmarking

We start with the ensemble prediction system run at the European Centre for Medium-Range Weather Forecasts (ECMWF). The control member and the 50-perturbed ensemble members serve as a basis for deriving point-forecasts. A point-forecast is defined as a single-valued forecast expressed in the unit of the weather variable of interest. A deterministic forecast for temperature at, say, Reading, UK, tomorrow is a point-forecast. Another example is the ensemble mean for the same quantity.

In this study, we propose to compare the performance and characteristics of different point-forecasts with a focus on daily precipitation and time ranges up to the medium range. As a benchmark, we consider the following point-forecasts:

• a deterministic forecast, here the ensemble control member (ctrl),
• the ensemble mean (em), which is typically a smoothed field at longer time ranges as predictability decreases,
• the maximum of all ensemble members (mx), which can also be interpreted as a quantile forecast at probability level ${\displaystyle \frac{M}{M+1}}$ with M the ensemble size1,
• the quantile at a probability level of 70% conditioned on that at least half of the ensemble members indicates rain ($q_{70}^c$). This latter is the consensus forecast used at Météo-France as a medium-range precipitation forecast.

These 4 types of point-forecasts are illustrated with a synthetic example in Figure 2(a).

2.2. Self-Adaptive Quantiles

Let’s now remind the reader of the definition of the crossing-point forecast. For this, consider a condition called single intersection condition. Namely, given F and G two cumulative distribution functions, F and G satisfy the single intersection condition if there is one unique q such that:

$$x\ge q\Rightarrow F\left(x\right)\ge G\left(x\right)$$

(1)

and

$$x<q\Rightarrow F\left(x\right)<G\left(x\right).$$

(2)

In this situation, the intersection between F and G is defined by the point $(q,F(q\left)\right)$. If F is a forecast and G a climatology, $F\left(q\right)$ is the so-called crossing-point forecast [19]. By analogy, q is called the crossing-point quantile which is the focus of this study.

The concept of crossing-point forecast and quantile is illustrated in Figure 2(b). The single intersection point between the ensemble distribution (F, represented by a solid grey line) and the climatology distribution (G, represented by a dashed grey line) is indicated by a circle. CPF is the projection of this point on the y-axis and so takes a value between 0 and 1 (not shown). In other words, CPF is a quantile level that varies for each forecast. Consequently, the crossing-point quantile q is a self-adaptive quantile forecast, a quantile forecast at a quantile level corresponding to the CPF of the day (indicated by a pink vertical line).

In the estimation of the CPF, forecast and climate distributions refer to the same space, the “model space”. In that case, the crossing-point quantile is denoted by $q_{\mathrm{cpf}}^M$ as it is based on the model climate. When issuing an adaptive quantile forecast, one can consider using an observation climatology instead of a model climatology. To be consistent and to ensure forecast reliability, the ensemble can be calibrated in the “observation space”. The calibrated forecast and observation climatology can differ from the raw forecast and the model climatology, respectively, as illustrated in Figure 2(c). When based on a calibrated forecast and the observation climatology, the crossing-point quantile is denoted by $q_{\mathrm{cpf}}^O$.

More details about how model and observation climate distributions are built in our experiments can be found in Appendix A.1 while a short description of the methodology followed to calibrate the ensemble is provided in Appendix A.2.

2.3. The Optimal Forecast in Terms of PSS

The Peirce skill score (PSS) is a verification metric for dichotomous (yes/no) forecasts assessed against dichotomous observations (an event happens or not). We can show that the crossing-point quantile forecast is the optimal forecast in terms of PSS for any event.

Let’s first define PSS. Consider a dichotomous forecast for a yes/no event (typically the observation being greater than a threshold). For verification, a contingency table is populated by counting the number of hits $\left(A\right)$, false alarms (B), misses (C), and correct negatives (D). The hit rate ($HR$) is defined as the ratio ${\displaystyle \frac{A}{A+C}}$ while the false alarm rate ($FAR$) is defined as the ratio ${\displaystyle \frac{B}{B+D}}$. The Peirce skill score definition follows:

$$PSS=HR-FAR.$$

(3)

A direct link exists between PSS and the relative operating characteristics (ROC) curve. For a dichotomous forecast, the ROC is defined by the point [$FAR$, $HR$]. In that case, [22] showed that PSS and the area under the ROC curve (AUC) are related as follows:

$$PSS=2AUC-1.$$

(4)

The right-hand side of Equation (4) is also know as the ROC skill score.

Consider now a forecast for a binary event that takes the form of a probability. In that case, the ROC curve is a set of points [$FAR$, $HR$] estimated for a range of probability thresholds. An illustration of a theoretical ROC curve for a probability forecast is provided in Figure 3(a). In this situation, the point closer to the left top corner of the plot corresponds to the probability threshold that maximizes PSS (i.e the difference between HR and FAR). If the forecast is reliable, it can be shown that the tangent to the ROC curve at that point has a slope equals to 1 and that this point corresponds to the results obtained with a probability threshold equals to the event base rate [23,24]. This latter characteristics (the optimal probability threshold is equal to the event base rate) is key in the following demonstration.

Now, coming back to the crossing-point forecast, let’s define more formally an event of interest say $x>\theta$ with $\theta$ the event threshold (e.g., rain observed greater than 10mm/24h). The base rate of this event, denoted $p_c$, is by definition equal to $1-G\left(\theta \right)$ with G the climate cumulated probability distribution. We denote $p_f$ the probability forecast for this event which is by definition $1-F\left(\theta \right)$ with F the forecast cumulated probability distribution. Using Equations (1) and (2), we can derive the following:

$$p_f<p_c\Rightarrow 1-F\left(\theta \right)<1-G\left(\theta \right)\Rightarrow F\left(\theta \right)\ge G\left(\theta \right)\Rightarrow \theta \ge q$$

(5)

and

$$p_f\ge p_c\Rightarrow 1-F\left(\theta \right)\ge 1-G\left(\theta \right)\Rightarrow F\left(\theta \right)<G\left(\theta \right)\Rightarrow \theta <q$$

(6)

In plain words, Equations (5) and (6) mean that taking action when the probability forecast $p_f$ is greater than the event base rate $p_c$ is equivalent to taking action when the crossing-point quantile q exceeds the event threshold $\theta$. The former corresponds to the optimal choice to maximize PSS. So, conditioned on reliability, the crossing-point quantile is the optimal forecast in terms of PSS (and equivalently in terms of ROC area) for any dichotomous event defined by a threshold.

3. A Qualitative Assessment

We now assess qualitatively the different types of point-forecasts using case studies. For this purpose, we show 2 types of plots: 1) time series of precipitation maxima within an area over consecutive summer days and 2) maps of precipitation fields for an interesting date in the series.

The observation used to compute statistics, build a climatology, and illustrate this article is a radar dataset called COMEPHORE (COmbination for Best Estimation of Hourly Precipitation). This hourly precipitation reanalysis product is based on radar and rain gauge data, covering metropolitan France [25].

The data, originally on a 1km grid, is aggregated at the spatial resolution of the ensemble forecast, that is 18km. The upscaling of the observation consists in averaging the precipitation at the new spatial scale. This step is applied consistently in the verification and in the observation climatology. This choice aims at a fair comparison of the different point-forecasts evaluated here, all interpreted as averaged precipitation over a grid box. For operational purposes, one could instead consider using a COMEPHORE climatology at the original scale to benefit from a downscaling while estimating the crossing point quantile.

3.1. Time Series

We define an area of $10\times 10$ model grid points centered over Toulouse, France. For each forecast as well as for the radar, the maximum precipitation within this area for each day is shown in Figure 4. The time series cover June, July, and August (JJA) 2022.

The ensemble mean ($em$) systematically underestimates the precipitation amount but larger $em$ values (in relative terms) are generally associated with larger observations. The crossing-point quantiles, both $q_{\mathrm{cpf}}^M$ and $q_{\mathrm{cpf}}^O$, show a good fit with the observations, with the former ($q_{\mathrm{cpf}}^O$) displaying larger values that match better the observations in this example. With the control member ($ctrl$), several large precipitation events are missed while the ensemble maximum ($mx$) is more prone to false alarms with sometimes large forecast values when no rain is observed. Finally, the conditional quantile ($q_{70}^c$) shows a systematic underestimation of large precipitation events.

The most extreme precipitation event observed in the series was on August, 14 (day 75). This event seems particularly well captured by the crossing-point quantiles with an intensity closer to the observation for $q_{\mathrm{cpf}}^O$. We now assess visually the forecasts as fields valid on that particular date.

3.2. Maps

The case study discussed here refers to August 14, 2024, identified as the most extreme event during the Summer of 2024 in the Toulouse area. The different point-forecasts are shown in Figure 5a–f, all have a forecast lead time of 5 days. In Figure 6a–e, we focus on the crossing-point quantile forecasts $q_{cpf}^O$ and compare forecasts at decreasing lead times but with the same validity time.

The radar observation for the verification date is shown at its native resolution in Figure 6(f). The radar captured the strongest precipitation signal over the South West of France with values exceeding 80$mm/24h$. High precipitation values are also registered along a South-West/North-East axis with larger scale precipitation in the North-East of France.

In this example, the crossing-point quantiles in Figure 5a,b exhibit heavy precipitation forecasts over the whole of France except Brittany and the South-East, with larger values in $q_{\mathrm{cpf}}^O$. In Figure 5(c), the conditional quantile shows a similar picture but with an intensity typically twice smaller than $q_{\mathrm{cpf}}^M$. In Figure 5(d), the maximum of all members forecast more than 5$mm/24h$ of rain over the whole country with large value (greater than 40$mm/24h$) all along a South-West/North-East axis. In Figure 5e,f, the ensemble mean and control forecast, respectively, performed poorly at this lead time with generally lower precipitation rates than in the radar and missed events in the North-East.

Figure 6a,e shows the evolution of a crossing-point quantile forecast over consecutive runs. Typically, the adaptive-quantile forecast would indicate larger areas with potentially extreme events at longer lead times (when the forecast uncertainty is large). As we approach the time of the event, the forecast is sharper with smaller areas indicating the potential occurrence of an extreme event. The forecast appears also less noisy at shorter lead times. More ensemble members would allow better capturing the tail of the forecast distribution and thus help reduce the forecast noise in the medium range.

3.3. Forecast Consistency

Is one type of forecast generally overestimating or underestimating the observed precipitation? The examples in Figure 4 and Figure 5 seem to indicate that this is the case but to answer this question systematically, we compare the distributions of the forecast and the observation. Here, we define equally-spaced bins and count the number of cases falling in each bin over the verification period. The results are shown in Figure 7.

In Figure 7(a), the forecast distributions at day 1 generally resemble the observation distribution with 2 striking exceptions: the ensemble mean ($em$) tends to systematically underestimate the highest precipitation while the maximum of all ensemble members ($mx$) tends to systematically overestimate it. It is worth noting that the control forecast ($ctrl$) shows good consistency with the observations. The control forecast is here the only point-forecast that is also a coherent scenario (in space and time).

As a consequence, at day 5, in Figure 7(b), the control forecast is the only forecast which still has some resemblance with the observation in terms of distribution. The other point-forecast distributions are somehow modulated by the uncertainty in the forecast at that time range. Ensemble mean ($em$) and conditional quantile ($q_{70}^c$) show a clear underestimation of larger precipitation values while the ensemble maximum tends to underestimate low precipitation amounts. Focusing on the tail of the observation distribution, we note that only the self-adaptive quantiles and the maximum of all members are able to generate the highest amount of precipitation observed in the radar (above 100mm/24h). By construction, it is expected that an optimal forecast in terms of PSS overestimate rare events (see Appendix A.3).

4. A Quantitative Assessment

A quantitative assessment complements the qualitative comparison based on case studies and the consistency check presented in Section 3. In the following, we discuss averaged scores computed for 3 consecutive summers, that is June, July, and August in 2022, 2023, and 2024.

4.1. Optimal Point-Forecasts

A first result was already presented in Figure 1: the ensemble mean is the best forecast in terms of RMSE (Figure 1a) while the ensemble maximum is the best forecast in terms of QSmax (Figure 1b). The crossing-point quantile forecasts $q_{cpf}^M$ and $q_{cpf}^O$ are “intermediate” forecasts for both metrics with a larger variability in $q_{cpf}^O$ leading to worse RMSE performance.

In theory, the crossing-point forecast should be the optimal forecast in terms of PSS for any event as discussed in Section 3.3. Whether this property holds in practice is explored in Figure 8. The Peirce skill score (equivalently the area under the ROC curve) are shown for a range of event thresholds, from 0.1mm/24h up to 40mm/24h, and for a range of forecast lead times, from day 1 to day 7. Figure 8(a) focuses on one lead time (day 5) while Figure 8(b) focuses on one event threshold (10mm/24h).

Figure 8 supports our theoretical findings: the crossing-point forecast is the best forecast in terms of PSS among the point-forecasts compared here. This result holds for all lead times. Importantly, we note that calibration plays a key role to reach this level of skill: calibration is a necessary condition for optimality.

Focusing on day 5, Figure 9 show ROC curves for 2 events: precipitation exceeding 1mm/24h and 10mm/24h in Figure 9a,b, respectively. Crossing-point forecasts have a ROC point close to the descending diagonal for both thresholds where results for optimal forecasts should lie. Typically, the ensemble maximum has a higher false alarm rate than optimal for smaller thresholds while the ensemble mean and control forecast tends to have a smaller hit rate than optimal for higher thresholds. Also, the impact of calibration is visible in Figure 9(b) with a ROC point for $q_{cpf}^O$ closer to the diagonal than $q_{cpf}^M$: calibration is needed to ensure optimality for higher event thresholds.

4.2. Economic Value

The economic value of a forecast is estimated here with the help of a simple cost-loss model. We consider the case where a forecast user can mitigate the loss associated with the occurrence of a weather event by taking a preventive action. The user decides to take action or not based on the forecast. The key parameters of this model are 1) the cost of taking action and 2) the loss encountered if no protective action is taken but the event occurs. The cost-loss ratio characterizes the user in the sense that it reflects a specific user-defined application. Building on this model, the so-called potential economic value is derived from the elements of the contingency table [26].

Figure 10 compares the economic value of different types of point-forecasts as a function of the user’s cost-loss ratio. Figure 10a,b focus on 2 different events defined as precipitation exceeding 1mm/24h and 10mm/24h, respectively. Both plots are results for forecasts at a lead time of 5 days. Note that a logarithmic scale is used for the x-axis in Figure 10 that put more emphasis on low cost-loss ratios. The ensemble maximum has a higher economic value for very low cost-ratios, adaptive quantile forecasts has a higher economic value than the other forecasts for cost-loss ratios in the range of the event base-rate (indicated by a vertical line). For applications with larger cost-loss ratios, the conditional quantile and ensemble mean are better choices for decision-making.

5. Summary and Outlook

The crossing-point is the intersection point between 2 cumulative probability distributions, a forecast and a climatology. The corresponding quantile varies with the forecast uncertainty of the day. This self-adaptive quantile forecast is called the crossing-point quantile forecast. In our study, we discussed the properties and performance of this new type of point-forecast. Our summary points and outlook are the following:

• Elicitation. The link between crossing-point forecast and PSS is similar to the link between mean forecast and RMSE, or median forecast and MAE. Indeed, we demonstrate that the crossing-point quantile is the optimal forecast in terms of PSS (and equivalently in terms of ROC area) for any event. Further, the crossing-point quantile links the PSS with the Diagonal score, which can be interpreted as a weighted average of AUC, as shown in Appendix A.4.
• Extreme events. Optimality for any event includes optimality for rare and extreme events. We showcase the crossing-point performance with case studies and statistical analysis that support our theoretical findings. The predictive power of the crossing-point forecast for extreme events is also explored in [27].
• Condition of optimality. The necessary condition for optimality is calibration. The ensemble or probabilistic forecast needs to be well-calibrated to ensure a theoretical guarantee of optimality. When necessary, statistical post-processing can be applied to reach that goal [28].
• Model climate. When post-processing is used to calibrate the forecast, there is no need to build an M-climate based on reforecasts (which can be expensive). After calibration in the observation space, the climate distribution used to build the CPF would be directly estimated from observations for consistency. In the case of a well-calibrated system, model and observation have the same climatology.
• Observation climatology. The estimation of climate percentiles can be a moving target due to natural variability and anthropogenic climate change. When one build a climatology with limited observation records, one could consider using extreme value theory to better capture the tail of the distribution. For example, non-negative precipitation amounts can be fitted with an Extended Pareto Distribution as in Naveau et al. [29].
• Interpretation. Like any other quantile forecast, a self-adaptive quantile forecast based on the CPF is not a physically consistent spatial or temporal scenario. A crossing-point quantile forecast should be interpreted at each location (grid-point) as a local “probabilistic worst-case scenario”. In simple terms, the forecast indicates the most extreme event such that its likelihood is larger in the forecast than in the climatology [19].
• Usage. Self-adaptive quantile forecasts are particularly well-suited for users whose cost-loss ratio is unfocused, that is with a cost-loss ratio that decreases as the rarity of the event increases. As revealed by our economic value analysis, the crossing-point quantile forecast is not suitable for all users. In particular, users sensitive to false alarms might consider instead using the ensemble mean which displays large values only when the forecast uncertainty is small.
• Loss function. In principle, weighted proper scores could be used as loss functions to train weather forecasting models based on machine learning. For example, one could consider the Diagonal score which is consistent with the crossing-point forecast as a performance metric. The link between Diagonal score, PSS, and the coefficient of predictive ability [CPA, 30] is developed in Appendix A.4.
• Outlook. The quality of the CPF predominantly depends on the quality of the underlying ensemble prediction system. So, improvement for this new type of prediction would lie in 1) better raw (or post-processed) ensemble forecasts, and 2) more ensemble members for a finer estimation of the intersection point between distributions. Machine learning could help achieve these objectives in a near future.

Appendix A.1. Model and Radar Climatology

The model climatology, also known as M-climate, is built on 20 years of reforecasts. A period of 5 weeks centered around the current day is used to account for seasonal variations as described in [? ]. Similarly, an observation climatology is estimated for each week of the year using a 5 week rolling period over 20 years of records, from 2001 to 2020.

Appendix A.2. Ensemble Calibration

Ensemble calibration is based on a member-by-member approach [MBM, [? ]] and tested successfully on ensemble precipitation forecasts [e.g. ? ]. A correction is applied to each ensemble member individually with a component common to all members and a component that adjusts the deviation of a member with respect to the ensemble mean. If we denote ${\widehat{x}}_i$ the corrected forecast for the $m^{\mathrm{th}}$ member of the ensemble, MBM formally consists in applying:

$${\widehat{x}}_i=\alpha +\beta \overline{x}+\gamma (x_i-\overline{x}),$$

(A1)

where $x_i$ is the ensemble member i and $\overline{x}$ the ensemble mean. A clipping to zero is applied to the corrected forecast. The parameter $\alpha$ is the bias parameter that nudges the ensemble mean, $\beta$ is the linear coefficient that scales the ensemble mean, and $\gamma$ is the scaling parameter that adjusts the spread of the ensemble. The coefficients $\alpha$, $\beta$, and $\gamma$ are estimated over the summer of 2021 for all grid points together but for each forecast lead time independently.

Appendix A.3. Optimal forecast and frequency bias

Based on the contingency table entries A, B, C, and D, defined in Section 2.3, let’s define the frequency bias index ($FBI$) and the event based rate ($p_c$):

$$FBI:=(A+B)/(A+C),$$

(A2)

$$p_c:={\displaystyle \frac{A+C}{A+B+C+D}}$$

(A3)

The ROC point that maximizes PSS is on the second diagonal such that $HR+FAR=1$. It follows:

$$\begin{array}{cc}\hfill FBI& =H+{\displaystyle \frac{A}{A+C}},\hfill \\ \hfill & =1+{\displaystyle \frac{B}{A+C}}-{\displaystyle \frac{B}{B+D}},\hfill \\ \hfill & =1+FAR\left({\displaystyle \frac{B+D}{A+C}}-1\right),\hfill \\ \hfill & =1+FAR\left({\displaystyle \frac{1-p_c}{p_c}}-1\right).\hfill \end{array}$$

(A4)

So if $FAR$ is positive, events such that $p_c<0.5$ would tend to be over-represented ($FBI>1$) by an optimal forecast.

Appendix A.4. Diagonal score, Peirce skill score, and universal ROC

Following Ben Bouallegue [19], the Diagonal score $D_G$ is defined using elementary scores $d_G$ as follows:

$$D_G(F,y):={\int }_0^1{d}_G(F^{-1}(1-\alpha ),y)\mathrm{d}\alpha$$

(A5)

with G the climatology probability distribution, F the forecast probability distribution, and

$$d_G(F^{-1}(1-\alpha ),y)=\left\{\begin{array}{cc}1-\alpha \hfill & \mathrm{if} y>G^{-1}(1-\alpha )\ge F^{-1}(1-\alpha )\hfill \\ \alpha \hfill & \mathrm{if} F^{-1}(1-\alpha )>G^{-1}(1-\alpha )\ge y\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(A6)

When the observation ranges the unconditional distribution G, the expectation of the elementary Diagonal score is a function of PSS:

$$\mathbb{E}\left[d_G(F^{-1}(1-\alpha ),Y)\right]=\alpha (1-\alpha )[1-(H_{\alpha }-F_{\alpha })],$$

(A7)

with $H_{\alpha }=\mathbb{P}(F^{-1}(1-\alpha )>G^{-1}(1-\alpha )|Y>G^{-1}(1-\alpha ))$ and $F_{\alpha }=\mathbb{P}(F^{-1}(1-\alpha )>G^{-1}(1-\alpha )|Y\le G^{-1}(1-\alpha ))$, respectively the hit rate and the false alarm rate associated to the event $G^{-1}(1-\alpha )$. For the climatological forecast, $\mathbb{E}\left[d_G(G^{-1}(1-\alpha ),Y)\right]=\alpha (1-\alpha )$.

In this framework, PSS can be written as the elementary diagonal skill score :

$$H_{\alpha }-F_{\alpha }=1-{\displaystyle \frac{\mathbb{E}\left[d_G(F^{-1}(1-\alpha ),Y)\right]}{\mathbb{E}\left[d_G(G^{-1}(1-\alpha ),Y)\right]}}.$$

(A8)

Similarly, the expected Diagonal score can be understood as a sum of quadratic weighted PSS:

$$\mathbb{E}\left[D_G(F,Y)\right]={\int }_0^1\alpha (1-\alpha )[1-(H_{\alpha }-F_{\alpha })]\mathrm{d}\alpha .$$

(A9)

Thus the expected Diagonal score of a climate forecast G is $1/6$. This score is obtained for any non-informative forecast, that is a forecast for which $H_{\alpha }=F_{\alpha }$. In other words, the Diagonal score is equitable.

Moreover, Gneiting and Walz [30] define the coefficient of predictive ability (CPA) as a general measure of potential predictive ability. CPA takes value between $1/2$ and 1, and is equivalent to the area under the universal ROC, obtained by convex combinations of ROC curves for any event of interest. In the case of point forecasts, the CPA can be informally defined as follows, up to a multiplicative factor:

$$\begin{array}{cc}\hfill 2CPA-1& ={\int }_0^{1}6\alpha (1-\alpha )(H_{\alpha }-F_{\alpha })\mathrm{d}\alpha \hfill \\ \hfill & =1-6\mathbb{E}\left[D_G(F,Y)\right].\hfill \end{array}$$

(A10)

As a result, the general measure of predictive ability of Gneiting and Walz [30] is equivalent to the Diagonal skill score:

$$2CPA-1=1-{\displaystyle \frac{\mathbb{E}\left[D_G(F,Y)\right]}{\mathbb{E}\left[D_G(G,Y)\right]}}.$$

(A11)