The ECOMAG (ECOlogical Model for Applied Geophysics) is a semi-distributed process-based hydrological model describing snow accumulation and melt, soil freezing and thawing, water infiltration into unfrozen and frozen soil, evapotranspiration, the thermal and water regime of soil and the overland, subsurface and channel flow with a daily time step (Motovilov et al., 1999). The model accounts for measurable watershed characteristics such as surface elevation, slope, aspect, land cover and land use, soil and vegetation properties. The parameters are spatially distributed by partitioning the watershed into sub-basins (elementary basins). Parameterization of the sub-grid processes is described by Motovilov (2016). The model is driven by time series of daily air temperature, air humidity and precipitation intensity.

The model was applied at an earlier time for hydrological simulations in many river basins with highly varying sizes and characteristics – from small- to medium-sized European basins (Gottschalk et al., 2001) to the large Volga, Lena and Mackenzie basins with watershed areas exceeding 1 million km2 (Motovilov 2016; Gelfan et al., 2017).

In this study, a digital elevation model with 1 km × 1 km spatial resolution was used for the basin discretization and river network construction. A total of 1045 elementary basins were delineated, with an average area of 340 km2. The model forcing data for each elementary basin were interpolated from the 157 weather stations' data (see Fig. 1), employing the inverse distance method. Most parameters are physically meaningful and were derived through available measurements of the basin characteristics (topography, soil and vegetation properties).

The model was calibrated and validated against the Cheboksary Reservoir daily water inflow observations beginning from 1 January 1982 (the first year after the reservoir was filled to capacity) to 31 December 2016: the calibration covered the period of 2000–2010; the rest of the data were used for the model evaluation. The ECOMAG calibration procedure is described in detail by Gelfan et al. (2015). It is worth emphasizing two specific aspects concerning this procedure. First, the values of several key parameters pre-assigned from literature or from the available measurements are considered as the initial approximations of the optimal values, and the latter are sought within the neighbourhood of the initial, pre-assigned values. Second, during the calibration process, the ratios between the initial values of the distributed parameter corresponding to different soils, landscapes and vegetation are preserved. This approach allows for the integration of important hydrological knowledge into the optimization procedure. The Nash and Sutcliffe (1970) efficiency criterion NSE is adopted to represent the goodness of fit of the simulated and measured variables.

The Multi-Site FRagment-based stochastic Weather Generator (MSFR_WG) is a stochastic model that uses a Monte Carlo simulation to generate time series of daily weather variables (precipitation, air temperature and air humidity deficit), retaining statistical properties, both spatial and temporal, of the corresponding observed variables. This modelling procedure is based on the so-called “spatial fragments' (SFR) resampling method” initially presented by Gelfan et al. (2015). The SFR method is a modification of the temporal fragments' (TFR) method proposed by Svanidze (1980) for the stochastic simulation of highly autocorrelated time series.

The SFR resampling method includes the following steps.

N normalized fields (spatial fragments, SFRs) of weather variables are computed on the basis of the available meteorological data. SFRs are computed for each of N years of observations by dividing each daily value of the specific variable by the corresponding spatially averaged annual value.

The proposed ensemble forecasting procedure utilized in this study was verified by producing hindcasts of water inflow into the Cheboksary Reservoir from 1 April for 3 months ahead (up to 30 June). The hindcasts cover a 35-year period between 1982 and 2016. (Hereafter, we use the term “forecasts” for these hindcasts.) For each ith year of the verification period (i = 1, 2, …, 35), the procedure consists of the following steps:

Spin-up of ECOMAG-based simulations (“warm start”) is conducted using meteorological observations data prior to the forecast issue date (31 March) in order to calculate the initial watershed hydrological state (soil, snow and channel water contents, groundwater level, soil freezing depth, etc.) that initializes the forecast. The simulations start from the end of the previous freshet, i.e. 8–9 months before the forecast issue date.

To verify deterministic and probabilistic model-based forecasts, as well as to compare them with each other and with the operational data-driven forecast of water inflow into the Cheboksary Reservoir, we used the following, quite traditional, measures of the forecasts' efficiency and skill.

For the deterministic forecast verification, the mean error, relative bias, root-mean-squared error (RMSE) and Pearson's correlation coefficient r were used. In addition, for presentation, we used a Taylor diagram (Taylor, 2001), which combines three forecast characteristics in one chart, namely the forecast standard deviation, the RMSE and the correlation coefficient between the observations and the forecasted values.

For categorical forecast verification, we used measures that can be calculated from a contingency table (Ferro and Stephenson, 2011), such as the probability of detection (POD, which shows the correct forecast fraction of the observed events), the false alarm ratio (FAR, which shows the fraction of forecasts that did not occur), the frequency bias (which shows correspondence of the observed and the forecasted events), the Heidke skill score (HSS, which shows the advantage of the forecast as compared to a random forecast), the Hansen and Kuipers score (KSS, which can detect if the forecast is hedging) and the Symmetric Extremal Dependency Index (SEDI, which evaluates the performance of the forecast of rare binary events).

The probabilistic ensemble forecasts' performance was assessed by several verification measures. The ability of forecasts to correctly predict the category of events that occurred within several categories was measured by the ranked probability score (RPS) (Wilks, 1995), which can also be treated as the mean squared error of the probabilistic forecast. The probability forecast efficiency relating to streamflow climatology was measured by the ranked probability skill score (Wilks, 1995). To visualize the specifics of probabilistic forecasts, three diagrams were employed. A predictive Q–Q (quantile–quantile) plot (Laio and Tamea, 2007) was used to assess the degree of correspondence between the cumulative distribution function of predictions and the observed values. A reliability diagram (Hartmann et al., 2002) was used to plot the forecast probability against the relative frequency of the observations in the corresponding forecast probability bin. Finally, the discrimination diagram (Wilks, 1995) was used to show the frequency of each forecast probability for events and non-events.

A full list of the aforementioned verification measures and their formulations, units and value ranges are presented in Table S1 (Supplement).

We verified the two types of the ensemble forecasts (ESP-based and WG-based) and compared them with each other and with the operational forecasts of water inflow into the Cheboksary Reservoir for April–June 1982–2016. To make a deterministic forecast, the forecasted inflow characteristics were averaged over the corresponding ensembles (51-member in the case of the ESP-based forecast and 1000-member for the WG-based forecast) to produce a single-value forecast of the desired characteristic: W, Qmax, Nq, Nqmax. Operational forecasts of inflow volume for the same April–June periods of 1982–2016 were obtained from official Roshydromet forecast bulletins (reports). All forecasts were analysed to assess the forecast performance measures: the mean absolute error, the bias and the RMSE. The results are presented in Table 1.

First, the deterministic forecasts of the inflow in April–June were compared to the operational forecasts for 1982–2016. As shown in Table 1, the mean error of the operational forecasts appears to be quite low (around 1 %) and close to those of the ESP and WG ensemble average values. However, the operational forecasts' RMSE values are significantly higher than those of the ESP and WG forecasts and account for almost 70 % of the observed inflow volume variability σW = 9.61 km3. For the ESP- and WG-based forecasts these values are around half of σW.

Figure 4 compares the inflow volume forecast errors of the operational forecasts (Fig. 4a) with the ESP-based forecast errors (Fig. 4b). The shaded area in the figures represents the area of the acceptable error [-0.674σW; 0.674σW] = [-6.48; 6.48 km3]. In Russian operational forecasting practice, a forecast is considered acceptable if its error falls into this area, and the forecast acceptability is calculated as the ratio of the acceptable forecasts to the whole number of forecasts. According to the assumption of the Gaussian distribution of the forecast errors, 50 % of the forecasts by climatology should fall into this interval. It can be seen from Fig. 4 that 5 of the 35 ESP-based forecasts (in 1985, 1994, 2002, 2005 and 2011) and every third (12 of 35) operational forecasts were not acceptable; i.e. the ESP-based forecast acceptability is 89 % and that of the operational forecast is 66 %. Note that the unacceptable forecasts in both cases occurred in the years when the spring precipitation amount was notably different from the corresponding climatic mean.

To compare the ESP-based and the WG-based forecasts, we present them in the form of a Taylor diagram (Fig. 5; Taylor, 2001), which combines three forecast characteristics in one chart, namely, the forecast standard deviation, RMSE and the correlation coefficient between the observed and the forecasted values of the inflow characteristics. The values of all characteristics are normalized by dividing the RMSE by the standard deviation of the observations. This normalization provides a demonstration of the forecast efficiency expressed in fractions of the observed standard deviation. As long as the forecast RMSE is less than the standard deviation of the observations, the forecast can be considered efficient against climatology.

It can be seen from Fig. 5 that the ESP-based forecasts of W, Qmax and Nqmax are slightly better correlated with the observations than the WG-based forecasts. Pearson's r values of the ESP-based forecasts are over 0.8 for all characteristics, except for Nq. Forecasts of Nq are less correlated with the observations, with r values for ESP-based and WG-based forecasts equal to 0.73 and 0.63, respectively. Forecasts of Qmax and Nqmax show normalized RMSE values around 58–67 % of the standard deviation of the corresponding observed characteristics.

For the purpose of reservoir management, it is often crucial to determine whether the expected inflow characteristic will exceed the corresponding mean value. To verify the methodology's capability of predicting this exceedance, the observations and forecasts were converted into binary vectors, with a value of 0 representing the event of non-exceedance of the mean annual value and a value of 1 representing the event occurrence. For example, for W, the event occurs with the exceedance of mean inflow volume during April–June. The forecast binary measures assessed with the use of the contingency tables and described in Table S1 are shown in Table 2.

The forecasts show good detection estimates (even perfect for Qmax) for both model-based methodologies. However, as the frequency bias is high, this might be the result of overprediction, as with the high values of the FAR and KSS. For W and Qmax, the forecast accuracy with a HSS of around 60 % is better than the accuracy of random chance; this means that the forecast is capable of detecting the occurrence of rare extreme events, which is shown by high values of the SEDI. Overall, the presented binary verification measures demonstrate a slight advantage of the ESP-based forecasts over the WG-based forecasts, though the differences are not substantial.

Binary measures of the operational forecasts of inflow volume are worse than those of the model-based forecasts. For instance, only 69 % of the observed events (exceedance of mean inflow volume) are correctly forecasted by the current operational methodology, and its accuracy relative to that of random chance is less then 50 %. The ability of a user to detect rare events on the basis of the operational forecast is also much lower than with the help of ensemble forecasts.

Thus, both continuous (Table 1) and binary (Table 2) model-based forecasts of inflow volume appear to be more preferable, in general, than the corresponding operational forecasts. However, as one can see from Fig. 4, there were several years when the operational forecasts were more accurate (in terms of the absolute error) than the ensemble ones. We found that most often the operational forecast outperforms the model-based forecast in those years when the modelled initial snow water equivalent (SWE) on the forecast issue date notably differed from the observed SWE. Since the latter is the main factor affecting the freshet volume, more accurate (observed) initial snow conditions used in Eq. (2) resulted in a more accurate forecast than the one initiated from the simulated SWE.

In the beginning of spring 2017, the basin's pre-melt conditions were close to climatology: snow water storage was 10 to 15 % above the long-term mean value, and soil water content and freezing depth were close to the corresponding mean values. However, the weather conditions during the spring freshet formation appeared to be significantly different from climatology. Anomalous warm and sunny weather that settled over the basin in the first half of March led to the commencement of snowmelt and river stage ascent at least half a month earlier than the mean dates. The last decade of months of March was, on the contrary, cold and damp, and the precipitation amount was twice above normal for this period. As a result, by the end of March the inflow volume (5.13 km3) into the reservoir exceeded mean March inflow by 32 %. Periods of intense snowmelt interchanged with cold spells and a large amount of precipitation, including snowfall during April and May 2017 (a number of stations even registered snowfall in June). Such diversity in weather conditions during the snowmelt period and their difference from the climatology resulted in a rather untypical regime of inflow into the Cheboksary Reservoir.

The ESP-based forecasting technique was tested in operational mode during the freshet period of 2017. The forecasts were issued on 1, 15 and 27 March for the period from 1 April till 30 June. Figure 6 shows daily forecast ensembles for this period compared to the observed inflow data.

Figure 6 shows the outcome of the anomalous weather conditions that led to an earlier increase of the inflow in mid-March (see Fig. 6a), which was not captured by the mean ensemble hydrograph of the forecast issued on 1 March. However, several scenarios of the ensemble show the behaviour of inflow to be similar to that observed. The forecast issued on 27 March showed the ongoing increase in inflow discharge; however the colder weather conditions led to inflow stabilization, not captured by the forecast. One can see visible improvement of the mean ensemble hydrograph issued on 27 March (Fig. 6b) compared with the one issued on 1 March (Fig. 6a).

Box plots of the ESP-based forecasts of different inflow characteristics are presented in Fig. 7. All forecasts of inflow volume showed low errors (Fig. 7a), unlike the maximum discharge forecasts (Fig. 7b); however, the Qmax forecast range envelops the observed maximum inflow discharge. Both forecasts of number of days over thresholds showed low errors (Fig. 7b and d); e.g. just before the beginning of April we correctly forecasted a low freshet with the absence of days when inflow discharge exceeds the mean maximum discharge for the period of observations.

In 2017, Roshydromet also issued a forecast of spring water inflow into the Cheboksary Reservoir on the basis of the methodology presented above (Fig. 8). In contrast with the results presented in Table 1 and Fig. 4, which demonstrate the general advantage of the ESP-based forecasts over the operational forecasts for 1982–2016, in 2017, the operational forecast of the inflow volume appears to be better. A possible explanation is again found in the simulation errors of the pre-melt SWE used as initial conditions for the ESP-based forecast.

In this section, the operational forecast, which is issued in deterministic form only, is not discussed.

One of the main advantages of ensemble forecasting is the ability to assess the uncertainty that is nested in the future possible behaviour of the hydrological system. The resulting ensemble is used to create cumulative distribution functions (CDFs) of the desired characteristic in jth forecast as 4FM(j)=∑i=1mfi(j),m=1…,M;j=1,…,N, where M refers to the forecast probability bins on the interval [0; 1], N is the total number of forecasts and fi is the probability of forecast in mth bin.

CDFs of the forecasted inflow volume W for the period from 1 April to 30 June of 35 years (1982–2016) are shown in Fig. 9. Three CDFs are combined in each plot: two CDFs of forecasts calculated under ESP-based and WG-based weather scenarios and the CDF of the observed inflow volume in the specific year (CDFs of observations can be represented as the Heaviside step function). One can see from Fig. 9 that for most of the years, the inflow is not far from the most probable one; in other words, the CDF of the forecasts crosses the CDF of observations at around 50 % probability. For almost all years observed inflow lies within the range of the ensemble. Exceptions are 1994, 2002, 2005 and 2011; i.e. once every 8–9 years, on average, the ensemble forecast range does not cover the observed inflow because of large forecast errors.

To quantify the ability of forecasts to predict the probability of an event occurring within the pre-assigned inflow categories, we used the RPS measure. The forecast efficiency was measured by the RPSS criterion, relating the verified forecast to streamflow climatology (both RPS and RPSS formulations are presented in Table S1).

Both forecasts demonstrate a moderate improvement over climatology: according to the RPSS value, around 30 % on average both for W and for Qmax (Table 3). Probably, accounting for a seasonal weather forecast and conditioning historical weather patterns on this forecast could result in a greater improvement over the streamflow climatology; however a reliable seasonal weather forecast for the study region is not available.

In addition, we compared the ESP-based and the WG-based forecasts by setting the former one as a reference forecast. The modified RPSS is formulated in this case as 5RPSSmodified=1-RPSWGRPSESP. As one might expect from a comparison of the unmodified RPSS measures, the modified one showed that the WG forecasts are less skilful than the ESP, with modified RPSS values of -0.16 for W and -0.13 for Qmax.

To compare quantiles of the forecasted characteristics with the quantiles of the corresponding observations, we used the predictive Q–Q plot (Laio and Tamea, 2007). As one can see from Fig. 10a, the predictive Q–Q plot of the inflow volume forecasts demonstrates good agreement with the distribution of the observations. This is fairly consistent for both methodologies and for all quantiles, but for rare events there is an underestimation of the predictive uncertainty, expressed as an offset from the 1 : 1 line in the upper right corner of the plot. For the maximum inflow discharge (Fig. 10b), one can see overprediction in both methodologies. However, the behaviour of ESP-based and WG-based forecasts of rare events is different in terms of predictive uncertainty. In particular, the WG-based forecasts of the events of low exceedance probability appear to be closer to the 1 : 1 line.

Additionally, comparisons between the ensemble forecasts of both types can be made based on the reliability and discrimination diagrams presented in Figs. S9–S12.

Overall, all presented measures of the probabilistic forecast performance are slightly better for the ESP-based forecasts than for the WG-based forecasts, though the differences are not significant and hardly interpretable. At the same time, verification measures obtained from the large ensemble of the WG-based forecasts are expected to be more statistically reliable, which is demonstrated in the next section for the two measures, CDF and RPSS.

It can be seen from Fig. 9 that the CDFs appear to be close to each other for both ensemble methodologies used. However, the sample variance of the CDF is significantly different due to a different number of scenarios in the ensembles: 51 in the ESP-based ensemble compared to 1000 in the WG-based ensemble. To illustrate this difference, we assessed confidence bands for CDFs derived from both forecasting approaches. Two-sided confidence bands were expressed through the Dvoretzky–Kiefer–Wolfowitz inequality as (e.g. Massart, 1990) 6Psup⁡x∈RF^n(x)-F(x)≥ε≤2exp⁡-2nε2, where F(x) and F^n(x) are the CDF's ordinate and its empirical estimation from a sample of size n, respectively; ε is the constant depending on the significance level α as 7ε=12nln⁡21-α. For the pre-assigned confidence probability p = (1 - α), the upper (U(x)) and the lower (L(x)) confidence bands of the empirical CDF F^n(x) are defined from Eqs. (5) and (6) as 8U(x)=min[F^(x)+ε,1]9L(x)=max[F^(x)-ε,0]. Figure 11 demonstrates the difference between 95 % confidence intervals of the ESP-based inflow volume forecast as compared to the corresponding intervals of the WG-based forecast. We believe that the presence of the mentioned difference should be taken into account by the ensemble forecast developers when they use statistical verification measures for the assessment of forecast performance, as well as by the users when they interpret the forecasts.

One can conclude from Table 3 that the RPSS criterion demonstrates the advantage of the ESP-based probabilistic forecast over the WG-based one as compared to climatology. However, it is important to take into account that the RPSS measure is strongly dependent on ensemble size and negatively biased (see, for instance, Müller et al., 2005; Weigel et al., 2007). A de-biased estimate of the RPSS can be formulated as by Weigel et al. (2007): 10RPSSD=1-RPSRPSref+D, where D is the correction term depending on the ensemble size, the climatological probabilities and the number of categories. For a very large ensemble size, the correction term D converges toward zero and the RPSSD converges towards the RPSS.

Figure 12 demonstrates the dependence of the RPSS bias on sample size built with the use of the approximation of D presented in Weigel et al. (2007).

One can see from this Fig. 12 that using the 51-member ensemble (i.e. the ESP-based ensemble), the bias can reach tens of percent depending on the RPSS estimate. Using the 1000-member ensemble, the bias is close to zero.