Twenty-three forecast locations are selected for this study, including forecast locations in all states: Queensland, New South Wales, Victoria, Tasmania, South Australia, and Western Australia, plus the Northern Territory. Flows at the forecast locations are a mixture of perennial, ephemeral and intermittent flows. Table 1 summarises information about the forecast locations, including long catchment name, short ID, state (in the jurisdictional sense), upstream catchment area, and approximate centroid latitude and longitude. The catchments range in size from 102 to 36 230 km2. Furthermore, the forecast locations are plotted on a map of Australia in Fig. 1, which also indicates whether the catchment is in a temperate, subtropical or tropical climate zone.

Daily streamflow data for the period 1982–2012 are sourced from the Bureau of Meteorology for all 23 forecast locations. The bureau sources the data from the respective data owners and performs quality control. Daily streamflow data records often contain missing values, for example, due to the failure of gauging equipment. The bureau partially infills missing data records. Records for all forecast locations are infilled by linear interpolation if the number of missing days is equal to or less than 3 days. For some forecast locations, records are infilled by linear regression with nearby forecast locations if the number of missing days is equal to or less than 14 days.

In this study, the daily streamflow data are aggregated to 28-day totals for use as predictors and 3-month totals (always beginning on the first day of the month) for use as predictands. Many of the data records still contain missing data after the infilling process; however the proportion of missing data is never greater than 10 %.

The bureau's current BJP seasonal streamflow forecasting models rely on predictors, including surface and subsurface ocean temperatures, the SOI and AAO. The predictors are identified through a rigorous predictor selection process (Robertson and Wang, 2012). In this study, we adopt the climate predictors used by the bureau in their operational models with some changes. The biggest change is that we restrict the set of climate predictors to SST predictors, substituting an ENSO SST index wherever the climate predictor in the bureau's operational model is subsurface ocean temperatures or SOI. SST climate predictors representing ENSO and the Indian Ocean state are likely to be stable across a period of several weeks and are known to have strong lagged relationships with Australian rainfall up to 3 months ahead (Schepen et al., 2012). Therefore, the predictors are likely to remain valid predictors as forecast lead time is increased by a few weeks. On the other hand, it is less certain that the AAO, representing the Southern Annular Mode, will remain a valid climate predictor as lead time is increased since it has a weak lagging relationship with Australian seasonal rainfall (Schepen et al., 2012). In cases where AAO is a climate predictor in the bureau's operational model, no climate predictor is used in this study.

Daily SST data are obtained from the NOAA 1 / 4∘ daily Optimum Interpolation Sea Surface Temperature (daily OISST) (Reynolds et al., 2007). Full years of data are available from 1982 onwards. The daily OISST is constructed by combining data from satellites, ships, buoys, probes and ocean-dwelling robots. Interpolation fills in missing gaps and ensures a complete historical record with no missing data. Compared to monthly ERSST records, which go back to 1854, the daily OISST record is relatively short, but it is necessary for generating timelier forecasts. To obtain SST climate predictors, we calculate 28-day average sea surface temperatures relative to 1982–2010 climatology. Area averaging is used to obtain the set of climate predictors: Niño3.4, Niño3, Niño4, the El Niño Modoki index (Ashok et al., 2007), the Indian Ocean Dipole (Saji et al., 1999) and the Indonesian index (Verdon and Franks, 2005). Figure 2 demonstrates that the monthly Niño3.4 values calculated from daily OISST and NOAA's monthly ERSSTv4 product are almost identical for the period 1982–2011 with R2 = 0.98.

Forecasting models are set up using the BJP modelling approach (Wang and Robertson, 2011; Wang et al., 2009). Separate forecasting models are established for each forecast location (23 locations), season (12 3-month seasons from the start of each month) and for 0–14- and 21-day lead time (16 lead times).

The BJP forecasting models make use of two types of predictors: predictors representing initial catchment conditions and predictors representing the climate state. For all models, the initial catchment condition predictor is fixed to the previous 28 days' total flow volume. This predictor will not be optimum for all forecast locations; however, it is a pragmatic choice that is likely to represent the overall catchment wetness reasonably well. Additionally, it gives a consistent model set-up without needing to undertake initial catchment condition predictor selection. SST climate predictors are adapted from the bureau's current operational forecasting models as described in Sect. 2.3. The same climate predictors are applied at all lead times to ensure a consistent model set-up.

The full mathematical formulation of the BJP modelling approach is presented in Wang et al. (2009) and Wang and Robertson (2011). Here, we note some key features of the BJP modelling approach. The BJP models are able to effectively handle missing and non-concurrent records. The BJP models are based upon the multivariate normal distribution after allowing for data transformation using either the log-sinh (Wang et al., 2012) or Yeo–Johnson (Yeo and Johnson, 2000) transformations. If a set of variables follows a multivariate normal distribution, then a subset of those variables also follows a multivariate normal distribution. Thus the many instances of missing data in streamflow records, as described in Sect. 2.2, are easily handled. Several of the forecast locations experience ephemeral and intermittent flows, which can result in a probability mass for zero flows. This problem is handled in the BJP modelling approach by treating zero flows as censored data, meaning that the observations of zero flow are treated as being of unknown precise value equal to or below zero. Uncertainty in the model parameters due to the short data records is handled by inferring parameters using Markov chain Monte Carlo methods (MCMC). Probabilistic (ensemble) forecasts are produced using conditional multivariate normal distribution equations. When predictor–predictand relationships are weak, the BJP modelling approach is designed to produce reliable forecasts that approximate climatology (i.e. the frequency distribution of historically observed streamflow).

Some aspects of the BJP implementation used in this study vary compared with those published in Wang et al. (2009) and Wang and Robertson (2011). We use fixed transformation parameters together with simplified parameter reparameterisations for more straightforward numerical inference (Zhao et al., 2016). The changes reflect our experience in achieving more robust and efficient BJP modelling.

In this study, we assess the performance of forecasts for the period from JFM 1982 to DJF 2011–2012. A separate BJP model is established for each season, forecast location, and lead time. Across different years, the model parameter inference and forecast process are cross-validated using leave-five-years-out cross-validation. For each historical forecast event to be tested, the data points for the year to be forecast plus the subsequent 4 years are left out. The leave-five-years-out procedure is designed to account for strong persistence in streamflows, potentially over many years.

Forecasts from the BJP modelling approach are probabilistic. The continuous ranked probability score (CRPS; Matheson and Winkler, 1976) is used to assess full forecast probability distributions, therefore involving forecast ensemble spread as well as forecast accuracy. The CRPS for a given forecast and observation is defined as CRPS=∫F(y)-Hy-yobs2dy, where y is the forecast variable; yobs is the observed value; F(.) is the forecast cumulative distribution function (CDF); and H(.) is the Heaviside step function, which equals 0 if y < yobs and equals 1 otherwise. Model forecasts are compared to reference forecasts by calculating skill scores: CRPSskillscore=CRPSref‾-CRPS‾CRPSref‾×100(%), where the overbar indicates averaging across a set of events. A climatology forecast is used as the reference in this study, although one may choose to use other reference forecasts. The CRPS skill score is positively oriented (whereas CRPS is negatively oriented). As a percentage, a maximum score of 100 is indicative of perfect forecasts. A score of 0 indicates no overall improvement compared to the reference forecast. A negative score indicates poor-quality forecasts in the sense that a naïve climatology forecast is more skilful. CRPS skill scores are calculated for each catchment and season.

Reliability refers to the statistical agreement of forecast probabilities with observed relative frequencies of events, which can be checked using probability integral transforms (PITs). At each lead time, the PIT values are calculated from the BJP-generated ensemble forecasts for every forecast event, forecast location and season, and pooled in the analysis. The PIT represents the non-exceedance probability of observed streamflow obtained from the CDF of ensemble forecast. If the ensemble spread is appropriate and the forecasts are free of bias, then observations will be contained within the forecast ensemble spread. Reliable forecasts are evidenced by PIT values that follow a uniform distribution between 0 and 1.

Sharpness refers to the concentration of the ensemble members and is a property of the forecast only. Forecast sharpness is desirable provided the forecast is reliable. A common measure of sharpness is the width or, more pertinently, the relative width of a forecast quantile range. Comparisons between forecasts can be made by comparing average widths or using a sharpness box plot (Gneiting et al., 2007). In our study, we compare the quantile range widths of forecasts with a lead time of 1–21 days with the quantile range widths of forecasts with a 0-day lead time. Averages are taken for each catchment and season.

We first assess the reliability of the forecasts for the different lead times. Histograms of PIT values are constructed for each lead time from 0 to 14 days (Fig. 3). For each lead time, the PIT histograms follow a similar distribution: approximately uniform but not perfectly so. There are a couple of reasons why the PIT histograms are not perfectly uniform. First, it is very difficult to model the shape of the forecast distributions perfectly, particularly under cross-validation. Second, a prior distribution over the data transformation parameters is influential in preventing very skewed distributions when there are limited data or many instances of zero flows. Thus, the model attempts to reliably reflect uncertainty but cannot tightly fit the shape of the data in all cases.

On the whole, the PIT values suggest that the forecasts capture the range of observations sufficiently well, and there is no strong evidence of bias. Therefore, the spread of the ensembles is deemed to be appropriate. However, note that the shape of the PIT histograms suggests that an alternative distribution, for example a fat-tailed distribution, may be able to fit the tails of high-flow events better. The PIT histograms exhibit a similar shape for all lead times, meaning that the forecasts are similarly reliable across all lead times.

We now assess the accuracy skill of the forecasts for the different lead times. To assess forecast accuracy, we calculate CRPS for each forecast location, season and lead time. It is possible to decompose the CRPS into components reflecting accuracy and reliability. Results in the previous section demonstrate that the forecasts are similarly reliable across all times. Therefore differences in the CRPS will mainly reflect differences in forecast accuracy. CRPS scores are in the original units of the data, and therefore it is difficult to compare across forecast locations and seasons. CRPS skill scores, as defined in Eq. (2), however, measure the percentage reduction in error relative to a reference forecast, and therefore it is easier to compare relative skill levels across forecast locations, seasons and lead times. Hereafter, we consider only CRPS skill scores with modelled leave-five-years-out climatological distributions as reference forecasts unless otherwise specified.

CRPS skill scores for forecasts with 0-day lead time are plotted in Fig. 4. Forecast locations form rows, and target seasons form columns. The CRPS skill scores vary highly between forecast locations and seasons, due to variations in catchment memory and climate predictability. The sources of predictability are not the main concern here though. Here, the aim is to analyse the changes in forecasting skill with increasing lead time. As an example, forecasts with 7-day lead time are plotted in Fig. 5. By visual comparison, skill is overall reduced; however there are individual cases where skill increases. Instances of positive CRPS skill score differences are conceivable given the small sample size (30 years) and associated uncertainty in the skill score. The patterns of skill across forecast locations and seasons are largely consistent at 0- and 7-day lead time. Overall, increasing the forecast lead time to 7 days does not appear to significantly decrease forecasting skill. To investigate further, CRPS skill scores for 7-day-lead-time forecasts are calculated with the 0-day-lead-time forecasts as the reference rather than climatology (Fig. 6). The median CRPS skill score (considering all catchments and seasons) is -1.8. Catchment by catchment, the median CRPS skill score across all seasons ranges from -5.1 to 0.8, indicating that CRPS scores typically increase by less than about 5 %, although larger increases in errors are observed in individual catchments and seasons.

Figure 7 illustrates for lead times of 0, 7, and 14 days the proportion of cases (catchments and seasons) where CRPS skill score thresholds are exceeded. CRPS skill scores between -5 and 5 are considered unskilful, below -5 is considered negative skill, and above 5 is considered skilful. The proportion of cases where certain CRPS skill score thresholds are exceeded decreases as the lead time increases. A CRPS skill score of -5 is exceeded in approximately 95, 94, and 92 % of cases for 0-, 7-, and 14-day lead time respectively, indicating only a small proportion of cases exhibit skill worse than climatology. The small difference in exceedance probabilities with increasing lead time is consistent with the knowledge that the BJP modelling approach should produce forecasts approximating climatology in the absence of any real forecasting skill. Under stringent leave-five-years-out cross-validation, instances of negative skill can occur for various reasons related to aridity, poor catchment memory, extreme events and data problems. A CRPS skill score of 5 is exceeded in approximately 66, 59, and 57 % of cases for 0-, 7-, and 14-day lead time respectively. The larger difference in exceedance probabilities suggests that the number of days of lead time is important for skilful forecast locations and catchments.

To evaluate the change in forecast skill for each 1-day increase in lead time, we calculate the difference between CRPS skill scores for 1–14- and 21-day lead times with CRPS skill scores at lead time 0. For each lead time, the skill score differences for all forecast locations and seasons are pooled. Simplified box plots of the CRPS skill score differences for each lead time are plotted (Fig. 8). In each plot, the boxes represent the [0.25, 0.75] and [0.10, 0.90] quantile ranges, with the median marked by the line crossing the box. Outliers are ignored. The median of the box plots indicates a clear monotonic trend, demonstrating that, on average, skill decreases as every 1-day forecast lead time increases. However, for individual forecast locations and seasons, the pattern of changes in skill is more variable. In some cases, the CRPS skill score can decrease by up to 18 percentage points (pp) as lead time increases up to 14 days. However, CRPS skill scores can increase by up to 10 pp as lead time increases up to 14 days. The trend of decreasing CRPS skill scores continues to 21-day lead time.

From Figs. 4 and 5, it is evident that many forecast locations and seasons have low skill. We now partition the catchment and seasons into two groups based on CRPS skill scores at 0-day lead time. If the CRPS skill score for a catchment and season at 0-day lead time is > 5, the case is assigned to the skilful group. Otherwise, the case is assigned to the unskilful group. Figure 9 plots the change in forecast skill for each 1-day increase in lead time, considering only skilful cases. The reduction in skill is more marked for skilful cases compared with all cases. Figure 10 plots the change in forecast skill for each 1-day increase in lead time, considering only unskilful cases. The average reduction in CRPS skill scores is near 0, with variation mainly within ±5 pp. The results in Fig. 9 (considering only skilful cases) are arguably the most important for analysing reductions in skill because the bureau issues climatology forecasts when hindcast skill is poor. From Fig. 9, at 7-day lead time the mean reduction in CRPS skill scores is about 4 pp, whereas at 14-day lead time it is about 6 pp. At 21 days the reduction in CRPS skill scores averages 7.5 pp, with few cases associated with skill improvements.

Forecast sharpness is a property of the forecasts only and relates to the narrowness of the forecast ensemble spread. That is, a forecast with a narrower spread is sharper than a forecast with a wider spread. We assess how forecast sharpness changes as forecast lead time is increased. For each forecast event, the [0.1, 0.9] quantile range of 1–21-day-lead-time forecasts is divided by the [0.1, 0.9] quantile range for the 0-day-lead-time forecast. An average is then calculated for each catchment and season, which is termed the average relative width (ARW). Simplified box plots of the ARW for each lead time are plotted (Fig. 11). In each plot, the boxes represent the [0.25, 0.75] and [0.05, 0.95] quantile ranges of ARW, with the median marked by the line crossing the box. Sharpness is seen to decrease gradually as forecast lead time is increased. At 7-day lead time, the forecasts are typically only marginally wider than at 0-day lead time, with the median ARW being approximately 1.02. However, the forecasts can be considerably wider in some cases, with the values of ARW up to 1.2 quite possible in some catchments and seasons. At 21-day lead time the median ARW increases to 1.08.