In the remainder of the text, we use the following definitions:

Statistics: summary statistics derived from a time series (e.g., the long-term mean of flow observations, the daily median flow).

We selected simulations from the National Water Model v3.0 (NWMv3.0) as a practical test case for our work, to investigate our hypothesis that deliberate use of benchmarks can help identify areas for model improvement. The National Water Model is used to generate operational forecasts across the United States, and is primarily designed to produce short-range and medium-range (18 h to 10 d) sub-daily streamflow forecasts. These forecasts are available for approximately 3.4 million river reaches, and complement the forecasts made by the various River Forecast Centres for approximately 3800 locations across the United States. The structure and setup of the NWMv3.0 are similar to those of NWMv2.1 and are described in more detail in .

We use the NWMv3.0 simulations from the NOAA National Water Model CONUS Retrospective Dataset for the period 1 January 1980 to 31 December 2022. Note that not all gauges have records for the entire period, and in some cases the period of analysis was thus shorter than the full length for which simulations are available. In the retrospective simulations, parameters for the NWM are obtained through a combination of calibration (i.e., parameter optimisation) on a subset of 1365 lightly regulated basins across CONUS and regionalization (i.e., parameter transfer) to the wider set of basins where either no streamflow observations are available or streamflow is more strongly impacted by water management . The model was calibrated for the period 1 October 2016 to 30 September 2021 (NOAA, personal communication, 2025). In contrast to the setup used for forecasting, retrospective runs do not include data assimilation.

For computational efficiency, we aggregated the hourly retrospective simulations to daily average values. This is not uncommon e.g.,, though we note the model runs operationally at an hourly timestep and is most commonly used to predict flood peaks in basins with a response time well below 24 h. The model skill in simulating diurnal patterns will thus not be visible nor assessed in this study. Moreover, the goal of this work is to demonstrate the use of benchmarks in model evaluation, and the average daily NWM simulations provides a useful test case to do so.

Though NWMv3.0 simulations are available without a need to run the model, we need certain meteorological data for the benchmarks used in this work (benchmarks are explained in Sect. ). The Analysis of Record for Calibration (AORC) is an hourly ∼ 800 m-resolution gridded meteorological forcing dataset used as input to NWM retrospective simulations , and thus used as input for the benchmarks in this work. We first aggregated the hourly gridded precipitation and 2 m air temperature to hourly basin averages using the areal mean. Precipitation was then aggregated from hourly to daily by summing the hourly amounts for each day from 1 February 1979 to 1 February 2023. For 2 m air temperature (used by the benchmark code to estimate snow fall and melt), we computed the daily mean. Streamflow observations from 1 January 1980 to 31 December 2023 were collected for approximately 4900 GAGES-II gauges for which streamflow simulations are available (i.e., the gauge was active for the full simulation period, and the stream reach the gauge is on is represented in the NWM) in the NWMv3.0 retrospective dataset .

The Kling-Gupta Efficiency KGE; was used to calibrate the NWMv3.0 on hourly timesteps (NOAA, personal communication, 2025): 2KGE=1-(r-1)2+(α-1)2+(β-1)23α=σsσo,β=μsμo, where r is the Pearson correlation coefficient and subscripts o and s indicate observations and simulations, respectively. To stay as close to the NWM setup as possible, we use the Kling-Gupta Efficiency to quantify model performance in the remainder of this paper (though again note that we perform our analysis at daily time steps whereas the NWM was calibrated at hourly resolution). We repeated our analysis with the Nash-Sutcliffe Efficiency presented in the Supplement to investigate if our conclusions hold for a different metric.

Hydrologic models are increasingly compared to different benchmarks than the long-term mean flow e.g., but outside the forecasting community see e.g. such work is still somewhat limited. Benchmarks also vary in their strengths and weaknesses, and what constitutes a strong benchmark can change regionally .

Sampling uncertainty can be quantified with bootstrapping methods as implemented in the gumboot R package . The gumboot package works by creating a collection of synthetic hydrographs and calculating the score(s) of interest (such as KGE) from the observations and each synthetic hydrograph. We ran gumboot with the default settings as given in . Briefly, this means that gumboot creates each synthetic hydrograph by dividing the period of record into water years (using October as the starting month and enforcing a minimum of 100 valid values within each water year) and sampling water years with replacement until the record length is reached. Using water years ensures that each sampled period is hydrologically independent, and the synthetic records are thus plausible hydrographs for the basin. With default settings gumboot returns 1000 synthetic hydrographs and associated NSE and KGE scores. We then define the sampling uncertainty as the difference between the 5th and 95th percentile of these scores.

We calculate the sampling uncertainty for each basin, for both the NWM simulations and each of the 17 benchmarks. This allows us to report both KGE scores and their associated uncertainty, and from this derive whether the accuracy of NWM simulations can be considered statistically different from the accuracy of the benchmarks. We report those results as Cumulative Distribution Functions (CDFs) that show that scores and uncertainty across the sample. We also report these results on a per-basin basis for the NWM and the best-performing benchmark. In this case, we use the Jaccard index (also known as the ratio of verification, critical success index, and Tanimoto index) to quantify the relative overlap of both uncertainty intervals. Assuming two uncertainty intervals, I1 and I2, defined as the difference between the 5th (Ip05) to 95th (Ip95) percentile estimates of KGE scores for the NWM (I1) and benchmark (I2): 4J(I1,I2)=|I1∩I2||I1∪I2|=overlapspan, overlap=max⁡{0,min⁡(I1p95,I2p95)-max⁡(I1p05,I2p05)}, span=max⁡(I1p95,I2p95)-min⁡(I1p05,I2p05).

When overlap = span, both sampling uncertainty intervals exactly overlap, and the performance of the NWM can be considered indistinguishable from the performance of the benchmark. When overlap = 0, the uncertainty intervals do not overlap, and the performance of the NWM and benchmark simulations can thus be considered to be clearly different. We then need to further distinguish whether the NWM performance can be considered higher or lower than that of the benchmarks. Here we make the simplifying assumption that the 50th percentile score estimate can be used to determine the relative positions of both uncertainty intervals. If the 50th percentile estimate of NWM performance is higher than the 50th percentile estimate of benchmark performance, we consider the NWM to perform better than the benchmark (and vice versa). How much better (or worse) the performance of the NWM is, can then be quantified using Eq. (1). High values of J indicate a large amount of overlap (with complete overlap at J=1) between the two distributions (i.e., smaller distinguishable differences), whereas low values of J indicate a small amount of overlap and clearer differences between the two distributions (no overlap at J=0). A schematic overview of the methodology can be found in Fig. .

Figure  shows the KGE scores obtained by the NWM as well as the 17 benchmark models, visualized as Cumulative Distribution Functions (CDFs) for straightforward comparison of performance aggregated across all locations. Results are presented for both the calibration period (up to water 5 years of data used, depending on data availability at each gauge) and the evaluation period (up to 37 water years). Calibration performance quantifies data fitting potential (i.e., how well can a given method – model or benchmark – capture the patterns in the data at all in a given basin?). Evaluation performance shows what sort of predictive power that data fit actually has (i.e., how well can a given method capture the underlying processes in a way that leads to accurate predictions for unseen data?).

First, for both the calibration and evaluation period, the NWM (black line) reaches higher KGE scores considerably more often than any of the benchmarks (colored lines). However, NWM performance also shows a tendency to decline quickly at lower KGE values, suggesting that there are locations where NWM performance is not as high as that of some of the benchmarks. For calibration, this suggests that the NWM (14 calibrated parameters in NWMv2.1, , assumed to be a similar number for the NWMv3.0 calibrations shown here), as may be expected, has greater flexibility than the benchmarks (0 to 2 parameters) to fit to the specific characteristics of the calibration data. For evaluation, the CDFs of both model and benchmark performances show a tendency towards lower scores. This is commonly seen in any modeling study and typically attributed to some degree of overfitting of the model to specifics of the calibration data, or to a change in conditions between calibration and evaluation periods that the model cannot effectively account for. Some benchmarks (e.g., BM11, Fig. k) show very limited performance change, suggesting that they capture the aggregated catchment response equally well (or poorly) during both data periods. Other benchmarks (e.g., BM07, Fig. g) show very large performance changes, suggesting that calibration conditions were not sufficient to let the benchmark accurately capture underlying catchment behavior. Compared to the benchmark ensembles, the NWM does not stand out as having particularly large or small performance changes.

Second, three benchmarks of note are BM01 (for performing quite poorly), and benchmarks BM07 and BM17 (for performing rather well). BM01 (the mean flow benchmark; Fig. a) can be found as a (nearly) vertical line at KGE=1-(2)≈-0.41. This is the traditional choice of benchmark model, derived from the original formulation of the Nash-Sutcliffe Efficiency, and it is the only benchmark that shows no spatial variability at all during calibration (there is some variability during evaluation, because the mean flow calculated from the calibration data is not always close to the actual mean flow during evaluation). Comparison of this CDF to all others highlights the point made by : the mean flow is not an equally hard-to-beat benchmark in all basins, and location-specific benchmarks are needed to set more locally appropriate expectations for models see also.

BM07 (the daily mean flow benchmark; Fig. g) is computed by taking the mean flow on each Julian day in the calibration period and appending these values to create a year-long timeseries, which is then repeated for each year of the full simulation period. While its CDF does not cover scores as high as the NWM CDF, this benchmark equally does not lead to KGE scores that are as low as some of those obtained by the NWM: during calibration, the NWM CDF covers a range of (roughly) <-5, 1], whereas the CDF of BM07 covers a more restricted range of (roughly) [0,1]. For unseen data (evaluation) the BM07 CDF does not stand out compared to the other benchmarks, possibly due to the somewhat limited amount of data (maximum 5 years) used to compute the benchmark.

BM17 (the adjusted smoothed precipitation benchmark; Fig. q) is a simple 2-parameter model that aims to capture three dominant facets of catchment functioning: partitioning of incoming precipitation into streamflow and sink terms, as well as time delay and attenuation of the resulting runoff . Its CDF is quite similar to that of the NWM but more constrained; the KGE values for this benchmark are neither as high nor as low as those obtained by the NWM. However, the benchmark requires calibration of only 2 parameters, suggesting that within this experimental setup relatively high KGE scores are obtainable with limited degrees of freedom.

In summary, the differences between the NWM and all benchmarks at the lower performance end of the CDF suggest that there are basins where the NWM performance is hindered in some way that the benchmarks are not. At the same time, the NWM obtains higher performance scores than the benchmarks much more often, suggesting that the NWM is able to simulate a wider range of hydrologic behavior with some degree of accuracy than any individual benchmark can. However, note that the CDFs mask the spatial distribution of performance differences. A direct comparison of NWM and benchmark performance will be presented in Sect. 3.3.

Figure  shows the sampling uncertainty associated with the benchmarks and NWM simulations using data from the evaluation period. To save space, a number of benchmarks have been omitted: BM01 and BM02 (mean and median flow) as well as BM10 (rainfall-runoff ratio to annual) have, in the majority of cases, limited performance and little can be learned from these; BM03 and BM04 (annual mean and median flow) use annual flow statistics as a predictor and can by definition not be used for unseen data; BM09 (rainfall-runoff ratio applied to all timesteps) is conceptually very similar to BM01 and has been omitted for the same reason.

As shown in earlier work , the sampling uncertainty of KGE scores can be substantial. In the case of the NWM (black line with grey uncertainty bounds) there is a broad inverse correlation between the KGE score and associated uncertainty bounds, though considerable scatter is present. This emphasizes the strong need to evaluate models while accounting for sampling uncertainty. In numerous basins, the KGE scores obtained by the NWM are strongly conditional on the idiosyncrasies of the evaluation period, and the same model instantiation might be evaluated quite differently if a different time period were to be used. The benchmarks show varying levels of sampling uncertainty. Some are mostly insensitive to data selection (e.g., BM13, BM14), whereas others are either highly sensitive (e.g., BM12, BM16), mostly robust but occasionally sensitive (e.g., BM06, BM08), or somewhere in between (e.g., BM07, BM17). The CDFs and uncertainty bounds should not be directly compared between the different subplots, but a general idea of the widths of these uncertainty intervals is helpful for understanding the results in the next section.

While CDFs of performance scores can be helpful to quickly compare performance differences across the full sample of basins, such approaches do not facilitate a basin-by-basin comparison of differences. Figure a and d therefore show a spatial overview of model and benchmark performance during the evaluation period, using gumboot's estimated 50th percentile KGE score for both. For simplicity, we only assess the evaluation performance of the best benchmark in each basin (in other words, Fig. d is a composite of different benchmarks selected for having the highest 50th percentile KGE score). Both maps confirm the broad statement suggested by the various CDFs, namely that the NWM spans a wider range of performance scores than the benchmarks. The spatial pattern of performance scores shown for the NWM is comparable with that of other modeling studies across this domain e.g.: performance is lowest in the drier central regions, and higher along the wetter west coast, the western mountain regions, and east of the 100th meridian. Benchmark performance is in many places lower than what is achieved by the NWM, but higher in the regions where the NWM already performs poorly.

Figure b, c, e and f clarify these performance difference by showing the relative overlap of the sampling uncertainty intervals of the NWM and best benchmark. Overlap is quantified with the Jaccard index (Eq. ) and separated into cases where the estimated 50th percentile KGE score of the NWM is higher than that of the best benchmark (Fig. b, c; here the NWM outperforms the benchmarks) and vice versa (Fig. e, f). These results are separated into basins used for calibration of the NWM parameters (Fig. b, e), and cases where NWM parameters were regionalized (Fig. c, f). For both sets of plots, the colored stations are complementary: a station plotted in green in Fig. b (or Fig. c) will appear as a yellow dot in Fig. e (or Fig. f) and vice versa. Note that no overlap (Jaccard index = 0; dark green and bright red) indicates that the distributions of KGE scores are clearly separate (in other words, the NWM score is either clearly higher or lower than the benchmark score), whereas lighter colors indicate that the performance of the NWM and benchmark are closer together.

Figure b shows that in approximately 70 % of calibration basins the NWM outperforms the benchmarks. In approximately 75 % of basins this is a clear improvement (Jaccard index ≈0). Basins where the KGE distributions of the NWM and best benchmark partly overlap are mostly found in the central mountainous and drier regions. Figure e shows the remaining 30 % of calibration basins where the benchmarks outperform the NWM. Here too the overlap between the KGE distributions is mostly low, showing that in approximately 60 % of basins the benchmarks obtain clearly higher scores than the NWM. Clusters of basins where the benchmarks outperform the NWM are mostly concentrated in the interior west (broadly inland of the western coastal mountain ranges until somewhat east of the 100th meridian) and the Appalachian Piedmont, with scattered occurrences elsewhere.

These patterns are reinforced in Fig. c and f, which show the performance of the NWM in basins where its parameters were regionalized (i.e., not calibrated). The NWM outperforms the benchmarks in approximately 50 % of basins, located mainly along the western coast and in the humid eastern part of the US. In contrast, the benchmarks perform better in the interior west and the Appalachians, with the appearance of a new cluster of strong performance in central Florida and an increase in scattered basins. Notably, the benchmarks outperform the NWM in almost half of the regionalization basins, with clear regional patterns. Performance distributions do not overlap in almost three-quarters of both cases (J=0 in 73.2 % and 72.3 % in Fig. c and f, respectively), suggesting that sampling uncertainty plays only a limited role in our analysis. Importantly, whereas a glance at Fig. a may suggest that NWM can be improved in the drier central and western regions where model performance is lower, the benchmarks suggest that improvements may be possible in much more widespread regions (Fig. e, f).

As shown in the Supplement, these findings generally hold when the Nash-Sutcliffe Efficiency is used to quantify model and benchmark performance, but with a few important caveats. First, the benchmarks show a tendency towards lower NSE scores and their CDFs are further away from the NWM CDF (Figs. S1, S2). Second, the NWM outperforms the benchmarks in more basins when NSE is used to quantify model performance (the NWM is better in 79.3 % of calibration basins and in 63.4 % of regionalization basins; Fig. S3). This is somewhat surprising, given that the benchmarks are identical in both cases and the NWM was not calibrated on NSE, and points to a need for further work on robust model evaluation practices. Preliminary analysis suggests that these differences are driven by the different sensitivities of NSE and KGE to the bias, variability and correlation components see e.g.,. In at least some basins, the benchmarks perform clearly better on bias and much worse on correlation than the NWM, and because correlation errors are weighted more heavily in NSE, this results in a larger difference in NSE scores than in KGE scores.