The models were all driven by the daily 0.5∘ WATCH Forcing Data ERA-Interim (WFDEI) meteorological dataset (1979–2012; ) with the precipitation (P) data adjusted using the monthly 0.5∘ gauge-based Climate Research Unit (CRU) TS3.1 dataset . Although the models all used the same P data, they used potential evaporation (PET) derived using diverse formulations, ranging from the temperature-based Hamon equation (PCR-GLOBWB) to various radiation-based approaches (WaterGAP3, SWBM, and HBV-SIMREG), the Penman–Monteith combination equation (HTESSEL, JULES, LISFLOOD, SURFEX, and W3RA), and a surface-energy balance approach (ORCHIDEE). The models also used different datasets for non-meteorologic inputs. For more details, see .

Table  lists the 10 state-of-the-art macro-scale hydrological models of which we evaluated the simulated daily unrouted runoff depths (mm d-1). The data used in this study have been named tier-1 and represent an initial run by all participating modeling groups . All data were acquired through the eartH2Observe Water Cycle Integrator (WCI; http://wci.earth2observe.eu), and for each model the sum of the subsurface and surface runoff components was calculated. Six of the models are GHMs (LISFLOOD, PCR-GLOBWB, SWBM, W3RA, WaterGAP3, and HBV-SIMREG) and four are LSMs (HTESSEL, JULES, ORCHIDEE, and SURFEX). The GHMs were all run at daily time steps and the LSMs at hourly and 15 min time steps. The models were run at a 0.5∘ spatial resolution, with the exception of LISFLOOD and WaterGAP3, which were run at 0.1∘ and 0.08∘, respectively. For the analysis, however, all model output was re-sampled to a common 0.5∘ spatial and daily temporal resolution. Four of the models were subjected to varying degrees of calibration to improve their parameters (LISFLOOD, SWBM, WaterGAP3, and HBV-SIMREG; see Sect.  for specifics). More details concerning the models can be found in Table 1 of .

Daily and monthly observed streamflow data were used in this study to evaluate the runoff estimates of the models. The observed streamflow and catchment boundary data used in this study originate from the same three sources as , namely (i) the Global Runoff Data Centre (GRDC; http://www.bafg.de/GRDC/), (ii) the Geospatial Attributes of Gages for Evaluating Streamflow (GAGES)-II database , and (iii) an Australian streamflow data compilation by . The following seven criteria were used to select suitable catchments for our analysis:

The streamflow record length was required to be ≥5 years (not necessarily consecutive) during 1979–2012 (the temporal span of the simulated runoff data).

The simulated runoff of the models were evaluated in five ways. First, for each catchment, we calculated the differences D (-) between simulated and observed values of several runoff signatures. Table  lists the six runoff signatures selected including their computation from the period with simultaneous simulated and observed runoff. The baseflow index (BFI), square-root-transformed 1st percentile exceedance flow (Q1), and square-root-transformed 99th percentile exceedance flow (Q99) require daily (rather than monthly) flow data. To compute the flow timing (T50) from monthly data, we first computed daily time series from monthly time series using linear interpolation. Some of the signature values were square-root transformed to give more weight to small values. D was computed according to Dq=Yqsim-Yqobsσq, where Y represent the values of the runoff signatures (-), the q subscript denotes the runoff signature, and the “sim” and “obs” subscripts refer to simulated and observed, respectively. The σ values (-) are constants that represent the spatial variability in the runoff signatures across the landscape and are used to normalize the D values (i.e., to make the D values of the different signatures intercomparable; see Table ). The σ values were computed by taking the standard deviation of the observed values. Next, the mean D value over all catchments was computed (expressed by D‾). D and D‾ values closer to zero correspond to better model performance (see Table ). It should be noted that, although D‾ provides a valuable estimate of the overall performance, a good D‾ value may reflect an overestimation in one region that is compensated by an underestimation in another region.

Second, to evaluate the temporal variability of the simulated runoff time series, we computed Pearson linear correlation coefficients (r) between daily, log-transformed daily, 5-day, monthly, monthly climatic, and annual time series of simulated and observed runoff (termed rdly, rdlylog, r5day, rmon, rmonclim, and ryr, respectively). The rdly, rdlylog, and r5day values were only computed for catchments with daily observations. If monthly data were not supplied by the data providers, monthly values were computed by simple averaging of the daily data only if >25 non-missing values were available. Annual values were computed by simple averaging of the monthly data (either supplied or computed) only if >10 non-missing values were available. We subsequently computed for each model and metric the mean r value over all catchments, expressed by r‾. The r and r‾ values range from -1 to 1, with higher values corresponding to better model performance (see Table ).

Third, to summarize the overall performance of each model, we computed for each catchment a summary performance statistic (termed OS) incorporating the previously mentioned metrics, and computed the mean value over all catchments (OS‾). The OS consists of two parts, of which the first (OSsig) considers the performance in terms of runoff signatures and is defined as OSsig=1-mean[|DRC|,|DMAR|,|DT50|,|DBFI|,|DQ1|,|DQ99|]. The second part (OSvar) evaluates the performance in terms of temporal variability, and is defined as OSvar=mean[rdly,rdlylog,r5day,rmon,rmonclim,ryr]. The summary score is subsequently computed following: OS=OSsig+OSvar2. The BFI, Q1, and Q99 components of Eq. () and the rdly and rdlylog components of Eq. () were omitted if daily observations were unavailable for a particular catchment. Higher OS values correspond to better model performance; the maximum attainable value is 1 (see Table ).

Fourth, to evaluate the ability of each model to simulate the variability among the catchments in the six previously mentioned runoff signatures, Spearman rank correlation coefficients (ρ) were computed between simulated and observed values of the runoff signatures. Spearman rank correlation coefficients rather than Pearson linear correlation coefficients were used to minimize the influence of outliers. The ρ values range from -1 to 1, with higher values corresponding to better model performance (see Table ).

Fifth, we computed trends in simulated and observed mean annual runoff time series (termed MAR trend) using the simple non-parametric approach of . We subsequently calculated the ρ between simulated and observed MAR trends (ρMARtrend), reflecting the agreement in spatial trend patterns.

Sixth and last, we produced density plots of grid cell values of aridity index (AI; ratio of long-term available energy to P) versus RC (ratio of long-term simulated runoff to P), revealing how the models behave in terms of RC under different climatic conditions. To estimate the available energy we used PET for four models (ORCHIDEE, PCR-GLOBWB, W3RA, and WaterGAP3) and net radiation for three models (HTESSEL, JULES, and SURFEX). For the remaining models estimates of the available energy were not available.

For the evaluation, we used for each catchment the simulated runoff time series of the 0.5∘ grid cell with its center located within the catchment. However, if multiple grid cell centers were located within the catchment, we calculated the mean simulated runoff time series, and if no grid cell center was located within the catchment, we used the simulated runoff time series of the grid cell with its center located closest to the catchment centroid.

Ensemble modeling using the outputs from multiple models or from different realizations of the same model typically improves predictive accuracy and is widely used in atmospheric, climate, and hydrological sciences . We tested two ways of combining the runoff estimates of the individual models into ensembles. First, for each 0.5∘ grid cell and day with non-missing values for all models, the mean simulated runoff of the 10 models was calculated (i.e., equal weights were assigned to the models). The resulting runoff estimates will be referred to hereafter as “MEAN-All”. Second, we computed the mean based on only the four models that performed best in terms of OS‾, to examine the effect of excluding less-accurate models. These runoff estimates will be referred to hereafter as “MEAN-Best4”.

There are a number of caveats that should be kept in mind when interpreting the results. First, some of the models (notably the LSMs) were not traditionally developed to estimate daily runoff for such small catchments. Some of the GHMs, on the other hand, have runoff estimation in small catchments among their primary aims (e.g., LISFLOOD, WaterGAP3, W3RA, and HBV-SIMREG), and four GHMs were even explicitly calibrated against observations (LISFLOOD, SWBM, WaterGAP3, and HBV-SIMREG; see Sect.  for specifics). Second, a model performing poorly in one respect may well perform better for other hydrological variables, climates, catchments, or performance metrics. Third, a poor model performance could simply be the result of suboptimal parameter values. Fourth, some studies have found that less-accurate models may still lead to a better ensemble mean , although this did not appear to be the case here (see Sect. ). Fifth, we stress that while some models may perform well, they are inherently unsuitable for specific types of impact assessments. For example, SWBM and HBV-SIMREG do not account for physical differences among land cover types and hence cannot be used for studies assessing the hydrological impacts of changes in land cover. Sixth and finally, the forcing data quality has an important influence on the evaluation results that should not be overlooked.

Table  shows, for the uncalibrated models, the calibrated models, and the ensembles: (i) the mean difference between simulated and observed values of the (normalized) runoff signatures (D‾), (ii) the mean temporal correlation between simulated and observed runoff time series (r‾), and (iii) the mean overall performance in terms of runoff signatures and temporal correlation coefficients (OS‾). HTESSEL obtained negative D values for the square-root-transformed RC and the square-root-transformed mean annual runoff (MAR), indicating it underestimates runoff. JULES performed moderately in terms of temporal correlation, as indicated by the low r values. Conversely, LISFLOOD performed good overall, particularly in terms of temporal correlation, although it tends to overestimate RC and MAR. ORCHIDEE appears to strongly underestimate runoff and performed fairly in terms of temporal correlation, whereas PCR-GLOBWB shows moderate to good scores for most metrics. Apart from a much too early bias in the flow timing (T50), SURFEX demonstrated moderate to good performance overall. Similar to SURFEX, W3RA exhibited a very early bias in T50, but generally obtained moderate to good scores. WaterGAP3 and particularly HBV-SIMREG outperformed the other models in most cases. JULES, ORCHIDEE, SURFEX, WaterGAP3, and especially SWBM displayed negative D values for the BFI and the square-root-transformed 99th flow percentile (Q99), and a positive D value for the square-root-transformed 1st flow percentile (Q1; Table ), suggesting they consistently overestimate quickflow. Conversely, LISFLOOD and particularly PCR-GLOBWB exhibited positive D values for BFI and Q99, and a negative D value for Q1, indicating they tend to underestimate quickflow.

Table  also presents, for the 10 models and the ensembles, the spatial correlation between simulated and observed values of the runoff signatures (ρ). HTESSEL, JULES, W3RA, WaterGAP3, and HBV-SIMREG performed good overall, while the remaining models performed moderately overall. PCR-GLOBWB, SURFEX, and WaterGAP3 performed poorly in terms of BFI, while SWBM obtained a poor score for Q99. WaterGAP3 performed good to excellent for all signatures except BFI, likely due to the empirical estimation of groundwater recharge and thus baseflow as a function of landscape characteristics . HBV-SIMREG attained good to excellent ρ values for all signatures. The models generally performed best for T50 and worst for BFI among the signatures.

Table  also shows, for the 10 models and the ensembles, OS‾ scores for the major Köppen–Geiger climate types. We used the newly produced Köppen–Geiger climate map from , which is based on the high-quality WorldClim climatic dataset supplemented with regional climatic datasets for the USA and New Zealand . All four LSMs (HTESSEL, JULES, ORCHIDEE, and SURFEX) generally demonstrated fair performance in cold and polar climates. Conversely, PCR-GLOBWB demonstrated poor performance in tropical and arid climates, likely due to the overestimation of baseflow. SWBM performed moderately only in arid catchments, probably at least partly due to the lack of baseflow under these conditions . Similarly, found that SWBM performs well during dry periods for eight small Swiss catchments (60–392 km2). Only LISFLOOD, WaterGAP3, and HBV-SIMREG exhibited at least moderate performance for all climates.

Figure  presents, for the 10 models and the ensembles, maps of simulated minus observed MAR for the catchments, revealing the data underlying the MAR D‾ and ρ values listed in Table . Maps of all other runoff signatures are presented in Supplement Figs. S1.2–1.8. HTESSEL and ORCHIDEE strongly underestimate runoff for most of the catchments, while LISFLOOD appears to strongly overestimate runoff for most of the globe with the exception of snow-dominated regions. All models showed negative MAR biases in snow-dominated regions such as Alaska, the Rocky Mountains, and southern Russia, while they consistently showed positive MAR biases for the Great Plains (USA) and southern Australia. Figure  shows, for the 10 models and the ensembles, maps of the correlation between simulated and observed monthly flows (rmon) for the catchments, showing the data underlying the rmon‾ values presented in Table . Maps of all other temporal variability metrics are presented in Figs. S1.9–1.14. In general, the GHMs obtained good rmon values for most catchments, while the LSMs obtained moderate rmon values for most catchments. All LSMs showed poor to fair rmon values for snow-dominated catchments.

Although the NSE has been widely criticized for being overly sensitive to the magnitude and timing of peak flows (e.g., ), we did calculate NSE scores to allow the present results to be put in the context of previous macro-scale studies (see Supplement Table S1). For most models negative median NSE scores were obtained, similar to , who evaluated the monthly and annual runoff estimates from 14 (uncalibrated) macro-scale models in 644 large Australian catchments (>2000 km2). Our scores are, however, slightly lower than those obtained by and , who evaluated the daily runoff estimates from four (uncalibrated) macro-scale models in about a thousand small-to-medium sized USA catchments (<10000 km2), but this is probably attributable to the high quality of the USA forcing data . They are also somewhat lower than those obtained by , who evaluated two (uncalibrated) macro-scale models in 80 large catchments (>100000 km2) around the globe, but this can be explained by their much larger catchment sizes.

Figure  shows, for the seven models with data on energy availability, density plots of grid cell values of aridity index (AI; ratio of long-term energy availability to P) versus RC (ratio of long-term mean runoff to P), revealing how the models respond in terms of RC to different climatic conditions. Also shown are the energy-limit line for which actual evaporation equals the available energy, the water-limit line for which runoff equals P, and the curve, the most well-known among several similar empirical relationships describing the competition between runoff and actual evaporation . Given its empirical nature, the Budyko curve should only be used for visual reference, and not to judge the performance of the different models. Besides the striking differences in behavior among the models, it can be seen that HTESSEL, JULES, PCR-GLOBWB, W3RA, and WaterGAP3 do not adhere to the water and/or energy limits (Fig. a, b, d, f, and g, respectively). For WaterGAP3, this may be due to the use of calibration factors, which have the potential to generate runoff that can go beyond the physical limits in an effort to compensate for errors in the P, PET, or streamflow data. For the other models this could be indicative of issues with the runoff and/or evaporation routines. The larger spread found for the models for which we used net radiation to estimate the available energy (HTESSEL, JULES, and SURFEX; Fig. a, b, and e, respectively) is because the majority of the net radiation is converted to sensible heat rather than latent heat in cold climates .

It is generally difficult to gain insight into why a particular model performs as it does due to the large number of interacting model components, equations, and parameters. Nevertheless, the underestimation of runoff by HTESSEL probably reflects the excessive evaporation by HTESSEL previously reported by . PCR-GLOBWB most likely suffers from suboptimal baseflow-related parameter values, since its structure is similar to that of LISFLOOD, which performs markedly better. SWBM clearly suffers from the absence of a baseflow routine outside (semi-)arid regions. Although W3RA and HBV-SIMREG use an identical snow routine, W3RA performs considerably worse in snow-dominated regions, probably because HBV-SIMREG uses a snowfall gauge undercatch correction factor. The unsatisfactory performance demonstrated by the LSMs in snow-dominated regions could be related to deficiencies in the snow routines or the energy balance estimates (see Sect. ). WaterGAP3 and particularly HBV-SIMREG performed quite well overall, likely because of their comprehensive calibration (see Sect. ). In any case, the pronounced inter-model performance spread found here suggests that model choice should be regarded as a critical step in any hydrological modeling study. Moreover, it underscores the importance of hydrological model uncertainty in addition to climate input uncertainty, as also emphasized in several other recent macro-scale studies . Currently, the large majority of studies assessing the hydrological impacts of climate change completely neglect hydrological model uncertainty .

The models displayed very similar MAR trends (Fig. S1.8), meaning they respond similarly to climate variability, given that none of the models account for land use or land cover changes, urbanization, reservoir construction, or increasing atmospheric CO2. However, the models obtained rather low spatial (Spearman) correlation coefficients (ρMARtrend) ranging from 0.32 (SURFEX) to 0.42 (LISFLOOD; Table ), indicating that the simulated MAR trends correspond fairly to moderately well to the observed ones. These values are lower than the (Pearson) correlation coefficients ranging from 0.52 to 0.63 obtained by , who evaluated MAR trends from seven models using observations from 293 small European catchments (100–1000 km2), presumably due to the better quality of the European meteorological forcing and observed streamflow data. evaluated MAR trends from a 12-model ensemble using observations from 165 large catchments (>50000 km2) around the globe, obtaining a (Pearson) correlation coefficient of 0.34, which is similar to ours. These low correlations, which were somewhat unexpected given the relative ease with which MAR can be estimated (e.g., ), may be indicative of changes in non-climatic drivers of hydrological change or drift errors in the forcing or observed streamflow data. We expect the inter-model variability in trends to be higher and the agreement with observations to be even lower for seasonal and monthly averages as well as runoff signatures sensitive to the shape of individual flow events (cf. ). Overall, these results suggest that studies using global-scale datasets to assess the impacts of past climate change on runoff in small-to-medium-sized catchments should be interpreted with considerable caution.

Similar to , the LSMs were found to produce less runoff overall (Table  and Fig. ), perhaps due to their use of physically based Richards–Darcy type equations, which neglect preferential flows. We further found that the GHMs perform, on average, worse than the LSMs in rain-dominated regions: the GHMs (excluding the comprehensively calibrated models WaterGAP3 and HBV-SIMREG; see Sect. ) obtained mean OS‾ scores of 0.28, 0.33, and 0.43 for tropical, arid, and temperate climates, respectively, while the same values for the LSMs are 0.39, 0.47, and 0.47, respectively (Table ). However, the lower performance of the GHMs is primarily attributable to PCR-GLOBWB and SWBM. As mentioned before, PCR-GLOBWB probably suffers from a suboptimal baseflow-related parameterization, while SWBM suffers from the absence of a baseflow routine.

The GHMs do appear to perform consistently better than the LSMs in snow-dominated regions: the GHMs (again excluding WaterGAP3 and HBV-SIMREG) obtained mean OS‾ scores of 0.46 and 0.32 for cold and polar climates, respectively, while the same values for the LSMs are 0.31 and 0.25, respectively (Table ). The performance of the LSMs appears to be mainly due to a very early bias in flow timing, a very low baseflow contribution, and a misrepresentation of the seasonal cycle (Figs. S1.4, S1.5, and S1.13, respectively). Our results are in agreement with , who found five GHMs to outperform, on average, four LSMs using observations from 252 temperate and cold catchments (64 to 1 350 000 km2) located in the central USA, and with , who found that two LSMs performed considerably worse than two GHMs in cold and polar regions using observations from 644 catchments (>2000 km2, upper limit not reported) around the globe. The poorer performance obtained by the LSMs is probably indicative of differences between the snow routines used by GHMs and LSMs. The GHMs use relatively simple conceptual temperature-index snow routines driven by air temperature, which can be estimated with relative ease, whereas the LSMs use more complex physically based energy balance snow routines driven by estimates of energy balance components, which are subject to considerable uncertainty, particularly in regions with complex topography . Although several previous studies have found that the two types of snow routines yield comparable performance (e.g., ), these studies used a very small number of relatively well-instrumented catchments (six, two, one, and three, respectively), which may have led to less-generalizable conclusions. Overall, it appears that the energy balance estimates and snow routines used by the LSMs require re-evaluation (cf. ).

Calibration is important for both conceptual and physically based hydrological models to provide more accurate runoff estimates, to account for (i) the impossibility of measuring all required model parameters at the model application scale, (ii) lack of process understanding, (iii) possibly overly simplistic process representations, (iv) the spatiotemporal discretization of highly heterogeneous rainfall–runoff processes, and (v) errors in the forcing data . Yet, despite the development of numerous calibration techniques over the last 50 years and the current widespread availability of streamflow observations , macro-scale models generally tend to be uncalibrated . This is perhaps mainly due to (i) the substantial amount of work involved with calibration (e.g., ), (ii) the risk of obtaining unrealistic parameters due to equifinality and data issues , and (iii) the lack of a commonly accepted regionalization technique . In addition, the modeler may feel that since their model is physically based, it does not require calibration . LSMs in particular are rarely calibrated against runoff, likely because (i) runoff estimation is generally not among the primary aims of LSMs; (ii) for water transport in the soil, LSMs typically use Richards–Darcy type equations, which are computationally expensive and require a fine vertical and temporal soil discretization; and (iii) LSMs often do not account for river routing, confounding the calibration of large catchments. Instead, the parameters in macro-scale models are usually based on “expert opinion” and thus founded on the bold assumption that the modeler sufficiently understands the hydrological processes, feedbacks, and parameter interactions taking place within the model for any location on Earth.

Nevertheless, out of the 10 models considered in this study, four use parameters derived by calibration (LISFLOOD, SWBM, WaterGAP3, and HBV-SIMREG all GHMs). LISFLOOD was calibrated against observed streamflow for 24 large catchments (84 230 to 4 680 000 km2) across the globe using the WFDEI forcing and an aggregate objective function incorporating bias, NSE, and log-transformed NSE computed from daily streamflow data. The calibration might have influenced the present evaluation; although we used much smaller catchments (1000 to 5000 km2), 47 % of our catchments are located within the calibration catchments. SWBM uses a spatially uniform parameter set based on calibration using the E-OBS forcing against European data on such key hydrologic variables as soil moisture, total water storage, evaporation, and runoff . For the calibration against runoff, they used observations from 436 small European catchments (mostly <1000 km2), and considered daily and monthly correlations as well as bias. The calibrated parameter set was subsequently applied globally. Besides the addition of a baseflow routine, SWBM would probably benefit from regionalized parameters that vary according to landscape characteristics. WaterGAP3 has been calibrated using the WFDEI forcing in terms of bias for the interstation regions (the catchment of a station excluding the catchments of nested upstream stations) of 2071 stations (catchment size ranging from 2830 to 966 321 km2) around the globe, some of which have also been used in the current evaluation. The calibrated parameters were subsequently regionalized to ungauged regions using multiple linear regression based on six predictors . The model does indeed perform very well for MAR and thus RC, but this did not necessarily translate into good performance for BFI (Table , and Figs.  and ). HBV-SIMREG also uses regionalized parameter fields, produced by transferring calibrated parameters from 674 small-to-medium sized “donor” catchments (10 to 10 000 km2) across the globe to “receptor” grid cells with similar climatic and physiographic characteristics . In their study, show that HBV using spatially uniform parameters performs within the range of the other models, confirming that the relatively good performance of HBV-SIMREG stems from the regionalization exercise. In addition, although did not use the WFDEI forcing for the calibration, they calibrated against several of the performance metrics also used here and used 179 of our catchments as parameter donors, further explaining the relatively good performance obtained by HBV-SIMREG (Table , and Figs.  and ).

Overall, it appears that the calibration exercises for WaterGAP3, HBV-SIMREG, and possibly LISFLOOD have resulted in markedly improved performance. However, WaterGAP3 performed poorly in terms of ρBFI (Table ), meaning the calibration of MAR did not translate into better BFI performance. These results underscore the benefits of calibrated parameters over a priori parameters (cf. ) and highlight the importance of using an objective function for the calibration that incorporates a broad range of metrics related to various important aspects of the hydrograph (cf. ). These results also emphasize the usefulness of regionalization techniques , which typically enhance performance over the entire model domain and are thus of particular value for macro-scale modeling, given that the majority of the land surface is ungauged or poorly gauged . However, although there are numerous studies performing regionalization at a regional scale (see reviews by ), only a few studies have attempted regionalization at a macro-scale (see review by ). We argue that more effort should be devoted to regionalizing the parameters of macro-scale models (cf. ).

It should be noted, however, that the potential performance improvement gained by calibration and regionalization will depend on the structure and flexibility of the model in question. Many current models have rigid structures and/or insufficient free parameters and thus cannot be calibrated successfully . Moreover, for climate projections one should bear in mind that calibrated parameters become less valid when the model is subjected to climatic conditions it has never seen before .

There are not only strong inter-model differences in the performance patterns but also clear inter-model similarities, suggesting that the forcing data quality imparts a strong limit on the performance. This is most notable for the MAR metric: all models showed negative biases in MAR in snow-dominated regions such as Alaska, the Rocky Mountains, and southern Russia, while they consistently showed positive biases in MAR for the Great Plains (USA) and southern Australia (Fig. ). The high spatial correlation in the performance patterns suggests that these consistent performance patterns may be due to biases in the WFDEI P data, rather than biases in the streamflow observations, which are unlikely to be spatially correlated.

It is conceivable that biases are present in the WFDEI P data, because (i) the monthly CRU dataset, which has been used to correct the WFDEI dataset, is based on only a subset of the available gauges and does not explicitly account for orographic effects; (ii) in sparsely gauged regions the correction using CRU is more likely to deteriorate rather than improve the P estimates; and (iii) the gauge undercatch correction factors are based on interpolation of a very sparse sample of gauges and thus subject to considerable uncertainty. For the conterminous USA we quantified the biases in the WFDEI P data using the high-quality Parameter-elevation Relationships on Independent Slopes Model (PRISM) climatic dataset , which is based on considerably more gauges than CRU and includes sophisticated corrections for orography. Figure a shows the bias in mean annual P from WFDEI relative to that from PRISM, suggesting that the WFDEI P data are indeed subject to large biases. Figure b shows the bias in MAR from the MEAN-All ensemble relative to MAR from the observations, revealing a comparable bias pattern, thus confirming that the biases in the WFDEI P propagate in the simulated runoff. The correlation coefficient between the MAR and P bias values is 0.58, indicating a moderately strong relationship. These P biases appear to translate into even more pronounced runoff biases in (semi-)arid regions (notably the northern Great Plains; Fig. b and c) due to the highly nonlinear response behavior in these environments . We were unable to quantify the P biases globally since no other independent, global-scale P dataset exists (the WorldClim and CHPclim datasets are likely to exhibit similar biases as the CRU TS3.1 dataset, given that they are based on similar sets of gauges). However, we expect the P biases to be at least similar, if not more severe, outside the well-instrumented conterminous USA (cf. ). It should be noted that biases in PET are probably of secondary importance as compared with biases in P .

The global-scale quantification and reduction of these P biases should be a priority for future research. Satellite-derived P offers unique opportunities in this regard (e.g., ) that extend beyond the tropics with the recent launch of the Global Precipitation Measurement (GPM) mission . Another little-explored way of reducing P uncertainty is by “doing hydrology backwards”; that is, to use information on other hydrological variables, for example, satellite-derived surface soil moisture (e.g., ), streamflow observations (e.g., ), and snow-depth observations (e.g., ) to reconstruct P through hydrological modeling. Arguably the most important obstacles to combining multiple data sources are the inconsistent temporal coverage and scale of different data sources and the general lack of error/uncertainty estimates.

Although the models all used the same P data, they used different formulations to compute PET, which has likely contributed to differences in simulated runoff among the models in energy-limited regions . However, PET data were available for only four models, which is insufficient to examine whether the PET formulation has had a discernible influence on the simulated runoff, given the numerous other differences in structure and parameterization among the models.

The multi-model ensemble MEAN-All incorporated all 10 models, while MEAN-Best4 incorporated only LISFLOOD, W3RA, WaterGAP3, and HBV-SIMREG (i.e., the four models that performed best in terms of OS‾; Table ). MEAN-All and MEAN-Best4 were found to perform better than all individual models (with the exception of HBV-SIMREG, which has been comprehensively calibrated; Table , and Figs.  and ). These results highlight the benefits of multi-model ensembles, in line with several previous studies . The similar OS‾ scores obtained by MEAN-All and MEAN-Best4 (0.57 and 0.60, respectively; Table ) suggests that the inclusion of less-accurate models has only limited adverse effects. It may be worthwhile for future studies to examine the benefits of more sophisticated multi-model combination techniques involving bias correction or model weighting (e.g., ). These weights can subsequently be transferred from gauged to ungauged areas using regionalization techniques typically used for hydrological model parameters .

HBV-SIMREG differs from the other models because it represents a “multi-parameterization ensemble”, which means the model was run multiple (10) times globally using different (regionalized) parameter sets representing different catchment response behaviors . HBV-SIMREG obtained slightly better performance than both MEAN-All and MEAN-Best4 overall (Table ), tentatively suggesting that a multi-parameterization ensemble for a single, sufficiently flexible model provides performance comparable to a multi-model ensemble (cf. ). If this is confirmed, it would negate the need to set up, run, and maintain multiple models, and incentivize the development of a single community hydrological model (cf. ) as well as modeling systems allowing for the selection of alternative model structures (cf. ), such as the Framework for Understanding Structural Errors (FUSE; ), Noah Multi-Parameterization (Noah-MP; ), and SUPERFLEX .

With the exception of ORCHIDEE and HBV-SIMREG, all models showed early T50 biases in snow-dominated regions (Fig. S1.3), indicating that the models produce the spring snowmelt peak early, as has also been reported in several previous studies using different models and forcing data . The early runoff timing is probably primarily due to P underestimation, which leads to insufficient snow accumulation that subsequently melts too quickly . The fact that HBV-SIMREG performs well in this regard is probably attributable to the snowfall gauge undercatch correction factor of the model. Indeed, Fig.  tentatively shows that catchments in which the models strongly underestimate runoff (i.e., negative DRC) generally tend to exhibit an early bias in T50 (i.e., negative DT50) and vice versa. The absence or misrepresentation of certain processes that delay snowmelt runoff in the models may have exacerbated the early runoff timing problem. Examples of such processes include the isothermal phase change of the snowpack, retainment of meltwater in the snowpack in pore spaces, infiltration of meltwater into the soil, meltwater refreezing during cold days and nights, and ice jams in rivers. On the whole, more research is needed to ascertain the exact reasons of the early runoff timing.