The study was based on twelve Australian catchments with minimal water resource development and land use disturbances. Although larger samples are desirable, here we restrict ourselves to sites that have nearby flux towers to quantify actual evapotranspiration, as discussed below. The catchments cover five Köppen–Geiger climate zones (Peel et al., 2007) and different hydroclimatic regimes (Fig. 1). The tropical catchments in the North have summer-dominated precipitation and discharge regimes arising from monsoon rainfall from January–March (Fig. 1b–d). The arid catchments in northern Australia also have some summer precipitation, but only little results in discharge due to high summer evapotranspiration rates (Fig. 1b–d). Most of the temperate catchments are in the Southeast and Southwest of Australia and for many of them precipitation and discharge is more evenly distributed over the year than in the tropical and the arid catchments (Fig. 1b–d). Overall, mean annual values of precipitation and discharge for the twelve study catchments range from 510–3200 mm, and 12–1720 mm, respectively. The catchments vary in size from 96–4794 km2 and are located between 39 and 902 m a.s.l. (meter above sea level).

Daily meteorological and discharge time series were extracted from version 2 of the CAMELS-AUS dataset (Fowler et al., 2024). It provides for all twelve study catchments data from at least 1977–2022. Among the multiple options provided within CAMELS-AUS, we chose the meteorological time series representing catchment averages calculated from the gridded precipitation product (∼5km×5km) of Evans et al. (2020) and the gridded Morton's wet environmental areal potential evapotranspiration estimates (∼5km×5km) of Jeffrey et al. (2001).

Actual evapotranspiration time series were used from two different sources to consider both remote sensing-based data and locally observed field data. The remote sensing-based time series were retrieved from the global gridded product (∼8km×8km) of the Numerical Terradynamic Simulation Group (NTSG) at the University of Montana (Zhang et al., 2010), which is available for the years 1982–2013. Their estimates of monthly actual evapotranspiration are based on biome-specific transpiration and soil evaporation calculated from a modified Penman–Monteith equation. For our study, we calculated a monthly catchment average value using the area-weighted mean of all grid cells covering a particular catchment. The catchment boundaries were taken from the CAMELS-AUS v2 dataset. The dataset was chosen because it has been a commonly used product in Australia (Gardiya Weligamage et al., 2023).

The locally observed time series of actual evapotranspiration were obtained from the OzFlux Research and Monitoring network (Beringer et al., 2016), part of the international Fluxnet program (Baldocchi et al., 2001), which maintains tower-based eddy covariance systems to collect data on energy, carbon, and water fluxes in Australia and New Zealand. Not all OzFlux sites are located within the boundaries of catchments with discharge gaugings. Thus, we matched each of the twelve OzFlux sites used in this study with the catchment having the closest catchment centroid within CAMELS-AUS v2. Table S1 in the Supplement provides the OzFlux site name and the corresponding catchment ID in the CAMELS-AUS dataset. The field-based daily actual evapotranspiration values were transferred to the catchments without modification assuming that they are a reasonable estimate of catchment average evapotranspiration (note that the objective function used is insensitive to bias, for reasons discussed in Sect. 2.3.1). The length of the available time series varies among the twelve OzFlux sites, spanning the most recent 7–21 years.

The temporal coverage differs between the three datasets described above, whereby the time periods of the remote sensing-based and field-based actual evapotranspiration datasets don't overlap. To make modelling results from the two actual evapotranspiration products more comparable, we created two separate model evaluation datasets using the following criteria: (i) the data period should ideally cover ten years, and (ii) only hydrological years with a complete discharge and actual evapotranspiration record are used. The start of the hydrological year depends on the hydroclimatic conditions of a catchment and was defined as the month with lowest average long-term monthly discharge (Wasko et al., 2020). A hydrological year was considered complete if less than 5 % of time steps were missing. As a result, the first model evaluation dataset (DETa.RS) including meteorological data, discharge, and remote sensing-based actual evapotranspiration covered the period 2003–2014 with nine out of twelve catchments having the full ten-year record. The second model evaluation dataset (DETa.Field) including meteorological data, discharge, and field-based actual evapotranspiration broadly covered the period 2011–2022. Time series length for the twelve catchments ranged between three and ten years with five out of twelve catchments having ten years of data.

SIMHYD is a lumped bucket-type hydrological model that has been applied in different hydroclimatic regions of Australia (Chiew et al., 2002; Fowler et al., 2020; Khatami et al., 2019). The model takes daily precipitation and potential evapotranspiration as inputs and simulates a catchment's rainfall–runoff response using three stores and seven parameters (Fig. S1 in the Supplement). The three stores represent interception by vegetation, soil water, and groundwater. The model parameters affect how the state of the stores varies as a function of rainfall, actual evapotranspiration, infiltration, recharge, and runoff generation. The streamflow generation mechanisms modelled by SIMHYD are infiltration excess flow, interflow and saturation excess flow, and baseflow. The three streamflow generation mechanisms are hereafter referred to as overland flow, interflow, and baseflow, and are used to create the runoff flux maps. A more detailed description of SIMHYD can be found in Chiew et al. (2002). The model code for SIMHYD is publicly available from the Modular Assessment of Rainfall–Runoff Models Toolbox (MARRMoT). We used MARRMoT version 2.1 (Trotter et al., 2022).

We defined three calibration cases to systematically analyse the impact of multivariate model calibration on simulated runoff generation and hydrographs. Note that we use the word “calibration” here to denote the process of selecting an ensemble of parameter sets based on their match with observed data. This is different to common usage in which a single “optimum” parameter set, not an ensemble, is chosen. The three calibration cases included a calibration to (i) daily discharge only, (ii) monthly actual evapotranspiration only, and (iii) a combination of both variables. Below we describe the objective functions used in calibration, followed by the calibration process in which parameter values were estimated, and the analysis of the modelling results.

Significant differences were found in terms of model performance when calibrating with discharge only versus discharge and evapotranspiration (Fig. 2; for more details see also Fig. S2 in the Supplement). The 100 best parameter sets selected in the discharge-based calibration clearly outperformed the uncalibrated model (i.e., median of all randomly sampled parameter sets) when evaluating discharge (OQ). However, the same parameter sets were only marginally better than the uncalibrated model when evaluated for actual evapotranspiration (OETa). Considering actual evapotranspiration in addition to discharge in model calibration strongly improved the performance for actual evapotranspiration, which can be attributed to more constrained interception and soil parameters that directly affect the simulation of actual evapotranspiration (Fig. S3 in the Supplement). However, the improved simulation of actual evapotranspiration comes at the expense of a slightly decreased performance for discharge, suggesting a trade-off between the two variables. Indeed, rank correlations between the model performance for discharge and actual evapotranspiration from the 100 000 simulations are for eight out of twelve catchments negative (Fig. S4 in the Supplement). The overall model performance (OQ,ETa) was highest when considering both variables in model calibration. Our results are consistent with findings of previous work, which concluded that multivariate calibration improves the overall realism of model simulations based on model performance and parameter sensitivity (Arciniega-Esparza et al., 2022; Dembélé et al., 2020a; Mei et al., 2023).

By exploring flux maps and hydrograph uncertainties, however, we get a more nuanced and different view of the effect of multivariate calibration on model realism. Figure 3a shows the flux map after different calibration cases for the Litchfield catchment. While interflow and baseflow were the main streamflow generation mechanisms when calibrating to discharge (OQ), overland flow gained importance when also considering actual evapotranspiration in calibration (OQ,ETa). Consequently, the percentage of the flux map area filled by the 100 best simulations increased from 14 % after a discharge-only calibration to 41 % when adding actual evapotranspiration to a discharge-based calibration. Such an increase in flux map area was observed for ten out of twelve catchments with flux map areas increasing between 2 % and 27 % (Fig. 3b). This suggests that multivariate calibration can increase simulated flux equifinality regardless of local conditions such as climate type. An interesting exception from this general observation is the Cow Bay catchment. Its flux map area changed by -22% leading to a more constrained map after multivariate calibration. However, this more constrained flux map is not a subset of the flux map of a discharge-based calibration as one would expect.

In terms of hydrograph uncertainty we found that the simulated hydrograph uncertainty of the Litchfield catchment considerably increased for many flow conditions when not only calibrating to discharge (OQ) but also considering actual evapotranspiration in calibration (OQ,ETa; Fig. 4a). This observation was common to all study catchments, whereby the increase in hydrograph uncertainty generally tended to be larger for low flows than high flows (Fig. 4b). Averaging over all flow conditions (i.e., simulated time steps), uncertainty increased in eleven out of twelve catchments. The only catchment experiencing a lower average hydrograph uncertainty after multivariate model calibration was Cow Bay. Interestingly, Cow Bay was also the only catchment with a much-reduced flux equifinality when adding actual evapotranspiration to a discharge-based calibration (Fig. 3b and c).

To explore why multivariate model calibration increased flux map area and hydrograph uncertainty, we first analysed the model parameter space after calibration. As mentioned previously, Monte Carlo-based multivariable calibration is a way of making an initially large parameter space smaller, whereby the goal is to seek the intersection between the separate behavioural clouds of two or more variables (here, discharge and actual evapotranspiration). It is therefore assumed that such an intersection exists. Here, we looked for these intersections between two clouds by comparing the parameter value distribution when calibrating SIMHYD solely to discharge and solely to actual evapotranspiration. Given SIMHYD has seven parameters, it is possible to assess the intersection of clouds in any number of dimensions between one (i.e., assessing overlap of histograms for each parameter individually) and seven (which is difficult to visualize but technically possible). Here, we focus on two-dimensional plots since they allow some consideration of parameter interactions but are still easy to visualize. We examined parameter distributions for all 21 parameter pairs (unique combinations of seven model parameters). For each catchment, we present the result for one parameter pair as a scatterplot (Fig. 5). The (subjectively) selected examples show different patterns of the lack of overlap (if any) between the separate clouds. Quantitative results for all 21 pairwise parameter combinations can be viewed in Fig. 6 showing the overlap of the convex hull of the behavioural parameter clouds of discharge and evapotranspiration.

Starting again with the Litchfield catchment as an example, Fig. 5a shows the values of the 100 best parameter sets for two model parameters, namely infiltration loss exponent (x axis) and maximum infiltration loss (y axis), when calibrating SIMHYD with discharge (OQ) and with actual evapotranspiration (OETa). We found that there was a strong separation between the two point clouds of behavioural parameter sets for the given parameter pair and no evidence of intersection as often assumed. Similar results, although with different separation patterns and different parameter pairs, were observed for the other study catchments (Fig. 5b). We expect that the separation patterns would get even more pronounced when moving from the two-dimensional parameter space to a higher dimensional space, but having demonstrated non-overlap in two dimensions for most catchments, we did not explicitly test this in higher dimensions, save by exception. An example of that is the Robson Creek catchment for which the point clouds of behavioural parameter sets overlapped in the two-dimensional parameter space (Fig. 5b) but became separated when analysed in the three-dimensional parameter space (not shown here). While we focused on presenting one example of separation pattern for each catchment, the patterns were typically present for several parameter pairs.

We further analysed how a calibration with discharge and a calibration with actual evapotranspiration affected the simulated contribution of streamflow generation mechanisms. In the case of the Litchfield catchment (Fig. 7a), a calibration with discharge (OQ) favoured parameter sets for which streamflow was largely composed of interflow and baseflow. In contrast, much of the total streamflow originated from overland flow when calibrating solely to actual evapotranspiration (OETa). Such considerable discrepancies in streamflow generation mechanisms were also observed for other study catchments (Fig. 7b). They could eventually lead to a larger flux map equifinality when simultaneously calibrating to discharge and actual evapotranspiration, which was indeed observed in Fig. 3.

Our results for twelve hydrologically contrasting catchments in Australia suggest that multivariate model calibration can lead to less constrained flux maps (i.e., streamflow generation mechanisms) and consequently increased hydrograph uncertainties relative to a discharge-based calibration (Figs. 3 and 4). These results may seem counterintuitive given the large number of studies reporting improved realism for a range of simulated variables and processes after multivariate calibration (e.g., Demirel et al., 2019; Mei et al., 2023; Meyer Oliveira et al., 2021). Thus, why does the inclusion of additional information not result in more constrained flux maps and hydrographs? The parameter distributions shown in Figs. 5 and 6 suggest that these symptoms could be caused by non-overlapping behavioural parameter distributions for the individual calibration variables. When simultaneously calibrating the model against discharge and actual evapotranspiration, overall model performance (i.e., the average performance over both variables) increases (by definition, since this is the objective function used). However, in the absence of an intersection between the individual behavioural clouds, the selected parameter sets represent compromise solutions that get spread across or between the two individual clouds (Figs. S5 and S13 in the Supplement), sometimes covering a wider space, which increases equifinality and finally leads to increased discharge uncertainty.

Under ideal conditions, there would, for each catchment, be one single parameter set resulting in a perfect match between several observed and simulated variables. However, the process of setting up a model for a catchment involves a large number of (subjective) decisions, including the choice of data, calibration procedure, and model structure, which all affect the study outcome (Ceola et al., 2015). Below we discuss some aspects of these decisions, which we think are relevant in the context of our work.

We used actual evapotranspiration data from two different sources. Field-based actual evapotranspiration data from flux towers are considered to be more accurate than remote sensing-based data and are used to benchmark the latter (Martens et al., 2017; Zhang et al., 2010). The near-point-scale nature of flux tower data means they are not necessarily representative of the entire (sub-) catchment, a fact known to affect the suitability of local observations to select catchment-scale parameter values (Gallart et al., 2007; Staudinger et al., 2021; Széles et al., 2020). Remote sensing-based actual evapotranspiration data may have a larger footprint and while many data products show reasonable seasonal dynamics, they can suffer from large volume bias (Elnashar et al., 2021; Wartenburger et al., 2018; Zhang et al., 2020). The choice of an evapotranspiration data product can significantly influence the selection of model parameter values even if a bias insensitive metric was used in calibration (Dembélé et al., 2020b; Nijzink et al., 2018). Interestingly, the remote sensing-based data and the field-based flux tower data led to similar observations for the twelve study catchments despite their different data characteristics: non-overlapping behavioural parameter distributions were present for both datasets, flux map patterns were similar for the individual calibration variables, and the direction of the correlation between the two calibration variables in a given catchment tended to be the same in both datasets. We take this as an indication that issues with actual evapotranspiration data are not the root cause of the results. Thus, changing the actual evapotranspiration datasets would be unlikely to result in significantly different conclusions. While gauging errors in discharge data influence results, these errors might be expected to differ from one station to the next, whereas the issues here occur across all twelve catchments tested. Thus, it appears data quality alone cannot explain the increased discharge uncertainty following multivariate calibration.

Another potential factor influencing our results is the calibration process. For example, the choice of an objective function defines which aspects of a time series receive most weight during calibration (Kiesel et al., 2017; Pool et al., 2017) and thus changes the distribution of the calibrated model parameter values (Choi and Beven, 2007; Garcia et al., 2017). We used objective functions that tend to simultaneously evaluate a range of data aspects as we were interested in a model being able to simulate multiple facets of catchment response. Both objective functions were based on the Kling–Gupta efficiency and represent a choice done in many modelling studies. Another commonly used objective function, the Nash–Sutcliffe efficiency (Nash and Sutcliffe, 1970), is in many aspects similar to KGE and its use should result in similar findings as made in this study. When conducting the multivariate calibration, we combined the individual objective functions for discharge and actual evapotranspiration into a single combined objective function (Eq. 3) as opposed to the simultaneous evaluation of two separate objective functions. This approach has the drawback that the selection of behavioural parameter sets can get biased towards one of the two calibration variables and increase the uncertainty of the other simulated variable, especially if pronounced trade-offs exist. While trade-offs have been reported in many previous studies irrespective of the multivariate calibration approach (Livneh and Lettenmaier, 2012; Mostafaie et al., 2018; Yassin et al., 2017), they were rarely shown to increase discharge uncertainty. We discuss this in more detail in Sect. 4.2.

A probably more impactful factor in our study was the decision to define the 0.1 % best performing parameter sets as behavioural. Loosening this threshold may increase the chance of finding overlapping parameter distributions for univariate calibrations with discharge and actual evapotranspiration. This would come at the expense of accepting lower model performance values for the calibration variables although the extent of these costs differs among catchments and the threshold (Fig. 8a). The choice of a threshold also affects how much parameter values are constrained. Results for the parameter values are more consistent among catchments than for model performance and generally indicate that parameter values get quickly less constrained with an increasing number of behavioural parameter sets (Fig. 8b). Most parameters in SIMHYD directly affect runoff generation and a wider range of their values can further increase flux equifinality and thus hydrograph uncertainty. This is similar to findings reported in a study by Lamb et al. (1998), where less constrained TOPMODEL parameter values and wider discharge uncertainty bounds were found for multivariate calibration than in a discharge-based calibration. Thus, working with the best 100 parameter sets seems to be a reasonable decision for this study. However, performance thresholds are only one relevant factor in a broad set of methodological choices that contributed to the result seen here, as discussed below in more detail in Sect. 4.2.

Although unplanned, the combination of two key methodological decisions led directly to the observed result being possible. Had either decision been made differently, the non-overlap of the individual parameter clouds (Figs. 5 and 6) would have presented with different symptoms. The two key methodological decisions are explained in Fig. 9. The first decision was whether to require parameter sets to fulfil acceptance criteria for both variables separately (horizontal axis in Fig. 9), which we did not do. Instead, acceptance was assessed based on average performance across variables. Had this decision gone the other way, no parameter set in the top 100 for discharge would have also been in the top 100 for actual evapotranspiration, and vice versa. In this case, the symptom of non-overlapping clouds would be that no behavioural parameter sets would be found. The second decision was whether to require a pre-defined performance threshold to be met (vertical axis in Fig. 9), which we did not do. Instead, the top 100 parameter sets were accepted, whatever their performance may be. Had this decision gone the other way, it is unlikely that any parameter set would have met the threshold because, at best, they might be located in one behavioural cloud but not the other, and never both. Thus, they would all have a poor average score across the two variables, and (once again) the symptom of non-overlapping clouds would be that no behavioural parameter sets would be found. Consequently, all methodological options except the one adopted would lead to no behavioural sets and make the observed result impossible. It is, however, important to note that (i) we are not advocating one of these methods over the others, but rather reflecting on the circumstances that gave rise to the results; and (ii) the issue of non-overlapping parameter clouds is present irrespective of the chosen calibration approach or whether the non-overlap is detected during calibration or not.

It is possible that many prior studies have also come across the problem of non-overlapping clouds using one of the other three methods outlined above, and thus they identified no behavioural parameter sets. This null result is sufficient to scuttle a study before it has even begun, and so it is likely that those authors altered the parameters of their study until a workaround was found. If this is true, it would explain why few papers have reported similar findings. However, we recommend that future studies recognise such problems, report them, and attempt to tackle them directly rather than seeking workarounds that leave the core problem unresolved and unreported. This is particularly important in the case of model structural inadequacy, whereby we need greater insights to distinguish among candidate models and provide evidence-based reasons to move away from legacy-based model selection (Addor and Melsen, 2019). These ideas are expanded further in the following sections.

Differences in parameter distributions or trade-offs in model performance when using multiple calibration variables can be a sign of model structural limitations. For example, Meyer Oliveira et al. (2021) found contrasting soil water storage capacity values after calibrating the MGB model with discharge or actual evapotranspiration for the Amazonas. They attributed the large differences to parameter compensation effects caused by a misrepresentation of processes related to deep root water uptake during the dry season. Vervoort et al. (2014) modelled four Australian sub-catchments with the CMD model and showed that adding actual evapotranspiration to a discharge-based calibration improved the simulated water balance, but negatively affected the timing of discharge simulations. They argued that multivariate calibration left less room for the parameters to compensate for structural errors in the conceptualization of potential flow paths. In the context of a discharge-only calibration but using multiple seasons and performance criteria, Choi and Beven (2007) also found that no TOPMODEL parameter sets were behavioural over all conditions defined for the Bukmoon catchment (South Korea). Their results suggest that adapting the model to allow for the seasonally changing recession and evapotranspiration characteristics may be needed to improve discharge simulations. In our work, the non-overlapping (but often rather well constrained) parameter distributions and contrasting flux maps suggest very different hydrological responses for the same catchment when calibrating with discharge or actual evapotranspiration. A common observation among many catchments was that a calibration with actual evapotranspiration resulted in a large fraction of (infiltration excess) overland flow, whereas a calibration with discharge resulted in a combination of interflow and baseflow. Such a contrasting behaviour could potentially be caused by limitations in the conceptualization of the soil store. However, to better identify and understand structural limitations of the SIMHYD model, a much more comprehensive analysis targeted to detect limitations would be needed.

Given the wide range of available models and their application for diverse purposes, it is desirable to apply tests to identify their suitability, and we believe the multivariate methods shown here could inform such a model testing. Some of the most widely applied model testing frameworks are based on the four-level testing scheme proposed by Klemeš (1986) in which the model's transposability is tested in time and space. Various variants of the split sample test (SST), such as the generalized SST (GSST; Coron et al., 2012) or a generalized differential SST (GDSST; Dakhlaoui et al., 2017), have been developed to thoroughly test the temporal robustness of models under changing conditions. There is also a long history of large sample catchment studies, which were conducted to better understand the applicability of a model across hydrologically contrasting catchments (Knoben et al., 2020; Merz and Blöschl, 2004; Perrin et al., 2001; Seibert et al., 2018). These tests are typically conducted using locally observed discharge data although they are applicable to any type and number of hydrological data. For example, one can test a model by calibrating it with one variable (e.g., discharge) and evaluating it with withheld data (e.g., actual evapotranspiration), as done in this study. Thus, the calibration and evaluation with multiple hydrological variables can be seen as an additional dimension of the tests proposed by Klemeš (1986).

Interestingly, the benefits of multivariate calibration are rarely evaluated in the context of testing (as outlined above) and potentially rejecting a model structure. Multivariate calibration studies are more often motivated by the specific need for discharge simulations in ungauged basins (Hulsman et al., 2020; Liu et al., 2022; Zhang et al., 2020) or the more general interest in improving model realism within the limits of a chosen model structure (Demirel et al., 2019; Finger et al., 2015; Széles et al., 2020). Our work suggests that the value of multivariate model calibration goes beyond estimating parameter values, and we think multivariate calibration can serve as an insightful test of a model's capacity to represent hydrological behaviour, especially if a range of modelled aspects are considered. Indeed, by evaluating model performance values, we would have concluded that multivariate calibration improves the overall realism of the SIMHYD model (Figs. 2 and S2). However, the flux maps and the parameter distributions (Figs. 5–7) provided important insights into model structural issues and perverse outcomes such as some adopted parameter sets being “compromise” solutions in subregions not associated with either variable's behavioural subspace.

Our results were based on a few catchments in Australia and a single model structure. Yet, the chosen modelling set-up allowed us to test and falsify the two, often implicitly assumed, hypotheses motivating this study. As discussed by the philosopher Karl Popper (Popper, 2002), the observation of a single example contradicting a hypothesis is sufficient to reject it. To additionally evaluate the generalizability of our findings, we encourage modellers to further explore the potential of multivariate model calibration for testing models on a larger catchment data set, a wider set of different models, and other types of data. Open-source model collections like MARRMoT (Trotter et al., 2022) facilitate the testing and comparison of several existing model structures. Data to set-up, run, and evaluate hydrological models for many catchments across the world have also become easily accessible in recent years through the publication of data sets such as CAMELS (Addor et al., 2017; Alvarez-Garreton et al., 2018; Coxon et al., 2020; Fowler et al., 2024). The preparation of remote sensing-based hydrological variables other than discharge for hydrological modelling is still more tedious, but modelers can often choose between several publicly available products for a particular variable (Awasthi and Varade, 2021; Beck et al., 2021; Elnashar et al., 2021). A major challenge though is access to data quality information, especially in the context of large-sample hydrology. The consideration of potential errors in data, for example by a limits of acceptability approach (Freer et al., 2004), can ideally form part of a thorough model testing scheme.

The more recent combined availability of models and data provides a great opportunity to evaluate the appropriateness and limitations of a particular model structure or its parameter values. While we used discharge and actual evapotranspiration to select behavioural parameter sets in a Monte Carlo-based approach, a similar concept can be applied with any variable of interest. In fact, a detailed model testing with multivariate calibration will be most insightful if conducted for variables most relevant for a particular scientific or operational purpose, such as the use of groundwater or terrestrial water storage in catchments showing multi-year “memory” (Fowler et al., 2020) or the need to match vegetation dynamics that might be driving catchment behaviour (Gardiya Weligamage et al., 2023; Peterson et al., 2021). As discussed by Andréassian et al. (2009), we need to be merciless towards our own models by applying demanding crash-tests. Calibrating with multiple variables, aside from being motivated by model realism, can be a powerful additional crash test for our models.