Uncertainty sources in all environmental systems models include (e.g., Fig. 1; ) the model structure (e.g., selection of process equations or grid cell resolution ()), initial condition values (e.g., groundwater and soil moisture storage volumes ()), model parameter values , and input data (e.g., precipitation and temperature in ). If employing a stochastic modeling approach to these deterministic models , additional uncertainty sources include the choice of residual error model shape (e.g., lognormal) , the error model parameter values, and the observation data that are used to compute the residual errors . Each of these uncertainty sources could be considered in a sensitivity analysis.

In this paper, the sensitivity analyses consider parametric uncertainty for a fixed model structure and input data time series (described in Sect. ). We do not consider stochastic methods because we evaluate sensitivity in ungauged locations where no data are available to inform an error model. However, we do evaluate the impact of considering model error for the regression model that was used to estimate total nitrogen concentrations, as described in Sect. . We address uncertainty in the initial conditions for RHESSys by employing a 5-year spin-up period before using simulated outputs for analysis. After 5 years, the water storage volume (saturation deficit) averaged over the watershed maintained a nearly stationary mean value for each of the evaluated parameter sets (Supplement item S3).

In many hydrological studies, sensitivity analysis is used to understand how input parameters influence model performance measures , such as the Nash–Sutcliffe efficiency. Performance measures temporally aggregate a time series into a single value that is indicative of model fit to the observed data e.g.,. note that using such performance measures as sensitivity metrics amounts to a parameter identification study to discover which parameters may be adjusted to improve model fit. Therefore, the calibration-relevant sensitivity metrics in this paper use such performance measures on the full time series. Evaluating performance measures for subsets of the time series that describe specific features of interest should identify those parameters that control processes that generate those features e.g., timing vs. volume metrics in. Therefore, the decision-relevant sensitivity metrics are evaluated on subsets of the time series that are relevant to decision-making objectives. While such subsets could be used for model calibration, that is uncommon because the model would be less likely to perform well on other data subsets e.g.,. The following subsections present the decision- and calibration-relevant sensitivity metrics, which are also summarized in Table .

Sensitivity analysis methods can be local about a single point or global to summarize the effects of parameters on model outputs across the specified parameter domain e.g.,. A global method is implemented for this study because the goal is to screen parameters for use in model calibration. The Method of derivative-based sensitivity analysis is employed as a computationally fast method whose parameter rankings have been shown to be similar to more expensive variance-based analyses for spatially distributed environmental models .

The Method of Morris is based on elementary effects (EEs) that approximate the first derivative of the sensitivity metric with respect to a change in a parameter value. The EEs are computed by changing one parameter at a time along a trajectory, and comparing the change in sensitivity metric from one step in the trajectory to the next. The change is normalized by the relative change in the parameter value (Eq. ). Assuming that the pth parameter is changed on the (s+1)th step in the jth trajectory, the EE for parameter p using the computed sensitivity metrics (SMs) (SAE, NSE, etc.) is computed as shown in Eq. (): 6EE[j,p]=SM[j,s+1]-SM[j,s]Δs+1,s,p7Δs+1,s,p=X[j,s+1,p]-X[j,s,p]|max(X[:,:,p])-min(X[:,:,p])|, where EE is the elementary effect matrix consisting of one row per trajectory and one column per parameter, Δs+1,s,p is the change in the value of the parameter as a fraction of the selected parameter range, and X is the matrix of parameter values. The EEs for each parameter are typically computed in tens to hundreds of locations in the parameter domain, and are then summarized to evaluate global parameter importance. The mean absolute value of the EEs computed over all of the r locations (one for each trajectory) is the summary statistic used to rank model sensitivity to each parameter, as recommended by . The sample estimator is provided in Eq. (): 8μ^p*=1r∑j=1r|EE[j,p]|.

We used 40 trajectories that were initialized by a Latin hypercube sample, and used the R sensitivity package to generate sample points and compute EEs. Each parameter had 100 possible levels that were uniformly spaced across its specified range. Step changes, Δ, in parameter values were set to 50 levels (i.e., 50 % of their range). For each parameter, this allows for a uniform distribution of parameter values across all samples (example sampling distributions for other percentages are provided in Supplement item S8). We adjusted some trajectory sampling points to satisfy inequality and simplex constraints within the RHESSys model (described in Supplement item S0).

After the hydrological model runs completed for all trajectories, we estimated 90 % confidence intervals for each parameter's μ^p* by bootstrapping. For each parameter, 1000 EE vectors of length r had their elements sampled with replacement from the original r EEs, and μ^p* was computed for each vector. We independently completed bootstrapping for each parameter as in the SALib implementation by instead of sampling whole Morris trajectories as in the STAR-VARS implementation by to allow greater variation in the resulting quantile estimates.

We used an EE cutoff to determine which parameters would be selected for calibration. For each sensitivity metric, we determined the bootstrapped mean EE value (Eq. ) corresponding to the top Xth percentile, after removing parameters whose EEs were equal to zero. All of the parameters whose estimated 95th percentile EE values were greater than this cutoff value would be selected for calibration for that metric. The union of parameters selected from all sensitivity metrics comprised the final set of calibration parameters. We evaluated the number of parameters selected as a function of the Xth percentile cutoff for basin and hillslope outlet sensitivity analyses in Sect. . Subsequent results are presented for the 10th percentile as an example cutoff; in practice, the cutoff value should be defined separately for each sensitivity metric based on a meaningful change for the decision maker (e.g., the ϵ-tolerance in optimization problems; ). To test the hypothesis of spatial variability in parameters that affect the sensitivity metrics, we compare parameters that would be selected based on each hillslope's EEs against each other and the basin outlet selection.

We compare the EEs for parameters that are traditionally adjusted by the same multiplier to determine if all parameter EEs are meaningfully large and not statistically significantly different from each other. This would suggest that a multiplier or another regularization method may be useful to reduce the dimensionality of the calibration problem. Parameters with large and statistically significantly different EEs are candidates for being calibrated individually, as this suggests the multiplier would not uniformly influence the model outputs across adjusted parameters. More investigation on the cause for different EEs could inform the decision to calibrate individually or use a multiplier (e.g., the difference in sensitivity could be caused by the parameters acting in vastly different proportions of the watershed area). We evaluate significance using the bootstrapped 90 % confidence intervals.

We used the Weighted Regression on Time Discharge and Season (WRTDS) method to estimate daily TN concentration as a function of simulated streamflows. Equation () provides the regression model 9ln(CTN,t)=β0+β1ln(Qt)+β2t+β3sin(2πt)+β4cos(2πt)+ϵ, where CTN,t is the TN concentration, βi is the ith regression model parameter, Qt is the streamflow (discharge), t∈R is a time index in years, and ϵ is residual error. The sin and cos terms model an annual cycle. We estimated regression model parameters using the observed basin outlet streamflow and TN data. The parameter estimation procedure employs a local window approach to weight observations by their proximity in t, Qt, and day of the year. Default values of these three WRTDS window parameters did not simulate the interquartile range of TN observations well, so we used a manual selection of WRTDS parameters to improve the model fit, as described in the Supplement (item S0). Furthermore, adding a quadratic log flow term did not result in a meaningful improvement, so we used the simpler Eq. () model.

In order to use WRTDS for any streamflow value within the observation time period, we created two-dimensional (t, Qt) interpolation tables for each of the five model parameters and the residual error (provided in Supplement item S6). Simulated flows that were outside of the observed range of values were assigned the parameters for the nearest flow value in the table. Extrapolation of the concentration–flow relationship to more extreme flows than were historically observed may provide inaccurate TN estimates, which is a limitation of this statistical prediction method. We expect the error from extrapolation in this basin to be low, as N loads appear to be dominated by effluent from septic systems as evidenced by isotopic sourcing p. 8229, and septic effluent supply should be fairly steady over time. Zero flows were assigned zero concentration. These interpolation tables apply only to the concentration–flow relationship at the basin outlet. We did not estimate TN for hillslopes due to a concern that this basin outlet relationship would overestimate TN in predominately forested hillslopes that would have different concentration–flow relationships and in this watershed do not have septic sources of TN. As a result, parameter selection for hillslopes is limited to the three streamflow sensitivity metrics.

We first compare the number of parameters selected for calibration based upon decision-relevant elementary effects (EEs) whose mean or 95th percentile estimates are larger than the Xth percentile cutoff. Figure  shows the total number of unique parameters (out of 102 with non-zero EEs) that would be selected for calibration as a function of the cutoff value. The plotted total is the union of the top X percent across the six decision-relevant metrics for the basin outlet, and across the three streamflow metrics for hillslope outlets, so more than X percent may be selected at each cutoff value. For hillslope outlets, the union is also computed over all hillslopes. The gap in number of parameters selected when using hillslope outlets instead of the basin outlet suggests that parameters that control physical processes captured by the streamflow sensitivity metrics are different across the watershed, as explored further in Fig. . For this problem, considering sensitivity metrics for hillslope outlets commonly results in an additional 10–20 parameters selected for calibration compared with only using the basin outlet. There can be as many as 40 more parameters near the X=50% cutoff. For basin and hillslope outlets, the gap between using the bootstrapped 95th percentile EE values instead of the mean values illustrates the importance of considering sampling uncertainty in parameter screening exercises. For this problem, sampling uncertainty commonly adds 5–15 additional parameters. Near the X=50% cutoff, almost all parameters would be selected for calibration using the hillslope outlets and 95th percentile EE values. If desired, these sampling uncertainty gaps can be reduced by evaluating more Morris trajectories (e.g., by using progressive Latin hypercube sampling to add new trajectory starting points, as in ). This should bring the mean and 95th percentile lines closer together in this figure.

For the selected 10 % cutoff in Fig. , 21 unique parameters would be selected for the basin outlet using the 95th percentile EE values. Of these parameters, 18 are selected based on the three streamflow metrics and 19 are selected based on the three TN metrics. This finding supports using sensitivity metrics for each of a decision maker's objectives to inform which parameters to calibrate.

Basin outlet EEs are displayed in Fig.  by parameter category (color) and type within each category (shape). Of the 237 parameters, 135 had EE values of exactly 0 for all metrics (i.e., these parameters do not affect model-predicted streamflow). These parameters primarily affect the RHESSys nitrogen cycle and vegetation growth (which are not used in static mode), buildings, and some snow parameters. For streamflow sensitivity metrics (left column), differences in the selected parameters and their EEs across metrics suggest that flows of different magnitudes are affected by different physical processes, as expected e.g.,. For example, hillslope groundwater controls (index 1) and saturation to groundwater controls for compacted other soil (index 93) that affect how water moves from groundwater to riparian areas are selected parameters for each metric, but their EEs for low flows are larger than for the other metrics. This is likely because groundwater would be the source of low flows. The EE magnitude for the specific rain capacity (interception storage capacity per leaf area index (LAI)) of trees (index 162) increases from flood flows to low flows. This result suggests that the impact of water intercepted by vegetation surfaces matters more for low flows, particularly in drought-stressed ecosystems, as that water alternatively reaching the ground would have a larger impact on the resulting stormflow hydrograph compared with non-drought conditions e.g.,. Septic water loads (index 13), which are modeled as constant throughout the year, have a higher mean EE for flood flows than the other streamflow metrics. This could result from uncertainty in saturated soil storage volumes leading to uncertainty in flood peaks. Similarly, the EE magnitude for tree maximum stomatal conductivity (index 119) is larger for flood flows, likely because of the impact on how quickly water can be transpired by trees. Finally, the EE for wind speed is largest for flood flows, which could be explained by the impact of wind on transpiration and the resulting reduction in the recessive limb of the hydrograph e.g.,. Other parameters with larger EEs generally describe soil properties that are selected or are near the cutoff point for each streamflow metric. The largest of these for all metrics is the coefficient that describes bypass flow for other soils (index 73) which cover the largest area of the lower elevations in the watershed (Fig. b).

For the three TN metrics (Fig. , right column), the parameters within the top 10 largest mean EEs are the same and their order is nearly identical when considering uncertainty. The largest EEs are close in magnitude to the 5th to 95th percentile streamflow metric. These results make sense because the TN metrics are all affected by the same streamflows, and sample collection is often limited to low and moderate flow conditions . The reason for differences in which parameters are selected for calibration is uncertainty in the mean EE. The EE error bars tend to be larger for the upper 95th percentile TN estimate, which results in the selection of more parameters to calibrate. This result demonstrates the value of considering both model error (different TN quantile estimates) and uncertainty in sensitivity (bootstrapped EE estimates) when selecting which parameters to calibrate. More parameters are found to be potentially influential when considering these sources of uncertainty.

For hillslope outlets, 37 unique parameters were selected using the 10 % cutoff and the 95th percentile EE values (Fig. ). This parameter set contained all of the parameters identified using only the basin outlet. Those 37 parameters are listed in Fig. a and b, which compare results for each hillslope and the basin outlet. Figure a provides the rank of mean EEs for the upper 95th percentile streamflow sensitivity metric. We provide plots for the other two streamflow sensitivity metrics in the Supplement (item S4). Figure b is aggregated over all decision-relevant sensitivity metrics (and spatial areas for hillslopes) and indicates whether or not the parameter would be selected for calibration.

Figure a for the flood flow sensitivity metric shows that the previously described parameters with high mean EE ranks based on the basin outlet tend to also have high mean EE ranks in all hillslopes. Septic water load and riparian soil m (hydraulic conductivity decay with saturation deficit) are exceptions, which only affect hillslopes with households and modeled riparian soils, respectively. Whether or not a hillslope is more forested or impervious explains many parameter rank differences among hillslopes (e.g., the percent impervious parameters). Tree parameters overall have higher ranks for more forested hillslopes, and grass parameters have higher ranks in more impervious hillslopes, which also have more grass areas. Compacted soils S108 and S109 have higher parameter ranks in more impervious hillslopes where these soils have larger proportions of the total hillslope area relative to more forested hillslopes. Coverage area of riparian soils is less than other soils and these soils tend to be wet regardless of the conductivity value due to spatial position, which could explain why riparian parameters tend to have smaller ranks than other soil parameters. While it is not surprising that parameter EE ranks vary across the watershed according to the hillslope features and respective processes that act in those areas e.g.,, this result demonstrates that evaluating sensitivity metrics across a watershed can lead to a different interpretation of which parameters are important to calibrate compared with evaluations completed for the outlet where calibration data are located.

Figure b further explores this point by showing which parameters would be selected for calibration using basin and hillslope analyses if aggregating the top 10 % over all decision-relevant sensitivity metrics. Comparing the parameters selected in Fig. b to their ranks for the flood flow sensitivity metric in Fig. a reveals that some lower-ranked parameters for flood flow are ultimately selected for calibration. This result supports selecting parameters based on multiple sensitivity metrics that represent all of a decision maker's objectives. Furthermore, several parameters that would be selected for hillslope analyses would not be selected for the basin analysis if sensitivity metrics were not aggregated over space, with riparian soil parameters being the most common. Three tree parameters and both grass parameters were also selected for a few hillslopes that are almost completely forested or have large grass areas, respectively, yet would not be selected for the basin analysis. Parameters that are selected for hillslopes but not for the basin would exert relatively smaller signals when calibrating to the basin outlet data, and would likely introduce equifinality to the calibration. However, there is value in considering such parametric uncertainty if the parameters have a meaningful contribution to the sensitivity of decision objectives nearer to the spatial scale of decision making i.e., within the representative elementary watershed. Specifically, engineering designs that would affect flows at these spatial scales and locations ought to be robust to the parametric uncertainty in flows that would likely result from calibration of these parameters. This point is discussed further in Sect. .

Figure  provides plots of parameter EEs for the four calibration-relevant sensitivity metrics. The parameters with the largest EEs are nearly identical for the NSE, LNSE, and pBias metrics, and the EE magnitudes are closest to the 5th to 95th percentile streamflow metric. (These metrics are highly correlated, as shown in item S7 in the Supplement.) Contrasting these results with Fig.  suggests that the NSE and LNSE are not sufficient to capture parameters that affect flood and low flows, contrary to reasoning often provided as justification for their use. The log-likelihood metric shows large EEs for many of the same parameters as other calibration and decision-relevant metrics; however, the magnitudes and rankings of parameters are different, and some new parameters are selected. Note that all parameters have non-zero EEs for the LogL metric as a result of equifinality in the parameters obtained from maximum likelihood estimation. The 10 % threshold cutoff used to select parameters for calibration is larger than the resulting noise that is introduced into the EE values.

Figure  presents a plot indicating whether or not each parameter would be selected for calibration using the calibration-relevant and decision-relevant sensitivity metrics. Note that the calibration-relevant metrics did not identify any new parameters than the decision-relevant metrics evaluated across hillslopes (All, H), so the y axis matches Fig. a and b. Considering only basin outlet evaluations (All, B), decision-relevant metrics identify five parameters that the calibration-relevant metrics do not identify. These parameters include two atmospheric parameters that were selected from the flood flow decision metric and a soil parameter that was selected from the low flow decision metric. The other two parameters were selected by considering model error in TN. Of the calibration-relevant metrics, only the log likelihood metric (LogL, B) identifies parameters that are unique from all other basin-evaluated metrics, but these parameters are selected for hillslopes using decision-relevant metrics (All, H). Of note is that a set of 10 parameters are selected for each of the calibration-relevant metrics and the aggregated decision-relevant metrics, and a set of 13 parameters are only selected from hillslope evaluation of the decision-relevant streamflow metrics. This result strengthens the recommendation to spatially evaluate sensitivity metrics to inform parameter selection of spatially distributed models.

When sensitivity analysis is used to inform model calibrations, a primary goal is usually to reduce the dimensionality of the search space by screening those parameters that most affect the outputs to be calibrated. How model outputs are considered in sensitivity analyses and subsequent screening exercises can affect which parameters are selected. We found that specifically evaluating high and low flows as decision-relevant metrics provided a different parameter selection than using the calibration-relevant metrics that are often used to capture parameters that control such flows. While the NSE is mathematically sensitive (i.e., not robust) to high flows, the EE magnitudes and parameters that are selected by the NSE sensitivity metric do not match well with those selected from the high flows decision-relevant metric. Instead, the EE magnitudes and selected parameters resemble the 5th to 95th percentile streamflow metric. A similar result is obtained for the LNSE metric. A possible explanation for these results is that the high and low flows sensitivity metrics each represent only 5 % of the time series used in the NSE and LNSE metrics, while the 5th–95th percentile metric represents 90 % of the time series. Another possibility is that in the Baisman Run watershed, flows greater than the 95th percentile are still relatively small, and so the model residuals are a similar order of magnitude for peak flows and other flows. Regardless of the cause, this analysis demonstrates that parameter selection based on decision-relevant metrics can result in different parameters than calibration-relevant metrics. Thus, these results support future studies that would evaluate which parameter screening method is ultimately preferable for various decision problems. For example, this could be assessed by optimizing engineering designs for controlling high and low flows using models that calibrate screened parameters from the two alternative approaches.

Calibration-relevant metrics have limited value for sensitivity analyses of spatially distributed models because they can only be computed at gauged locations. The sensitivity analyses that we completed for ungauged hillslope outlets led to the identification of more parameters to calibrate than were selected based on sensitivity analysis at the gauged basin outlet. Calibrating additional parameters that have smaller impact at the gauged location is likely to exacerbate equifinality in simulated outputs. Equifinality at the basin outlet will often result in variability in outputs at ungauged locations, such that calibration of these additional parameters should be important to better capture the physical processes in hillslopes where engineering controls could be located. Even if parameter values are unchanged from their prior distributions after calibration, locations of engineering control measures can be optimized to be robust to the resulting uncertainty in model outputs across the watershed. Spatially distributed monitoring of model parameters and streamflow gauges within sub-catchments could help to reduce this uncertainty, particularly for catchments with spatially heterogeneous characteristics. In summary, spatial evaluation of sensitivity metrics for spatially distributed models allows for the discovery of parametric sources of uncertainty across the watershed to which engineering designs would have to be robust.

Spatial sensitivity analyses also reveal opportunities to reduce parametric uncertainty by using additional data and data types. Parametric uncertainty could be reduced for any parameter by better constraining its prior range. For example, septic water loads could be constrained with household water consumption surveys. Surveys and data collection efforts for other parameters can target those hillslopes for which model sensitivity is largest. Alternatively, some of the parameters could instead be specified by additional input datasets to reduce the dimensionality of the calibration. For example, impervious surface percentage could be specified spatially from the land cover dataset, and time series of wind speed may be obtained from weather gauges or satellite data and then be processed to the spatial scale of the model. These approaches would transfer parametric uncertainty to input data uncertainty, which would ideally be negligible. Finally, uncertainty may be reduced by better capturing spatial trends in parameter values, for example, using finer-resolution soils data products, such as POLARIS estimates , or implementing different vegetation species composition in riparian and non-riparian areas. However, both of these approaches change the RHESSys model structure and add more parameters, so it is unclear if total uncertainty would be reduced, even if local hillslope performance is improved. Nevertheless, preliminary analysis with an uncalibrated RHESSys model in dynamic mode found that simulated streamflow and nitrogen were better aligned with observations when a more spatially explicit soil and vegetation parameterization was used (; vegetation by plant functional type is described in ). Similar performance was observed for soils data by using RHESSys and by using a SAC-SMA model. This lends support to future analyses that consider sensitivity analysis of alternative model structures and parameters to discover dominant processes, as in and . The selected parameters across water quantity and quality-focused metrics would likely be different if TN concentrations were estimated from a process-based model, as in the dynamic mode of RHESSys, instead of statistically as a function of streamflow using WRTDS (e.g., RHESSys and WRTDS estimations are compared in ).

Parameter multipliers and other regularization methods are a common dimensionality reduction choice for spatially distributed models. A comparison of model sensitivity results for parameters that can be adjusted by built-in RHESSys multipliers revealed opportunities for dimensionality reduction by a multiplier, and also identified some parameters that may be better to calibrate individually for this problem. Future research is needed to formally test these recommendations for their impact on model calibration.

For RHESSys streamflow simulations, the global sensitivity analysis identified some parameters for calibration that are not commonly calibrated and should therefore be assigned priors that are adjusted to local site conditions. Studies of other models, such as NOAH-MP , have reached similar conclusions about the need to calibrate parameters that are commonly fixed. For example, in RHESSys, zone (atmospheric) parameters are typically assigned fixed site values, but this analysis suggests careful examination should be given to parameters that adjust the estimated average temperature based on the supplied minimum and maximum temperature time series. For vegetation species simulated in static mode, this analysis revealed that stomatal and leaf conductivity parameters, interception storage capacity parameters, and the parameter that sets the first day leaves show on deciduous trees were among the most important for modeling streamflow. For primarily forested hillslopes, parameters describing the length of time that leaves open and fall are also important. In addition to these parameters that are not adjusted by built-in RHESSys multipliers, many of the soil and groundwater parameters that are adjusted by multipliers were also identified as important to calibrate, which is common practice.

This paper focuses on the importance of evaluating sensitivity analyses at the spatial scale and magnitude that is appropriate for decision making. Selecting the appropriate temporal resolution for the sensitivity metric and the time period of sensitivity analysis is also important to inform parameter selection. All of the sensitivity metrics in this paper are temporally aggregated measures instead of time-varied. With this approach, two model runs could have very different simulated time series, yet could have similar metric values. Additionally, parameters that arise from different generating processes (e.g., floods from spring snowmelts vs. summer hurricanes) would not necessarily be parsed out from any one model run. For engineering problems, a magnitude-varying sensitivity analysis could be useful to identify those parameters that control specific extreme events. A time-varying sensitivity analysis could discover more seasonally important parameters. Related to this point, this sensitivity analysis was completed for a short 6-year period. For engineering designs that will last several decades, model sensitivity to alternative climate futures would be useful to identify additional parameters to calibrate that could become important in future climates, even if they are not historically important. Considering uncertainty in these parameters for optimizations under future climatic conditions would allow engineering designs to be robust to their uncertainty. Outside of an engineering context, showed that selecting parameters that control processes within sub-catchments is important when using calibrated models for climate change forecasts.

A final consideration for risk-based decision making is the use of deterministic or stochastic watershed models. We found that sensitivity metrics for TN model residual error resulted in a different set of parameters to calibrate than using the mean of TN. This result suggests that sensitivity analysis of stochastic watershed models could lead to different parameter selection. Future work is needed to compare sensitivity analysis and resulting parameter selection for deterministic and stochastic watershed models.