Model structure uncertainty is known to be one of the three main sources of hydrologic model uncertainty along with input and parameter uncertainty. Some recent hydrological modeling frameworks address model structure uncertainty by supporting multiple options for representing hydrological processes. It is, however, still unclear how best to analyze structural sensitivity using these frameworks. In this work, we apply the extended Sobol' sensitivity analysis (xSSA) method that operates on grouped parameters rather than individual parameters. The method can estimate not only traditional model parameter sensitivities but is also able to provide measures of the sensitivities of process options (e.g., linear vs. non-linear storage) and sensitivities of model processes (e.g., infiltration vs. baseflow) with respect to a model output. Key to the xSSA method's applicability to process option and process sensitivity is the novel introduction of process option weights in the Raven hydrological modeling framework. The method is applied to both artificial benchmark models and a watershed model built with the Raven framework. The results show that (1) the xSSA method provides sensitivity estimates consistent with those derived analytically for individual as well as grouped parameters linked to model structure. (2) The xSSA method with process weighting is computationally less expensive than the alternative aggregate sensitivity analysis approach performed for the exhaustive set of structural model configurations, with savings of 81.9 % for the benchmark model and 98.6 % for the watershed case study. (3) The xSSA method applied to the hydrologic case study analyzing simulated streamflow showed that model parameters adjusting forcing functions were responsible for 42.1 % of the overall model variability, while surface processes cause 38.5 % of the overall model variability in a mountainous catchment; such information may readily inform model calibration and uncertainty analysis. (4) The analysis of time-dependent process sensitivities regarding simulated streamflow is a helpful tool for understanding model internal dynamics over the course of the year.

Hydrologic processes such as infiltration of water into soil or water interception by the canopy of
trees are often too complex to parameterize or insufficiently understood at scales of interest to be
represented in every detail in computer models. The consequence of this is that simplified
conceptual or empirical models are often used to represent these physical processes; such models are
typically computationally expedient and possess a relatively smaller number of parameters than
continuum models based upon the

Model structural uncertainty is commonly recognized

To date, there have been limited attempts to simultaneously estimate model parameter, input, and
structural sensitivities. One notable attempt is introduced by

In all the cases above, the resultant sensitivity metrics may be useful for (e.g.,) differentiating between the magnitude of model sensitivity to structure vs. that of parameters. However, these methods cannot be used to provide insight into the sensitivity of individual model structural choices, nor can they be used to disentangle the complex relations between model parameter and structure sensitivities or the interplay between interacting model structures. It is therefore difficult to use such methods to identify preferred model structures to inform the process of model calibration. In addition, none of the methods that derive sensitivities across multiple model structures recognize the fact that model parameters may be present or absent conditional upon the model structure. Lastly, the above-mentioned methods are generally computationally expensive, are only available for a small number of process parameterization options, and only determine the sensitivity of the parameterization in general without providing insight into what is causing this sensitivity by analyzing, for example, the sensitivity of individual parameterization options or individual parameters. While potentially useful for some applications, the available approaches have either not been applied or do not allow for such an in-depth analysis of model structure and hence might provide only limited support for improvements in hydrologic modeling.

Two main contributions of this work are (a) to reformulate a hydrologic modeling framework so that it can define model structure by weighting or blending of discrete model process options
continuously for simulating process-level hydrologic fluxes and (b) to propose a technique, the extended Sobol' sensitivity analysis (xSSA) method, based on the existing concept of grouping parameters when applying the Sobol' method

The xSSA method allows us to efficiently estimate not only the global sensitivity of model
parameters independently and hence unconditionally of the chosen model structure, but also to evaluate the sensitivity of alternative model process options (e.g., that of different snowmelt algorithms)
and the sensitivity of hydrological process components (e.g., snowmelt vs. infiltration). We here
pose these as four distinct sensitivity metrics.

Conditional parameter sensitivity.

Unconditional parameter sensitivity.

Process option sensitivity.

Process sensitivity.

Below, we define these metrics explicitly and introduce the xSSA methodology for calculating them. The xSSA method is tested using two artificial benchmark models to check for consistency between analytically and numerically derived sensitivity index estimates. The proposed method is also compared to the existing DVM revealing limitations that can be resolved using the xSSA method. The xSSA method is then applied to a hydrologic modeling case study using the Raven hydrologic modeling framework, demonstrating the insights that may be gained through the simultaneous in-depth analysis of model parameters and model structures to improve hydrologic modeling practices.

We propose a method for estimating how sensitive a simulated model output is to groups of parameters. We have chosen here streamflow as this model output as it is the fundamental and most important and common output variable in hydrologic studies. The sensitivities of the groups of parameters are hence obtained regarding streamflow. The groups defined here are either individual parameters (metric b) or the set of parameters that is used in an individual process option (metric c) or all parameters used in any available process option for a modeled process (metric d). We acknowledge that the definition of these groups is subjective and has been chosen here to demonstrate a novel approach to how to evaluate process and process option sensitivities, i.e., how sensitive the simulated streamflow is regarding the choice of a specific infiltration process description or how sensitive the simulated streamflow is regarding infiltration in general. We also wish to mention that the terms “sensitive” and “influential” are used interchangeably throughout this work.

The section will first introduce the models and their setups
(Sect.

This section will briefly introduce the three test cases used to demonstrate the functioning of the
xSSA. The first two test cases are artificial benchmark models where the sensitivity index values
can be derived analytically (Sect.

The artificial benchmark models are employed to demonstrate that the proposed method is capable of
deriving the sensitivities of not only individual model parameters but also of grouped parameters
linked to individual process options (e.g., the linear baseflow algorithm) or processes regardless
of available options (e.g., baseflow). The benchmark models are further used to demonstrate
limitations of existing methods that were previously used to analyze model structure
sensitivities. We use two hypothetical models here: one model where each model parameter is only used in distinct processes and process options (disjoint-parameter benchmark;
Fig.

The three model setups used in this study. The first two serve as an artificial benchmark model since sensitivities of parameters, process options and processes can be derived
analytically. The benchmark examples consist of three processes:

The model that is built using the first process options

The Sobol' sensitivity indexes of all 12 model configurations can be derived analytically following
closely the description in

A reasonable approach for evaluating the sensitivity of 12 individual models involves choosing exactly one process option for each process in Eq. (

To sample the continuum of all process options, the weights need to be independently and identically distributed (iid). Therefore, random numbers

The approach of weighted model options hence comes at the expense of introducing additional parameters

The analytically derived Sobol' indexes for the remaining three sensitivity metrics (b–d) can be derived using the revised model description (Eq.

The Raven hydrologic modeling framework developed by James R. Craig at the University of Waterloo

In Raven, the user defines the model as a list of hydrologic processes which move water between
storage compartments corresponding to physical stores (e.g., topsoil, canopy, snowpack). The list
determines the state variables, connections between stores and the parameters required. For each
hydrologic process, several options of process algorithms are implemented. There are, for example,
14 infiltration process options available. Amongst others, the GR4J

Raven has another unique feature relative to other modular frameworks: Rather than selecting one process option (e.g., HMETS method for estimation of infiltration runoff fluxes) one can specify multiple process options (e.g., HMETS, VIC/ARNO, and HBV) and define weights for each option (e.g., 0.4, 0.5, and 0.1, respectively). Raven then uses the weighted sum of the fluxes calculated by the process options internally. Raven is run only once and not multiple times to obtain the outputs for the multiple process options. Based on this feature, we chose Raven as model for our study. Please note that the proposed sensitivity method is applicable for any multi-model framework that allows to mix-and-match process descriptions. However, in the case of a framework without weights for process options, the application of the method would be much less efficient.

For the case study used herein, Raven is applied in lumped mode, and the models are solved using the ordered series numerical scheme defined in

This selection of process options results in

Figure

The Salmon River catchment located in the Canadian Rocky Mountains in British Columbia is selected
as the study watershed. The domain is depicted in Fig.

Meteorological inputs are obtained from Natural Resources Canada on an approximately

The lumped model was set up for the simulation period from 1 January 1989 to 31 December 2010, while the first 2 years were discarded as warm-up. Hence, 20

In this section we briefly describe the Sobol' method

Traditionally, the Sobol' sensitivity analysis – as all other methods – focuses on the sensitivity of model parameters. In the case of multiple models, one would typically run the analysis individually for each model and might aggregate the sensitivity index estimates for parameters that are active in multiple models. This, however, may underestimate the sensitivity of parameters that are active in only a few models, while parameters that are active in almost all the models might be overestimated in their aggregated sensitivity.

We here briefly revisit the implementation of the traditional Sobol' method to emphasize the differences with the extended method we propose that will be able to handle multiple process options and derive (overall) parameter sensitivities, sensitivities of process options, and sensitivities of whole processes.

Usually two Sobol' indexes are derived: the main and total Sobol' indexes. The main Sobol' index

Similarly, the total Sobol' index

For the numerical estimation of the indexes

Out of the 12 possible shared-parameter benchmark models (Eq.

The Sobol' method is here generalized to groups of parameters

The numerical approximation of these indexes is similar to the traditional approach. It is again
based on the two matrices

The xSSA method can be used to derive conventional model parameter sensitivities by using groups that contain exactly one of the model parameters

As an example (see Fig.

The shared-parameter benchmark model consists of 7 model parameters

Four experiments will be performed for each of the two benchmark models to demonstrate that the
proposed method is able to obtain the analytically derived values available for the the benchmark
examples (Sect.

The artificial benchmark models are used to prove that the proposed method of extended Sobol' sensitivity indexes and its implementation is working. They are furthermore employed to demonstrate
some limitations of the existing method proposed by

The xSSA method is tested using different budgets to show that numerical values indeed converge. The
number of reference sets used are

The DVM also applied to the two benchmark models was set up using the same computational budgets

The errors between approximated main effects

The Salmon watershed model is analyzed using the xSSA method with

The budget for the sensitivity metric b analyzing unconditional sensitivity of parameters
independent of choice of model options results in

The sensitivities are determined for simulated streamflow

We further analyze the time-dependent behavior of process sensitivities to reveal temporal patterns in the importance of processes at different times of the year. Therefore, the total process
sensitivity

We will present the results of the extended Sobol' sensitivity analysis (xSSA) applicable not only to model parameters, but also to model process options and processes. First, the xSSA method will be
compared to the existing discrete value method (DVM) to derive sensitivities of groups of parameters introduced by

In this section the proposed xSSA method based on grouped parameters and weighted process options
will be compared against analytically derived Sobol' sensitivity indexes for both benchmark problems (Fig.

The DVM defines the term “sources of uncertainty” (

To demonstrate this major difference of the DVM and the proposed xSSA concept, we define three groups (sources of uncertainty) as the processes

Error between approximated and analytically derived total Sobol' sensitivity index estimates for processes of the benchmark models

The shared-parameter benchmark model (Fig.

We also tested several numbers of pre-sampled parameter sets

The xSSA method does not pre-sample parameters. When one group is analyzed, all parameters contained
in this group get perturbed. The

It is notable that several publications that are considering structural sensitivities so far have been limited by the disjoint definition of parameter groups

The shared-parameter benchmark setup is utilized to compare the xSSA-derived numerical sensitivity metric values with the analytically derived, correct sensitivity metric values for all three
metrics: parameters, process options, and processes. We have chosen the shared-parameter model over the disjoint-parameter benchmark model here as it appears to be the more difficult model to analyze. The
errors converge to zero in every analysis, and this proves that the implementation of the extended Sobol' sensitivity analysis is coherent with the analytical theory
(Fig.

Error between xSSA approximated and analytically derived Sobol' sensitivity index estimates

It holds for all three analyses that the model parameters/options/processes with the largest sensitivities converge slowest. For example, model parameter

It can be noted that the weight-generating random numbers

Each subsection here focuses on one experiment performed using the hydrologic modeling framework
Raven. The subsections will address the results of the unconditional parameter sensitivities
(Sect.

It is important to note that the results of the analysis for the hydrologic framework are the product of an iterative process where the intermediate results are not quantitatively reported on. Qualitatively, we wish to emphasize that the intermediate xSSA-based results helped us to improve the modeling framework by identifying sources of model instabilities and non-intuitive model results. It was especially helpful to have estimates for aggregated model compartments, i.e., process options and processes. We strongly believe that this kind of analysis will help to analyze the hydrologic realism of models since the estimates are easier to interpret and to compare to experience and known evidence.

The variance-weighted main and total Sobol' sensitivity estimates

Results of the Sobol' sensitivity analysis of the hydrologic modeling framework Raven.

Besides that, the most influential parameters are the weight-generating random variables associated with processes that are most sensitive (indicated by the same color in
Fig.

A sensitivity analysis regarding model parameters is often performed prior to model calibration to
identify the most sensitive parameters, which are in turn the parameters that are most likely to be identifiable during calibration. The analysis shows that 13 of the 35 parameters (

This analysis helps to identify the most influential parameters independently of model structure and therefore helps to identify the main sources of parametric uncertainty in models despite structural configuration, presuming that individual structures are equally viable. It likewise determines non-identifiable parameters – as a traditional sensitivity analysis does – with respect to streamflow.

The results of the sensitivity analysis of model parameters are consistent with the analysis of
process options (Fig.

The potential melt process – the algorithm used to determine incoming melt energy – is used in
this study only with one process option (POTMELT_HMETS), and it is still the hydrologic process that the simulated streamflow is most sensitive to (orange bar). The three infiltration options are all
equally sensitive (light blue bars). The same holds for the two options of the evaporation process
(dark blue bars) which are slightly more influential than the infiltration processes (light blue
bars). The quickflow options BASE_VIC and BASE_TOPMODEL (medium blue bars) are the second-most
sensitive after the potential melt. The quickflow option BASE_LINEAR_ANALYTIC however is much less
sensitive. The two baseflow options (dark green bars) as well as the convolution options for surface
and delayed runoff (yellow and light green bars) exhibit almost no influence with sensitivity metrics near zero (

In summary, it can be deduced that the potential melt, the quickflow options BASE_VIC and BASE_TOPMODEL, and the evaporation options are most influential upon modeled streamflow. The interpretation and use of this process option sensitivity is open, and depends upon the purpose of the sensitivity analysis. As an example of interpretation, we can consider whether or not we wish to maximize the flexibility of our models in calibration, and if so, we may wish to discard insensitive processes. The three infiltration options are equally sensitive and hence are all able to achieve the same amount of variability in simulated streamflow time series. This similarity is an indicator that the choice of the infiltration option will therefore not influence the model performance.

This analysis of process options allows us, for the first time, to objectively compare model process options by mixing-and-matching all of them through the approach of a weighted mean of all outputs. It can assist the setup of models by guiding choices of process options and hence guides model structure decisions depending on the purpose of the model built.

The sensitivity analysis of the 11 processes (Fig.

Processes associated with the surface are quickflow

The soil-related processes of infiltration

Baseflow

This model structure-based sensitivity analysis can help to guide model development by targeting the dominant model processes. It derives a high-level sensitivity of the main model components, i.e., processes. It reveals, for the first time, the sensitivity of model processes independently of model structure chosen and hence is one step towards sensitivity analyses regarding model structure using a true model ensemble by mixing-and-matching a variety of model process options.

The previous analysis estimated the time-aggregated sensitivities

Results of the Sobol' sensitivity analysis of the hydrologic modeling framework Raven. The
sensitivities of 11 processes are shown as their averages per day of the year (colored bars). The simulation period is 1991 to 2010. The sensitivities are normalized such that they sum
up to 1.0 at every day of the year (Eq.

Infiltration (light blue) has an almost constant but minor sensitivity throughout the whole year. Quickflow (medium blue) is most of the time the dominating process – especially in summer – but not during the high-flow melt season. Evaporation (dark blue) is consistently responsible for about 35.4 % of the sensitivity during summer (June to October) and is, expectedly, less sensitive during winter. Snow balance (medium green) and potential melt (orange) are sensitive as long as snow is present (November to May). Potential melt is about twice as influential than snow balance process. Percolation (light red) is almost constant in its sensitivity but nearly negligible. Baseflow (dark green) and the convolution of the surface runoff as well as the delayed runoff (light green and yellow) are not even visible in the graph and have negligible sensitivities throughout the whole year.

These results highlight the importance of the weighting procedure when deriving the aggregated
sensitivities shown in Fig.

The traditional method to derive sensitivity index estimates for model parameters conditional on a fixed model structure is of limited applicability when the model is allowed to vary in its structure. First, the number of model runs can be massive when each model is analyzed independently. Second, the analysis derives a unique model-dependent sensitivity index for each parameter but no overall parameter sensitivity across the ensemble. Third, aggregated sensitivities of model processes or the sensitivity of a set of process options may lead to more useful insights than analyzing individual model parameters.

In this work we introduce two new concepts. The first is the idea of reformulating a hydrologic
modeling framework such that it is able to weight or blend discrete model process options for simulating process-level fluxes. This converts the countable, discrete model ensemble space into an infinite, continuous model space. The method of weighted process options is shown to significantly
reduce the number of model runs required to run a sensitivity analysis based on model parameters. For the shared-parameter benchmark model, 81.9 % fewer model runs are required (A:

The second key contribution here is the application of the conventional Sobol' sensitivity analysis method based on grouped parameters and interpreting these groups as process options and hydrologic processes. The extended Sobol' sensitivity analysis (xSSA) method uses these groups of parameters to perturb them simultaneously rather than individually, allowing the simultaneous assessment of model output sensitivity to model parameters, model process options, and model processes. While grouping of parameters is not a new concept for Sobol' analyses, they have to our knowledge not yet been interpreted in the context of sensitivity assessment to model structural choices. The method was successfully tested using two artificial benchmark models based upon the Ishigami–Homma function. The estimated sensitivity indexes are proven to converge against the analytically derived Sobol' sensitivity indexes for model parameters, process options, and processes. The xSSA method is shown to resolve limitations of an existing method that also derives sensitivities of groups of parameters but that cannot handle overlapping parameter groupings.

The extended Sobol' sensitivity analysis method was also applied to a hydrologic modeling framework that supports the representation of internal model fluxes using a weighted sum of fluxes calculated from individual process algorithms/options. The sensitivity analysis of the hydrologic modeling framework used here identified potential melt processes and other surface processes as the most influential processes regarding streamflow in a mountainous, energy-limited, and snow-dominated catchment, while all subsurface and routing processes were insensitive. This information helps to guide further model development and model calibration and can inform the incorporation of additional observations to reduce model uncertainty. Three processes (potential melt, rain–snow partitioning, precipitation correction) handle solely inputs to the hydrologic system and can hence be attributed as input uncertainty or, in other words, model components adjusting forcing functions.

The presented methods of weighted process options and the application of the extended Sobol' sensitivity analysis method present a simultaneous analysis of model structure, model parameters, and forcing adjustments in a frugal way consistent with known methods based on the Sobol' method.

In this work we define a model that uses the weighted average of a set of process options instead of choosing one fixed process option (Eq.

The sampling of such weights needs to lead to independent and identical (not necessarily uniform) distribution for each of the weights

This sampling leads to the following CDF

Python and R implementations of the sampling algorithm of Eq. (

All sensitivity indexes for the benchmark setups (Sect.

The analytically derived sensitivity indexes of sensitivity

The analytically derived results of the overall parameter sensitivities (independent of the model options chosen;

Analytically derived Sobol' indexes

The Raven hydrologic modeling framework

Processes and process options used for the Raven setup. In total

The model parameters

The code and data used for this analysis will be made available on GitHub
(

The supplement related to this article is available online at:

JM set up the analyses, implemented the sensitivity analysis based on groups of parameters, implemented the proper sampling of weights used in this study, wrote the main parts of the manuscript, and prepared all figures and tables; JRC contributed to the writing of the manuscript, implemented the weighting of process options in Raven, provided ranges for the parameters included in the analysis, helped to set up the model with the selected options and resolved inconsistencies in Raven detected by earlier versions of the sensitivity analysis, and helped with the hydrologic interpretation of the results; BAT contributed to the writing of the manuscript, provided feedback on the manuscript and the setup of all experiments, including the benchmark models, as well as helped with the hydrologic interpretation of the results.

The authors declare that they have no conflict of interest.

This research was undertaken thanks in part to funding from the CANARIE research software funding
program (project RS-332). The work was made possible by the facilities of the Shared Hierarchical
Academic Research Computing Network (SHARCNET;

This research was undertaken thanks in part to funding from the CANARIE research software funding program (project RS-332).

This paper was edited by Jim Freer and reviewed by two anonymous referees.