HESS Hydrology and Earth System Sciences HESS Hydrol. Earth Syst. Sci. 1607-7938 Copernicus Publications Göttingen, Germany 10.5194/hess-11-793-2007 Comparing sensitivity analysis methods to advance lumped watershed model identification and evaluation Tang Y. 1 Reed P. 1 Wagener T. 1 van Werkhoven K. 1 Department of Civil and Environmental Engineering, The Pennsylvania State University, University Park, Pennsylvania, USA 05 02 2007 11 2 793 817 Copyright: © 2007 Y. Tang et al. 2007 This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 2.5 Generic License. To view a copy of this licence, visit https://creativecommons.org/licenses/by-nc-sa/2.5/ This article is available from https://hess.copernicus.org/articles/11/793/2007/hess-11-793-2007.html The full text article is available as a PDF file from https://hess.copernicus.org/articles/11/793/2007/hess-11-793-2007.pdf

This study seeks to identify sensitivity tools that will advance our understanding of lumped hydrologic models for the purposes of model improvement, calibration efficiency and improved measurement schemes. Four sensitivity analysis methods were tested: (1) local analysis using parameter estimation software (PEST), (2) regional sensitivity analysis (RSA), (3) analysis of variance (ANOVA), and (4) Sobol's method. The methods' relative efficiencies and effectiveness have been analyzed and compared. These four sensitivity methods were applied to the lumped Sacramento soil moisture accounting model (SAC-SMA) coupled with SNOW-17. Results from this study characterize model sensitivities for two medium sized watersheds within the Juniata River Basin in Pennsylvania, USA. Comparative results for the 4 sensitivity methods are presented for a 3-year time series with 1 h, 6 h, and 24 h time intervals. The results of this study show that model parameter sensitivities are heavily impacted by the choice of analysis method as well as the model time interval. Differences between the two adjacent watersheds also suggest strong influences of local physical characteristics on the sensitivity methods' results. This study also contributes a comprehensive assessment of the repeatability, robustness, efficiency, and ease-of-implementation of the four sensitivity methods. Overall ANOVA and Sobol's method were shown to be superior to RSA and PEST. Relative to one another, ANOVA has reduced computational requirements and Sobol's method yielded more robust sensitivity rankings.

 References Abbaspour, K C., Schulin, R., and van Genuchten M Th.: Estimating unsaturated soil hydraulic parameters using ant colony optimization, Adv. Water Resour., 24, 827–841, 2001. Anderson, E A.: National Weather Service River Forecast System - Snow Accumulation and Ablation Model, Tech. rep., NOAA Technical Memorandum NWS Hydro-17, Dept. of Commerce, Silver Spring, Maryland, USA, 1973. Anderson, E A.: Calibration of Conceptual Hydrologic Models for Use in River Forecasting, Tech. rep., NOAA Technical Report, NWS 45, Hydrology Laboratory,Silver Spring, Maryland, USA, 2002. Andres, T H.: Sampling method and sensitivity analysis for large parameter sets, J. Statist. Comput. Simul., 57, 77–110, 1997. Andres, T H. and Wayne, C H.: Using Iterated Fractional Factorial Design to Screen Parameters in Sensitivity analysis of a Probabilistic Risk Assessment Model, in: Proceedings of the Joint International Conference on Mathematical Methods and Supercomputing in Nuclear Applications, edited by: Kusters, H., Stein, E., and Werner, W., vol 2, pp 328–337, 1993. Archer, G. E B., Saltelli, A., and Sobol', I M.: Sensitivity measures, ANOVA-like techniques and the use of bootstrap, J. Statist. Comput. Simul., 58, 99–120, 1997. Beven, K.: Uniqueness of place and process representations in hydrological modeling, Hydrol. Earth Syst. Sci., 4, 203–214, 2000. Beven, K. and Freer, J.: Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems using the GLUE methodology, J. Hydrol., 249, 11–29, 2001. Box, G. E P., Hunter, W G., and Hunter, J S.: Statistics for Experiments: An Introduction to Design, Data Analysis, and Modeling Building, Wiley, New York, 1978. Boyle, D., Gupta, H., and Sorooshian, S.: Toward improved calibration of hydrologic models: Combining the strengths of manual and automatic methods, Water Resour. Res., 36, 3663–3674, 2000. Bratley, P. and Fox, B L.: Implement Sobol's Quasirandom Sequence Generator, ACM Transactions on Mathematical Software, 14, 88–100, 1988. Burnash, R. J C.: The NWS river forecast system-Catchment model, in: Computer Models of Watershed Hydrology, edited by Singh, V P., Water Resour. Publ., Highlands Ranch, CO, 1995. Christiaens, K. and Feyen, J.: Use of sensitivity and uncertainty measures in distributed hydrological modeling with an application to the MIKE SHE model, Water Resour. Res., 38, 1169, https://doi.org/10.1029/2001WR000 478, 2002. Demaria, E., Nijssen, B., and Wagener, T.: Monte Carlo sensitivity analysis of land surface parameters using the variable infiltration capacity model, J. Geophys. Res. – Atmos., in press, 2007. Doherty, J.: Groundwater model calibration using pilot points and regularization, Ground Water, 41, 170–177, 2003. Doherty, J.: PEST-Model Independent Parameter Estimation User Manual: 5th Edition, Watermark Numerical Computing, Brisbane, Australia, 2004. Doherty, J. and Johnston, J M.: Methodologies for calibration and predictive analysis of a watershed model, J. Amer. Water Resour. Assoc., 39, 251–265, 2003. Duan, Q., Gupta, V K., and Sorooshian, S.: Effective and efficient global optimization for conceptual rainfall-runoff models, Water Resour. Res., 28, 1015–1031, 1992. Duffy, C J.: A two-state integral-balance model for soil moisture and groundwater dynamics in complex terrain, Water Resour. Res., 32, 2421–2434, 1996. Duffy, C. J.: Semi-Discrete Dynamical Model for Mountain-Front Recharge and Water Balance Estimation (Rio Grande of Southern Colorado and New Mexico), in: Groundwater Recharge in a Desert Environment: The Southwestern United States, edited by: Hogan, J., Phillips, F., and Scanlon, B., Water Science and Application Monograph, 9, American Geophysical Union, pp 236–255, 2004. Efron, B. and Tibshirani, R.: An introduction to the bootstrap, Chapman Hall, New York, USA, 1993. Fieberg, J. and Jenkins, K J.: Assessing uncertainty in ecological systems using global sensitivity analyses: a case example of simulated wolf reintroduction effects on elk, Ecological Modelling, 187, 259–280, 2005. Freer, J., Beven, K J., and Ambroise, B.: Bayesian estimation of uncertainty in runoff prediction and the value of data: An application of the GLUE approach, Water Resour. Res., 32, 2161–2173, 1996. Frey, H. and Patil, S.: Identification and Review of Sensitivity Analysis Methods, Risk Analysis, 22, 553–578, 2002. Hall, J., Tarantola, S., P.D.Bates, and M.S.Horritt: Distributed sensitivity analysis of flood inumdation model calibration, J. Hydraulic Eng., https://doi.org/10.1061/(ASCE)0733-9429(2005)131:2(117), 2005. Hamby, D.: A review of techniques for sensitivity analysis of environmental models, Environmental Modelling and Assessment, 32, 135–154, 1994. Helton, J. and Davis, F.: Illustration of sampling-based methods for uncertainty and sensitivity analysis, Risk Analysis, 22, 622–691, 2002. Helton, J. and Davis, F.: Latin hypercube sampling and the propagation of uncertainty in analyses of complex systems, Reliability Engineering and System Safty, 81, 23–69, 2003. Henderson-Sellers, A., Yang, Z.-L., and Dickinson, R.: The project of intercomparison of land-surface parameterization schemes, Bull. Am. Meteorol. Soc., 74, 1335–1349, 1993. Hornberger, G. and Spear, R.: An approach to the preliminary analysis of environmental systems, J. Environ. Manage., 12, 7–18, 1981. Koren, V., S., R., Smith, M., Zhang, Z., and Seo, D J.: Hydrology laboratory research modeling system (HL-RMS) of the US national weather service, J. Hydrol., 291, 297–318, 2004. Kottegoda, N. and Rosso, R.: Statistics, Probability and Reliability for Civil and Environmental Engineering, McGraw-Hill, New York, 1997. Langbein, W B.: Overview of Conference on Hydrologic Data Networks, Water Resour. Res., 15, 1867–1871, 1979. Lence, B. and Takyi, A.: Data requirements for seasonal discharge programs: an application of a regionalized sensitivity analysis, Water Resour. Res., 28, 1781–1789, 1992. Levenberg, K.: A method for the solution of certain non-linear problems in least squares, Q. Appl. Math., 2, 164–168, 1944. Liang, X. and Guo, J.: Intercomparison of land-surface parameterization schemes: sensitivity of surface energy and water fluxes to model parameters, J. Hydrol., 279, 182–209, 2003. Marquardt, D W.: An algorithm for least-squares estimation of nonlinear parameters, Journal of the Society of Industrial and Applied Mathematics, 11, 431–441, 1963. Mckay, M., Beckman, R., and Conover, W.: A comparison of three methods for selecting values of input variables in the analysis of output from a computer code, Technometrics, 21, 239–245, 1979. Misirli, F., Gupta, H V., Sorooshian, S., and Thiemann, M.: Bayesian recursive estimation of parameter and output uncertainty for watershed models, in: Calibration of Watershed Models, edited by: Duan, Q. and Gupta, H. V., Vol 6, pp 113–124, American Geophysical Union, Washington, D.C., 2003. Mokhtari, A. and Frey, H C.: Sensitivity Analysis of a Two-Dimensional Probabilistic Risk Assessment Model Using Analysis of Variance, Risk Analysis, 25, 1511–1529, 2005. Moore, C. and Doherty, J.: Role of the calibration process in reducing model predictive error, Water Resour. Res., 41, W05020, https://doi.org/10.1029/2004WR003 501, 2005. Moreda, F., Koren, V., Zhang, Z., Reed, S., and Smith, M.: Parameterization of distributed hydrological models: learning from the experiences of lumped modeling, J. Hydrol., 320, 218–237, 2006. Moss, M E.: Space, Time, and the Third Dimension (Model Error), Water Resour. Res., 15, 1797–1800, 1979. Muleta, M. and Nicklow, J W.: Sensitivity and uncertainty analysis coupled with automatic calibration for a distributed watershed model, J. Hydrol., 306, 1–4, 127–145, 2005. Neter, J., Kutner, M., Nachtsheim, C., and Wasserman, W.: Applied Linear Statitical Models, 4th ed., McGraw-Hill, Chicago, IL, 1996. Oakley, J. and O'Hagan, A.: Probabilistic sensitivity analysis of complex models: a Bayesian approach, Journal of Royal Statistical Society Series B - statistical Methodology, 66, 751–769, 2004. Osidele, O. and Beck, M B.: Identification of model structure for aquatic ecosystems using regionalized sensitivity analysis, Water Sci. Technol., 43, 271–278, 2001. Panday, S. and Huyakorn, P.: A fully coupled physically-based spatially-distributed model for evaluating surface/subsurface flow, Adv. Water Resour., 27, 361–382, 2004. Pappenberger, F., Iorgulescu, I., and Beven, K J.: Sensitivity analysis based on regional splits and regression trees (SARS-RT), Environmental Modelling and Software, 21, 976–990, 2005. Pappenberger, F., Iorgulescu, I., and Beven, K.: Sensitivity analysis based on regional splits and regression trees (SARS-RT), Environmental Modelling & Software, 21, 976–990, 2006. Patil, S. and Frey, H.: Comparison of sensitivity analysis methods based upon applications to a food safety risk model, Risk Analysis, 23, 135–154, 2004. Peck, E L.: Catchment modeling and initial parameter estimation for the Natioanl Weather Service river forecast system, Tech. rep., Tech. Memo, NWS Hydro-31, Natl. Oceanic and Atmos. Admin., Silver Spring, Maryland, USA, 1976. Ratto, M., Young, P C., Romanowicz2, R., Pappenberge, F., Saltelli1, A., and Pagano1, A.: Uncertainty, sensitivity analysis and the role of data based mechanistic modeling in hydrology, Hydrol. Earth Syst. Sci., 3, 3099–3146, 2006. Reed, P., Brooks, R., Davis, K., DeWalle, D R., Dressler, K A., Duffy, C J., Lin, H S., Milller, D., Najjar, R., Salvage, K M., Wagener, T., and Yarnal, B.: Bridging River Basin Scales and Processes to Assess Human-Climate Impacts and the Terrestrial Hydrologic System, Water Resour. Res., 42, W07418, https://doi.org/10.1029/2005WR004 153, 2006. Reed, S., Koren, V., Smith, M., Zhang, Z., Moreda, F., Seo, D., and Participants, D.: Overall distributed model intercomparison project results, J. Hydrol., 298, 27–60, 2004. Saltelli, A.: Making best use of model evaluations to compute sensitivity indices, Computer Physics Communications, 145, 280–297, 2002. Saltelli, A., Andres, T H., and Homma, T.: Sensitivity analysis of model output. Performance of the iterated fractional factorial design method, Computational Statistics $&$ Data Analysis, 20, 387–407, 1995. Saltelli, A., Tarantola, S., and Chan, K. P.-S.: A quantitative model-independent method for global sensitivity analysis of model output, Technometrics, 41, 39–56, 1999. Saltelli, A., Tarantola, S., and Campolongo, F.: Sensitivity Analysis as an Ingredient of Modeling, Statistical Science, 15, 377–395, 2000. Saltelli, A., Tarantola, A., Campolongo, F., and Ratto, M.: Sensitivity Analysis in Practice-A Guide to Assessing Scientific Models, John Wiley and Sons, Chichester, 2004. Sieber, A. and Uhlenbrook, S.: Sensitivity analyses of a distributed catchment model to verify the model structure, J. Hydrol., 310, 216–235, 2005. Singh, V. and Woolhiser, D.: Mathematical modeling of watershed hydrology, J. Hydrol. Eng., 7, 270–292, 2002. Smith, M., Seo, D., Koren, V., Reed, S., Zhang, Z., Duan, Q., Moreda, F., and Cong, S.: The distributed model intercomparison project (DMIP): motivation and experiment design, J. Hydrol., 298, 4–26, 2004. Sobol', I.: Sensitivity estimates for nonlinear mathematical models, Math Model Comput. Exp., 1, 407–417, 1993. Sobol', I.: Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates, Mathematics and Computers in Simulation, 55, 271–280, 2001. Sobol', I M.: On the distribution of points in a cube and the approximate evaluation of integrals, USSR Comput. Math. Math. Phys., 7, 86–112, 1967. Sobol', I M.: A primer for the Monte Carlo method, CRC Press, Boca Raton, Fla., 1994. Spear, R., Grieb, T M., and Shang, N.: Parameter uncertainty and interaction in complex environmental models, Water Resour. Res., 30, 3159–3169, 1994. Tang, Y., Reed, P., and Kollat, J.: Parallelization Strategies for Rapid and Robust Evolutionary Multiobjective Optimization in Water Resources Applications, Adv. Water Resour., 30(3), 335–353, 2007. Tang, Y., Reed, P., and Wagener, T.: How effective and efficient are multiobjective evolutionary algorithms at hydrologic model calibration, Hydrol. Earth Syst. Sci., 10, 289–307, 2006. Tonkin, M J. and Doherty, J.: A hybrid regularized inversion methodology for highly parameterized environmental models, Water Resour. Res., 41, W10412, https://doi.org/10.1029/2005WR003 995, 2005. Vandeberghe, V., Bauwens, W., and Vanrolleghem, P.: Evaluation of uncertainty propagation into river water quality predictions to guide future monitoring campaigns, Environmental Modelling & Software, 22(5), 725–732, 2007. Vrugt, J., Gupta, H V., Bastidas, L A., Bouten, W., and Sorooshian, S.: Effective and efficient algorithm for multiobjective optimization of hydrologic models, Water Resour. Res., 39, 1214, https://doi.org/10.1029/2002WR001 746, 2003. Wagener, T. and Kollat, J.: Numerical and visual evaluation of hydrological and environmental models using the Monte Carlo analysis toolbox, Environmental Modeling and Software, in press, 2007. Wagener, T., Boyle, D., Lees, M J., Wheater, Gupta, H S., and H V Sorooshian, S.: A framework for development and application of hydrological models, Hydrol. Earth Syst. Sci., 5, 13–26, 2001. Wagener, T., McIntyre, N., Lees, M J., Wheater, H S., and Gupta, H V.: Towards reduced uncertainty in conceptual rainfall-runoff modelling: Dynamic identifiability analysis, Hydrol. Processes, 17, 455–476, 2003. Wagener, T., Wheater, H S., and Gupta, H V.: Rainfall-runoff modeling in gauged and ungauged catchments, Imperial College Press, London, UK, 2004. Wagener, T., Liu, Y., Gupta, H., Springer, E., and Brookshire, D.: Multi-resolution integrated assessment modeling for water resources management in arid and semi-arid regions, in: Regional hydrologic impacts of climate change - Impact assessment and decision making, edited by: Wagener, T., Franks, S., Bøgh, E., Gupta, H., Bastidas, L., Nobre, C., and Oliveira~Galväo, C., IAHS Redbook Publ., 295, pp 265–272, 2005. William, H., Teukolsky, S A., William, T V., and Brian, P F.: Numerical recipes in C (2nd edition), Cambridge University Press, New York, USA, 1999. Young, P C.: A general theory of modelling for badly defined dynamic systems, in: Modeling, Identification and Control in Environmental Systems, edited by: Vansteenkiste, G C., North Holland, Amsterdam, pp 103–135, 1978.