HESS Hydrology and Earth System Sciences HESS Hydrol. Earth Syst. Sci. 1607-7938 Copernicus Publications Göttingen, Germany 10.5194/hess-17-2827-2013 Legitimising data-driven models: exemplification of a new data-driven mechanistic modelling framework Mount N. J. 1 Dawson C. W. 2 Abrahart R. J. 1 School of Geography, University of Nottingham, Nottingham, NG7 2RD, UK Department of Computer Science, Loughborough University, Loughborough, LE11 3TU, UK 17 07 2013 17 7 2827 2843 Copyright: © 2013 N. J. Mount et al. 2013 This work is licensed under the Creative Commons Attribution 3.0 Unported License. To view a copy of this licence, visit https://creativecommons.org/licenses/by/3.0/ This article is available from https://hess.copernicus.org/articles/17/2827/2013/hess-17-2827-2013.html The full text article is available as a PDF file from https://hess.copernicus.org/articles/17/2827/2013/hess-17-2827-2013.pdf

In this paper the difficult problem of how to legitimise data-driven hydrological models is addressed using an example of a simple artificial neural network modelling problem. Many data-driven models in hydrology have been criticised for their black-box characteristics, which prohibit adequate understanding of their mechanistic behaviour and restrict their wider heuristic value. In response, presented here is a new generic data-driven mechanistic modelling framework. The framework is significant because it incorporates an evaluation of the legitimacy of a data-driven model's internal modelling mechanism as a core element in the modelling process. The framework's value is demonstrated by two simple artificial neural network river forecasting scenarios. We develop a novel adaptation of first-order partial derivative, relative sensitivity analysis to enable each model's mechanistic legitimacy to be evaluated within the framework. The results demonstrate the limitations of standard, goodness-of-fit validation procedures by highlighting how the internal mechanisms of complex models that produce the best fit scores can have lower mechanistic legitimacy than simpler counterparts whose scores are only slightly inferior. Thus, our study directly tackles one of the key debates in data-driven, hydrological modelling: is it acceptable for our ends (i.e. model fit) to justify our means (i.e. the numerical basis by which that fit is achieved)?

 References Abrahart, R. J. and See, L. M.: Neural network modelling of non-linear hydrological relationships, Hydrol. Earth Syst. Sci., 11, 1563–1579, <a href="http://dx.doi.org/10.5194/hess-11-1563-2007">https://doi.org/10.5194/hess-11-1563-2007</a>, 2007. Abrahart, R. J., See, L. M., and Kneale, P. E.: Using pruning algorithms and genetic algorithms to optimise neural network architectures and forecasting inputs in a neural network rainfall-runoff model, J. Hydroinform., 1, 103–114, 1999. Abrahart, R. J., See, L. M., and Kneale, P. E.: Investigating the role of saliency analysis with a neural network rainfall-runoff model, Comput. Geosci., 27, 921–928, 2001. Abrahart, R. J., Ab Ghani, N., and Swan, J.: Discussion of "An explicit neural network formulation for evapotranspiration", Hydrolog. Sci. J., 54, 382–388, 2009. Abrahart, R. J., Mount, N. J., Ab Ghani, N., Clifford, N. J., and Dawson, C. W.: DAMP: a protocol for contextualising goodness-of-fit statistics in sediment-discharge data-driven modelling, J. Hydrol., 409, 596–611, 2011. Abrahart, R. J., Anctil, F., Coulibaly, P., Dawson, C. W., Mount, N. J., See, L. M., Shamseldin, A. Y., Solomatine, D. P., Toth, E., and Wilby, R. L.: Two decades of anarchy? Emerging themes and outstanding challenges for neural network river forecasting, Prog. Phys. Geog., 36, 480–513, 2012a. Abrahart, R. J., Dawson, C. W., and Mount, N. J.: Partial derivative sensitivity analysis applied to autoregressive neural network river forecasting, in: Proceedings of the 10th International Conference on Hydroinformatics, Hamburg, Germany, 14–18 July 2012, p. 8, 2012b. Ahmadi, A., Han, D., Karamouz, M., and Remesan, R: Input data selection for solar radiation estimation, Hydrol. Process., 23, 2754–2764, 2009. American Institute of Aeronautics and Astronautics: Guide for the Verification and Validation of Computational Fluid Dynamics Simulations, AIAA-G-077-1998, Reston, Virigina, USA, 1998. Anctil, F., Michel, C., Perrin, C., and Andreassian, V.: A soil moisture index as an auxiliary ANN input for stream flow forecasting, J. Hydrol., 286, 155–167, 2004. Aytek, A., Guven, A., Yuce, M. I., and Aksoy, H.: An explicit neural network formulation for evapotranspiration, Hydrolog. Sci. J., 53, 893–904, 2008. Babovic, V.: Data mining in hydrology, Hydrol. Process., 19, 1511–1515, 2005. Balci, O.: Verification, validation and testing, in: Handbook of Simulation, John Wiley and Sons, Chichester, UK, 335–396, 1998. Baxter, C. W., Stanley, S. J., Zhang, Q., and Smith, D. W.: Developing artificial neural network process models: A guide for drinking water utilities, in: Proceedings of the 6th Environmental Engineering Society Specialty Conference of the CSCE, 376–383, 2000. Beven, K. J.: Towards a coherent philosophy for modelling the environment, P. R. Soc. London A, 458, 2465–2484, 2002. Beven, K. J. and Binley, A.: The future of distributed models: model calibration and uncertainty prediction, Hydrol. Process., 6, 279–298, 1992. Carson, J. S.: Convincing users of a model's validity is a challenging aspect of a modeler's job, Ind. Eng., 18, 74–85, 1986. Caswell, H.: The validation problem, in: Systems Analysis and Simulation in Ecology, Vol. IV., Academic Press, New York, 313–325, 1976. Coulibaly, P., Anctil, F., and Bobe, B.: Daily reservoir inflow forecasting using artificial neural networks with stopped training approach, J. Hydrol., 230, 244–257, 2000. Curry, G. L., Deuermeyer, B. L., and Feldman, R. M.: Siscrete Simulation, Holden-Day, Oakland, California, 297 pp., 1989. Davis, P. K.: Generalizing concepts of verification, validation and accreditation for military simulation, R-4249-ACQ, October 1992, RAND, Santa Monica, CA, 1992. Dawson, C. W., Abrahart, R. J., Shamseldin, A. Y., and Wilby, R. L.: Flood estimation at ungauged sites using artificial neural networks, J. Hydrol., 319, 391–409, 2006. Dawson, C. W., Abrahart, R. J., and See, L. M.: HydroTest: a web-based toolbox of evaluation metrics for the standardised assessment of hydrological forecasts, Environ. Modell. Softw., 22, 1034–1052, 2007. Dawson, C. W., Abrahart, R. J., and See, L. M.: HydroTest: further development of a web resource for the standardised assessment of hydrological models, Environ. Modell. Softw., 25, 1481–1482, 2010. Dawson, C. W., Mount, N. J., Abrahart, R. J., and Shamseldin, A. Y.: Ideal point error for model assessment in data-driven river flow forecasting, Hydrol. Earth Syst. Sci., 16, 3049–3060, <a href="http://dx.doi.org/10.5194/hess-16-3049-2012">https://doi.org/10.5194/hess-16-3049-2012</a>, 2012. de Vos, N. J.: Echo state networks as an alternative to traditional artificial neural networks in rainfall-runoff modelling, Hydrol. Earth Syst. Sci., 17, 253–267, <a href="http://dx.doi.org/10.5194/hess-17-253-2013">https://doi.org/10.5194/hess-17-253-2013</a>, 2013. Dibike, B. Y. and Solomatine, D. P.: River flow forecasting using artificial neural networks, Phys. Chem. Earth, 26, 1–7, 2001. Fernando, D. A. K. and Shamseldin, A. Y.: Investigation of internal functioning of the radial-basis-function neural network river flow forecasting models, J. Hydrol. Eng., 14, 286–292, 2009. Firat, M.: Comparison of Artificial Intelligence Techniques for river flow forecasting, Hydrol. Earth Syst. Sci., 12, 123–139, <a href="http://dx.doi.org/10.5194/hess-12-123-2008">https://doi.org/10.5194/hess-12-123-2008</a>, 2008. Fraedrich, D. and Goldberg, A: A Methodological framework for the validation of predictive simulations, Eur. J. Oper. Res., 124, 55–62, 2000. Giustolisi, O. and Laucelli, D.: Improving generalization of artificial neural networks in rainfall–runoff modelling, Hydrolog. Sci. J., 50, 439–457, 2005. Hamby, D. M.: A review of techniques for parameter sensitivity analysis of environmental models, Environ. Monit. Assess., 32, 135–154, 1994. Hashem, S.: Sensitivity analysis for feedforward artificial networks with differentiable activation functions, in: Proceedings of the International Joint Conference on Neural Networks, Baltimore, USA, 7–11 June, 1, 419–424, 1992. Helton, J. C.: Uncertainty and sensitivity analysis techniques for use in performance assessment for radioactive waste disposal, Reliab. Eng. Syst. Safe., 42, 327–367, 1993. Hill, M. C. and Tiedeman, C. R.: Effective Groundwater Model Calibration with Analysis of Sensitivities, Predictions, and Uncertainty, Wiley, New York, 2007. Hipel, K. W., McLeod, A. I., and Lennox, W. C.: Advances in Box-Jenkins modeling 1. model construction, Water Resour. Res., 13, 567–575, 1977. Holvoet, K., van Griensven, A., Seuntjens, P., and Vanrolleghem, P. A.: Sensitivity analysis for hydrology and pesticide supply towards the river in SWAT, Phys. Chem. Earth, 30, 518–526, 2005. Howes, S. and Anderson, M. G.: Computer simulation in geomorphology, in: Modeling Geomorphological Systems, John Wiley and Sons Ltd, Chichester, 1988. Hu, T. S., Lam, K. C., and Ng, S. T.: River flow time series prediction with a range-dependent neural network, Hydrolog. Sci. J., 46, 729–745, 2001. Huang, W., Xu, B., and Chan-Hilton, A.: Forecasting flows in Apalachicola River using neural networks, Hydrol. Process., 18, 2545–2564, 2004. Imrie, C. E., Durucan, S., and Korre, A.: River flow prediction using artificial neural networks: generalisation beyond the calibration range, J. Hydrol., 233, 138–153, 2000. Jain, A. and Kumar, S.: Dissection of trained neural network hydrologic models for knowledge extraction, Water Resour. Res., 45, W07420, <a href="http://dx.doi.org/10.1029/2008WR007194">https://doi.org/10.1029/2008WR007194</a>, 2009. Jain, A., Sudheer, K. P., and Srinivasulu, S.: Identification of physical processes inherent in artificial neural network rainfall runoff models, Hydrol. Process., 18, 571–581, 2004. Jakeman, A. J., Letcher, R. A., and Norton, J. P.: Ten iterative steps in development and evaluation of environmental models, Environ. Modell. Softw., 21, 602–614, 2006. Kingston, G. B., Maier, H. R., and Lambert, M. F.: Understanding the mechanisms modelled by artificial neural networks for hydrological prediction, in: Modsim 2003 – International Congress on Modelling and Simulation, Modelling and Simulation Society of Australia and New Zealand Inc, Townsville, Australia, 14–17 July, 2, 825–830, 2003. Kingston, G. B., Maier, H. R., and Lambert, M. F.: Calibration and validation of neural networks to ensure physically plausible hydrological modelling, J. Hydrol., 314, 158–176, 2005. Kingston, G. B., Maier, H. R., and Lambert, M. F.: A probabilistic method to assist knowledge extraction from artificial neural networks used for hydrological prediction, Math. Comput. Model., 44, 499–512, 2006. Kingston, G. B., Maier, H. R., and Lambert, M. F.: Bayesian model selection applied to artificial neural networks used for water resources modelling, Water Resour. Res., 44, W04419, <a href="http://dx.doi.org/10.1029/2007WR006155">https://doi.org/10.1029/2007WR006155</a>, 2008. Kişi, Ö.: River flow forecasting and estimation using different artificial neural network techniques, Hydrol. Res., 39, 27–40, 2008. Kişi, Ö. and Cigizoglu, H. K.: Comparison of different ANN techniques in river flow prediction, Civ. Eng. Environ. Syst., 24, 211–231, 2007. Kleijnen, J. P. C.: Verification and validation of simulation-models, Eur. J. Oper. Res., 82, 145–162, 1995. Kleijnen, J. P. C. and Sargent, R. G.: A methodology for fitting and validating metamodels in simulation, Eur. J. Oper. Res., 120, 14–29, 2000. Klemes, V.: Operational testing of hydrological simulation models, Hydrolog. Sci. J., 31, 13–24, 1986. Krieger, T. J., Durston, C., and Albright, D. C.: Statistical determination of effective variables in sensitivity analysis, Trans. A. Nuc. Soc., 28, 515–516, 1977. LeBaron, B. and Weigend, A. S.: A bootstrap evaluation of the effect of data splitting on financial time series, IEEE T. Neural Networ., 9, 213–220, 1998. Maier, H. R. and Dandy, G. C.: Determining inputs for neural network models of multivariate time series, J. Comp. Aid. Civ. Infrastr. Eng., 5, 353–368, 1997. Maier, H. R. and Dandy, G. C.: Application of artificial neural networks to forecasting of surface water quality variables: issues, applications and challenges, in: Artifical Neural Networks in Hydrology, Kluwer, Dordrecht, Netherlands, 287–309, 2000. Maier, H. R. and Dandy, G. C.: Neural network based modelling of environmental variables: a systematic approach, Math. Comput. Model., 33, 669–682, 2001. Maier, H. R., Dandy, G. C., and Burch, M. D.: Use of artificial neural networks for modelling cyanobacteria <i>Anabaena spp.</i> in the Murray River, South Australia, Ecol. Model., 105, 257–272, 1998. Marti, P., Manzano, J., and Royuela, A: Assessment of a 4-input neural network for ET$_o$ estimation through data set scanning procedures, Irrigation Sci., 29, 181–195, 2011. May, R. J., Maier, H. R., Dandy, G. C., and Fernando, T. M. K. G.: Non-linear selection for artificial neural networks using partial mutual information, Environ. Modell. Softw., 23, 1312–1326, 2008. McCuen, R. H.: The role of sensitivity analysis in hydrologic modelling, J. Hydrol., 18, 37–53, 1973. Minns, W. and Hall, M. J.: Artificial neural networks as rainfall-runoff models, Hydrolog. Sci. J., 41, 399–417, 1996. Mishra, S.: Uncertainty and sensitivity analysis techniques for hydrologic modelling, J. Hydroinform., 11, 282–296, 2009. Mount, N. J. and Abrahart, R. J.: Discussion of "River flow estimation from upstream flow records by artificial intelligence methods" by M. E. Turan, M. A. Yurdusev [J. Hydrol. 369 (2009) 71–77], J. Hydrol., 396, 193–196, 2011a. Mount, N. J. and Abrahart, R. J.: Load or concentration, logged or unlogged? Addressing ten years of uncertainty in neural network suspended sediment prediction, Hydrol. Process., 25, 3144–3157, 2011b. Mount, N. J., Abrahart, R. J., Dawson, C. W., and Ab Ghani, N.: The need for operational reasoning in data-driven rating curve prediction of suspended sediment, Hydrol. Process., 26, 3982–4000, 2012. Muleta, M. K. and Nicklow, J. W.: Sensitivity and uncertainty analysis coupled with automatic calibration for a distributed watershed model, J. Hydrol. 306, 127–145, 2005. National Oceanic and Atmospheric Administration: National Weather Service Middle Atlantic River Forecast Center: The models and the final product, available at: <a href="http://www.erh.noaa.gov/marfc/Science/models.html">http://www.erh.noaa.gov/marfc/Science/models.html</a> (last access: 15 July 2013), 2011. Nourani, V. and Fard, M. S.: Sensitivity analysis of the artificial neural network outputs in simulation of the evaporation process at different climatologic regimes, Adv. Eng. Softw., 47, 127–146, 2012. Olden, J. D. and Jackson, D. A.: Illuminating the `black box': a randomization approach for understanding variable contributions in artificial neural networks, Ecol. Model., 154, 135–150, 2002. Oreskes, N., Shrader-Frechette, K., and Belitz, K.: Verification, validation and confirmation of numerical models in the Earth Sciences, Science, 263, 641–646, 1994. Pappenberger, F., Beven, K. J., Ratto, M., and Matgen, P.: Multi-method global sensitivity analysis of flood inundation models, Adv. Water Resour., 31, 1–14, 2008. Refsgaard, J. C. and Knudsen, J.: Operational validation and intercomparison of different types of hydrological models, Water Resour. Res., 32, 2189–2202, 1996. Robinson, S.: Simulation model verification and validation: increasing the users' confidence, in: Proceedings of the 1997 Winter Simulation Conference, Atlanta, Georgia, 53–59, 1997. Rumelhart, D. E., Hinton, G. E., and Williams, R. J.: Learning internal representations by error propagation, in: Parallel Distributed Processing: Explorations in the Microstructures of Cognition, Volume 1, The MIT Press, Cambridge, Massachusetts, USA, 318–362, 1986. Rykiel, E. J.: Testing ecological models: the meaning of validation, Ecol. Model., 90, 229–244, 1996. Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., and Tarantola, S.: Global Sensitivity Analysis, The primer, Wiley, Chichester, 304 pp., 2008. Sargent, R. G.: Verification and validation of simulation models, in: Proceedings of the Winter Simulation Conference 1998, Washington DC, USA, 121–130, 1998. Sargent, R. G.: Verification and validation of simulation models, in: Proceedings of the 2010 Winter Simulation Conference, Baltimore, Maryland, USA, 166–183, 2010. Sargent, R. G.: Verification and validation of simulation models, in: Proceedings of the 2011 Winter Simulation Conference, Informs Simulation Society, 183–197, 2011. See, L. M., Jain, A., Dawson. C. W., and Abrahart, R. J.: Visualisation of hidden neuron behaviour in a neural network rainfall-runoff model, in: Practical Hydroinformatics: Computational Intelligence and Technological Developments in Water Applications, Springer, Berlin, 87–99, 2008. Shamseldin, A. Y. and O'Connor, K. M.: A non-linear neural network technique for updating of river flow forecasts, Hydrol. Earth Syst. Sci., 5, 577–598, <a href="http://dx.doi.org/10.5194/hess-5-577-2001">https://doi.org/10.5194/hess-5-577-2001</a>, 2001. Shrestha, R. R. and Nestmann, F.: Physically-based and data-driven models and propagation of uncertainties in flood prediction, J. Hydrolog. Eng., 14, 1309–1319, 2009. Smith, E. D., Szidarovszky, F., Karnavas, W. J., and Bahill, A. T.: Sensitivity analysis, a powerful system validation technique, Open Cybernetics System. J., 2, 39–56, 2008. Snee, R. D.: Validation of regression models: methods and examples, Technometrics, 19, 415–428, 1977. Solomatine, D., See, L. M., and Abrahart, R. J.: Data-driven modelling: concept, approaches, experiences, in: Practical Hydroinformatics: Computational Intelligence and Technological Developments in Water Applications, Springer-Verlag, 2008. Spear, R. C. and Hornberger, G. M.: Eutrophication in Peel Inlet, II, Identification of critical uncertainties via generalized sensitivity analysis, Water Resour. Res., 14, 43–49, 1980. Spruill, C. A., Workman, S. R., and Taraba, J. L.: Simulation of daily and monthly stream discharge from small watersheds using the SWAT model, T. Am. Soc. Civ. Eng., 43, 1431–1439, 2000. Sudheer, K. P.: Knowledge extraction from trained neural network river flow models, J. Hydrolog. Eng., 10, 264–269, 2005. Sudheer, K. P. and Jain, A.: Explaining the internal behaviour of artificial neural network river flow models, Hydrol. Process., 18, 833–844, 2004. Sun, F., Chen, J., Tong, Q., and Zeng, S.: Structure validation of an integrated waterworks model for trihalomethanes simulation by applying regional sensitivity analysis, Sci. Total Environ., 408, 1992–2001, 2009. Tsai, M.-J., Abrahart, R. J., Mount, N. J., and Chang, F.-J.: Including spatial distribution in a data-driven, rainfall-runoff model to improve reservoir inflow forecasting in Taiwan, Hydrol. Process., <a href="http://dx.doi.org/10.1002/hyp.9559">https://doi.org/10.1002/hyp.9559</a>, in press, 2013. Turanayi, T. and Rabitz, H.: Local methods, in: Sensitivity Analysis, Wiley Series in Probability and Statistics, Wiley, Chichester, 2000. Wilby, R. L., Abrahart, R. J., and Dawson, C. W.: Detection of conceptual model rainfall-runoff processes inside an artificial neural network, Hydrolog. Sci. J., 48, 163–181, 2003. Wu, W., May, R., Dandy, G. C., and Maier, H. R.: A method for comparing data splitting approaches for developing hydrological ANN models, in: Proceedings of the 6th Biennial Meeting of the International Environmental Modelling and Software Society, 2012 International Congress on Environmental Modelling and Software Managing Resources of a Limited Planet, Leipzig, Germany, 2012. Yeung, D. S., Cloete, I., Shi, D., and Ng, W. W. Y.: Sensitivity Analysis for Neural Networks. Springer, Berlin, 86 pp., 2010. Young, P. C. and Beven, K. J.: Databased mechanistic modelling and the rainfall flow nonlinearity, Environmetrics, 5, 335–363, 1994. Young, P. C., Chotai, A., and Beven, K. J.: Data-based mechanistic modelling and the simplification of environmental systems, in: Environmental Modelling: Finding Simplicity in Complexity, Wiley, Chichester, 371–388, 2004. Zealand, C. M., Burn, D. H., and Simonovic, S. P.: Short term streamflow forecasting using artificial neural networks, J. Hydrol., 214, 32–48, 1999.