Bouaziz, L. J. E., Fenicia, F., Thirel, G., de Boer-Euser, T., Buitink, J., Brauer, C. C., De Niel, J., Dewals, B. J., Drogue, G., Grelier, B., Melsen, L. A., Moustakas, S., Nossent, J., Pereira, F., Sprokkereef, E., Stam, J., Weerts, A. H., Willems, P., Savenije, H. H. G., and Hrachowitz, M.: Behind the scenes of streamflow model performance, Hydrol. Earth Syst. Sci., 25, 1069–1095, https://doi.org/10.5194/hess-25-1069-2021, 2021.
Charlesworth, S. M.: A review of the adaptation and mitigation of global
climate change using sustainable drainage in cities, J. Water Clim. Chang.,
1, 165–180, https://doi.org/10.2166/wcc.2010.035, 2010.
Chen, H., Janizek, J. D., Lundberg, S., and Lee, S. I.: True to the model or
true to the data?, arXiv [preprint], arXiv:1805.11783, 2020.
Chen, T. and Guestrin, C.: XGBoost: A scalable tree boosting system, in:
Proceedings of the ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining, 13–17 August, 785–794, San Francisco, CA, USA, 2016.
Chen, T. and He, T.: xgboost: eXtreme Gradient Boosting, available
at:
https://cran.r-project.org/web/packages/xgboost/vignettes/xgboost.pdf, last access: 29 June 2020.
Damodaram, C., Giacomoni, M. H., Prakash Khedun, C., Holmes, H., Ryan, A.,
Saour, W., and Zechman, E. M.: Simulation of combined best management
practices and low impact development for sustainable stormwater management,
J. Am. Water Resour. Assoc., 46, 907–918,
https://doi.org/10.1111/j.1752-1688.2010.00462.x, 2010.
Darner, R. A. and Dumouchelle, D. H.: Hydraulic Characteristics of
Low-Impact Development Practices in Northeastern Ohio, 2008-2010: U.S.
Geological Survey Scientific Investigations Report 2011–5165, available at:
https://pubs.usgs.gov/sir/2011/5165/ (last access: 7 July 2020),
2011.
Darner, R. A., Shuster, W. D., and Dumouchelle, D. H.: Hydrologic
Characteristics of Low-Impact Stormwater Control Measures at Two Sites in
Northeastern Ohio, 2008–2013: U.S. Geological Survey Scientific
Investigations Report 2015-5030, U.S. Geological Survey, Reston, VA, USA, 2015.
DeBusk, K. M., Hunt, W. F., and Line, D. E.: Bioretention Outflow: Does It
Mimic Nonurban Watershed Shallow Interflow?, J. Hydrol. Eng., 16,
274–279, https://doi.org/10.1061/(ASCE)HE.1943-5584.0000315, 2011.
Demirdjian, D., Taycher, L., Shakhnarovich, G., Grauman, K., and Darrell, T.:
Avoiding the “streetlight effect”: Tracking by exploring likelihood modes,
in: Proceedings of the IEEE International Conference on Computer Vision, vol.
I, 357–364, San Diego, CA, USA, 20 to 26 June 2005, 2005.
Eckart, K., McPhee, Z., and Bolisetti, T.: Performance and implementation of
low impact development – A review, Sci. Total Environ., 607–608, 413–432,
https://doi.org/10.1016/j.scitotenv.2017.06.254, 2017.
Elliott, A. H. and Trowsdale, S. A.: A review of models for low impact urban
stormwater drainage, Environ. Model. Softw., 22, 394–405,
https://doi.org/10.1016/j.envsoft.2005.12.005, 2007.
EPA: Flow and Rainfall Data used for SHC Headwatershed SWMM Calibration, EPA [data set], https://doi.org/10.23719/1378947, 2017.
Eric, M., Li, J., and Joksimovic, D.: Performance Evaluation of Low Impact
Development Practices Using Linear Regression, Br. J. Environ. Clim. Chang.,
5, 78–90, https://doi.org/10.9734/bjecc/2015/11578, 2015.
Fassman-Beck, E., Hunt, W., Berghage, R., Carpenter, D., Kurtz, T., Stovin,
V., and Wadzuk, B.: Curve number and runoff coefficients for extensive living
roofs, J. Hydrol. Eng., 21, 04015073,
https://doi.org/10.1061/(ASCE)HE.1943-5584.0001318, 2016.
Fletcher, T. D., Shuster, W., Hunt, W. F., Ashley, R., Butler, D., Arthur,
S., Trowsdale, S., Barraud, S., Semadeni-Davies, A., Bertrand-Krajewski, J.
L., Mikkelsen, P. S., Rivard, G., Uhl, M., Dagenais, D., and Viklander, M.:
SUDS, LID, BMPs, WSUD and more – The evolution and application of
terminology surrounding urban drainage, Urban Water J., 12, 525–542,
https://doi.org/10.1080/1573062X.2014.916314, 2015.
Frazier, P. I.: A tutorial on bayesian optimization, arXiv [preprint], arXiv:1807.02811,
2018.
Friedman, J. H.: Greedy function approximation: A gradient boosting machine,
Ann. Stat., 29, 1189–1232, https://doi.org/10.1214/aos/1013203451, 2001.
Gimenez-Maranges, M., Breuste, J., and Hof, A.: Sustainable Drainage Systems
for transitioning to sustainable urban flood management in the European
Union: A review, J. Clean. Prod., 255, 120191,
https://doi.org/10.1016/j.jclepro.2020.120191, 2020.
Gauch, M., Kratzert, F., Klotz, D., Nearing, G., Lin, J., and Hochreiter, S.: Rainfall–runoff prediction at multiple timescales with a single Long Short-Term Memory network, Hydrol. Earth Syst. Sci., 25, 2045–2062, https://doi.org/10.5194/hess-25-2045-2021, 2021.
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., and
Pedreschi, D.: A Survey of Methods for Explaining Black Box Models, ACM
Comput. Surv., 51, 1–42, https://doi.org/10.1145/3236009, 2019.
Guo, Y. and Senior, M. J.: Climate model simulation of point rainfall
frequency characteristics, J. Hydrol. Eng., 11, 547–554,
https://doi.org/10.1061/(ASCE)1084-0699(2006)11:6(547), 2006.
Hastie, T., Tibshirani, R., and Friedman, J.: The Elements of Statistical
Learning, Springer New York, New York, NY, USA, 2009.
Hoghooghi, N., Golden, H. E., Bledsoe, B. P., Barnhart, B. L., Brookes, A.
F., Djang, K. S., Halama, J. J., McKane, R. B., Nietch, C. T., and Pettus, P.
P.: Cumulative effects of Low Impact Development on watershed hydrology in a
mixed land-cover system, Water, 10, 991,
https://doi.org/10.3390/w10080991, 2018.
Hopkins, K. G., Bhaskar, A. S., Woznicki, S. A., and Fanelli, R. M.: Changes
in event-based streamflow magnitude and timing after suburban development
with infiltration-based stormwater management, Hydrol. Process., 34,
387–403, https://doi.org/10.1002/hyp.13593, 2020.
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., and Madry, A.:
Adversarial examples are not bugs, they are features, in: Advances in Neural
Information Processing Systems, vol. 32, GitHub [data set], available at:
http://git.io/adv-datasets (last access: 28 June 2021), 2019.
Johannessen, B. G., Hanslin, H. M., and Muthanna, T. M.: Green roof
performance potential in cold and wet regions, Ecol. Eng., 106, 436–447,
https://doi.org/10.1016/j.ecoleng.2017.06.011, 2017.
Jones, P. and Macdonald, N.: Making space for unruly water: Sustainable
drainage systems and the disciplining of surface runoff, Geoforum, 38,
534–544, https://doi.org/10.1016/j.geoforum.2006.10.005, 2007.
Khan, U. T., Valeo, C., Chu, A., and He, J.: A data driven approach to
bioretention cell performance: Prediction and design, Water,
5, 13–28, https://doi.org/10.3390/w5010013, 2013.
Kirchner, J. W.: Getting the right answers for the right reasons: Linking
measurements, analyses, and models to advance the science of hydrology,
Water Resour. Res., 42, W03S04, https://doi.org/10.1029/2005WR004362, 2006.
Kratzert, F., Herrnegger, M., Klotz, D., Hochreiter, S., and Klambauer, G.:
NeuralHydrology – Interpreting LSTMs in Hydrology, in: Lecture Notes in
Computer Science (including subseries Lecture Notes in Artificial
Intelligence and Lecture Notes in Bioinformatics), vol. 11700 LNCS, 347–362, Springer, Cham, Switzerland, 2019.
Kuhn, M. and Johnson, K.: Feature Engineering and Selection: a Practical
Approach for Predictive Models., Chapman and Hall/CRC, available
at:
https://www.routledge.com/Feature-Engineering-and-Selection-A-Practical-Approach-for-Predictive-Models/Kuhn-Johnson/p/book/9781138079229
(last access: 24 July 2020), 2019.
Kuhn, M. and Wickham, H.: Tidymodels: a collection of packages for modeling
and machine learning using tidyverse principles, available at:
https://www.tidymodels.org (last access: 8 November 2021),
2020.
Lee, J. G., Nietch, C. T., and Panguluri, S.: Drainage area characterization for evaluating green infrastructure using the Storm Water Management Model, Hydrol. Earth Syst. Sci., 22, 2615–2635, https://doi.org/10.5194/hess-22-2615-2018, 2018a.
Lee, J. G., Nietch, C. T., and Panguluri, S.: SWMM Modeling Methods for
Simulating Green Infrastructure at a Suburban Headwatershed: User's Guide,
U.S. Environ. Prot. Agency, October, 157, available at:
https://nepis.epa.gov/Exe/ZyPDF.cgi/P100TJ39.PDF?Dockey=P100TJ39.PDF%0A
(last access: 11 July 2020b), 2018b.
Li, S., Kazemi, H., and Rockaway, T. D.: Performance assessment of stormwater
GI practices using artificial neural networks, Sci. Total Environ., 651,
2811–2819, https://doi.org/10.1016/j.scitotenv.2018.10.155, 2019.
Liu, J., Sample, D., Bell, C., and Guan, Y.: Review and Research Needs of
Bioretention Used for the Treatment of Urban Stormwater, Water, 6,
1069–1099, https://doi.org/10.3390/w6041069, 2014.
Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., and Zhang, G.: Learning under
Concept Drift: A Review, IEEE Trans. Knowl. Data Eng., 31, 2346–2363,
https://doi.org/10.1109/TKDE.2018.2876857, 2019.
Lundberg, S. M. and Lee, S. I.: A unified approach to interpreting model
predictions, in: Advances in Neural Information Processing Systems, December 2017, 4766–4775, available at:
https://github.com/slundberg/shap (last access: 30 June 2020), 2017.
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B.,
Katz, R., Himmelfarb, J., Bansal, N., and Lee, S.-I.: From local explanations
to global understanding with explainable AI for trees, Nat. Mach. Intell.,
2, 56–67, https://doi.org/10.1038/s42256-019-0138-9, 2020.
Maier, H. R. and Dandy, G. C.: Neural networks for the prediction and
forecasting of water resources variables: a review of modelling issues and
applications, Environ. Model. Softw., 15, 101–124,
https://doi.org/10.1016/S1364-8152(99)00007-9, 2000.
Mitchell, R. and Frank, E.: Accelerating the XGBoost algorithm using GPU
computing, PeerJ Comput. Sci., 2017, e127, https://doi.org/10.7717/peerj-cs.127,
2017.
Montalto, F., Behr, C., Alfredo, K., Wolf, M., Arye, M., and Walsh, M.: Rapid
assessment of the cost-effectiveness of low impact development for CSO
control, Landscape Urban Plan., 82, 117–131,
https://doi.org/10.1016/j.landurbplan.2007.02.004, 2007.
Morton, A.: Mathematical models: Questions of trustworthiness, Br. J.
Philos. Sci., 44, 659–674, https://doi.org/10.1093/bjps/44.4.659, 1993.
Muthanna, T. M., Viklander, M., and Thorolfsson, S. T.: Seasonal climatic
effects on the hydrology of a rain garden, Hydrol. Process., 22,
1640–1649, https://doi.org/10.1002/hyp.6732, 2008.
Nash, J. E. and Sutcliffe, J. V.: River flow forecasting through conceptual
models part I – A discussion of principles, J. Hydrol., 10, 282–290,
https://doi.org/10.1016/0022-1694(70)90255-6, 1970.
Nearing, G. S., Kratzert, F., Sampson, A. K., Pelissier, C. S., Klotz, D.,
Frame, J. M., Prieto, C., and Gupta, H. V: What Role Does Hydrological
Science Play in the Age of Machine Learning?, Water Resour. Res., 57, e2020WR028091, https://doi.org/10.1029/2020WR028091, 2021.
Niazi, M., Nietch, C., Maghrebi, M., Jackson, N., Bennett, B. R., Tryby, M., and Massoudieh, A.: Storm Water Management Model: Performance Review and Gap
Analysis, J. Sustain. Water Built Environ., 3, 04017002,
https://doi.org/10.1061/JSWBAY.0000817, 2017.
Nielsen, A.: Practical Time Series Analysis, O'Reilly Media, Inc., available at:
https://www.oreilly.com/library/view/practical-time-series/9781492041641/
(last access: 30 June 2020), 2019.
Nielsen, D.: Tree Boosting With XGBoost: Why does XGBoost win every machine
learning competition?, Master's Thesis, Norwegian University of Science and Technolgy,
http://hdl.handle.net/11250/2433761 (last access: 10 November 2021), Norwegian University of Science and Technology, Norway, 2016.
Oreskes, N., Shrader-Frechette, K., and Belitz, K.: Verification, validation,
and confirmation of numerical models in the earth sciences, Science, 80, 641–646, https://doi.org/10.1126/science.263.5147.641, 1994.
Pelletier, A. and Andréassian, V.: Hydrograph separation: an impartial parametrisation for an imperfect method, Hydrol. Earth Syst. Sci., 24, 1171–1187, https://doi.org/10.5194/hess-24-1171-2020, 2020.
Rosa, D. J., Clausen, J. C., and Dietz, M. E.: Calibration and Verification
of SWMM for Low Impact Development, J. Am. Water Resour. Assoc., 51, 746–757,
https://doi.org/10.1111/jawr.12272, 2015.
Ross, A., Hughes, M. C., and Doshi-Velez, F.: Right for the Right Reasons:
Training Differentiable Models by Constraining their Explanations, available at:
https://github.com/dtak/rrr (last access: 2 September 2021),
2017.
Rudin, C.: Stop explaining black box machine learning models for high stakes
decisions and use interpretable models instead, Nat. Mach. Intell., 1,
206–215, https://doi.org/10.1038/s42256-019-0048-x, 2019.
Schmidt, L., Heße, F., Attinger, S., and Kumar, R.: Challenges in
Applying Machine Learning Models for Hydrological Inference: A Case Study
for Flooding Events Across Germany, Water Resour. Res., 56, e2019WR025924, https://doi.org/10.1029/2019WR025924, 2020.
Selbig, W. R., Buer, N., and Danz, M. E.: Stormwater-quality performance of
lined permeable pavement systems, J. Environ. Manage., 251, 109510, https://doi.org/10.1016/j.jenvman.2019.109510, 2019.
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., and De Freitas, N.:
Taking the human out of the loop: A review of Bayesian optimization, Proc.
IEEE, 104, 148–175, https://doi.org/10.1109/JPROC.2015.2494218, 2016.
Snoek, J., Larochelle, H., and Adams, R. P.: Practical Bayesian optimization
of machine learning algorithms, in: Advances in Neural Information Processing
Systems, vol. 4, 2951–2959, arXiv [preprint], arXiv:1206.2944v2, 2012.
Shapley, L. S.: A value of n-person games. Contributions to the Theory of
Games, 307–317, Princeton University Press, Princeton, NJ, USA, 1953.
Solomatine, D. P. and Dulal, K. N.: Model trees as an alternative to neural
networks in rainfall-runoff modelling, Hydrol. Sci. J., 48, 399–411,
https://doi.org/10.1623/hysj.48.3.399.45291, 2003.
Starn, J. J., Kauffman, L. J., Carlson, C. S., Reddy, J. E., and Fienen, M.
N.: Three-Dimensional Distribution of Groundwater Residence Time Metrics in
the Glaciated United States Using Metamodels Trained on General Numerical
Simulation Models, Water Resour. Res., 57, e2020WR027335,
https://doi.org/10.1029/2020WR027335, 2021.
stsfk: stsfk/ExplainableML_SuDS: (v1.0), Zenodo [code], https://doi.org/10.5281/zenodo.5652719, 2021.
Sundararajan, M. and Najmi, A.: The many shapley values for model
explanation, in: 37th International Conference on Machine Learning, ICML
2020, vol. PartF16814, 9210–9220, 13 to 18 July 2020, 2020.
Sundararajan, M., Taly, A., and Yan, Q.: Axiomatic attribution for deep
networks, in 34th International Conference on Machine Learning, ICML 2017,
vol. 7, pp. 5109–5118, Sydney, Australia, 6 to 11 August 2017, 2017.
Teetor, N.: zeallot: Multiple, Unpacking, and Destructuring Assignment, R
package version 0.1.0, available at:
https://CRAN.R-project.org/package=zeallot (last access: 8 November 2021), 2018.
Trinh, D. H. and Chui, T. F. M.: Assessing the hydrologic restoration of an urbanized area via an integrated distributed hydrological model, Hydrol. Earth Syst. Sci., 17, 4789–4801, https://doi.org/10.5194/hess-17-4789-2013, 2013.
Ushey, K.: RcppRoll: Efficient Rolling/Windowed Operations, R package
version 0.3.0, available at:
https://CRAN.R-project.org/package=RcppRoll (last access: 8 November 2021), 2018.
Wani, O., Beckers, J. V. L., Weerts, A. H., and Solomatine, D. P.: Residual uncertainty estimation using instance-based learning with applications to hydrologic forecasting, Hydrol. Earth Syst. Sci., 21, 4021–4036, https://doi.org/10.5194/hess-21-4021-2017, 2017.
Yang, Y. and Chui, T. F. M.: Hydrologic Performance Simulation of Green
Infrastructures: Why Data-Driven Modelling Can Be Useful?, in: New Trends in
Urban Drainage Modelling, 480–484, Springer International Publishing, Cham, Switzerland, 2019.
Yang, Y. and Chui, T. F. M.: Reliability Assessment of Machine Learning
Models in Hydrological Predictions through Metamorphic Testing, Water
Resour. Res., 57, 1–27, https://doi.org/10.1029/2020wr029471, 2021.
Yong, C. F., McCarthy, D. T., and Deletic, A.: Predicting physical clogging
of porous and permeable pavements, J. Hydrol., 481, 48–55,
https://doi.org/10.1016/j.jhydrol.2012.12.009, 2013.
Zambrano-Bigiarini, M.: hydroGOF: Goodness-of-fit functions for comparison
of simulated and observed hydrological time series R package version 0.4-0, Zenodo [code], https://doi.org/10.5281/zenodo.840087,
2020.
Zeng, X. and Martinez, T. R.: Distribution-balanced stratified
cross-validation for accuracy estimation, J. Exp. Theor. Artif. Intell.,
12, 1–12, https://doi.org/10.1080/095281300146272, 2000.
Zhang, K. and Chui, T. F. M.: A review on implementing infiltration-based
green infrastructure in shallow groundwater environments: Challenges,
approaches, and progress, J. Hydrol., 579, 124089,
https://doi.org/10.1016/j.jhydrol.2019.124089, 2019.
Zhou, Q.: A Review of Sustainable Urban Drainage Systems Considering the
Climate Change and Urbanization Impacts, Water, 6, 976–992,
https://doi.org/10.3390/w6040976, 2014.
