Deep learning rainfall-runoff predictions of extreme events

Frame, Jonathan; Kratzert, Frederik; Klotz, Daniel; Gauch, Martin; Shelev, Guy; Gilon, Oren; Qualls, Logan M.; Gupta, Hoshin V.; Nearing, Grey S.

doi:https://doi.org/10.5194/hess-2021-423

Preprints

https://doi.org/10.5194/hess-2021-423

Preprints

18 Aug 2021

Review status: this preprint is currently under review for the journal HESS.

Deep learning rainfall-runoff predictions of extreme events

Jonathan Frame^1,2, Frederik Kratzert³, Daniel Klotz³, Martin Gauch³, Guy Shelev⁴, Oren Gilon⁴, Logan M. Qualls², Hoshin V. Gupta⁵, and Grey S. Nearing^6,7 Jonathan Frame et al.

Show author details

¹National Water Center, National Oceanic and Atmospheric Administration, Tuscaloosa, AL, United States
²University of Alabama, Tuscaloosa, AL, United States
³LIT AI Lab & Institute for Machine Learning, Johannes Kepler University, Linz, Austria
⁴Google Research, Tel Aviv, Israel
⁵The University of Arizona, Tucson, AZ, United States
⁶Google Research, Mountain View, CA, United States
⁷University of California Davis, Department of Land, Air & Water Resources, Davis, CA, United States

Received: 10 Aug 2021 – Accepted for review: 12 Aug 2021 – Discussion started: 18 Aug 2021

Abstract. The most accurate rainfall-runoff predictions are currently based on deep learning. There is a concern among hydrologists that data-driven models based on deep learning may not be reliable in extrapolation or for predicting extreme events. This study tests that hypothesis using Long Short-Term Memory networks (LSTMs) and an LSTM variant that is architecturally constrained to conserve mass. The LSTM (and the mass-conserving LSTM variant) remained relatively accurate in predicting extreme (high return-period) events compared to both a conceptual model (the Sacramento Model) and a process-based model (US National Water Model), even when extreme events were not included in the training period. Adding mass balance constraints to the data-driven model (LSTM) reduced model skill during extreme events.

Jonathan Frame et al.

Status: open (until 16 Oct 2021)

Post a comment Subscribe to comment alert

CC1: 'Comment on hess-2021-423', John Ding, 20 Aug 2021 reply

I’m intrigued by the opening sentence in the Abstract that “(t)he most accurate rainfall-runoff predictions are currently based on deep learning. “ And after a quick read, I’d have to concur with the authors in their findings.
One question I have is about one of their previous works. Kratzert et al. (2019a, Section 3.1, paragraph 2) wrote that “This [LSTM] is not dissimilar to any standard hydrological simulation model (i.e., is it not a one-step-ahead forecast model).“
The word “it” inside the parentheses seems out of place. I’d appreciate a clarification of this.
The conceptual model the authors adopt for benchmarking is the Sacramento Soil Moisture Accounting model (SAC-SMA). This includes “a [linear] unit hydrograph routing function” (Line 122).
As a proponent of using a nonlinear response function to simulate what I’ve called Childs-Minshall phenomenon (Ding, 2011, Figures 1 and 2), I feel the SAC-SMA can be improved by moving to a nonlinear store or storage. But then advancing the state of the art of a standard conceptual model is a separate issue.
References

Ding, J. Y.: A measure of watershed nonlinearity: interpreting a variable instantaneous unit hydrograph model on two vastly different sized watersheds, Hydrol. Earth Syst. Sci., 15, 405–423, https://doi.org/10.5194/hess-15-405-2011, 2011.

Reply

Citation: https://doi.org/10.5194/hess-2021-423-CC1
RC1: 'Comment on hess-2021-423', Anonymous Referee #1, 05 Sep 2021 reply

This is an interesting paper that addresses a most relevant issue of incorporating background knowledge about processes in question into machine learning (ML) algorithms.

Clearly, in ML we are hoping to program computers by telling them what we want to achieve without having to explicitly instruct them how to achieve such goals. But, what is it that we want from those programs? Do they just need to be accurate or should we also be able to interpret them? In the scientific contexts, the ambition is clear: we are looking for a learning machine capable of finding an accurate approximation of a natural phenomenon, as well as expressing it in the form of a meaningful or an interpretable model. The authors adopt this view, and I fully subscribe to such a working hypothesis.

At the same time, this bias towards meaningfulness and interpretability opens several additional issues. The computer-generated hypotheses should take advantage of the already existing body of knowledge about the domain in question. In the case of two equally good approximations of the same data set: the one which blindly fits the data and the other which, in addition to the fit, also respects the background knowledge, we should be biased toward the latter one.

However, there are few questions that I would appreciate authors could address in the manuscript to a greater depth.

1.) The fashion in which we express knowledge about the processes and make it available to the learning machine remains rather unclear. One can insist on strict adherence to the background knowledge principles - such as 100% mass balance accuracy. We declare this desire and hence this is referred to as declarative bias. Alternatively, one can treat the bias as an additional objective that should be treated simultaneously with the goodness of fit in the learning process. This is referred to as preferential bias. Declarative bias reduces search space but results in so-called broken ergodicity. Preferential bias results in a Pareto-optimal set of solutions. For further discussion in the context of water management see:

M Keijzer and V Babovic, 2002, Declarative and preferential bias in GP-based scientific discovery, Genetic Programming and Evolvable Machines 3 (1), 41-79

In the present paper, it would appear that authors prefer declarative treatment of background knowledge. However, I would appreciate further analysis, comparison, and, if that is not possible, at least a discussion on preferential vs. declarative bias in the case studies described in theirs work.

2.) Bias Variance Tradeoff. Arguably incorporation of the knowledge bias affects model variance. In this case, bias denotes the difference between the average prediction of a model and the correct value which it is trying to predict. Variance is the variability of model prediction for a given data point or a value that tells us the spread of our data. For in-depth discussion see:

Hastie, T; Tibshirani, R; Friedman, J. H. (2009). The Elements of Statistical Learning, Springer

I would love to see a more in-depth analysis of the bias-variance tradeoff in the present case, and am looking forward to reading more about it in the revised version of the manuscript.

3.) Models vs. Predictions.

LSTM-type of ML models are extremely good at forecasting. The authors have eloquently argued in favour of the approach in this (as well as in previous) published research works. At the same time, one must consider if such a n ML approaches induce models or forecasters. On this topic I would advise the following recent works:

HMVV Herath, et al, 2021, Hydrologically informed machine learning for rainfall–runoff modelling: towards distributed modelling, Hydrology and Earth System Sciences 25 (8), 4373-4401

HMVV Herath, J et al, 2021, Genetic programming for hydrological applications: to model or forecast that is the question, Journal of Hydroinformatics

In general, this is an interesting and potentially valuable contribution to the hydrological society. I am looking forward to reading revised version of the manuscript.

Reply

Citation: https://doi.org/10.5194/hess-2021-423-RC1

Jonathan Frame et al.

Data sets

CAMELS return period analysis Jonathan M. Frame https://doi.org/10.4211/hs.c7739f47e2ca4a92989ec34b7a2e78dd

Model code and software

Model results analysis Jonathan M. Frame https://github.com/jmframe/mclstm_2021_extrapolate/tree/main/results

NeuralHydrology Frederick Kratzert https://github.com/neuralhydrology/neuralhydrology

Code for calibrating SAC-SMA Grey S. Nearing https://github.com/Upstream-Tech/SACSMA-SNOW17

Jonathan Frame et al.

Viewed

Total article views: 985 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	BibTeX	EndNote
711	265	9	985	4	4

HTML: 711
PDF: 265
XML: 9
Total: 985
BibTeX: 4
EndNote: 4

Views and downloads (calculated since 18 Aug 2021)

Month	HTML	PDF	XML	Total
Aug 2021	521	213	4	738
Sep 2021	190	52	5	247

Cumulative views and downloads (calculated since 18 Aug 2021)

Month	HTML	PDF	XML	Total
Aug 2021	521	213	4	738
Sep 2021	190	52	5	247

Viewed (geographical distribution)

Total article views: 908 (including HTML, PDF, and XML) Thereof 908 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 19 Sep 2021

Short summary

The most accurate rainfall-runoff predictions are currently based on deep learning. There is a concern among hydrologists that deep learning models may not be reliable in extrapolation or for predicting extreme events. This study tests that hypothesis. The deep learning models remained relatively accurate in predicting extreme events compared traditional models, even when extreme events are not included in the training set.


Total:	0
HTML:	0
PDF:	0
XML:	0