23 Nov 2021

23 Nov 2021

Review status: this preprint is currently under review for the journal HESS.

Hydrological Concept Formation inside Long Short-Term Memory (LSTM) networks

Thomas Lees1,5, Steven Reece2, Frederik Kratzert6, Daniel Klotz3, Martin Gauch3, Jens De Bruijn5,7, Reetik Kumar Sahu5, Peter Greve5, Louise Slater1, and Simon Dadson1,4 Thomas Lees et al.
  • 1School of Geography and the Environment, University of Oxford, South Parks Road, Oxford, United Kingdom, OX1 3QY
  • 2Department of Engineering, University of Oxford, Oxford, United Kingdom
  • 3LIT AI Lab & Institute for Machine Learning, Johannes Kepler University Linz, Linz, Austria
  • 4UK Centre for Ecology and Hydrology, Maclean Building, Crowmarsh Gifford, Wallingford, United Kingdom, OX10 8BB
  • 5International Institute for Applied Systems Analysis (IIASA), Laxenburg, Austria
  • 6Google Research, Vienna, Austria
  • 7Institute for Environmental Studies, VU University, De Boelelaan 1087, 1081HV, Amsterdam, The Netherlands

Abstract. Neural networks have been shown to be extremely effective rainfall-runoff models, where the river discharge is predicted from meteorological inputs. However, the question remains, what have these models learned? Is it possible to extract information about the learned relationships that map inputs to outputs? And do these mappings represent known hydrological concepts? Small-scale experiments have demonstrated that the internal states of Long Short-Term Memory Networks (LSTMs), a particular neural network architecture predisposed to hydrological modelling, can be interpreted. By extracting the tensors which represent the learned translation from inputs (precipitation, temperature) to outputs (discharge), this research seeks to understand what information the LSTM captures about the hydrological system. We assess the hypothesis that the LSTM replicates real-world processes and that we can extract information about these processes from the internal states of the LSTM. We examine the cell-state vector, which represents the memory of the LSTM, and explore the ways in which the LSTM learns to reproduce stores of water, such as soil moisture and snow cover. We use a simple regression approach to map the LSTM state-vector to our target stores (soil moisture and snow). Good correlations (R2 > 0.8) between the probe outputs and the target variables of interest provide evidence that the LSTM contains information that reflects known hydrological processes comparable with the concept of variable-capacity soil moisture stores.

The implications of this study are threefold: 1) LSTMs reproduce known hydrological processes. 2) While conceptual models have theoretical assumptions embedded in the model a priori, the LSTM derives these from the data. These learned representations are interpretable by scientists. 3) LSTMs can be used to gain an estimate of intermediate stores of water such as soil moisture. While machine learning interpretability is still a nascent field, and our approach reflects a simple technique for exploring what the model has learned, the results are robust to different initial conditions and to a variety of benchmarking experiments. We therefore argue that deep learning approaches can be used to advance our scientific goals as well as our predictive goals.

Thomas Lees et al.

Status: open (until 18 Jan 2022)

Comment types: AC – author | RC – referee | CC – community | EC – editor | CEC – chief editor | : Report abuse
  • CC1: 'Comment on hess-2021-566', John Ding, 24 Nov 2021 reply

Thomas Lees et al.

Thomas Lees et al.


Total article views: 483 (including HTML, PDF, and XML)
HTML PDF XML Total BibTeX EndNote
349 130 4 483 0 2
  • HTML: 349
  • PDF: 130
  • XML: 4
  • Total: 483
  • BibTeX: 0
  • EndNote: 2
Views and downloads (calculated since 23 Nov 2021)
Cumulative views and downloads (calculated since 23 Nov 2021)

Viewed (geographical distribution)

Total article views: 474 (including HTML, PDF, and XML) Thereof 474 with geography defined and 0 with unknown origin.
Country # Views %
  • 1
Latest update: 01 Dec 2021
Short summary
Despite the accuracy of deep learning rainfall-runoff models, we are currently uncertain of what these models have learned. In this study we explore the internals of one deep learning architecture and demonstrate that the model learns about intermediate hydrological stores of soil moisture and snow water, despite never having seen data about these processes during training. Therefore, we find evidence that the deep learning approach learns a physically realistic mapping from inputs to outputs.