Hydrological concept formation inside long  short-term memory (LSTM) networks

Lees, Thomas; Reece, Steven; Kratzert, Frederik; Klotz, Daniel; Gauch, Martin; De Bruijn, Jens; Kumar Sahu, Reetik; Greve, Peter; Slater, Louise; Dadson, Simon J.

doi:https://doi.org/10.5194/hess-26-3079-2022

Articles | Volume 26, issue 12

Hydrol. Earth Syst. Sci., 26, 3079–3101, 2022

https://doi.org/10.5194/hess-26-3079-2022

© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.

Hydrol. Earth Syst. Sci., 26, 3079–3101, 2022

https://doi.org/10.5194/hess-26-3079-2022

© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 26, issue 12

Research article

20 Jun 2022

Research article | 20 Jun 2022

Hydrological concept formation inside long short-term memory (LSTM) networks

Thomas Lees et al.

Download

Final revised paper (published on 20 Jun 2022)
Preprint (discussion started on 23 Nov 2021)

Interactive discussion

Status: closed

CC1:
'Comment on hess-2021-566', John Ding, 24 Nov 2021

S-hydrograph vs. the unit hydrograph
The LSTM uses a hyperbolic tangent (tanh) as an activation function for the cell input g[t] and the recurrent input h[t], though not discussed in the manuscript, but elsewhere (e.g., Frame et al., 2021, Fig. A1.). This is similar in shape to a summation or S-curve hydrograph in unit hydrograph theory, e.g., Chow (1964, Fig. 14-5(a)).
To map some hydrologic realism onto the LSTM, I suggest the authors consider, as an alternative, using a kernel, i.e. a unit hydrograph model, my work (Ding, 1974, Figs. 1 & 4) being but one. Since a kernel has typically a unimodal distribution, a new LSTM-kernel variant will have to track whether the discharge is rising, falling, or remains steady.
References
Chow, V. T. , 1964. Handbook of applied hydrology, Section 14 - Runoff, McGraw-Hill, New York, ISBN 07-010774-2.
Ding, J. Y., 1974. Variable unit hydrograph. J. Hydrol., 22: 53-69.

Citation: https://doi.org/10.5194/hess-2021-566-CC1
- CC2: 'Supplement on CC1', John Ding, 07 Dec 2021
  
  A kernel or impulse response function, u[t], has typically a rising limb, a peak, and a falling limb.
  At time step, t-1, the discharge Q[t-1] had a corresponding u[t-1]. But at time step t, Q[t] has two u[t] values depending on the sign of dQ=Q[t]-Q[t-1], i.e. whether the discharge is rising (+), falling (-), or steady (0). During the rising stage, u[t]>u[t-1]; the falling stage, u[t]<u[t-1]; and u[t]=u[t-1], otherwise.
  In the LSTM (long short-term memory) networks, one need know only u[t-1] and u[t], i.e. the short(est)-term memory, and not further apart.
  
  Citation: https://doi.org/10.5194/hess-2021-566-CC2
- AC1: 'Reply on CC1', Thomas Lees, 16 Feb 2022
  
  We thank John Ding for his comments on our manuscript. Ultimately, our study was not focused on altering the LSTM architecture but rather on how we can best interpret the architecture that has already been shown to exhibit state of the art performance for rainfall runoff modelling. Other work has explored alterations to the LSTM architecture, and of particular interest is the Mass Conserving LSTM (Hoedt et al 2021) that some of the authors of this paper were involved in. That being said, the lead author, Thomas Lees, would be more than happy to discuss this further if you would like to reach out to him and organise a time to discuss. Thank you again for your interest in our work.
  Hoedt, Pieter-Jan, et al. "MC-LSTM: Mass-Conserving LSTM." arXiv preprint arXiv:2101.05186 (2021).
  
  Citation: https://doi.org/10.5194/hess-2021-566-AC1
RC1:
'Comment on hess-2021-566', Lukas Gudmundsson, 15 Dec 2021
The paper submitted by Lees et al. aims at advancing the interpretability of neural networks used for rainfall-runoff modelling, focussing in particular on Long Short Term Memory (LSTM) architectures. LSTMs (a special type of neural network) have in recent years been popularized for rainfall-runoff modelling. The resulting models perform very well but lack a stringent physical interpretation. An interesting property of LSTMs is that they contain internal states – similar to internal states (storages) in classical hydrological models. Lees et al use statistical “probes” to link these LSTM states to independent estimates of soil moisture and snow storage. The analysis shows that LSTM states mimic soil moisture and snow storage dynamics although these quantities were not used for model calibration.

This paper is to my knowledge the first systematic attempt to interpret the internals of an LSTM used for rainfall-runoff modelling in a systematic manner, although un-systematic examples have been emerging in the literature. The application of LSTMs for rainfall-runoff modelling is state of the art and the use of “probes” based on linear regression makes intuitively sense. Beside this, I personally value the authors effort to assess the robustness of their results by (i) applying the “probes” to both re-analysis based and remote-sensing based soil moisture estimates and (ii) by randomization based statistical testing.

The objective of the study is clearly stated: I.e., to test if internal states of an LSTM used for rainfall-runoff modelling at many catchments at once can be linked to independent estimates of soil moisture and snow. The paper is well structured to meet this objective and the data and methods are chosen accordingly.

Nonetheless the study leaves some open questions which the authors may want to further explore:

An obvious limitation of the proposed approach is that it relies on independent estimates of soil-moisture and snow (or other variables). Therefore, it does not allow for a self-contained interpretation of LSTM states.

It would be interesting to know how much of the variance of the internal state vectors is captured in the resulting soil moisture estimates.

How many independent signals are present in the 64 states? (e.g. estimated as the number of dominant principal components)

Minor issues:

Inconsistency: Equation 1-2 use i_t as input. Equations 3-4 use x_t

The paper somehow requires that the reader is familiar with how LSTMs work. I acknowledge the authors choice not to repeat the LSTM definition, also since it is available in many other publications. Nonetheless, this made it a bit more difficult to fully understand the paper.

Elastic net: I assume that the description of the elastic net regularisation might be quite cryptic to readers who are not familiar with this tool (I am). Also: Given the large number of samples (# catchments x # time steps) and the relatively low number of predictors I wondered if a linear regression would perform equally well.
Citation: https://doi.org/10.5194/hess-2021-566-RC1
- AC2: 'Reply on RC1', Thomas Lees, 16 Feb 2022
  
  We thank the reviewer for their positive and constructive comments on the manuscript. Please find attached the PDF with our complete responses to the reviewers comments.
  
  Citation: https://doi.org/10.5194/hess-2021-566-AC2
RC2:
'Comment on hess-2021-566', Anonymous Referee #2, 17 Jan 2022
Lees et al. adapted a novel method used in Natural Language Processing, “probe”, to examine the internal function of the Long Short Term Memory (LSTM) model in rainfall-runoff predictions. Their results over 669 catchments in Great Britain show a good correlation between the LSTM internal states with re-analysis and independent soil moisture and snow cover products.

I agree with the authors that this paper could be a stepping stone to a myriad of interesting explorations in the field of hydrology. I also appreciate the authors effort in providing additional analysis in the appendices. However, I have some minor comments about some parts of the manuscript, mostly about the clarity and the tone toward traditional hydrologic models.

I feel the structure of the Introduction is a bit difficult and redundant for me to follow. I could not get the logical flow here. I found the main objective was stated in both the beginning and the end of the introduction. Why do we need a separate and long paragraph about the interpreting machine learning from other fields? This paragraph disrupts my focus on LSTM interpretability.

I think the authors don’t have to state that LSTM is the best rainfall-runoff model multiple times in the paper (Introduction and Conclusion). While this statement is still debatable, in my opinion, each rainfall-runoff model has its place in the modeling world. LSTM is increasing its popularity because of its robustness, computational efficiency and accuracy. Period. There is no need for bashing one over another.

Section 2.3 ERA5-Land Data: there is an imbalance between the descriptions of soil moisture and snow depth. I would expect to see more information about snow depth and its accuracy over GB.

Figure 2: no y label

Figure 5: no y label

Line 249: I thought there are only two meteorological drivers (temperature and precipitation (line 6))?

Line 268: See the second opinion

Line 282: See the second opinion

Line 319: I found a recent paper (Tran et al, Development of a Deep Learning Emulator for a Distributed Groundwater–Surface Water Model: ParFlow-ML. Water. 2021) in which the spatial information is included in the LSTM architecture. Do the authors think the probing technique could be use in this architecture? Can the probing technique map between predicted and observed spatially-distributed soil moisture?
Citation: https://doi.org/10.5194/hess-2021-566-RC2
- AC3: 'Reply on RC2', Thomas Lees, 16 Feb 2022
  
  We thank the reviewer for their positive and constructive comments on the manuscript. Please find attached the PDF with our complete responses to the reviewers comments.
  
  Citation: https://doi.org/10.5194/hess-2021-566-AC3

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision

ED: Publish subject to minor revisions (further review by editor) (07 Mar 2022) by Alexander Gruber

Dear authors,

thank you for your submission. The referees are, overall, very positive about your manuscript and I share their opinion that the presented study provides an interesting and valuable novel approach to tackle the problem of interpretability in ML for Earth science problems.

I thus invite you to carefully revise your manuscript by implementing your proposed changes to the manuscript, which I believe do, in general, address most of their concerns. I, personally, share the following concerns of the reviewers in particular:

(i) Referee #1: I agree that it might be difficult for readers not too familiar with LSTMs and some other concepts (such as elastic nets) to understand all that is presented. While I also acknowledge that some background is provided and that it is difficult to strike a good balance, I tend to think that a little more detail (perhaps as am illustrative figure) might be beneficial for a broader readership.

(ii) Referee #2: I share the opinion that statements such as "the LSTM produces more accurate discharge simulations than any other hydrological model" need to be toned down a bit. In my view, there isn't a large enough body of evidence that would proof that LSTMs universally perform better than hydrological models, which I don't believe is what the authors want to say, but the text somewhat reads as so...

Would it perhaps be more fair to say that physically based models should - in theory - be more generalisable (by definition), yet they often require model calibration to account for representativeness errors; whereas ML-based approaches learn whatever data you throw at them and MAY be generalisable provided that actual physical processes are learned? I believe this study makes a good case that it could, and I believe this is also its intention... The fact that a well-trained non-linear ML model might, in the presence of good training data, often - if not always - outperform a perhaps not perfectly calibrated physical model with various simplifying assumptions etc. solely in terms of predictive accuracy is, from how I see it, not really a relevant statement in any context.

Best regards,
Alexander Gruber

Hide

AR by Thomas Lees on behalf of the Authors (15 Mar 2022) Author's response Author's tracked changes Manuscript

ED: Publish as is (04 Apr 2022) by Alexander Gruber

Short summary

Despite the accuracy of deep learning rainfall-runoff models, we are currently uncertain of what these models have learned. In this study we explore the internals of one deep learning architecture and demonstrate that the model learns about intermediate hydrological stores of soil moisture and snow water, despite never having seen data about these processes during training. Therefore, we find evidence that the deep learning approach learns a physically realistic mapping from inputs to outputs.