Parameter-state ensemble thinning for short-term hydrological prediction

Davison, Bruce; Fortin, Vincent; Pietroniro, Alain; Yau, Man K.; Leconte, Robert

doi:https://doi.org/10.5194/hess-23-741-2019

Volume 23, issue 2

Hydrol. Earth Syst. Sci., 23, 741–762, 2019
https://doi.org/10.5194/hess-23-741-2019
© Author(s) 2019. This work is distributed under
the Creative Commons Attribution 4.0 License.

Volume 23, issue 2

Research article 08 Feb 2019

Research article | 08 Feb 2019

Parameter-state ensemble thinning for short-term hydrological prediction

Bruce Davison et al.

Download

Final revised paper (published on 08 Feb 2019)
Preprint (discussion started on 20 Sep 2017)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC1: 'Review of hess-2017-482: Parameter-state ensemble data assimilation using Approximate Bayesian Computing for short-term hydrological prediction', Anonymous Referee #1, 20 Nov 2017
- AC1: 'Response to RC1', Bruce Davison, 18 Feb 2018
RC2: 'Referee comment', Anonymous Referee #2, 27 Nov 2017
- AC2: 'Response to RC2', Bruce Davison, 18 Feb 2018
RC3: 'Review', Anonymous Referee #3, 05 Dec 2017
- AC3: 'Response to RC3', Bruce Davison, 18 Feb 2018

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision

ED: Reconsider after major revisions (further review by editor and referees) (18 Feb 2018) by Harrie-Jan Hendricks Franssen

AR by Bruce Davison on behalf of the Authors (30 Apr 2018) Author's response Manuscript

ED: Referee Nomination & Report Request started (02 May 2018) by Harrie-Jan Hendricks Franssen

RR by Anonymous Referee #2 (07 Jun 2018)

RR by Anonymous Referee #1 (13 Jun 2018)

RR by Jasper Vrugt (02 Jul 2018)

Suggestions for revision or reasons for rejection

I was asked to look again at a revision of this paper. To be honest, at this point, I am not sure whether I commented on an earlier draft of this paper. I think I did. In short, I looked at the revision and have substantial concerns about the presented work. I did confirm with the Editor that I am reviewing the "right" version of the manuscript as I was surprised seeing relatively few changes to earlier comments on the methodology.

1. The authors use the word "filtering" - this suggests a state estimation exercise. In practice, no effort is made to discuss/describe the filtering methodology. Likelihood, incremental likelihood, and normalized likelihood. Which state variables are being included in the analysis The responses of the authors' to previous comments on this matter were not helpful as they did not lead to changes in the paper. What is the model error that is used - the stochastic perturbation of the state/output forecast? What is the measurement error of the discharge data that is being used - all this determines the likelihood, incremental likelihood and normalized likelihood of the particles - and so their time evolution. This is all crucial information to understand the implementation and evaluate the results. As I guess the authors pick at each time from a predefined ensemble those parameter vectors that perform "best" for a short past time period - and use those with state estimation to generate a forecast. I am not sure about my interpretation.

2. If filtering is not used re: state estimation then the authors should make this clear. In their algorithmic recipe nothing is mentioned about state estimation - nor details given. So, if filtering refers to picking out the "best" parameters at each time step then this should be made clear. If this is true, then the authors are missing other important work on this topic such as the DYNIA approach - and the PIMLI methodology that have elements in common with what the authors are doing in present paper. What is also relevant is the BARE approach of Thiemann et al. (2000) - Bayesian recursive estimation - also many elements in common with present approach if authors did indeed not do state estimation.

3. The authors implement an ABC methodology. They use the model as likelihood - but no likelihood is presented - OK - if the model output is the likelihood then this does not suffice - the likelihood (simulated discharge) requires perturbation to make the model stochastic rather than deterministic. ABC method cannot work with a deterministic likelihood - otherwise the posterior would converge to a point in the limit of "epsilon" going to zero. So, what is the stochastic perturbation that the authors are using to compute/define the likelihood? One cannot simply use the present model as likelihood. Instead, assumptions are required about the expected probabilistic properties of the model error - and one needs to sample from this distribution to corrupt the deterministic forecasts - this will then converge to the exact target - if the assumptions about the residual errors are honored by the data. See Vrugt and Beven (2018) for more detailed comments on this matter - and our earlier papers published in 2013, 2014 and 2015 in WRR, HESS, etc..

4. The authors use Latin Hyper Cube Sampling to sample 10,000 parameter vectors - and at each time they simple use for their forecasts the best "M" parameter vectors from the past few days. Is this what the authors are doing? This involves no filtering as per state estimation. If state estimation is used on top then I am worried about the initial states of the members that are not used. Lets say that at time 1 one uses members 1-10 to generate a forecast for time 2 - those members were found to produce the best forecasts for time 0. Then state estimation is used for those members 1-10. Then I would expect the initial states at time 2 of these members to be better than those remaining members (11-10000) of the ensemble s their states have not been estimated - so then at time 2 I would expect the first 10 members to do better for time 3 - as their states were estimated - and not those of the other members. So, again, I have simple but important questions about the methodology.

I cannot evaluate the results without understanding in detail the methodology.
I hope these comments are useful to further enhance the manuscript. It may very well be possible that I completely misread/misunderstand the authors intentions. Even then, it may benefit the authors from addressing some of these comments/concerns as other readers may experience similar issues with the presented material.

Jasper Vrugta
Irvine, July 2, 2018

Hide

ED: Reconsider after major revisions (further review by editor and referees) (08 Jul 2018) by Harrie-Jan Hendricks Franssen

AR by Bruce Davison on behalf of the Authors (13 Aug 2018) Author's response Manuscript

ED: Referee Nomination & Report Request started (16 Aug 2018) by Harrie-Jan Hendricks Franssen

RR by Anonymous Referee #1 (21 Sep 2018)

RR by Jasper Vrugt (25 Oct 2018)

Suggestions for revision or reasons for rejection

1. The authors use the word "Data Assimilation" in the name of their methodology - and also use this term at various places throughout the paper. Yet, their method does not use data assimilation at all. Instead, they now clarified what I thought last time and that is that for the prediction period they simple use the initial states which are linked intrinsically to the parameters. They use a screening period to determine the best N parameter,state combinations to be used for the prediction period. So, in essence this is a down-sampling exercise - those initial state & parameter combinations that are not used in prediction are still evaluated for the same period to get values of the initial states of this realization for the next prediction period (after the current one). In any case, at no point do the authors actually assimilate data - they simply do open loop simulation with M parameter vectors and for prediction they each time pick the N most desirable parameter vectors - derived from an earlier screening period.

The authors did respond to earlier comments on the use of DA in the title of their methodological framework. I thought I repeat my concerns as the paper is in much better shape now. Point is - no data is assimilated - the state&parameters are tight together. As this has important ramifications for the presented methodology and introduction this issues needs to be clarified.

2. It is still unclear to me after one read why the ABCDE methodology is needed at all. The authors mention ABC - add the deterministic equivalent ( = ABC without stochastic operator) but I am not sure whether this approach is implemented at all. In fact, the authors refer to it on Page 5, Line 3 as a way to interpret their P-SEDA approach.
Thus, I am wondering whether the exercise on ABC - with DE in this paper is needed after all. I do not think that Section 2 is needed to provide justification for their P-SEDA method - only if the ABC method is used to find an initial distribution of suitable parameter values that honor some user-defined summary statistics for the entire period.

3. More fundamental question is whether you can have ABC with a deterministic model operator. In fact, you cannot. The ABC methodology requires use of a stochastic model operator, otherwise it is impossible to approximate the likelihood via repeated simulation.

4. The ABC-DE acronym coined by the authors has been used before by Turner et al. in a publication in Journal of Mathematical Psychology. Here is the link:
http://faculty.sites.uci.edu/bmturner/files/2011/01/Turner.Sederberg.2012.pdf
The ABCDE approach of Turner uses Differential Evolution to sample the ABC posterior; this approach, however, does not work well with increasing number of summary statistics and increasing dimensionality ( = number of parameters) of the target distribution. The DREAM_ABC method of Sadegh and Vrugt provides a remedy against this. The latter is not relevant for this paper - but as paper is cited anyway, I thought I'd mention it here.

5. The authors have to fix some literature references - that is - use round brackets if reference appears at end of sentence - and use round brackets around year if authors of paper are part of the sentence. For example, see Line 8 on Page 4.

Altogether, I would recommend a moderate and final revision to give the authors the opportunity to respond to these remaining and somewhat lingering issues.

Hide

ED: Publish subject to revisions (further review by editor and referees) (25 Oct 2018) by Harrie-Jan Hendricks Franssen

AR by Bruce Davison on behalf of the Authors (04 Dec 2018) Author's response Manuscript

ED: Referee Nomination & Report Request started (09 Dec 2018) by Harrie-Jan Hendricks Franssen

RR by Jasper Vrugt (09 Dec 2018)

ED: Publish as is (10 Dec 2018) by Harrie-Jan Hendricks Franssen

Journal metrics

Abstracted/indexed

Parameter-state ensemble thinning for short-term hydrological prediction

Download

Interactive discussion

Peer review completion