Exploring hydrologic post-processing of ensemble streamflow forecasts based on affine kernel dressing and non-dominated sorting genetic algorithm II

Xu, Jing; Anctil, François; Boucher, Marie-Amélie

doi:https://doi.org/10.5194/hess-26-1001-2022

Articles | Volume 26, issue 4

https://doi.org/10.5194/hess-26-1001-2022

© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/hess-26-1001-2022

© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 26, issue 4

Research article

|

22 Feb 2022

Research article |

| 22 Feb 2022

Exploring hydrologic post-processing of ensemble streamflow forecasts based on affine kernel dressing and non-dominated sorting genetic algorithm II

Jing Xu, François Anctil, and Marie-Amélie Boucher

Download

Final revised paper (published on 22 Feb 2022)
Supplement to the final revised paper
Preprint (discussion started on 17 Jul 2020)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC1: 'Review remarks on the paper "Exploring hydrologic post-processing of ensemble stream flow forecasts based on Affine kernel dressing and Nondominated sorting genetic algorithm II"', Anonymous Referee #1, 04 Oct 2020
- AC1: 'Reply on 'Review remarks on the paper "Exploring hydrologic post-processing of ensemble stream flow forecasts based on Affine kernel dressing and Nondominated sorting genetic algorithm II"'', Jing Xu, 11 Nov 2020
RC2: 'Referee comment on "Exploring hydrologic post-processing of ensemble stream flow forecasts based on Affine kernel dressing and Nondominated sorting genetic algorithm II"', Anonymous Referee #2, 14 Oct 2020
- AC2: 'Reply on 'Referee comment on "Exploring hydrologic post-processing of ensemble stream flow forecasts based on Affine kernel dressing and Nondominated sorting genetic algorithm II"'', Jing Xu, 11 Nov 2020
EC1: 'Editor's comment', Dimitri Solomatine, 19 Nov 2020

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

ED: Publish subject to revisions (further review by editor and referees) (19 Nov 2020) by Dimitri Solomatine

AR by Jing Xu on behalf of the Authors (16 Feb 2021) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (17 Feb 2021) by Dimitri Solomatine

RR by Anonymous Referee #3 (12 Apr 2021)

Suggestions for revision or reasons for rejection

Summary
This study aims at improving the quality of probabilistic forecasts and facilitating the uncertainty communication by using two post-processing approaches within a hydrologic ensemble prediction system framework. The methods are clearly described and I find the results convincing. I think the article is ready for publication after some minor corrections.
Some minor comments
L53: It might be useful to give a short description of what “post-processing” means, something like “a correction of predictive distributions based on a comparison with past observations”.
L68: A reference to “Another IMPREX project” is needed.
L120: It might be useful to add the location of the gauging stations.
L148: “Five random hydrologic models from HOOPLA are exploited in this study.”: what do you mean by “random” ? I find it very surprising that no other reason explains the choice of using those models.
L225: It find it difficult to understand how the NSE is computed: with probabilistic streamflow forecast or with “average forecast and the average observation”? In addition, what do you mean by: “observed ones”? Did you use the same uncertainty estimation for observed values as in the EnKF (L160)? If not, please explain why.
L390: I find it a bit difficult to understand the link between the post-processing settings and the impact of some probabilistic scores. In particular, reliability is clearly improved, at the cost of dispersion, but I am not sure to understand how and why the weights obtained with NSGA-II can lead to those results. I would find it useful that you discuss this point in more depth.
L413: While I find the study convincing, I would appreciate a bit more of discussion, especially regarding the limits of the H-EPS and the various ways to improve them. It might be surprising that a H-EPS based on weather forecasts and data assimilation (EnKF) still needs some post-processing methods to be reliable. It might indicate that more work is needed to more appropriately define the data assimilation settings and inflate the ensemble within the data assimilation step. Or that a multimodel approach is required ?

Hide

RR by Anonymous Referee #2 (02 Oct 2021)

Suggestions for revision or reasons for rejection

Dear authors,

Thank you for the interesting manuscript. The Introduction with literature review has been rewritten with more focus on the use of ensemble post-processing in practice and user requirements, and I appreciate the benefits of multi-objective optimisation methods in communicating with practitioners/end users on trade-offs. At the end of the Conclusion section you reflect again on this point.

The methodology has been further clarified on several points, e.g. on the particular implementation of the NSGA-II, and the discussion of results has been expanded.

General comments:
The body of the experiments and results is in my understanding a comparison of performance of two post-processing methods AKD and NSGA-II multi-objective optimisation. In the Introduction and Conclusion sections now a reflection has been added on user requirements of ensemble forecasts and benefits of being able to communicate trade-offs in which skill aspect(s) to increase with a hydrologic post-processor. The multi-objective optimisation has that benefit through presenting of pareto fronts and discussing these with end users. The current write-up of the Introduction and Conclusions, however, leaves the reflection on end user perspective still quite disconnected from the experimental set-up and results (and the reflection on end user perspective is not mentioned at all in the Abstract). A suggestion, for consideration, to further clarify the connection between objective and experimental set-up and results is to take as objective something like: "In this paper the performance of evolutionary multi-objective optimisation (with NSGA-II) as hydrological ensemble post-processor is tested and compared with a conventional state-of-the-art post-processor, AKD, because of the benefit of NSGA-II method in communicating trade-offs with end users on which performance aspects to improve." This is rather a long sentence and can be split-up in several sentences if that makes the message more clear. In the Conclusion sections the authors can then reflect on this objective, and objective and conclusions can be taken up in the abstract as well.

The choice to train the post-processors on day-4 lead time and test/validate on days 1-3 and 5-7 has been clearly described, and the option to train for each lead time separately instead has been mentioned. The main point of discussion, however, is that by the approach taken, the observational data set for training and validation is the same. Question to be discussed in the paper by the authors is whether and how a validation of the methods to a period unseen would affect the results. Unless there is a miss-understanding on my side, because I do not quite understand the 30-day moving window mentioned as training period in lines 320-321. In other sections years 1985-2017 are referred to as the focus of this study (e.g. line 132), and 2011-2016 as "committed to forecasting" (line 150). It could be that the longer time period was used in earlier studies for calibrating and validating the hydrological models used, and then perhaps the years 2011-2016 have been used to emulate re-forecasting using the trained post-processing methods, and then perhaps the training of the post-processor is done only on the past 30-days (emulating an operational setting over the period 2011-2016 and re-training every next day within that period). After which the performance metrics reported in figures 7 onwards report the performance over 2011-2016. While these elements are all stated somewhere in the paper, I would request the authors to further clarify. This can be at the lines referred to in my comment here, and in addition perhaps by adding analysis periods to the captions of the result figures.

If there is any overlap between training/calibration and validation period in hydrological model development and/or re-forecast analysis and/or post-processor training, the potential impact of that overlap on the results and interpretation should be discussed.

Detailed comments and editorials.
l26-27: suggest to leave out colloquial sentence about insufficient single deterministic forecast
l34: delete "national" (because ECMWF is not a national organisation)
l46-47: Edit, sentence is not correct.
l57: Edit: "value of a hydrological forecasts"
l59: "She also demonstrated.." ? Colloquial. Provide specific name/reference
l64-69: this is an important paragraph leading-up to the objective of the paper, but it contains some unclear sentences. Please reformulate to clarify. (see suggestion in General Comment above)
l68: "..another IMPREX product conduct.." ? (not a correct sentence, but also IMPREX not introduced before I think. Leave out if possible)
l71: should be "This study is a contribution.."
l71: ".. to probe this topic.." Instead of "this topic" explicitly state the topic here.
I assume it is the remaining challenge mentioned in l54-56: "how to improve the human interpretation of probabilistic forecasts and the communication of integrated ensemble forecast products to end-users (e.g., operational hydrologists, water managers, local conservation authorities, stakeholders and other relevant decision makers)." But then in l71 and further it should be explained how the testing of these two post-processing methods contributes to improving interpretation and communication. Or "this topic" refers to the paragraph l64-69, which needs reformulation to be more clear, but is directed towards harmonising forecast improvement and user-specific requirements and use in decision making, in which case l71 onwards also has to explain how the comparison of these two post-processing methods is contributing to this. In the present formulation of l71-75 it seems more as if the authors are addressing the benefits of distribution-free postprocessing methods. See my suggestion above under General Comment.
l314: ".. for each day."
l320-321: A training period of 30-days with moving window is mentioned here. Please kindly clarify, including whether that applies to re-training every next day both the AKD and NSGA-II post-processor, using only the past 30-days observational and forecast data (day-4 lead time).
l354: I do not think the heading "Uncertainty analysis" covers what is presented and discussed here. I would suggest a separate heading for the NSGAII result, e.g. "4.2: NSGA-II convergence", and then for the comparison, e.g. "4.3 AKD and NSGA-II performance comparison".
l356: remove the space in "wi thout"
l359-361: I do not understand what is done here and why, when referring to 'random selection from the pareto front'
l367-369: Remove. I suggest to start the new section 4.3 with "The reliability of the raw.."

Hide

ED: Publish subject to revisions (further review by editor and referees) (04 Oct 2021) by Dimitri Solomatine

AR by Jing Xu on behalf of the Authors (15 Nov 2021) Author's response

ED: Referee Nomination & Report Request started (18 Nov 2021) by Dimitri Solomatine

RR by Anonymous Referee #3 (21 Dec 2021)

ED: Publish as is (30 Dec 2021) by Dimitri Solomatine

Short summary

The performance of the non-dominated sorting genetic algorithm II (NSGA-II) is compared with a conventional post-processing method of affine kernel dressing. NSGA-II showed its superiority in improving the forecast skill and communicating trade-offs with end-users. It allows the enhancement of the forecast quality since it allows for setting multiple specific objectives from scratch. This flexibility should be considered as a reason to implement hydrologic ensemble prediction systems (H-EPSs).