Review of HESS Manuscript

“Testing data assimilation strategies to enhance short-range AI-based discharge forecasts”

Please find attached my review of the manuscript.

1. Scope

The scope of the article is within the scope of HESS.

1. Summary

The authors use the outputs of an LSTM and a SAC model, as inputs to a feed-forward neural network (together with historical discharge and meteorological variables) to produce hydrological forecasts 1, 3 and 7 days ahead. They ran their experiments in CAMELS US.

I believe there are important deficiencies in the study that should be addressed before moving forward in the review process. See my comments below.

General comments

Section 2.1.

The authors indicate that they are benchmarking against Kratzert2019 and Newman2017, but they have different training / testing periods as the benchmarks.  In section 2.1 you indicate that you are training from 1989-2006 and testing in 2006-2008. However, Kratzert used 1999-2008 for training and 1989-1999 for testing. This means you are training much more than them, and testing in only a 2-year period. Why this difference?

Moreover, you are also indicating that you re-simulated the 1989-2008 period with the pretrained models from Kratzert and Newman, but again, you are not respecting the training/testing split that the authors did in their original study.

Using benchmarks is an extremely valuable technique, because it automatically places your method in existing literature, however the conditions of the original studies need to be respected. One should, if possible, adapt the new experiment to the existing benchmark, otherwise it does not make any sense to do a benchmark.

Also, why are you using only two years of testing?

Lastly, you should benchmark your study against other studies that used data assimilation methods. In CAMELS US there is the study of Nearing (2022), Feng (2020) or more recently Yang (2025).

Section  2.3

In section 2.3 you indicate that a specific model should be calibrated for each lead time, and that the alternative is inefficient. But this is not true. The LSTM has a linear layer at the end that transforms the hidden states to discharge.

Your model is a simple extension of this, but instead of a linear layer that go from hidden states to discharge, you have a feed forward neural network (so a couple of linear layers), in which, besides the simulated discharge you concatenate some past meteorological variables and discharge. I do not have anything against the simplicity of the model, because if it is simple and it works then great, but there are multiple things that need to be considered.

You can run the LSTM as a seq-seq model, roll it over the forecast horizon (so if you are predicting 7 day forecast just do seq-7) and in each step concatenate the hidden states with whatever you want and pass it through the feed forward neural network. This way you have a consistent and generalizable model that does not require a different embedding for each case.

Also why are you evaluating only on days 1, 3 and 7?  Why not all seven days?

Furthermore, you are using climatological ensembles to create a possible forecast, but I do not believe this is the best strategy to do that. Climatology is used, normally, in medium to seasonal range (so couple of weeks to some months) where the forecast models of the meteorological variables are no longer reliable, and for variables that present a cyclic pattern (temperature, radiation…)  but you are using it in precipitation 1 or 3 days ahead, which I do not believe is a good practice. How is the precipitation of the 1^st of November for the last 18 years related to the 1^st of November of this year?  I do not think there is a strong relationship between these values that can be used for short-term forecasting, especially for precipitation, which is the most important variable to drive the forecast. If you want to use it for temperature or radiation, that can be an (non-ideal but defendable) option, but I would highly recommend to not use it for precipitation.

In this section, you also indicate the absence of operational weather forecast archives, but this is also not fully correct. Shalev (2024) release a historical weather forecast for CARAVAN, which includes the CAMELS US dataset. It is true that not all the products are available in the testing periods that you have but:  One option is to use CHIRPS-GEFS which is only precipitation but is available in your period of interest. Another, better option, is to benchmark a simple LSTM against Kratzert 2019, and once that is working well, you evaluate your model and the new LSTM in the periods where you have a historical forecast (2016-2024) from Shalev. This is more work but would actually give you a robust study to evaluate your model under real forecast conditions.

Section 2.2 and 2.5.2

Here you indicate that you use an ensemble of 60 runs. Why 60?  Most studies use between 5 and 10. Do you get a significant different with 60? If you want to use 60 that is your choice, but in section 2.5.2 you indicate that because of the 60 ensembles you have unreasonable computational cost, and that you are only going to use a subset of basins. If 5 to 10 ensembles give you the same as 60, then you can reduce the computational costs, and then do the study in the full region, which would produce more robust result. 

Also, the 18 ensembles members can be accommodated in the batch dimension of the tensor and the different seeds can be run in parallel (even in a single GPU) so most of the computational overhead you are reporting can be overcome with some technical tweaking.

Also, from what I understood, you are not training the LSTM, just the feed-forward neural network. Is that right? If it is so, this should be extremely fast, I do not understand where the unreasonable computational cost is coming from.

1. Specific comment

Line 38: I do not agree that discharge simulation and discharge forecasting are fundamentally different tasks. You are trying to model the same system and the same rainfall-runoff response. In forecasting mode you have the increased uncertainty of the meteorological input, however that is more of a limitation and not a fundamentally different task. Multiple operational models are calibrated with observed data in pseudo-forecast mode and later incorporated in forecasting pipelines, and they work well. Data-driven methods give the advantage that, if trained with real forecast, they can learn to compensate for systematic biases, but again, this is more of training strategies to compensate for data quality limitations and not because the task is fundamentally different.

Line 66. The tittle of Fig 1 should be more self-explanatory.

Line 127-128: What do you mean by: “The direct forecasts from the benchmark models were assumed to be unchanged for the tested lead time; therefore, no further running was necessary”.

Line 136: Is the reference to figure 1 correct here?

Figure 7. You should explain the colors also on the legend of the figure and not only on the text above. The figure plus the legend should be self-explanatory. Also as a suggestion the message of this figure would be better explain by a boxplot per lead time graph. The boxplot would give the distribution along the basins and because you have one for each model and for each lead time it can be easily compared. Something similar to your Figure 8, but for the different lead times (you can also see Figure 3 from Nearing 2024).

Section 3.2.2: Can you explain in detail how did you constructed this figure? How did you construct the 10 classes? What does a lower or a higher rank indicates?

Line 330-335:  Can you explain in detail how are you evaluating the LSTM here to produce these results?  Also you are indicating that “This result suggests that the LSTM model is insufficiently responsive to recent meteorological inputs”.  Can it be that because the LSTM is driven only by meteorology, and because the climatological forecast are not good (see my comment above about that) then the predictions are biased? The other models have the advantage of having discharge, which is a highly autoregressive variable, so they somehow compensate. However, if all you have is meteorological forecast and these are non-sense, how can the model perform well? I think this point is important an can biased the results you are presenting.

References:

Nearing, G. S., Klotz, D., Frame, J. M., Gauch, M., Gilon, O., Kratzert, F., Sampson, A. K., Shalev, G., & Nevo, S. (2022). Technical note: Data assimilation and autoregression for using near-real-time streamflow observations in long short-term memory networks. Hydrology and Earth System Sciences, 26(21), 5493–5513. https://doi.org/10.5194/hess-26-5493-2022

Nearing, G., Cohen, D., Dube, V. et al. Global prediction of extreme floods in ungauged watersheds. Nature 627, 559–563 (2024). https://doi.org/10.1038/s41586-024-07145-

Feng, D., Fang, K., & Shen, C. (2020).Enhancing streamflow forecast andextracting insights using long‐shortterm memory networks with dataintegration at continental scales. WaterResources Research, 56,e2019WR026793. https://doi.org/10.1029/2019WR026793

Yang, Y., Pan, M., Feng, D., Xiao, M., Dixon, T., Hartman, R., Shen, C., Song, Y., Sengupta, A., Delle Monache, L., & Ralph, F. M. (2025). Improving streamflow simulation through machine learning-powered data integration and its potential for forecasting in the Western U.S. Hydrology and Earth System Sciences, 29(20), 5453–5476. https://doi.org/10.5194/hess-29-5453-2025

Shalev, G., & Kratzert, F. (2024). Caravan MultiMet: Extending Caravan with multiple weather nowcasts and forecasts. arXiv preprint arXiv:2411.09459. https://arxiv.org/abs/2411.09459

Review of Manuscript: 

“Testing data assimilation strategies to enhance short-range AI-based discharge forecasts” by Saint-Fleur at al.

Scope:

The manuscript is in the scope of the journal.

Summary:

The authors exploring two MLP-based data assimilation strategies to improve the discharge forecasts of two different hydrological models, an AI based LSTM and the mechanistic SAC-SMA model. The resulting forecasting setups are applied for three different lead times within two forecast scenarios, a deterministic optimal forecast and an ensemble forecast based on a climatological ensemble approach.

In my opinion the manuscript needs improvements in content and structure.

General comments:

The manuscript focuses on DA strategies. But why do the authors choose exclusively MLP-based strategies? This choice is not sufficiently justified in the introduction. How would established DA strategies such as Ensemble Kalman filters perform in comparison? Are there already studies that address this question? And what are the advantages of using MLP-based strategies over established DA strategies? 

In this regard, it would also be necessary to examine the required amount of training data. Is 18 years really necessary for this? And are there any catchment-specific differences? 

The chosen training method also seems rather random. Why are 60, 8 and 10 seeds used? Would smaller numbers also be suitable? Why are the reference models treated differently?

In terms of structure: the authors use result figures in the introduction and conclusion, which in my opinion is not good style. On the one hand, they refer to content introduced later on, and on the other hand, they add new content when a summary is required. Figure 1 is unnecessary, as its content is repeated in Figure 7. Figure 14 shows examples of the analysed data and should be presented at the beginning of the results section to give an impression of the data. A similar figure should be presented for the SAC-SMA model at least in the appendix.

Minor comments

Chapter 1 intro

- L38: I wouldn’t argue that forecasting is  “a fundamentally different task” since hydrologic discharge simulation models are used for forecasting but in a wider framework

- L49: “model structures”: of what?

- L55-65: manuscript structure, see comment above 

Chapter 2 Materials and methods

2.1 dataset

- style issue: repeating citations are not necessary

2.2 data assimilation

- L99-104: to be placed and elaborated in the introduction

- L129-L134: difficult to understand, “In both forecasting approaches --> this is not clear. You describe three DA approaches. But is MLP alone a DA approach or rather an alternative ML approach compared to LSTM? 

2.3. forecasting setup 

- L154: climatological ensemble --> unusual for daily data, provide discussion and reference

2.4. evaluation metric

Metric choice: You use CRPS. Would CRPSS also be an option for comparison with the reference?

- L186: Please provide reference for this statement. 

- I am not familiar with rank diagrams. How are they created? Why does the number of ranks differ between Figures 4 and 10?

- L197: You can also provide a formula for the spread skill ratio

- L215: I am not sure what you mean by “ensemble rank”. Do you mean each ensemble member or a quantile of the ensemble?

2.5. Experimental settings

- L230: typos: '.', RR-> RQ? In Figure 5 you are using Rainfall - Discharge cross-correlation.

- Figure 5: Please avoid overlapping colour areas, which will appear as a new colour. Perhaps just draw the lines.

- L239: Why 56 basins? Is the choice only motivated by NSE-values or also by different hydrological regimes?

Chapter 3 Results

- L340: Why should the observation be calibrated? Please provide reference.

- fig 13: Are the low flows over- or underestimated? 

Chapter 4 Conclusions

- fig 14: see comment above

- fig 14: Why is there a spread on the left side for the deterministic optimal forecast? Is this due to the forecast initialisation times and different lead times as a ‘poor man's ensemble’??

- L397: ”numerous publications”: Please provide examples.