Is the scientific field mired with improper methodologies adopted by journal offices?

Sivarajah Mylevaganam

Alumnus, Spatial Sciences Laboratory, Texas A&M University, College Station, USA.

I presume that the lifecycle of a manuscript goes through the steps outlined  below.

1. Performing a modeling work (e.g., mathematical modeling or physical modeling)
2. Writing a manuscript based what has been driven out from the modeling work.
3. Submitting the manuscript to a journal office that has advertised to have achieved a high impact factor.
4. Reviewing of the manuscript (probably a rigorous review) based on what has been presented in the manuscript by the authors.
5. Accepting or Rejecting based on what has been presented in the manuscript by the authors.

Does step 4 assert that the authors have done the modeling work correctly? In my opinion, even a seasoned guy with extensive experience and many qualifications may not be able to catch a bug if the authors are smart enough to present the manuscripts in a way to get through the system of processing. The following questions would be pertinent to be asked at this level to understand the methodologies adopted by journal offices.

• Is it possible to assert that there are no PhD students who have not tweaked modeling results or analysis to attain their titles and ambitions? Would we need hypothesis testing on this?
• Is it possible to assert that faculty members who are on the edge of a knife have never tweaked modeling results or analysis to meet their needs?
• Is it possible to assert that faculty members have never tweaked modeling results or analysis to progress in their careers?
• Is it possible to assert that the faculty members who supervise students are conversant with all the nuts and bolts of the work carried out by the students?
• Is it possible to assert that a manuscript accepted by a journal office with a high impact factor has no tweaked modeling results or analysis?

In my opinion, considering this loophole in the methodology adopted by journal offices, neither an advancement in the technologies nor in the scientific theories would yield the best of what is desirable for the betterment of the scientific field.

Acknowledgement and Disclaimer

The author is an alumnus of Texas A&M University, Texas, USA. The views expressed here are solely those of the author in his private capacity and do not in any way represent the views of Texas A&M University, Texas, USA.

Sivarajah Mylevaganam

Alumnus, Spatial Sciences Laboratory, Texas A&M University, College Station, USA.

The history of hydrological models goes to many decades. The progress that has been made to improve hydrological models through scientific findings has been showing the endless road to guide the next generation of hydrologist and associated specialists to quest for the betterment and mark the dead end. In this manuscript, the authors, who are not from the same hydrological basin, have employed an ML model to predict streamflow by training the model with hydrologically diverse data that is spatially and temporally large in extent. Based on the research, the authors draw a concrete conclusion that the previous modeling works (that is found based on ad-sense free search) using ML models have failed to understand the underlying philosophy in training and testing ML models.

In my opinion, the current version of the manuscript has many flaws. Moreover, the way the manuscript has been presented gives an impression that the authors are far off from the field of hydrology. Therefore, the current version of the manuscript needs an expert in hydrology to go through in detail in an unbiased manner. Furthermore, the language that has been used in the manuscript needs to be edited by a language specialist as the way the manuscript has been presented and the words that have been coined throughout the manuscripts are beyond from what is expected in a scientific manuscript that is submitted to a journal office that advertises to have achieved a high impact factor based on rigorous reviews by a panel of experts in the fields of specializations listed in the scope of the journal.

The following comments are posted.  

• In my opinion, the abstract of the manuscript needs to be re-written. I would say that an ML model with limited and poor data has been employed in writing the abstract. Moreover, what has been highlighted (“there is one MISTAKE in particular that is common”) is not well documented.
• Line 10-Line 14:

Hydrology models based on machine learning (ML) are different – ML models work best when trained on data from many watersheds (Nearing et al., 2021). This citation needs to be evaluated against the conclusion (i.e., ML models are best when trained with a large amount of hydrologically diverse data) that the authors draw from the manuscript that is submitted to this journal office.

• Line 10-Line 14:

Because ML models are trained with data from multiple watersheds, they are able to learn hydrologically diverse rainfall–runoff responses (Kratzert et al., 2019b). This citation goes to 2019. The comment 2 goes to 2021. Are the cited manuscripts giving the same thoughts? If so, the rationales in citing these manuscripts are not well understood.

• Line 17-Line 23:

The paragraph needs to be critically reviewed by a specialist. The sentences need to be evaluated. The paragraph gives an impression that the authors lack fundamental knowledge in the subject. The terminologies and words from an English dictionary are thrown without understanding the exact meaning.

• The crux of the manuscript that is highlighted by the authors (i.e., LSTM stream models are best when trained with a large amount of hydrologically diverse data) is a well established fact in the scientific field. Basically, the authors are hitting the concept of SAMPLING SIZE of an experiment. Therefore, instead of going through a painful path of running models to determine the number of basins, it would be a wise man thought to go through some statistical methods to answer this question (i.e., sampling size). In fact, even the current version of this manuscript does not give the exact number that would be required. It is a random number (531) that the authors have ended up with based on what has been analyzed (see Fig.5).
• Line 30-Line 36:

Does the order of the KEYWORDS have an influence in your search? Is the search from the engine not prioritized by the engine provider based on the business model employed?  What was the reason to limit the search to 2021. Based on Fig.1, it is understood that there are more than 3500 publications. Even if we consider the authors’ statement that review was initiated in September 2022, an iota of incompleteness surface.

• Line 25-Line 29:

The authors claim that the use of LSTMs for rainfall–runoff modeling has increased exponentially in the last several years. A figure (Fig.1) to support the claim is found in the manuscript. As per the figure, a rough estimation considering the heights of the bars gives an indication that around 8500(=3500+2500+1500+500+400+100) manuscripts have been found in the topic that the authors have invested. Referring to the previous comment, the authors have considered 100 manuscripts based on the search from a search engine of their interest. In other words, this manuscript is based on 100/8500*100%=1% of the manuscript found in the literature. What can be inferred from the training dataset that is employed in reviewing the literature? Will it lead to conclude that similar to an ML model the limited publications reviewed lead to draw wrong conclusions?

• The title of the manuscript needs to be assessed by a specialist. What is a basin? What is a watershed? What is a catchment? What is a region? What is the amount of data that a single basin possesses? What is the spatial and temporal extent of the basin that the authors are defining in the title of the manuscript? What is the heterogeneity level of the basin that the authors are defining in the title of the manuscript?
• Refer to Part III

Acknowledgement and Disclaimer

The author is an alumnus of Texas A&M University, Texas, USA. The views expressed here are solely those of the author in his private capacity and do not in any way represent the views of Texas A&M University, Texas, USA.

Sivarajah Mylevaganam

Alumnus, Spatial Sciences Laboratory, Texas A&M University, College Station, USA.

9)            Line 90-Line 94:

We have some evidence that there might be ways to construct training sets that could result in better models than simply training on all available streamflow data. We do not have results that support this directly.

These statements are not understood.

10)          Line 79-Line 84: Figure 5 shows how test period performance increases as more basins are added to the training set. Performance continues to increase up to the maximum size of the CAMELS data set (531 basins). In other words, even these 531 basins are most likely not enough to train optimal LSTM models for streamflow.

In my opinion, these statements do not make sense. To better understand your methodology, assume that we have 16 basins (not 531 basins as in your analysis) as shown in the figure in the attached PDF file. For simplicity, let us forget the shapes of the basins. Moreover, assume that you have added the basins A, B, and C to have a training set to derive the associated NSE. Likewise you add more basins to your training set to have an array of NSE values to show a plot like the one that you have shown in Fig.5. Based on this figure, is it meaningful to conclude that the performance of the model increases as more basins are added to the training set?  From a hydrological point of view, does it make sense to have a streamflow that is sourced by basins A, B, and C? Would the stream network that sources the flow at a location of interest become discontinuous?

The NSE value that you have reported for a training set size of 100 basins (see Fig.5) may not actually represent the hydrology although the reported NSE value is very close to the NSE value for a training set size of 531 basins that may represent the actual hydrology.  In ML models, you need to understand the theory that governs the system of equations. An in-depth understanding on the system of equations and how they are formulated will lead to understand the physics. Do the catchment attributes that you have chosen in your analysis play a role in the NSE values that you have reported in Fig.5?

Acknowledgement and Disclaimer

The author is an alumnus of Texas A&M University, Texas, USA. The views expressed here are solely those of the author in his private capacity and do not in any way represent the views of Texas A&M University, Texas, USA.

Sivarajah Mylevaganam

Alumnus, Spatial Sciences Laboratory, Texas A&M University, College Station, USA.

11)          Line 12-Line 13:

Because ML models are trained with data from multiple watersheds, they are able to learn hydrologically diverse rainfall–runoff responses (Kratzert et al., 2019b) in a way that is useful for example for prediction in ungauged basins (Kratzert et al., 2019a).

First In First Out (FIFO) is the concept that is implemented in citing (i.e., 2019a should come first). Since I don’t know what is being implemented by this journal office, I would let the journal office to pay an attention on this.

12)          Line 21-Line 23:

Then, in the bottom-up approach, after a model is developed, we might work on regionalization strategies to extrapolate parameters and parameterizations to larger areas (e.g., Samaniego et al., 2010; Beck et al., 2016).

Using e.g., in citing previous research works is considered an evading language in the scientific field. This gives an indication that the authors have failed to document a comprehensive review of the literature.

13)          Line 34-Line 36:

We collected these 100 papers for review in September, 2022, nearly three years after the original regional LSTM rainfall–runoff modeling papers (Kratzert et al., 2019a, b) were published.

The critical review of these 100 papers is not found the current version of the manuscript. Throughout this manuscript, the authors have freely cited their own works. Considering the level of their knowledge in the field of hydrology and the other related disciplines that are reflected in the current version of the manuscript, I would have my reservation in the cited manuscripts.

13)          Line 106-Line 107:

We selected a k-means cluster model based on a maximin criterion on silhouette scores, which resulted in a model with 6 clusters ranging from 59 to 195 basins per cluster.

I would say that your clustering methodology is inappropriate for this task as it would completely destroy the stream network and the underlying hydrology. This is one of the reasons why we have HUCs in the datasets that you have used in your analysis. Do you know the exact definition of HUC and the rationale behind developing HUCs? Your Fig.A4 is completely meaningless considering the purpose of the manuscript. I would suggest you have an in-depth look at your Fig.A3 and Fig.A4. What do you learn from those figures? 

14)          Line 235-Line 240:

For a particular basin of your interest, would you be able to show the values of Ws and the associated hs? 

Acknowledgement and Disclaimer

The author is an alumnus of Texas A&M University, Texas, USA. The views expressed here are solely those of the author in his private capacity and do not in any way represent the views of Texas A&M University, Texas, USA.

Does the number of basins influence the modeling outcome using ML models?

A systematic methodology to answer the question that is being raised is presented. This method ensures that the underlying hydrology is conserved at the best to answer the question.

Step-1:

Draw an artificial stream network. The stream network shall have one limb or line per basin. In other words, if you have 531 basins in your dataset, your stream network will have 531 limbs or lines.

Step-2:

Use one of the existing techniques to order or name the stream developed in step 1. For example, if you are using Strahler's method to order your stream network, you will end up with a network like the one shown below. There are many methods to order or name a stream network. Therefore, you will have to do some research to determine the best method that is suited for the problem that you are solving.

Step-3:

Run your ML model, considering all the basins that you have in your dataset. Report the coefficient that is of your interest (e.g., NSE).

Step-4:

Run your ML model, considering the basins after removing the smallest stream number. If you are referring to the above figure, the smallest number will be 1. You can randomly remove the streams from the network.

Step-4-1:

Assume that you have 50 basins with "1". You can run your ML model after removing those 50 basins. In other words, you will have 531–50=481 basins in your simulation. Report the coefficient that is of your interest (e.g., NSE).

Step-4-2:

Assume that you have 50 basins with “1”. You can run your ML model after removing one of those 50 basins. You can randomly remove the basin from the network. In other words, you will have 531-1=530 basins in your simulation. You repeat this procedure by removing them one by one. In other words, your final simulation will have 531–50=481 basins in your simulation. Report the coefficient that is of your interest (e.g., NSE). You will have an array of NSEs.

Step-5:

Run your ML model, considering the basins after removing the next smallest stream number. If you are referring to the above figure, the next smallest number will be 2. You can randomly remove the streams from the network.  

Step-5-1: Refer to Step-4-1

Step-5-2: Refer to Step-4-2

Step-6:

Continue your simulation work until you reach the largest stream number in your network. If you are referring to the above figure, the largest stream number will be 3.

Step-7:

Plot your NSEs and answer the question that has been raised.

In summary, a reverse algorithm needs to be developed to address the problem. Moreover, being conversant with spatial operations using products from ESRI or other vendors is required.

The concept of summation vs. unit hydrograph

By the title alone of their opinion paper, I admire the frankness of the authors admitting the shortcomings of the LSTM  networks when applied to a SINGLE basin.

In the LSTM, the form of the conversion function for the hidden gate h_t,  is tanh, e. g., Kratzert et al. (2018, Equation 7).  As observed previously by me, this is similar in shape to a summation or S-curve hydrograph in unit hydrograph theory, e.g., Lees (2022, CC1 and CC2 therein).

For the hidden gate, I suggest the authors consider taking one further step of using the first derivative of the conversion function. This is equivalent to using the form of  an instantaneous unit hydrograph or impulse response function in convolution integral, e.g., Ding (1974). This, I believe, will inject some hydrologic realism into the LSTM.

The bottom half of Figure 4 for Buffalo Fork, Wyoming (USGS Gage 13011900) for the single LSTM basin model clearly indicates that an impulse response model having a distinct peak time and magnitude characteristic, e.g., Jeong and Kim (2023, Figure 2), would outperform the LSTM as now configured.

A reconfigured LSTM as suggested above may perform as well as, if not better than,  the impulse response ones.

References

Ding, J.Y., 1974. Variable unit hydrograph. Journal of Hydrology, 22(1-2), pp.53-69.

Jeong, M. and Kim, D.H., 2023. Instantaneous physical rainfall–runoff prediction technique using a power–law relationship between time to peak and peak flow of an instantaneous unit hydrograph and the rainfall excess intensity. Journal of Hydroinformatics, 25(2), pp.415-431.

Kratzert, F., Klotz, D., Brenner, C., Schulz, K., and Herrnegger, M.: Rainfall–runoff modelling using Long Short-Term Memory (LSTM) networks, Hydrol. Earth Syst. Sci., 22, 6005–6022, https://doi.org/10.5194/hess-22-6005-2018, 2018.

Lees, T., Reece, S., Kratzert, F., Klotz, D., Gauch, M., De Bruijn, J., Kumar Sahu, R., Greve, P., Slater, L., and Dadson, S. J.: Hydrological concept formation inside long short-term memory (LSTM) networks, Hydrol. Earth Syst. Sci., 26, 3079–3101, https://doi.org/10.5194/hess-26-3079-2022, 2022.

The manuscript was well written and the topic is very interesting. I really enjoy reading this manuscript. The discussion is thorough, providing a deep understanding of LSTM and good practice. I think there is still room for discussion (related to the questions below). Please feel free to include it if the authors think these points are relevant. 

1. Why is the performance of physical or conceptual models (e.g., mHM and VIC) when calibrated for a single basin (“basin model”) better than the one calibrated for multiple basins (“regional model”) while that is the opposite with LSTM?
2. "Never train an LSTM on a single basin". What makes LSTM so special that we should not train on a single basin? What if we want to model new processes with LSTM and we just have data for a single (or just a few) basin(s)? – we do not have other publicly available data (as for streamflow modeling). 

Blow are possible discussions regarding these questions (just my opinion, please correct me if I am wrong)

Discussion of question 1:

Physically-based (or conceptual) model formulation: The physical processes are conceptualized using mathematical /empirical equations that were based on experimental results or the modeler’s understanding of reality. In this sense, PB models already used some kinds of data (because the equations in PB models are based on experimental results or the modeler’s understanding of reality – so equations in PB models are also a kind of data).

PB models: “basin” and “regional” models are often calibrated for a few dozen (or a few hundreds) calibration parameters (I am not sure how many calibration parameters there are for the VIC and mHM  that used in this study?). There are uncertainties in the data, model structure, and imprecise description of the physical process via mathematical equations in PB models, the calibration process will adjust calibraiton parameters to compensate for these errors/uncertainties. However, when applying to a large number of basins, the number of parameters might not be large enough to compensate for these errors/uncertainty => leading to a reduce model performance of the “regional” model compared to “basin” model.

ML models: “basin” and “regional” both have a high number of “trainable” parameters (similar to the term calibration parameters in PB) compared to PB models. For example, a basin (regional) model with 16 (256) hidden states, 1 layer of LSTM, and the model head consisting of 1 dense layer could have more than 1,400 (250,000) “trainable” parameters (these numbers were roughly calculated for a native LSTM network in PyTorch). With such models, it can easily overfit the train data and provide poor performance for test data. While the “regional” model with more data the overfitting problem might be avoided (but still possible) and provide a better performance for test data compared to “basin” model. This could be a reason why the performance of the “regional” model is better than the “basin” model with the LSTM network. 

Now if we calibrate  “basin” and "regional" PB model by adjusting parameters for individual model grid cells (or many groups of grid cells based on certain criteria), which results in a much larger number of calibration parameters – up to the level that the number of calibrated parameters of PB models and the number of trainable parameters of LSTM are the same, will PB behave similar to ML model? (I mean will the performance of the “basin” model is worse than the “regional” model?).

Discussion of question 2:

What makes LSTM so special that we should not train on a single basin? Assume that we have one basin and train the model for daily streamflow simulation for 10 years. In this case, we have 3650 pairs of inputs and outputs– this is already “big data” compared to other models that use a much lower number data points  (e.g., Table 1 – in Zhu et al., 2022; https://doi.org/10.1016/j.eehl.2022.06.001). I would be curious to know if this could be because the number of trainable parameters of the LSTM networks is high or something related to the long-short term memory (that we need more data to “warm-up” the model?).

What if we want to model new processes with LSTM and we just have data for a single (or very few) basin? Could we impose our understanding of the process of interest on the structure of the LSTM model or somehow make LSTM applicable even for a single basin? For example in Figure 4 (page 5), if we know that there are seasonal and trend components, can we model these two components separately and then combine them which might improve the model prediction, etc.. I would very helful  it if the authors could give some comments on this.

Thank you

Tam

Does the number of catchments/watersheds/sub-watersheds/subbasins influence the modeling outcome using ML models?

A systematic methodology to answer the question that is being raised is presented. This method ensures that the underlying hydrology is conserved at the best to answer the question.

Step-1:

Identify the basin of your interest. This should be based on a proper research methodology (Refer to PART IX)

Step-2:

Extract the associated catchments and the stream network using ESRI products or other vendors. These catchments and the stream network cover the basin of your interest. See the attached PDF file.

Step-3:

Use one of the existing techniques to order or name the stream network developed in step 2. For example, if you are using Strahler's method to order your stream network, you will end up with a network like the one shown below. There are many methods to order or name a stream network. Therefore, you will have to do some research to determine the best method that is suited for the problem that you are solving.

Step-4:

Run your ML model, considering all the catchments that you have in your basin. Report the coefficient that is of your interest (e.g., NSE).

 Step-5:

Run your ML model, considering the catchments after removing the smallest stream number. If you are referring to the above figure, the smallest number will be 1. You can randomly remove the streams from the network.

Step-5-1:

Assume that you have 500 catchments with "1". Assume that you have 10,000 catchments within your basin. You can run your ML model after removing those 500 catchments. In other words, you will have 10,000–500=9,500 catchments in your simulation. Report the coefficient that is of your interest (e.g., NSE).

Step-5-2:

Assume that you have 500 catchments with "1". Assume that you have 10,000 catchments within your basin. You can run your ML model after removing one of those 500 catchments. You can randomly remove the catchment from the network. In other words, you will have 10,000-1=9,999 catchments in your simulation. You repeat this procedure by removing them one by one. In other words, your final simulation will have 10,000–500=9,500 catchments in your simulation. Report the coefficient that is of your interest (e.g., NSE). You will have an array of NSEs.

Step-6:

Run your ML model, considering the catchments after removing the next smallest stream number. If you are referring to the above figure, the next smallest number will be 2. You can randomly remove the streams from the network. 

Step-6-1: Refer to Step-5-1

Step-6-2: Refer to Step-5-2

Step-7:

Continue your simulation work until you reach the largest stream number in your network. If you are referring to the above figure, the largest stream number will be 3.

Step-8:

Plot your NSEs and answer the question that has been raised.

In summary, a reverse algorithm needs to be developed to address the problem. Moreover, being conversant with spatial operations using products from ESRI or other vendors is required.