the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 02 Sep 2022
Hydrologic Interpretation of Machine Learning Models for 10-daily streamflow simulation in Climate sensitive Upper Indus Catchments
Abstract. Machine learning for hydrologic modeling has seen significant development and has been suggested as a valuable augmentation to physical hydrological modeling, especially in data scarce catchments. In Pakistan, surface water flows predominantly originate from the transboundary Upper Indus sub-catchments of Chenab, Jhelum, Indus and Kabul rivers. These are high elevation data scarce catchments and generated streamflows are highly seasonal and prone to climate change. Given the catchment characteristics, there is utmost need to develop machine learning models that are hydrologically robust. Thus, the current study besides evaluating the potential of three machine learning models for streamflow simulation also focused on the hydrologic interpretation of machine learning models using SHapley Additive exPlananations(SHAP).XGBOOST, RandomForest and Classification and Regression Trees(CART) were evaluated. All of these models performed well and range of R 2 and Nash Efficiency for all three models lies between 0.65 to 0.90. Our study’s most crucial contribution is SHapley Additive exPlananations (SHAP) method which gives extensive insights into the influence of each variable on simulated streamflow. SHAP analysis highlighted the significance of minimum temperature in high elevation zones for both Indus and Chenab catchment where streamflows are dominated by snow and glacier melt. We strongly believe that the findings of this study will promote the use of SHAP analysis for streamflow forecasting in data scarce and high elevation catchments in Pakistan.
This preprint has been withdrawn.
-
Withdrawal notice
This preprint has been withdrawn.
-
Preprint
(30457 KB)
Interactive discussion
Status: closed
-
RC1: 'Comment on hess-2022-213', Anonymous Referee #1, 29 Oct 2022
This study applied several machine learning models to streamflow forecast and investigated the model explanation with SHAP. The study is of potential value due to the unique features of hydrological process in Pakistan basins. However, the writing requires careful revision. And the contribution of this study should be further strengthened. I have the following comments for the authors’ consideration.
Line 4-5ã“Given the catchment characteristics, there is an utmost need to develop machine learning models that are hydrologically robust”ï¼ The logic is not convincing. Why there is an utmost need to develop machine learning models under the described catchment characteristics?
Line 5-8 Please split it into two short sentences.
Line 12 “We strongly believe….”. In my personal opinion, it’s better to remove emotional word.
Line 43 “Data-driven”. The initial should be lower-case.
Line 68 “These streamflows are highly seasonal, variant, and prone to extreme events and climate change”. I agree with the authors. The catchments in Pakistan have some special features. Development of machine learning for these catchments provides good chance to explore the potential of machine learning technique. It’s better if the authors could expand the challenges of machine learning on modeling the streamflow in these catchments. In other words, does the machine learning have the potential to forecast streamflow which may be highly seasonal, variant, and prone to extreme events and climate? It is an interesting point to attract the readers and an important contribution for hydrological modeling.
Line 69 Please correct the citation formatLine 87-117 The authors summarized the existing studies of several machine learning models. It will be better if the features (advantages and disadvantages) of these models could be briefly summarized.
Line 118 It is weird to set a “1.1 Hydrologic interpretation” in the Introduction section. I suggest to remove the subtitle.
Line 139 There is typo
Line 131-137 These two paragraphs can be merged.
Line 177-185 Please merge these paragraphs.
Line 187 Please reorganize this sentence.Line 363-366 Please reorganize this long sentence.
Figure 6 Please reorganize this figure to make it more formal.
Figure 7-9 Is it possible to specifically display the comparison under extreme events and climate (or other unique conditions in Pakistan) during 2005-2014? The long-term comparison in one figure may cover some interesting streamflow patterns, which however can be the new finding of this study.
Citation: https://doi.org/10.5194/hess-2022-213-RC1 -
AC1: 'Reply on RC1', Haris Mushtaq, 08 Feb 2023
We sincerely thank the referee for insightful comments that should surely help us in improving the quality of our manuscript
Line 4-5　“Given the catchment characteristics, there is an utmost need to develop machine learning models that are hydrologically robust”． The logic is not convincing. Why there is an utmost need to develop machine learning models under the described catchment characteristics?
Response: We agree with reviewer suggestion. We understand the use of word ‘Utmost’ was not appropriate here.
We have replaced this with the following sentence and will revise the manuscript accordingly. “As ground data is not available, we use remote sensing data, which are prone to errors and hence become source of uncertainty in hydrological models. This makes ML algorithms relevant.”
Line 5-8 Please split it into two short sentences.
Response: We will make the required change accordingly
Line 12 “We strongly believe….”. In my personal opinion, it’s better to remove emotional word.
Response: We have changed the sentence to "We expect that the findings of this study will promote the use of SHAP analysis for streamflow forecasting in data-scarce and high-elevation catchments in Pakistan."
Line 43 “Data-driven”. The initial should be lower-case.
Response: We will do the correction in revised manuscript.
Line 68 “These streamflows are highly seasonal, variant, and prone to extreme events and climate change”. I agree with the authors. The catchments in Pakistan have some special features. Development of machine learning for these catchments provides good chance to explore the potential of machine learning technique. It’s better if the authors could expand the challenges of machine learning on modeling the streamflow in these catchments. In other words, does the machine learning have the potential to forecast streamflow which may be highly seasonal, variant, and prone to extreme events and climate? It is an interesting point to attract the readers and an important contribution for hydrological modeling.
Response: We will explain the challenges of machine learning in modeling the streamflow in these catchments. The models were developed on 10-daily time scale as planning and operations in Pakistan are based on 10-daily time scale. Thus, the objective was not to predict extreme events, rather to establish ML models that are suitable for data-scarce catchments and catchments with mix of rainfall and snowmelt contributions to the runoff. We have, however, included seasonal analysis attached in Figure 1. Figure 1 compares the yearly seasonal volume for Kharif(April- September) and Rabi (October- March) between the Actual and simulated/predicted results. The results are displayed for XGBOOST, and it is evident that the model has captured seasonal trends well, apart from the year 2010, which was a year of super floods in Pakistan. We will include seasonal analysis in the revised manuscript
Line 69 Please correct the citation format
Response: We will correct the format in the revised manuscript
Line 87-117 The authors summarized the existing studies of several machine learning models. It will be better if the features (advantages and disadvantages) of these models could be briefly summarized.
Response: We will add a brief description of the features of ML models in the revised manuscript.
Line 118 It is weird to set a “1.1 Hydrologic interpretation” in the Introduction section. I suggest to remove the subtitle.
Response: We will make the required changes accordingly in the revised manuscript
Line 139 There is typo
Response: We will correct this in the revised manuscript
Line 131-137 These two paragraphs can be merged.
Response: We will make the required change accordingly
Line 177-185 Please merge these paragraphs.
Response: We will make the required change accordingly
Line 187 Please reorganize this sentence.
Response: We have reorganized the sentence to the following: Figure 5 presents the methodology of the study, and discussed in forthcoming sections.
Line 363-366 Please reorganize this long sentence.
Response: We have split this sentence into two sentences as follows and will do the change in the revised manuscript:
“Moreover, SHAP analysis allows reason-based predictions and thus analyse the learned representations. Hence, SHAP analysis not only increases the transparency of machine learning, but it can also eventually promote the application of machine learning methods in hydrological studies.”
Figure 6 Please reorganize this figure to make it more formal.
Response: We have reorganized the figure in more presentable and formal way now attached as Figure 2.
Figure 7-9 Is it possible to specifically display the comparison under extreme events and climate (or other unique conditions in Pakistan) during 2005-2014? The long-term comparison in one figure may cover some interesting streamflow patterns, which however can be the new finding of this study.
Response: We thank the referee for this comment and agree that this would be a valuable addition to our work. We have prepared river discharge density plots. Plots for XGBoost model are attached in Figure 3. We will include these results in the revised manuscript. It is evident that ML does not capture extreme flows, however, this may not be critical as 10-daily models are mostly used for operations and planning purposes and do not need to account for extreme flows.
Figure 1: Seasonal volume comparison of simulated and actual streamflow for four catchments of XGBOOST: Indus at Tarbela, Jhelum at Mangla, Chenab at Marala, and Kabul at Nowshera
Figure 2: Deviation score of hydrologic signature for each ML algorithm for four catchments: Indus at Tarbela, Jhelum at Mangla, Chenab at Marala, and Kabul at Nowshera
Figure 3: Probability density functions of Streamflow at four catchments: Indus at Tarbela, Jhelum at Mangla, Chenab at Marala, and Kabul at Nowshera
Citation: https://doi.org/10.5194/hess-2022-213-AC1
-
AC1: 'Reply on RC1', Haris Mushtaq, 08 Feb 2023
-
RC2: 'Comment on hess-2022-213', Elena Toth, 16 Dec 2022
Dear Authors,
I do apologise for the delay in closing the discussion: unfortunately not all the referees who accepted the review uploaded their comments, despite several reminders, and I will therefore provide a comment myself, as Referee as well as Editor (on the other hand, you should have contributed to the discussion replying at least to the comments uploaded by Ref#1 more than 6 weeks ago, since you had all the time…).
I will also send separately to you the annotated manuscript with additional details of my revision.
REFEREE’S COMMENTS
The work presents an analysis of different data-driven methods for predicting streamflow in a set of catchments in the Upper Indus. Such a comparison is not innovative, but the study region is extremely interesting, also given the importance of snow-generated flow and the main novelty is the application of a novel method (SHAP) for understanding the role of the different meteorological predictors.
GENERAL COMMENTS:
-check all the references, both in the text (also amending format, according to journal guidelines) and in the final list (where some cited works are missing)
-revise the English that is still often not clear and editorial review on the English is needed
MAIN COMMENTS
Abstract: I agree with Ref#1’s comment that you should better explain and justify why ML approaches should be particularly suitable for data-scarce catchments and why you think that “Given the catchment characteristics, there is an utmost need to develop machine learning models that are hydrologically robust”. To do so, on the first point (and first line of the abstract), I would perhaps suggest replace “Data-scarce” with a phrase explaining that there is a recent abundance of meteorological data, but little information on the catchment properties and on the characterisation of detailed hydrological processes.
The Introduction should be re-organised to better present and separate the two main issues of your work, i.e. i) ML methods and their potential also for identifying the influence of the inputs and ii) the challenges in snow-driven areas like the Indus upper catchments
In particular:
ll 45-52 and 53-64 should go together with ll 24-35
ll 53-64 may be removed (not needed: you may limit the state-of-the-art to the models you use and the applications in your region (already included in the following)
ll 35-39 should go with together with ll. 19-24 (and ll. 19-24 that should be rephrased/shortened or even removed, since there is no need to highlight, in a general way. the importance of streamflow prediction, that is obvious and certainly not limited to the last decades…)
Study area: clarify which is the closure sections of each catchment (adding also the drainage area) and denote them always with the same name (not mixing name of river and name of section)
Data sets: add more information on the choice and justification in reference to previous studies in the region
Predictors: the main features are too many (5 multiplied by the number of elevation zones) and they are certainly strongly correlated (for the same feature between elevation zones, but also between PET and temperature for each zone): why not performing a Principal Component analysis (or other, possibly non-linear, methods), in order to get a smaller set of predictors, to simplify also the interpretation of the results?
Methodologies: the description of the SHAP method (see Eq. 1), that is the core of the work, is far from clear: try to explain better, possibly with the help of figures/flowchart?
Results: clarify over which period (training/calibration or testing/validation?) the goodness-of-fit indexes and the difference in observed/modelled signatures are calculated (text Section 5.1+ Table 4 + Figure 6)
Move section 5.3 together with 5.1 (and in 5.1 elaborate more on the results and comment on differences between the models in capturing signatures)
Section 5.2 is far from clear and needs to be thoroughly revised, starting from clarifying the meaning of the figures you present: first explain and discuss the Feature Importance Order you obtained for each basin and for each model. Then explain the meaning of the shape, colors and position on the plot. Finally present the interpretation for each basin (or couple of basin) in separate subsections. This section is the core of the paper, representing the novelty of your research and as it is in not understandable, so it is impossible to assess the value of the work.
Citation: https://doi.org/10.5194/hess-2022-213-RC2 -
AC2: 'Reply on RC2', Haris Mushtaq, 09 Feb 2023
We apologize for the late response to the comments. Due to unforeseen circumstances, we were not able to submit our response in time. We would also like to thank the referee for valuable comments and suggestions that will help us improve the quality of the manuscript.
Referee Comments
The work presents an analysis of different data-driven methods for predicting streamflow in a set of catchments in the Upper Indus. Such a comparison is not innovative, but the study region is extremely interesting, also given the importance of snow-generated flow and the main novelty is the application of a novel method (SHAP) for understanding the role of the different meteorological predictors.
General Comments
-check all the references, both in the text (also amending format, according to journal guidelines) and in the final list (where some cited works are missing)
-revise the English that is still often not clear and editorial review on the English is needed
Response: We will review the manuscript thoroughly to revise reference format and the final list. We will also put in an effort to improve the readability of the manuscript.
Main Comments
Abstract: I agree with Ref#1’s comment that you should better explain and justify why ML approaches should be particularly suitable for data-scarce catchments and why you think that “Given the catchment characteristics, there is an utmost need to develop machine learning models that are hydrologically robust”. To do so, on the first point (and first line of the abstract), I would perhaps suggest replace “Data-scarce” with a phrase explaining that there is a recent abundance of meteorological data, but little information on the catchment properties and on the characterisation of detailed hydrological processes.
Response: We agree with the referee's comments and as suggested by referee#1 we understand that the use of ‘utmost’ was not appropriate. Moreover, will add the following:
“Very limited research is available on these catchments, related to understanding and modeling hydrologic response to catchment characteristics. Machine Learning models could, thus, be very useful in modeling hydrologic response via a data-driven approach.”
The Introduction should be re-organised to better present and separate the two main issues of your work, i.e. i) ML methods and their potential also for identifying the influence of the inputs and ii) the challenges in snow-driven areas like the Indus upper catchments
In particular:
ll 45-52 and 53-64 should go together with ll 24-35
Response: We will make the required changes accordingly in revised manuscript
ll 53-64 may be removed (not needed: you may limit the state-of-the-art to the models you use and the applications in your region (already included in the following)
Response: We will make the required changes accordingly in the revised manuscript
ll 35-39 should go with together with ll. 19-24 (and ll. 19-24 that should be rephrased/shortened or even removed, since there is no need to highlight, in a general way. the importance of streamflow prediction, that is obvious and certainly not limited to the last decades…)
Response: We will make the required changes accordingly in the revised manuscript
Study Area: clarify which is the closure sections of each catchment (adding also the drainage area) and denote them always with the same name (not mixing name of river and name of section)
Following are the closure sections and respective drainage areas in Square kilometers of each catchment:
Kabul: Nowshera (91,297 Square kilometers)
Jhelum: Mangla Reservoir (33,531.31 Square kilometers)
Chenab: Marala Barage(24,247.93 Square kilometers)
Indus: Tarbela Reservoir (175,362 Square kilometers)
We will also incorporate this information in the revised manuscript.
Datasets add more information on the choice and justification in reference to previous studies in the region
Response: APHRODITE dataset for precipitation has been used in various studies (Lutz et al. 2014, Rao et al. 2022, Faiz et al. 2020, Kundeti et al. 2020) over the Upper Indus basin. Moreover, Baudouin et al. (2020) identified that the APHRODITE is the finest observational dataset over the Indus basin in terms of daily and monthly variability.
CPC temperature: We used the CPC daily maximum and minimum temperature data for the analysis mainly because of the lack of multiple daily temperature observations for the reference comparison. This dataset has been used in various studies for Upper Indus catchments (Rao et al. 2022, Kundeti et al. 2020).
GLDAS: GLDAS Snow water equivalent data has been used in the number of studies for Upper Indus catchments (Charles et al. 2018, Hussain et al. 2021, Ougahi et al. 2022).
We will include these explanations in the revised manuscript as well.
Predictors: the main features are too many (5 multiplied by the number of elevation zones) and they are certainly strongly correlated (for the same feature between elevation zones, but also between PET and temperature for each zone): why not performing a Principal Component analysis (or other, possibly non-linear, methods), in order to get a smaller set of predictors, to simplify also the interpretation of the results?
Response: Thank you for your suggestion. We observed that removing PET from the features list has not made any difference in our results, since PET is derived from temperature. Moreover, we have also reduced the elevation zones to half. So the number of features has been significantly reduced while maintaining satisfactory results. Following tables present updated elevation zones, the number of features for each catchment, and results obtained with the reduced number of features:
Catchments
Number of elevation zones
Number of features
Indus
4
16
Jhelum
5
20
Chenab
4
16
Kabul
4
16
Model
Chenab(NSE/R2)
Indus(NSE/R2)
Kabul(NSE/R2)
XGBOOST
0.83/0.85
0.89/0.89
0.59/0.65
RandomForest
0.83/0.77
0.91/0.90
0.70/0.68
CART
0.77/0.66
0.89/0.89
0.63/0.72
We will incorporate the updated results in the revised manuscript.
Methodologies:the description of the SHAP method (see Eq. 1), that is the core of the work, is far from clear: try to explain better, possibly with the help of figures/flowchart?
Response: We thank the referee for this comment and agree that this section needs more explanation as the core work of the study.
We will include the following information :
There are two type of explanations:
Local explanations: Local explanation is presented using SHAP force plot in Figure 1. This plot presents SHAP value at any single instance of our streamflow data. The plot shows how each feature contributes to push the model output from the baseline prediction (without influence of features) to the actual prediction in bold text. Features pushing the prediction higher are shown in red, while those pushing the prediction lower are shown in blue.
Global Explanations: Figure 2 presents global Explanations. These explanations combine many local explanations, we can represent global structure(summary plots) while retaining local faithfulness to the original model. This allows for analyzing which meta-information has the highest global influence at each point. We will include these explanations and try to further expand on them in the revised manuscript.
Results: clarify over which period (training/calibration or testing/validation?) the goodness-of-fit indexes and the difference in observed/modelled signatures are calculated (text Section 5.1+ Table 4 + Figure 6) Move section 5.3 together with 5.1 (and in 5.1 elaborate more on the results and comment on differences between the models in capturing signatures)
Response: For text Section 5.1+ Table 4 + Figure 6 validation/testing results were presented. Validation/testing of the models was performed from 2005-2014. We will also clarify the validation period in the respective sections/figures/tables. We will also move Section 5.3 with Section 5.1 and elaborate results.
Section 5.2: is far from clear and needs to be thoroughly revised, starting from clarifying the meaning of the figures you present: first explain and discuss the Feature Importance Order you obtained for each basin and for each model. Then explain the meaning of the shape, colors and position on the plot. Fin,ally present the interpretation for each basin (or couple of basin) in separate subsections. This section is the core of the paper, representing the novelty of your research and as it is in not understandable, so it is impossible to assess the value of the work.
Response: We thank the referee for the suggestion to significantly revise section 5.2. Results will also be updated based on the updated number of elevation zones and features. Hence, we will incorporate the updated results and explanations as per your suggested flow.
References:
Baudouin, J.P., Herzog, M. and Petrie, C.A., 2020. Cross-validating precipitation datasets in the Indus River basin. Hydrology and Earth System Sciences, 24(1), pp.427-450.
Charles, S.P., Wang, Q.J., Ahmad, M.U.D., Hashmi, D., Schepen, A., Podger, G. and Robertson, D.E., 2018. Seasonal streamflow forecasting in the upper Indus Basin of Pakistan: an assessment of methods. Hydrology and Earth System Sciences, 22(6), pp.3533-3549.
Faiz, M.A., Liu, D., Tahir, A.A., Li, H., Fu, Q., Adnan, M., Zhang, L. and Naz, F., 2020. Comprehensive evaluation of 0.25° precipitation datasets combined with MOD10A2 snow cover data in the ice-dominated river basins of Pakistan. Atmospheric Research, 231, p.104653.
Hussain, D., Khan, A.A., Hassan, S.N.U., Naqvi, S.A.A. and Jamil, A., 2021. A time series assessment of terrestrial water storage and its relationship with hydro-meteorological factors in Gilgit-Baltistan region using GRACE observation and GLDAS-Noah model. SN Applied Sciences, 3(5), pp.1-11.
Koteswara Rao, K., Lakshmi Kumar, T.V., Kulkarni, A., Chowdary, J.S. and Desamsetti, S., 2022. Characteristic changes in climate projections over Indus Basin using the bias corrected CMIP6 simulations. Climate Dynamics, 58(11), pp.3471-3495.
Kundeti, K., TV, L.K., Kulkarni, A., Chowdary, J.S. and Desamsetti, S., 2021. Climate change projections over Indus basin using CMIP6 model simulations.
Lutz, A.F., Immerzeel, W. and Kraaijenbrink, P., 2014. Gridded meteorological datasets and hydrological modelling in the Upper Indus Basin. Final Report, for International Centre for Integrated Mountain Development (ICIMOD), FutureWater, Costerweg, 1, p.6702.
Lundberg, S.M., Erion, G., Chen, H., DeGrave, A., Prutkin, J.M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N. and Lee, S.I., 2020. From local explanations to global understanding with explainable AI for trees. Nature machine intelligence, 2(1), pp.56-67.
Ougahi, J.H., Cutler, M.E. and Cook, S.J., 2022. Assessing the Karakoram Anomaly from long-term trends in earth observation and climate data. Remote Sensing Applications: Society and Environment, 28, p.100852.
Figure 1: SHAP Force plot shows exactly which features had the most influence on the model's prediction for a single observation.
Figure 2: By computing several local explanations SHAP enables us to understand the global model structure(summary plots) (Lundberg et al. 2020)
Citation: https://doi.org/10.5194/hess-2022-213-AC2
-
AC2: 'Reply on RC2', Haris Mushtaq, 09 Feb 2023
Interactive discussion
Status: closed
-
RC1: 'Comment on hess-2022-213', Anonymous Referee #1, 29 Oct 2022
This study applied several machine learning models to streamflow forecast and investigated the model explanation with SHAP. The study is of potential value due to the unique features of hydrological process in Pakistan basins. However, the writing requires careful revision. And the contribution of this study should be further strengthened. I have the following comments for the authors’ consideration.
Line 4-5ã“Given the catchment characteristics, there is an utmost need to develop machine learning models that are hydrologically robust”ï¼ The logic is not convincing. Why there is an utmost need to develop machine learning models under the described catchment characteristics?
Line 5-8 Please split it into two short sentences.
Line 12 “We strongly believe….”. In my personal opinion, it’s better to remove emotional word.
Line 43 “Data-driven”. The initial should be lower-case.
Line 68 “These streamflows are highly seasonal, variant, and prone to extreme events and climate change”. I agree with the authors. The catchments in Pakistan have some special features. Development of machine learning for these catchments provides good chance to explore the potential of machine learning technique. It’s better if the authors could expand the challenges of machine learning on modeling the streamflow in these catchments. In other words, does the machine learning have the potential to forecast streamflow which may be highly seasonal, variant, and prone to extreme events and climate? It is an interesting point to attract the readers and an important contribution for hydrological modeling.
Line 69 Please correct the citation formatLine 87-117 The authors summarized the existing studies of several machine learning models. It will be better if the features (advantages and disadvantages) of these models could be briefly summarized.
Line 118 It is weird to set a “1.1 Hydrologic interpretation” in the Introduction section. I suggest to remove the subtitle.
Line 139 There is typo
Line 131-137 These two paragraphs can be merged.
Line 177-185 Please merge these paragraphs.
Line 187 Please reorganize this sentence.Line 363-366 Please reorganize this long sentence.
Figure 6 Please reorganize this figure to make it more formal.
Figure 7-9 Is it possible to specifically display the comparison under extreme events and climate (or other unique conditions in Pakistan) during 2005-2014? The long-term comparison in one figure may cover some interesting streamflow patterns, which however can be the new finding of this study.
Citation: https://doi.org/10.5194/hess-2022-213-RC1 -
AC1: 'Reply on RC1', Haris Mushtaq, 08 Feb 2023
We sincerely thank the referee for insightful comments that should surely help us in improving the quality of our manuscript
Line 4-5　“Given the catchment characteristics, there is an utmost need to develop machine learning models that are hydrologically robust”． The logic is not convincing. Why there is an utmost need to develop machine learning models under the described catchment characteristics?
Response: We agree with reviewer suggestion. We understand the use of word ‘Utmost’ was not appropriate here.
We have replaced this with the following sentence and will revise the manuscript accordingly. “As ground data is not available, we use remote sensing data, which are prone to errors and hence become source of uncertainty in hydrological models. This makes ML algorithms relevant.”
Line 5-8 Please split it into two short sentences.
Response: We will make the required change accordingly
Line 12 “We strongly believe….”. In my personal opinion, it’s better to remove emotional word.
Response: We have changed the sentence to "We expect that the findings of this study will promote the use of SHAP analysis for streamflow forecasting in data-scarce and high-elevation catchments in Pakistan."
Line 43 “Data-driven”. The initial should be lower-case.
Response: We will do the correction in revised manuscript.
Line 68 “These streamflows are highly seasonal, variant, and prone to extreme events and climate change”. I agree with the authors. The catchments in Pakistan have some special features. Development of machine learning for these catchments provides good chance to explore the potential of machine learning technique. It’s better if the authors could expand the challenges of machine learning on modeling the streamflow in these catchments. In other words, does the machine learning have the potential to forecast streamflow which may be highly seasonal, variant, and prone to extreme events and climate? It is an interesting point to attract the readers and an important contribution for hydrological modeling.
Response: We will explain the challenges of machine learning in modeling the streamflow in these catchments. The models were developed on 10-daily time scale as planning and operations in Pakistan are based on 10-daily time scale. Thus, the objective was not to predict extreme events, rather to establish ML models that are suitable for data-scarce catchments and catchments with mix of rainfall and snowmelt contributions to the runoff. We have, however, included seasonal analysis attached in Figure 1. Figure 1 compares the yearly seasonal volume for Kharif(April- September) and Rabi (October- March) between the Actual and simulated/predicted results. The results are displayed for XGBOOST, and it is evident that the model has captured seasonal trends well, apart from the year 2010, which was a year of super floods in Pakistan. We will include seasonal analysis in the revised manuscript
Line 69 Please correct the citation format
Response: We will correct the format in the revised manuscript
Line 87-117 The authors summarized the existing studies of several machine learning models. It will be better if the features (advantages and disadvantages) of these models could be briefly summarized.
Response: We will add a brief description of the features of ML models in the revised manuscript.
Line 118 It is weird to set a “1.1 Hydrologic interpretation” in the Introduction section. I suggest to remove the subtitle.
Response: We will make the required changes accordingly in the revised manuscript
Line 139 There is typo
Response: We will correct this in the revised manuscript
Line 131-137 These two paragraphs can be merged.
Response: We will make the required change accordingly
Line 177-185 Please merge these paragraphs.
Response: We will make the required change accordingly
Line 187 Please reorganize this sentence.
Response: We have reorganized the sentence to the following: Figure 5 presents the methodology of the study, and discussed in forthcoming sections.
Line 363-366 Please reorganize this long sentence.
Response: We have split this sentence into two sentences as follows and will do the change in the revised manuscript:
“Moreover, SHAP analysis allows reason-based predictions and thus analyse the learned representations. Hence, SHAP analysis not only increases the transparency of machine learning, but it can also eventually promote the application of machine learning methods in hydrological studies.”
Figure 6 Please reorganize this figure to make it more formal.
Response: We have reorganized the figure in more presentable and formal way now attached as Figure 2.
Figure 7-9 Is it possible to specifically display the comparison under extreme events and climate (or other unique conditions in Pakistan) during 2005-2014? The long-term comparison in one figure may cover some interesting streamflow patterns, which however can be the new finding of this study.
Response: We thank the referee for this comment and agree that this would be a valuable addition to our work. We have prepared river discharge density plots. Plots for XGBoost model are attached in Figure 3. We will include these results in the revised manuscript. It is evident that ML does not capture extreme flows, however, this may not be critical as 10-daily models are mostly used for operations and planning purposes and do not need to account for extreme flows.
Figure 1: Seasonal volume comparison of simulated and actual streamflow for four catchments of XGBOOST: Indus at Tarbela, Jhelum at Mangla, Chenab at Marala, and Kabul at Nowshera
Figure 2: Deviation score of hydrologic signature for each ML algorithm for four catchments: Indus at Tarbela, Jhelum at Mangla, Chenab at Marala, and Kabul at Nowshera
Figure 3: Probability density functions of Streamflow at four catchments: Indus at Tarbela, Jhelum at Mangla, Chenab at Marala, and Kabul at Nowshera
Citation: https://doi.org/10.5194/hess-2022-213-AC1
-
AC1: 'Reply on RC1', Haris Mushtaq, 08 Feb 2023
-
RC2: 'Comment on hess-2022-213', Elena Toth, 16 Dec 2022
Dear Authors,
I do apologise for the delay in closing the discussion: unfortunately not all the referees who accepted the review uploaded their comments, despite several reminders, and I will therefore provide a comment myself, as Referee as well as Editor (on the other hand, you should have contributed to the discussion replying at least to the comments uploaded by Ref#1 more than 6 weeks ago, since you had all the time…).
I will also send separately to you the annotated manuscript with additional details of my revision.
REFEREE’S COMMENTS
The work presents an analysis of different data-driven methods for predicting streamflow in a set of catchments in the Upper Indus. Such a comparison is not innovative, but the study region is extremely interesting, also given the importance of snow-generated flow and the main novelty is the application of a novel method (SHAP) for understanding the role of the different meteorological predictors.
GENERAL COMMENTS:
-check all the references, both in the text (also amending format, according to journal guidelines) and in the final list (where some cited works are missing)
-revise the English that is still often not clear and editorial review on the English is needed
MAIN COMMENTS
Abstract: I agree with Ref#1’s comment that you should better explain and justify why ML approaches should be particularly suitable for data-scarce catchments and why you think that “Given the catchment characteristics, there is an utmost need to develop machine learning models that are hydrologically robust”. To do so, on the first point (and first line of the abstract), I would perhaps suggest replace “Data-scarce” with a phrase explaining that there is a recent abundance of meteorological data, but little information on the catchment properties and on the characterisation of detailed hydrological processes.
The Introduction should be re-organised to better present and separate the two main issues of your work, i.e. i) ML methods and their potential also for identifying the influence of the inputs and ii) the challenges in snow-driven areas like the Indus upper catchments
In particular:
ll 45-52 and 53-64 should go together with ll 24-35
ll 53-64 may be removed (not needed: you may limit the state-of-the-art to the models you use and the applications in your region (already included in the following)
ll 35-39 should go with together with ll. 19-24 (and ll. 19-24 that should be rephrased/shortened or even removed, since there is no need to highlight, in a general way. the importance of streamflow prediction, that is obvious and certainly not limited to the last decades…)
Study area: clarify which is the closure sections of each catchment (adding also the drainage area) and denote them always with the same name (not mixing name of river and name of section)
Data sets: add more information on the choice and justification in reference to previous studies in the region
Predictors: the main features are too many (5 multiplied by the number of elevation zones) and they are certainly strongly correlated (for the same feature between elevation zones, but also between PET and temperature for each zone): why not performing a Principal Component analysis (or other, possibly non-linear, methods), in order to get a smaller set of predictors, to simplify also the interpretation of the results?
Methodologies: the description of the SHAP method (see Eq. 1), that is the core of the work, is far from clear: try to explain better, possibly with the help of figures/flowchart?
Results: clarify over which period (training/calibration or testing/validation?) the goodness-of-fit indexes and the difference in observed/modelled signatures are calculated (text Section 5.1+ Table 4 + Figure 6)
Move section 5.3 together with 5.1 (and in 5.1 elaborate more on the results and comment on differences between the models in capturing signatures)
Section 5.2 is far from clear and needs to be thoroughly revised, starting from clarifying the meaning of the figures you present: first explain and discuss the Feature Importance Order you obtained for each basin and for each model. Then explain the meaning of the shape, colors and position on the plot. Finally present the interpretation for each basin (or couple of basin) in separate subsections. This section is the core of the paper, representing the novelty of your research and as it is in not understandable, so it is impossible to assess the value of the work.
Citation: https://doi.org/10.5194/hess-2022-213-RC2 -
AC2: 'Reply on RC2', Haris Mushtaq, 09 Feb 2023
We apologize for the late response to the comments. Due to unforeseen circumstances, we were not able to submit our response in time. We would also like to thank the referee for valuable comments and suggestions that will help us improve the quality of the manuscript.
Referee Comments
The work presents an analysis of different data-driven methods for predicting streamflow in a set of catchments in the Upper Indus. Such a comparison is not innovative, but the study region is extremely interesting, also given the importance of snow-generated flow and the main novelty is the application of a novel method (SHAP) for understanding the role of the different meteorological predictors.
General Comments
-check all the references, both in the text (also amending format, according to journal guidelines) and in the final list (where some cited works are missing)
-revise the English that is still often not clear and editorial review on the English is needed
Response: We will review the manuscript thoroughly to revise reference format and the final list. We will also put in an effort to improve the readability of the manuscript.
Main Comments
Abstract: I agree with Ref#1’s comment that you should better explain and justify why ML approaches should be particularly suitable for data-scarce catchments and why you think that “Given the catchment characteristics, there is an utmost need to develop machine learning models that are hydrologically robust”. To do so, on the first point (and first line of the abstract), I would perhaps suggest replace “Data-scarce” with a phrase explaining that there is a recent abundance of meteorological data, but little information on the catchment properties and on the characterisation of detailed hydrological processes.
Response: We agree with the referee's comments and as suggested by referee#1 we understand that the use of ‘utmost’ was not appropriate. Moreover, will add the following:
“Very limited research is available on these catchments, related to understanding and modeling hydrologic response to catchment characteristics. Machine Learning models could, thus, be very useful in modeling hydrologic response via a data-driven approach.”
The Introduction should be re-organised to better present and separate the two main issues of your work, i.e. i) ML methods and their potential also for identifying the influence of the inputs and ii) the challenges in snow-driven areas like the Indus upper catchments
In particular:
ll 45-52 and 53-64 should go together with ll 24-35
Response: We will make the required changes accordingly in revised manuscript
ll 53-64 may be removed (not needed: you may limit the state-of-the-art to the models you use and the applications in your region (already included in the following)
Response: We will make the required changes accordingly in the revised manuscript
ll 35-39 should go with together with ll. 19-24 (and ll. 19-24 that should be rephrased/shortened or even removed, since there is no need to highlight, in a general way. the importance of streamflow prediction, that is obvious and certainly not limited to the last decades…)
Response: We will make the required changes accordingly in the revised manuscript
Study Area: clarify which is the closure sections of each catchment (adding also the drainage area) and denote them always with the same name (not mixing name of river and name of section)
Following are the closure sections and respective drainage areas in Square kilometers of each catchment:
Kabul: Nowshera (91,297 Square kilometers)
Jhelum: Mangla Reservoir (33,531.31 Square kilometers)
Chenab: Marala Barage(24,247.93 Square kilometers)
Indus: Tarbela Reservoir (175,362 Square kilometers)
We will also incorporate this information in the revised manuscript.
Datasets add more information on the choice and justification in reference to previous studies in the region
Response: APHRODITE dataset for precipitation has been used in various studies (Lutz et al. 2014, Rao et al. 2022, Faiz et al. 2020, Kundeti et al. 2020) over the Upper Indus basin. Moreover, Baudouin et al. (2020) identified that the APHRODITE is the finest observational dataset over the Indus basin in terms of daily and monthly variability.
CPC temperature: We used the CPC daily maximum and minimum temperature data for the analysis mainly because of the lack of multiple daily temperature observations for the reference comparison. This dataset has been used in various studies for Upper Indus catchments (Rao et al. 2022, Kundeti et al. 2020).
GLDAS: GLDAS Snow water equivalent data has been used in the number of studies for Upper Indus catchments (Charles et al. 2018, Hussain et al. 2021, Ougahi et al. 2022).
We will include these explanations in the revised manuscript as well.
Predictors: the main features are too many (5 multiplied by the number of elevation zones) and they are certainly strongly correlated (for the same feature between elevation zones, but also between PET and temperature for each zone): why not performing a Principal Component analysis (or other, possibly non-linear, methods), in order to get a smaller set of predictors, to simplify also the interpretation of the results?
Response: Thank you for your suggestion. We observed that removing PET from the features list has not made any difference in our results, since PET is derived from temperature. Moreover, we have also reduced the elevation zones to half. So the number of features has been significantly reduced while maintaining satisfactory results. Following tables present updated elevation zones, the number of features for each catchment, and results obtained with the reduced number of features:
Catchments
Number of elevation zones
Number of features
Indus
4
16
Jhelum
5
20
Chenab
4
16
Kabul
4
16
Model
Chenab(NSE/R2)
Indus(NSE/R2)
Kabul(NSE/R2)
XGBOOST
0.83/0.85
0.89/0.89
0.59/0.65
RandomForest
0.83/0.77
0.91/0.90
0.70/0.68
CART
0.77/0.66
0.89/0.89
0.63/0.72
We will incorporate the updated results in the revised manuscript.
Methodologies:the description of the SHAP method (see Eq. 1), that is the core of the work, is far from clear: try to explain better, possibly with the help of figures/flowchart?
Response: We thank the referee for this comment and agree that this section needs more explanation as the core work of the study.
We will include the following information :
There are two type of explanations:
Local explanations: Local explanation is presented using SHAP force plot in Figure 1. This plot presents SHAP value at any single instance of our streamflow data. The plot shows how each feature contributes to push the model output from the baseline prediction (without influence of features) to the actual prediction in bold text. Features pushing the prediction higher are shown in red, while those pushing the prediction lower are shown in blue.
Global Explanations: Figure 2 presents global Explanations. These explanations combine many local explanations, we can represent global structure(summary plots) while retaining local faithfulness to the original model. This allows for analyzing which meta-information has the highest global influence at each point. We will include these explanations and try to further expand on them in the revised manuscript.
Results: clarify over which period (training/calibration or testing/validation?) the goodness-of-fit indexes and the difference in observed/modelled signatures are calculated (text Section 5.1+ Table 4 + Figure 6) Move section 5.3 together with 5.1 (and in 5.1 elaborate more on the results and comment on differences between the models in capturing signatures)
Response: For text Section 5.1+ Table 4 + Figure 6 validation/testing results were presented. Validation/testing of the models was performed from 2005-2014. We will also clarify the validation period in the respective sections/figures/tables. We will also move Section 5.3 with Section 5.1 and elaborate results.
Section 5.2: is far from clear and needs to be thoroughly revised, starting from clarifying the meaning of the figures you present: first explain and discuss the Feature Importance Order you obtained for each basin and for each model. Then explain the meaning of the shape, colors and position on the plot. Fin,ally present the interpretation for each basin (or couple of basin) in separate subsections. This section is the core of the paper, representing the novelty of your research and as it is in not understandable, so it is impossible to assess the value of the work.
Response: We thank the referee for the suggestion to significantly revise section 5.2. Results will also be updated based on the updated number of elevation zones and features. Hence, we will incorporate the updated results and explanations as per your suggested flow.
References:
Baudouin, J.P., Herzog, M. and Petrie, C.A., 2020. Cross-validating precipitation datasets in the Indus River basin. Hydrology and Earth System Sciences, 24(1), pp.427-450.
Charles, S.P., Wang, Q.J., Ahmad, M.U.D., Hashmi, D., Schepen, A., Podger, G. and Robertson, D.E., 2018. Seasonal streamflow forecasting in the upper Indus Basin of Pakistan: an assessment of methods. Hydrology and Earth System Sciences, 22(6), pp.3533-3549.
Faiz, M.A., Liu, D., Tahir, A.A., Li, H., Fu, Q., Adnan, M., Zhang, L. and Naz, F., 2020. Comprehensive evaluation of 0.25° precipitation datasets combined with MOD10A2 snow cover data in the ice-dominated river basins of Pakistan. Atmospheric Research, 231, p.104653.
Hussain, D., Khan, A.A., Hassan, S.N.U., Naqvi, S.A.A. and Jamil, A., 2021. A time series assessment of terrestrial water storage and its relationship with hydro-meteorological factors in Gilgit-Baltistan region using GRACE observation and GLDAS-Noah model. SN Applied Sciences, 3(5), pp.1-11.
Koteswara Rao, K., Lakshmi Kumar, T.V., Kulkarni, A., Chowdary, J.S. and Desamsetti, S., 2022. Characteristic changes in climate projections over Indus Basin using the bias corrected CMIP6 simulations. Climate Dynamics, 58(11), pp.3471-3495.
Kundeti, K., TV, L.K., Kulkarni, A., Chowdary, J.S. and Desamsetti, S., 2021. Climate change projections over Indus basin using CMIP6 model simulations.
Lutz, A.F., Immerzeel, W. and Kraaijenbrink, P., 2014. Gridded meteorological datasets and hydrological modelling in the Upper Indus Basin. Final Report, for International Centre for Integrated Mountain Development (ICIMOD), FutureWater, Costerweg, 1, p.6702.
Lundberg, S.M., Erion, G., Chen, H., DeGrave, A., Prutkin, J.M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N. and Lee, S.I., 2020. From local explanations to global understanding with explainable AI for trees. Nature machine intelligence, 2(1), pp.56-67.
Ougahi, J.H., Cutler, M.E. and Cook, S.J., 2022. Assessing the Karakoram Anomaly from long-term trends in earth observation and climate data. Remote Sensing Applications: Society and Environment, 28, p.100852.
Figure 1: SHAP Force plot shows exactly which features had the most influence on the model's prediction for a single observation.
Figure 2: By computing several local explanations SHAP enables us to understand the global model structure(summary plots) (Lundberg et al. 2020)
Citation: https://doi.org/10.5194/hess-2022-213-AC2
-
AC2: 'Reply on RC2', Haris Mushtaq, 09 Feb 2023
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 967 | 439 | 28 | 1,434 | 18 | 15 |
- HTML: 967
- PDF: 439
- XML: 28
- Total: 1,434
- BibTeX: 18
- EndNote: 15
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1