1. There is a problem with the meteorological data reference. For the meteorological data, the authors still need to specify the data source. The resolution for some variables seems coarse. Authors can use the following documentation to get more details:

https://power.larc.nasa.gov/docs/methodology/data/sources/

Response:

Thank you for your suggestions. We have provided the following detailed information about the data source and emphasize the limitations of the variable resolution in the manuscript.

Data Source: Specifically, the meteorological data applied in this work is sourced from the NASA POWER project (https://power.larc.nasa.gov/), which provides a wide range of meteorological data, including temperature, precipitation, solar radiation, and more.

Data Resolution Issue: The horizontal resolution of the primary solar data source (longwave and shortwave downward irradiance) is a global 1° x 1° latitude/longitude grid while the meteorological data sources (air temperature, relative humidity, wind speed, and precipitation) are ½° x ⅝° latitude/longitude grid. Detailed information can be found at (https://power.larc.nasa.gov/data-access-viewer/). The temporal resolutions of the data sets are daily. The meteorological data are used as an auxiliary component for soil moisture prediction in our work. Therefore, even though the resolution of some variables appears coarse, we can safely disregard the potential influence of resolution on our research findings and conclusions.

1. Relying solely on time series data may not be sufficient. Could other attributes be introduced, such as vegetation and soil information? Such additions might help improve the model's performance.

Response:

Thank you for your suggestions. Static attributes such as vegetation land cover and soil information are indeed necessary for generating transferable model predictions. There are several reasons why we do not consider static properties in our work: 1) Given our research objectives, which aim to understand how different deep learning model structures adapt to spatio-temporal variations in soil moisture data, introducing coarse static attributes will make the question more complicated. The static characteristics of different sites are finally captured by the model within the network parameters through training;

2) Static attributes derived from site information cannot completely describe the local situation, such as root water uptake, preferential flow, etc. An ideal transferable model should encompass both static properties and the incorporation of limited local in-situ data. This represents a highly meaningful area of research that can serve as a primary focus for future studies.

1. For the Transformer model, with time series data, relying solely on positional embeddings might make it difficult to reflect seasonality and long-term sequential information. How did the authors address this issue?

Response:

Thank you for your comments. Because of the nonlocal effect in the Transformer, appropriate positional encoding is essential (Vaswani et al., 2017; Devlin et al., 2019; Shaw et al., 2018). Positional encodings enable the encoding of absolute or relative information in sequential inputs as required. Our work considers that the soil moisture on the fifth day can be entirely determined by the conditions of the first four days, which is reasonable from the perspective of soil moisture dynamics. In this case, the provided absolute positional encoding is adequate.

When addressing seasonality, absolute encoding, such as the sinusoidal positional encodings, is generally acknowledged to have the ability to encode this kind of long-term sequential information (Devlin et al., 2019). Additionally, the incorporation of time-dependent variables into the model can further enhance its capacity to capture seasonality and long-term dependencies.

1. The authors conducted a SHAP analysis. Under what conditions does the model perform well? Can feature analysis determine which features enhance the model's performance? How does it differ from the feature importance of Random Forest? Do both have similar conclusions?

Response:

Thank you for your comments. Incorporating the model performances and SHAP analysis results, we believe that the SHAP analysis of a well-performing model should be as follows:

1) Strongly reflecting on important characteristics, such as LSTM and CNN, leads to better results in surface soil moisture or short-term prediction.

2) Avoiding overlearning unimportant features ensures that established correlations remain sufficiently accurate. False correlations tend to complement each other, leading to the learning of incorrect rules and resulting in poor performance for long-term forecasts.

Because input features have varying effects on soil water prediction results across different scenarios (sites, depths), there is no standardized format for SHAP analysis. However, the SHAP analysis can help us discover the characteristics of various basic and hybrid neural networks in processing soil moisture data.

Generally, the results of feature importance ranking based on SHAP analysis are consistent with that of feature importance analysis of Random Forest. SHAP calculates the feature importance rooted in the concept of Shapley value (in Appendix B), while Random Forest determines feature importance by evaluating the reduction in impurity when each feature is split in the tree. Both of them can reflect the feature importance.

However, sometimes the feature importance ranking results may be different. SHAP provides a more detailed and comprehensive calculation of the influence of each feature on the prediction result, with high computational costs. In contrast, Random Forest's feature importance estimation is a more general measure. In addition, SHAP technology can also detect the interaction between features, thus providing a more comprehensive and refined feature importance ranking result, as depicted in Figure R2. Each point shows the Shapley value (impact on the results) of a specific feature in a sample, with the color indicating the value of the input feature. In Figure R3, the feature importance rankings perform consistently on significant features but exhibit slight differences on less significant factors, indicating that both feature analysis methods are credible.

The selection of the two methods depends on the research requirements. If the emphasis is on the relative importance of input features, feature importance from Random Forest is sufficient. However, if a detailed explanation of individual forecasts is required, SHAP analysis may be more suitable. Generally, a combined utilization of the two methods can be employed to achieve a more comprehensive model understanding. In our work, SHAP analysis is more suitable because of the interaction between the input features and the need for detailed insights.

We have made clearer expressions about the SHAP analysis in the revised manuscript.

Figure R2. SHAP value analysis for Random Forest at the Monahans site at 0.0500m

Figure R3. Mean |SHAP value| bar analysis (a) and feature importance analysis (b) for Random Forest at the Monahans site at 0.0500m

1. (~Line 595) In Figure 13, what do the x and y axes represent? Does the input data consist solely of soil moisture or does it also include other time series data? What does the distribution shape of points in the model signify? Can the authors infer model performance or the most critical variables from the results? How do the authors explain the phenomena in the figure? What caused these phenomena?

Response:

Thank you for your comments. We have added more details about the t-SNE visualizations in the manuscript.

1) When conducting t-SNE visualizations, the x and y axes make no sense, as depicted in Figure R4 of Van der Maaten and Hinton’s work (2008). Only the relative distance between sample points matters.  

2) The input data in Figure 13(a) from the manuscript denotes the flattened form of the four days’ inputs I, {x_t-3, x_t-2, x_t-1, x_t}, where x_t ={P_t, T_t, LW_t, SW_t, RH_t, WS_t, ST_t, SM_t}, mentioned in Line 128.

3) The closer the points are to each other, the higher the similarity is. The color of each sample point corresponds to the soil moisture content value (Line 605). Based on this, the distribution of points should be analyzed from two aspects: the regularity of color distribution from light to dark; and the specific shape generated through models. The regularity of the colored plots exhibits two forms: linearly separable or curved. We infer that the linearly separable pattern holds greater utility in this soil moisture regression task. The specific shape reveals the distinct encoding strategies of models, and it appears that the points by more complex models tend to cluster. However, it appears that this clustering is not the primary determinant of accuracy in regression tasks. We could observe that models with specific shapes in t-SNE visualization do not perform the superior accuracy, such as Figure 13(e) (g) (j).

4) For a soil moisture prediction regression task, we infer that in t-SNE visualizations of models with great forecasting capacity, the sample points can be arranged vertically from light to dark in color, such as the Figure 13(h). Additionally, these visualizations enable us to discern the impact of the attention mechanism and adversarial training on LSTM in Figure 13(k)(h), ultimately leading to enhanced accuracy.

5) Generally, from the t-SNE visualizations, it can be summarized that different deep learning models capture distinct intrinsic characteristics of input data and encode them into various vectors for making predictions. We analyze the models from the perspective of soil moisture prediction tasks, and we believe that this phenomenon is likely associated with the complexity and data utilization characteristics. Further validation through mathematical research will be necessary to substantiate this hypothesis in the future.

Figure R4 The t-SNE Visualizations of 6,000 handwritten digits from the MNIST data set from Van der Maaten and Hinton (2008)

1. (~Line 305) "To enhance the accuracy of deep learning models and address the issue of lack of interpretability, attention mechanisms have been incorporated into LSTM models to weigh the importance of different." Can the authors specify how the Attention mechanism addresses this issue?

Response:

Thank you for your comments. When applying an attention mechanism into the LSTM model at feature (FA-LSTM) or time dimensions (TA-LSTM), it will generate attention weights for the input features  or the temporal hidden states  through training, as illustrated in Figure R5. These attention weights enable the model to assign importance to various elements within the input sequence, allowing it to select the most crucial factors for making more accurate predictions.

Additionally, these attention weights offer a visualized representation, as demonstrated in Figure 10 of the manuscript, which provides insights into the sections of the input sequence most essential for a specific prediction. The feature importance depicted in Figure 10 represents the attention weights generated through the self-learning of the attention mechanism. Higher feature importance values signify that the corresponding features are more significant within the model. Moreover, in our work, the feature importance in Figure 10 shows a reasonable adaptation to the varying depth, demonstrating the effective feature selection capability of attention mechanisms (Line 515). This visualization facilitates a deeper understanding of how the model learns and utilizes data, contributing to a more comprehensive comprehension of the deep learning model's decision-making process.

Figure R5 Structures of feature attention mechanisms and temporal attention mechanisms.

Code problems:

1. There is an issue with the code. It can not run directly and requires reader debugging.

We feel very sorry for the code problems. We have reorganized and uploaded the codes for readers to use. (https://doi.org/10.5281/zenodo.10060492)

1. Why did the authors choose only about five years of data for training and testing?

Thank you for your comments. The duration for training and testing the model is determined based on the available data from ISMN. When collecting data, the data quality is our focus, so the length of the dataset may vary for each site. As we mentioned in Line 388 in the manuscript, the collected data in Section 2 is divided into training, validation, and test sets in a 6:2:2 ratio in time order. This allocation process is implemented in our code at Lines 71-72.

1. It seems that the code didn't utilize GPU. With only a CPU, every 100 epochs take about 20 seconds. With GPU acceleration, computation cost comparison holds more significance.

Thank you for your suggestions. The utilization of GPUs indeed improves the training speed. We have clarified in the revised manuscript that the computational costs are evaluated using CPUs. In the newly uploaded codes, we have provided a GPU version. Besides, we conducted a comparison of the training speed between CPU and GPU, as illustrated in Figure R6. It is evident that almost all models benefit from significantly improved training speed when utilizing a GPU. Notably, the transformer model in our study also exhibits fast training performance when running on a CPU.

Figure R6 Comparisons of training time for every 100 epochs of ten models with CPU and GPU.

1. During testing, the authors set the batch_size to 1, which is not efficient for large datasets.

Thank you for your comments. As previously stated in Line 388, the collected data is split into training, validation, and test sets following a 6:2:2 ratio in time order. For each result, all the data in the test set (20% of the collected data) are evaluated. The test batch_num is 1, but the batch_size encompasses all the available data in the test set.

1. The code seems to use four days of data to forecast the next day, but the paper needs to explicitly mention whether the authors use 4 days of historical data to forecast the next 1, 3, and 7 days.

Thank you for your suggestions. In the code, we conducted soil moisture predictions iteratively. After completing the prediction for the first day, we utilize the generated soil moisture data for the first day, along with the corresponding observed meteorological data on that day, combined with historical three-day data. This reconstructed the new four-day input, which is used to predict soil water for the second day, and this process is repeated to complete predictions for 1, 3, and 7 days, as stated in Line 382. We have made clearer descriptions in the revised manuscript.

1. Each forecasting step overlaps when forecasting for 3 or 7 days. How did the authors handle these overlapping data to calculate metrics?

Thank you for your comments. When calculating metrics for 3 or 7 days, we compute them by comparing all the predictions with the ground truth. This is implemented in our code at Lines 223-225.

1. After normalizing the data, did the authors denormalize it when calculating metrics?

Thank you for your comments. When calculating metrics, it is necessary to denormalize the data. In our code, this denormalization is performed at the final stage at Lines 215-216. It's worth noting that the unit for soil moisture, used in the calculation of metrics, is expressed in percentage (%), according to the work of Gill et al. (2006).

Reference:

Abbaszadeh, P., Moradkhani, H., and Zhan, X.: Downscaling SMAP radiometer soil moisture over the CONUS using an ensemble learning method, Water Resour. Res., 55, 324–344, 2019.

Carranza, C., Nolet, C., Pezij, M., and van der Ploeg, M.: Root zone soil moisture estimation with Random Forest, J. Hydrol., 593, 125840, https://doi.org/10.1016/j.jhydrol.2020.125840, 2021.

Datta, P. and Faroughi, S. A.: A multihead LSTM technique for prognostic prediction of soil moisture, Geoderma, 433, 116452, https://doi.org/10.1016/j.geoderma.2023.116452, 2023.

Devlin, J., Chang, M. W., Lee, K., and Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding, NAACL HLT 2019 - 2019 Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol. - Proc. Conf., 1, 4171–4186, 2019.

Fang, K., Pan, M., and Shen, C.: The Value of SMAP for Long-Term Soil Moisture Estimation with the Help of Deep Learning, IEEE Trans. Geosci. Remote Sens., 57, 2221–2233, https://doi.org/10.1109/TGRS.2018.2872131, 2019.

Feng, D., Beck, H., de Bruijn, J., Sahu, R. K., Satoh, Y., Wada, Y., Liu, J., Pan, M., Lawson, K., and Shen, C.: Deep Dive into Global Hydrologic Simulations: Harnessing the Power of Deep Learning and Physics-informed Differentiable Models (δHBV-globe1. 0-hydroDL), Geosci. Model Dev. Discuss., 2023, 1–23, 2023.

Gill, M. K., Asefa, T., Kemblowski, M. W., and McKee, M.: Soil moisture prediction using support vector machines, J. Am. Water Resour. Assoc., 42, 1033–1046, https://doi.org/10.1111/j.1752-1688.2006.tb04512.x, 2006.

Jiang, H. and Li, Q.: Approximation theory of transformer networks for sequence modeling, arXiv Prepr. arXiv2305.18475, 2023.

Li, Q., Hao, H., Zhao, Y., Geng, Q., Liu, G., Zhang, Y., and Yu, F.: GANs-LSTM Model for Soil Temperature Estimation from Meteorological: A New Approach, IEEE Access, 8, 59427–59443, https://doi.org/10.1109/ACCESS.2020.2982996, 2020.

Li, Q., Zhu, Y., Shangguan, W., Wang, X., Li, L., and Yu, F.: An attention-aware LSTM model for soil moisture and soil temperature prediction, Geoderma, 409, 1–17, https://doi.org/10.1016/j.geoderma.2021.115651, 2022a.

Li, Q., Li, Z., Shangguan, W., Wang, X., Li, L., and Yu, F.: Improving soil moisture prediction using a novel encoder-decoder model with residual learning, Comput. Electron. Agric., 195, https://doi.org/10.1016/j.compag.2022.106816, 2022b.

Liu, J., Rahmani, F., Lawson, K., and Shen, C.: A multiscale deep learning model for soil moisture integrating satellite and in situ data, Geophys. Res. Lett., 49, e2021GL096847, 2022.

Liu, Y., Mei, L., and Ki, S. O.: Prediction of soil moisture based on Extreme Learning Machine for an apple orchard, CCIS 2014 - Proc. 2014 IEEE 3rd Int. Conf. Cloud Comput. Intell. Syst., 400–404, https://doi.org/10.1109/CCIS.2014.7175768, 2014.

Van der Maaten, L. and Hinton, G.: Visualizing data using t-SNE., J. Mach. Learn. Res., 9, 2008.

Orth, R.: Global soil moisture data derived through machine learning trained with in-situ measurements, Sci. Data, 8, 1–14, 2021.

Orth, R., Weber, U., and Park, S. K.: High-resolution European daily soil moisture derived with machine learning (2003–2020), Sci. Data, 9, 1–13, 2022.

Shaw, P., Uszkoreit, J., and Vaswani, A.: Self-attention with relative position representations, NAACL HLT 2018 - 2018 Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol. - Proc. Conf., 2, 464–468, https://doi.org/10.18653/v1/n18-2074, 2018.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.: Attention is all you need, Adv. Neural Inf. Process. Syst., 30, 2017.

Wang, S., Li, Z., and Li, Q.: Inverse Approximation Theory for Nonlinear Recurrent Neural Networks, arXiv Prepr. arXiv2305.19190, 2023.

Yu, J., Zhang, X., Xu, L., Dong, J., and Zhangzhong, L.: A hybrid CNN-GRU model for predicting soil moisture in maize root zone, Agric. Water Manag., 245, 106649, https://doi.org/10.1016/j.agwat.2020.106649, 2021.