Soil moisture at the catchment scale exhibits a huge spatial variability. This suggests that even a large amount of observation points would not be able to capture soil moisture variability.

We present a measure to capture the spatial dissimilarity and its change over time. Statistical dispersion among observation points is related to their distance to describe spatial patterns. We analyzed the temporal evolution and emergence of these patterns and used the mean shift clustering algorithm to identify and analyze clusters. We found that soil moisture observations from the 19.4 km

By combining uncertainty propagation with information theory, we were able to calculate the information content of spatial similarity with respect to measurement uncertainty (when are patterns different outside of uncertainty margins?). We were able to prove that the spatial information contained in soil moisture observations is highly redundant (differences in spatial patterns over time are within the error margins). Thus, they can be compressed (all cluster members can be substituted by one representative member) to only a fragment of the original data volume without significant information loss.

Our most interesting finding is that even a few soil moisture time series bear a considerable amount of information about dynamic changes in soil moisture. We argue that distributed soil moisture sampling reflects an organized catchment state, where soil moisture variability is not random. Thus, only a small amount of observation points is necessary to capture soil moisture dynamics.

Although soil water is by far the smallest freshwater stock on earth, it plays a key role in the functioning of terrestrial ecosystems. Soil moisture controls (preferential) infiltration and runoff generation and is a limiting factor for vegetation growth. Plant-available soil water affects the Bowen ratio, i.e., the partitioning of net radiation energy in latent and sensible heat, and last but not least it is an important control for soil respiration and related trace gas emissions. Technologies and experimental strategies to observe soil water dynamics across scales have been at the core of the hydrological research agenda for more than 20 years

Although large spatial variability seems to be a generic feature of soil moisture, there is also evidence that ranks of distributed soil moisture observations are largely stable in time, as observed at the plot

Soil moisture dynamics have been subject to numerous review works

Soil moisture responds to two main forcing regimes, namely rainfall-driven wetting or radiation-driven drying. The related controlling factors and processes differ strongly and operate at different spatial and temporal scales, and the soil moisture pattern reflects thus the multitude of these influences

Landscape heterogeneity is thus a perquisite for temporarily persistent spatial patterns found in a set of soil moisture time series. While most catchments are strongly heterogeneous, it is striking how spatially organized they are

A spatial covariance function describes how linear statistical dependence of observations declines with increasing separating distance up to the distance of statistical independence. This is often expressed as an experimental variogram. Geostatistics relies on several assumptions, such as second-order stationarity (see, e.g.,

However, here we take a different avenue, as we do not intend to interpolate.
One of our goals is to detect dynamic changes in the spatial soil moisture pattern. Following

The core objective of this study is to provide evidence that distributed soil moisture time series provide, despite their strong spatial variability, representative information on soil moisture dynamics. More specifically, we test the following hypotheses.

We test these hypotheses using a distributed soil moisture data set collected in the Colpach catchment in Luxembourg. In Sect.

We base our analyses on the CAOS data set, which was collected in the Attert experimental watershed between 2012 and 2017 and is explained in

Attert experimental catchment in Luxembourg and Belgium. The purple dots show the sensor cluster stations installed during the CAOS project. Here we focus on those cluster stations within the Colpach catchment. Figure adapted after

The experimental design is based on spatially distributed, clustered point measurements within replicated hillslopes. Typical hillslope lengths vary between 400 and 600 m, showing maximum elevations of 50 to 100 m above stream level. For further details on the hillslopes, we refer the reader to Fig. 6a in

Soil moisture data overview. Soil moisture observations in 10 cm

Soil moisture in the 19.4 km

We focus on spatial patterns of soil moisture and how they change over time.
For our analysis the data set was aggregated to mean daily soil moisture values

We tested different window sizes, as we expect that different processes control the emergence of spatial dependence at different temporal scales. The chosen window size was most suitable for detecting seasonal effects.

.To estimate the spatial dependence structure between observations, we relate their pairwise separation distance to a measure of pairwise similarity. Here, we further define the statistical spatial dispersion as a measure of spatial similarity. We compare the empirical distribution of pairwise value differences at different distances. Statistically, a more dispersed empirical distribution is less well described by its mean value. Thus, observations taken at a specific distance are more similar in value if they are less dispersed.

To estimate the dispersion, we use the Cressie–Hawkins estimator

Observation point pairs further apart than 1200 m are most likely located on different hillslopes. These points might share similar soil, topographic and terrain aspect characteristics. Soil moisture dynamics might thus be similar, although they are located at rather large separating distances

. The lag classes are not equidistant, but with a fixedWe analyzed how and whether meaningful spatial dispersion functions emerge and whether those converge into stable configurations. To tackle the hypotheses formulated in the introduction, a clustering is applied to the dispersion functions derived for each window. The clustering algorithm should form groups of functions that are more similar to each other than to members of other clusters. The similarity between two dispersion functions is calculated by the Euclidean vector distance between the dispersion values forming the function. This distance is defined by Eq. (

The next step is to generate a representative dispersion function for each cluster. The straightforward representative function is the cluster centroid (the dispersion function closest to the point of highest cluster member density; see Sect.

We suggest that the number of clusters needed to represent all observed spatial dispersion functions over a calendar year can be used as a measure of spatial organization (fewer clusters needed means a higher degree of organization, because dispersion functions are redundant in time). Additionally, it is insightful to judge the information loss that goes along with this compression, as a high compression with little information loss is understood as a manifestation of spatial and temporal organization of soil moisture dynamics.

In line with

Soil moisture measurements have a considerable measurement uncertainty of 1–3 cm

Next we propagate these uncertainties into the dispersion functions and the distances among those. As we assume the measurement uncertainties to be statistically independent, we use the Gaussian uncertainty propagation to calculate error bands/margins. In a general form, for any function

Equation (

The Shannon entropy

For a discrete frequency distribution of

To ensure comparability, we use one binning for all calculations of

The Kullback–Leibler divergence

We use the Kullback–Leibler divergence to measure and quantify the information loss due to compression. To compress the series of dispersion functions, each cluster member is expressed by its centroid function. Now, we need to calculate the amount of information lost in this process. To calculate the mean information content of the compressed series, each cluster member is substituted by the respective cluster centroid. This substitution is obviously not a compression in a technical sense, but it is necessary to calculate the Kullback–Leibler divergence. Then a frequency distribution for the compressed series

Figure

Spatial dispersion functions in 30 cm for 2016 based on a window size of 30 d.

The spatial dispersion functions take several distinct shapes, with each of these shapes occurring during a certain period in time. More specifically, from Fig.

To provide further insight into the temporal occurrence of cluster members, we colored the soil moisture time series according to the color codes of the identified clusters (Fig.

As the spatial dispersion functions in the presented example are redundant in time, we compressed the information by replacing the dispersion function within one cluster by the cluster centroid. All four representative functions shown in Fig.

In the vegetation period observations are similar even at large separating distances. Interestingly, dispersion functions in the orange cluster start with small values that only gently increase with separating distance. That means soil moisture becomes more homogeneous. Outside of the vegetation period, different spatial patterns can be observed, with increasing dissimilarity with separating distances. The part of the blue cluster overlapping with the vegetation period shows still higher soil moisture values. The transition to the orange cluster sets in as the soil moisture drops (Fig.

Figure

Soil moisture time series of 2016 in all three depths with respective cluster centroids. The three rows show the data from 10 cm

In comparison to the dispersion functions in 30 cm (Fig.

The green clusters emerge with strong rainfall events after longer previous dry spells (Fig.

Observations at 50 cm depth show a clear spatial dependency throughout the whole year. We cannot identify a summer cluster, mean shift yielded two clusters and rainfall forcing does not have a clear influence on their occurrence or transition. The two 50 cm dispersion functions (Fig.

Table

Qualitative description of method success in all years and depths. The results from years other than 2016 and all depths were inspected visually and are summarized here for the sake of completeness. The first three columns identify the year, sensor depth and number of clusters found by mean shift. The remaining three columns state whether specific features existed in the given result.

To further illuminate interannual changes in soil moisture patterns, we present the time series of cluster memberships for the sensors in 30 cm for the entire monitoring period in Fig.

Soil moisture time series of all years in 30 cm depth

Distinct summer recessions in soil moisture are only identified in 2015 and 2016. Evapotranspiration (indicated by the cumulative temperature curves in Fig.

We calculated the Shannon entropy for all soil moisture time series for all years and depths (Table

Information content and information loss due to compression. The information content is given as Shannon entropy

The clusters obtained in 30 cm for the year 2016 (compare Sect.

According to Eq. (

In line with our central hypothesis

In the following we will discuss our main findings that similarity in space leads to dynamic similarity in time, the way we utilized the measurement uncertainty to determine the information content and how two different processes forcing soil moisture dynamics induce two fundamentally different spatial patterns.

We related the dispersion of pairwise point observations to their separating distance. For brevity and due to their shape we called these relationships

We found spatial similarity to persist in time. This is reflected in the temporal stability in cluster membership. In line with

Although cluster memberships occur in temporally continuous blocks in all depths throughout all years, for a few cases we could not relate their emergence to distinct changes in forcing. This implies that

Dispersion functions in 50 cm show a clear spatial dependency throughout the year, with distinct differences within and outside the vegetation period. In 50 cm of 2016 this is different. We find essentially two clusters that do not separate the data series by vegetation period. The shape of the two centroids (Fig.

We related the evaluation of compression quality directly to the measurement uncertainty. This was achieved by Gaussian error propagation of measurement uncertainty into the dispersion functions and their distances. The latter allowed definition of a minimum separable vector distance between two dispersion functions that are different with respect to the error margin. We based the bin width for calculating the Shannon entropy on this minimum distance, because this ensured that the Shannon entropy gives the information content of each cluster

In line with

From Eq. (

We provided an example of how the quality of a compression can be assessed. Instead of considering the number of clusters (compression rate) only, we linked the compression rate to the resulting information loss. We could show that in the majority of the cases substantial compression rates could be achieved, which are accompanied by negligible information losses. We thus suggest that the trade-off between compression rate and information loss should be used as a compression quality measure.

Outside of the vegetation period, we found a recurring picture of spatial dispersion functions with characteristic lengths clearly smaller than the typical extent of hillslopes. Dispersion functions were calculated in three depths for every day throughout 4 years. In most cases there is an characteristic length at which the dispersion function shows a sudden rise in dispersion. For spatial lags smaller than this distance the dispersion is usually very small. Higher lags show much higher and more variable dispersion values. This characteristic length is approx. 500 m. This corresponds to a common hillslope length for the Colpach catchment. During the vegetation period variability at a large separating distance was smoothed out. Dispersion was low also at large distances, suggesting similarity even at distances larger than the typical slope length. We thus conclude that there is dependence of the dispersion on the rainfall pattern, which is reflected in the dispersion function's shape and characteristic length. This confirms

Outside the vegetation period we observed multiple cluster transitions.
Although more than one cluster was identified, the clusters were more similar in shape to each other than to the clusters in the “dry” summer period. In many cases these cluster transitions coincided with a shift in rainfall regimes. Either the first stronger rainfall event after a longer period without rainfall sets in, or one of the heaviest rainfall events of that year occurs. There are also instances with recurring clusters that develop more than once (e.g., Figs.

Many other works also tried to link soil moisture pattern to forcing.

We conclude that cluster transitions were often triggered by rainfall events.
Not all of the strongest rainfall events caused a cluster transition and not every cluster transition could be related to a rainfall sum or frequency within the window of the transition. The characteristics provided in Sect.

We used mean shift mainly as a diagnostic tool to cluster dispersion functions based on their similarity. Similarity is measured by the Euclidean distance between two dispersion function vectors. This Euclidean distance does, however, not provide information on the underlying cause of dissimilarity, and thus a minor difference in the values of the dispersion functions, even though characterized by a very similar shape, could result in the same level of dissimilarity as a change in the shape of the dispersion function. We observed some cluster separations that were caused by minor differences in mean dispersion, while essentially describing the same spatial dependency.

It is possible to train better mean shift algorithm instances. As described in the methods, we selected the bandwidth parameter for mean shift to yield meaningful results for the entire data set. The same parameter was used for all subsets to cluster dispersion functions on the same basis. This makes the clustering procedure itself comparable and, thus, the number of identified clusters can support result interpretation. Nevertheless, it is likely that better bandwidth parameters can be found for each data subset individually and overcome misclassifications as described above. Our objective, however, was to find clustering results that can directly be compared to each other (instead of comparing hyper-parameters).

Dispersion functions operate in a higher-dimensional space and might be affected by the curse of dimensionality. Mean shift clusters data points based on their distance to each other. Following the theory of the curse of dimensionality, with each added dimension (of these points), the difference of maximum and minimum distance between points becomes less significant

Mean shift is sensitive for the bandwidth parameter. As described in the methods (Sect.

Successful clustering does not point out spatial dependency. Mean shift can cluster functions without spatial dependency, as it uses their distance and no actual covariance between the functions. In this case the clustering is based on differences in the mean, which may not even be statistically significant. The mean shift algorithm is not meant to test clusters for statistical independence. Whether two groups of points are separated or not depends only on the bandwidth parameter. Therefore the centroid functions of each cluster have to be checked for their shape and the information on spatial dependency that follows from that shape.

Our approach to find suitable bins to calculate the Shannon entropy is sensitive to outliers. We decided to rather define the width of a bin instead of their number. The reasons and necessity to do so were discussed in detail in Sect.

From the point of view of the monitoring network, it has to be mentioned that the analysis of the 2013 data is likely to be less reliable, as during this period of installation the number of sensors was still lower than in the following years.

Due to the sampling design and the amount of observation points, we did not systematically test for differences of forest vs. pasture plots, but ran our analyses across the two land covers. The fundamentally different shapes of cluster centroids in the summer clusters and, thus, the strong effect of vegetation-altering soil moisture patterns might be partly more pronounced due to the sampling design and not easy transferable to other sites. In our opinion, we would have made the same observations with a more stratified sampling design, as this is systematic catchment behavior, but we can neither confirm nor reject this.

We presented a new method to identify periods of similar spatial dispersion present in a data set. While soil moisture observations might be spatially heterogeneous, spatial patterns are much more persistent in time. We found two fundamentally different states: on the one hand, rainfall-driven cluster formations, usually characterized by strong relationships between dispersion and separating distance and a characteristic length roughly matching the hillslope scale. On the other hand, we found clusters forming during the vegetation period. A drying and then dry soil exhibits dispersion functions which are much flatter, indicating homogeneity across space. Interestingly, these functions flatten out by minimizing the dispersion on large distance lags, which implies that dissimilarities do not increase with separating distance. We can thus see how the soil acts as a low-pass filter.

While these long-lasting periods of similar spatial patterns help us to understand how and when the soil is wetting or dying in an organized manner, there are possible applications beyond this. One could use the identification of clusters to stratify data based on spatial dispersion for combined modeling. Then, for example, a set of spatio-temporal geostatistical models or hydrological models applied to each period separately might in combination return reasonable catchment responses.

Our most interesting finding is that even a few soil moisture time series bear a considerable amount of predictive information about dynamic changes in soil moisture. We argue that distributed soil moisture reflects an organized catchment state, where soil moisture variability is not random and only a small amount of observation points is necessary to capture soil moisture dynamics.

Mean shift starts by forming a cluster for each sample on its own. Here, a

Schematic procedure of the mean shift algorithm in

Mean shift is sensitive to the selected bandwidth. Two clusters whose centroids are within one bandwidth length will be shifted into a combined cluster before convergence is met. As a result a bandwidth parameter chosen too big might classify all samples as a single cluster as indicated in Fig.

Quantitative results summary. For each depth and cluster of 2016 different cluster characteristics were calculated. The duration of each cluster is given in the third column. To compare rainfall forcing with the emergence of clusters, the rainfall characteristics were based on the same moving window as the clusters. The mean rainfall frequency

Mean rolling rainfall sum

Spatial dispersion functions in 30 cm for 2014 based on a window size of 30 d.

Spatial dispersion functions in 30 cm for 2015 based on a window size of 30 d.

Major parts of the analysis are based on the scipy

The methodology was developed by MM, supervised by EZ and discussed with SKH. The data were provided by TB and MW. All the code was developed by MM. The manuscript was written by MM, with contributions by EZ in the introduction, discussion and formulas. SKH supplied the field and data descriptions. The structure, narrative and language of the manuscript were revised and significantly improved by TB.

The authors declare that they have no conflict of interest.

We thank the German Ministerium für Wissenschaft, Forschung und Kunst, Baden-Württemberg, for funding the V-FOR-WaTer project. We thank the German Research Foundation (DFG) for funding of CAOS research unit FOR 1598. We especially thank Britta Kattenstroth and Tobias Vetter, the technicians in charge of the maintenance of the monitoring network. The authors also acknowledge support by the Deutsche Forschungsgemeinschaft and the Open Access Publishing Fund of the Karlsruhe Institute of Technology (KIT).

The article processing charges for this open-access publication were covered by a Research Centre of the Helmholtz Association.

This paper was edited by Alberto Guadagnini and reviewed by two anonymous referees.