Agricultural production is highly dependent on the weather. The mechanisms of action are complex and interwoven, making it difficult to identify relevant management and adaptation options. The present study uses random forests to investigate such highly non-linear systems for predicting yield anomalies in winter wheat at district levels in Germany. In order to take into account sub-seasonality, monthly features are used that explicitly take soil moisture into account in addition to extreme meteorological events. Clustering is used to show spatially different damage potentials, such as a higher susceptibility to drought damage from May to July in eastern Germany compared to the rest of the country. In addition, relevant heat effects are not detected if the clusters are not sufficiently defined. The variable with the highest importance is soil moisture in March, where higher soil moisture has a detrimental effect on crop yields. In general, soil moisture explains more yield variations than the meteorological variables. The approach has proven to be suitable for explaining historical extreme yield anomalies for years with exceptionally high losses (2003, 2018) and gains (2014) and the spatial distribution of these anomalies. The highest test

Extreme weather conditions have increased over the last 2 decades over Germany, leading to an amplification of inter-annual crop yield variations in the agricultural sector. These include years with above-average wet years (2002, 2007, 2010) but also the droughts of 2003, 2015, and 2018 and the year 2012 with a longer period of bare frost

In previous studies we have tried to approximate this non-linear and complex damage spectrum by considering the sub-seasonal effects of hydro-meteorological variables such as temperature and soil moisture, applying however an econometric linear model neglecting sub-seasonal interaction of the features. This approach was very well able to project long-term mean yield changes but not the inter-annual variations caused by extreme conditions

Disentangling the non-linear spectrum of extreme conditions harmful to plant growth and identifying the causes of yield loss will help improve decision support systems in the agricultural sector. Machine learning focuses primarily on predictive accuracy, while econometricians focus on inference, i.e. deriving statistical properties of estimators for hypothesis testing within a classical parametric and linear approach

The annual yield data for winter wheat are provided by the Federal Statistical Office for the districts from 1999 to 2018

Table of the indicators of seven extreme weather conditions as well as an index for soil moisture for different soil depths. Column 2 shows the corresponding meteorological conditions and months of occurrence. The indicators (first column) are generated by counting the days above or below the thresholds of certain meteorological variables for specific months (second column). The variable names of the resulting features are displayed in the last column. The number indicates the month. For example, Frost10 represents the number of days with black frost in October of the previous year and Heat6 the number of days with heat in June.

The daily temperature and precipitation data are obtained from a network of stations of the German Weather Service

The soil moisture simulation was obtained from the German Drought Monitor

Soil moisture is presented here as an index because an index configuration supports the reduction of systematic errors of data that are simulated as well as spatially processed, such as in the present study

We apply the machine-learning method random forests to explain the variation of winter wheat anomalies by the hydro-meteorological features introduced above. Random forests (RFs) have been used to analyze the effect of meteorological determinants on crop yields on a global scale

The crop yield potential varies regionally in Germany due to differences in climate and soils, among other factors. To take account of these differences, a spatial clustering was implemented to identify different sub-regions within Germany. The clustering methods used are representatives of centroid-based ones, such as k-means (KMEANS,

The random forest algorithm allows us to study the relationships between hydro-meteorological extremes and yield anomalies by assessing the relative importance of the variables and the functional relationship between each predictor and the response variable

Table with the average

Spatial structure of clusters shown in Table

Scatter

To evaluate the cluster algorithm and the number of clusters, the test

The main variables for each sub-cluster and the corresponding average marginal effects are presented below in order to understand the range of adverse effects on yield variation in winter wheat. To generate variable importance and ALE plots, no split is made between test and training data. The non-cluster results are compared with the spatial clusters generated with the PAM clustering algorithm for a cluster size of 4 – PAM (4). The detailed ALE plots for the overall best algorithm cluster size combination, i.e. the HIERARCHICAL cluster algorithm with two clusters considering only the top 25 cm of the soil column, are provided in the Appendix (Fig.

Accumulated local effects plots of the nine most important features for no cluster

The ALE plots in Fig.

Figure

For cluster 1 (Fig.

Previous studies showed that water deficit has no limiting effect on wheat yield in North Rhine-Westphalia

A strong drought signal can only be found in the data if the model is applied to a sub-region such as cluster 2 (Fig.

In general, it is difficult to disentangle the compounding effects of heat and water supply on plant growth

Maps of the observed, the predicted (for PAM clustering with four sub-regions), and the difference between these two for winter wheat yield anomalies for the in-sample years 2003

Figure

Here we show that random forests are very suitable for assessing the non-linear damaging effects of different environmental conditions on winter wheat yield anomalies. Explicit consideration is given to soil moisture at various depths. In addition, the crop yield potential and other spatially related environmental conditions are taken into account, which helps to improve predictive power. Different clustering algorithms and cluster sizes have been applied to improve the predictive capacity of the model from 64 % in average test

We use a spatiotemporal data set that includes 412 districts and 20 years. All districts with less than 12 years of reported yields (green areas) are excluded from the analysis (Fig.

Map showing the number of available winter wheat yield observations for each district used in the analysis for the period 1999–2018. Green districts were removed because 8 or more years of winter wheat data were not reported by regional statistics. Grey areas are districts with no non-irrigated agricultural land.

Correlation plot (Pearson correlation coefficient) of the soil moisture index for the entire root zone (Lall) for all months of the season of winter wheat. The SMI variables of October to December (10–12) refer to the previous year, since winter wheat is usually planted in late autumn and harvested in the summer of the following year.

Figure

Internal validation measures for clusters with different sizes between 2 and 16. The measures depicted are connectivity, Dunn index, and silhouette width.

Here, we use internal validation measures to assess the quality of the clustering, which employ only the data set and the clustering partition for the assessment

Variable importance of the 12 most important features for no cluster

Here, importance is defined as the factor by which the model's mean absolute error (mae), a measure of model performance, changes when the feature is shuffled

The detailed ALE plots for the overall best algorithm cluster size combination, i.e. the HIERARCHICAL cluster algorithm with two clusters considering only the top 25 cm of the soil column, are provided in Fig.

ALE plots for the best combination of cluster algorithm and cluster size (HIERARCHICAL with two clusters) for a soil moisture index configuration that only considers the uppermost 25 cm. For both clusters, the nine ALE plots with the highest feature importance are shown. The importance ranking is established with 50 repetitions. We have chosen a grid size of 50 to estimate the ALE plots, which allows us to reveal the true complexity of the model at the expense of shakiness. Therefore a non-linear smoothing function (LOESS – locally estimated scatterplot smoothing) is added in blue (with the confidence interval in grey). SMI represents the soil moisture index for the uppermost 25 cm of the soil column, PS stands for days without rain in a given month, and Heat stands for days with a maximum temperature of more than 30

ALE plots for the second best combination of cluster algorithm and cluster size (HIERARCHICAL with six clusters) for a soil moisture index configuration that only considers the uppermost 25 cm. For the six clusters, the nine ALE plots with the highest feature importance are shown. The importance ranking is established with 50 repetitions. We have chosen a grid size of 50 to estimate the ALE plots, which allows us to reveal the true complexity of the model at the expense of shakiness. Therefore a non-linear smoothing function (LOESS – locally estimated scatterplot smoothing) is added in blue (with the confidence interval in grey). SMI represents the soil moisture index for the uppermost 25 cm of the soil column, PS stands for days without rain in a given month, Heat stands for days with a maximum temperature of more than 30

ALE plots for the best combination of cluster algorithm and cluster size (HIERARCHICAL with two clusters) for a soil moisture index configuration that considers both the uppermost 25 cm as well as the entire soil column. For both clusters, the nine ALE plots with the highest feature importance are shown. The importance ranking is established with 50 repetitions. We have chosen a grid size of 50 to estimate the ALE plots, which allows us to reveal the true complexity of the model at the expense of shakiness. Therefore a non-linear smoothing function (LOESS – locally estimated scatterplot smoothing) is added in blue (with the confidence interval in grey). SMI represents the soil moisture index for the uppermost 25 cm of the soil column, PS stands for days without rain in a given month, and Heat stands for days with a maximum temperature of more than 30

ALE plots for the second best combination of cluster algorithm and cluster size (PAM with four clusters) for a soil moisture index configuration that considers both the uppermost 25 cm as well as the entire soil column. For the four clusters, the nine ALE plots with the highest feature importance are shown. The importance ranking is established with 50 repetitions. We have chosen a grid size of 50 to estimate the ALE plots, which allows us to reveal the true complexity of the model at the expense of shakiness. Therefore a non-linear smoothing function (LOESS – locally estimated scatterplot smoothing) is added in blue (with the confidence interval in grey). SMI represents the soil moisture index for the uppermost 25 cm of the soil column, PS stands for days without rain in a given month, and Heat stands for days with a maximum temperature of more than 30

ALE plots for the third best combination of cluster algorithm and cluster size (PAM with six clusters) for a soil moisture index configuration that considers both the uppermost 25 cm as well as the entire soil column. For the six clusters, the nine ALE plots with the highest feature importance are shown. The importance ranking is established with 50 repetitions. We have chosen a grid size of 50 to estimate the ALE plots, which allows us to reveal the true complexity of the model at the expense of shakiness. Therefore a non-linear smoothing function (LOESS – locally estimated scatterplot smoothing) is added in blue (with the confidence interval in grey). SMI represents the soil moisture index for the uppermost 25 cm of the soil column, PS stands for days without rain in a given month, and Heat stands for days with a maximum temperature of more than 30

The input data and the script for processing the data and for analysis are available in the following UFZ repository:

AM and ST prepared the historical meteorological data. AM applied the hydro-meteorological simulations. ST was responsible for the spatial processing of the data. MP developed the research idea, prepared the data, and developed the statistical crop model. MP and AM analysed the results. MP composed the text. MP, ST, LS, BH, and AM contributed to interpreting results.

The authors declare that they have no conflict of interest.

Publisher’s note: Copernicus Publications remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the special issue “Understanding compound weather and climate events and related impacts (BG/ESD/HESS/NHESS inter-journal SI)”. It is not associated with a conference.

The yield data are provided by Regional Statistics Germany (

This work was partially supported by funds through the project CLIMALERT (project no. ERA4CS/0005/2016). The article processing charges for this open-access publication were covered by the Helmholtz Centre for Environmental Research – UFZ.

This paper was edited by Bart van den Hurk and reviewed by two anonymous referees.