The ECMWF's Hydrology Tiled ECMWF Scheme for Surface Exchanges over Land (HTESSEL, Balsamo et al., 2011) land surface model is an integral component of the ECMWF Integrated Forecasts System (IFS) that is used in the different forecast and data assimilation systems, ranging from deterministic 10-day forecasts to the ensemble seasonal forecasts. The surface model is responsible for providing the atmospheric boundary conditions (heat, moisture, and momentum) by simulating the surface water and energy budgets and the temporal evolution of the underlying soil (temperature and moisture), snowpack, and vegetation interception.

The surface energy budget is computed in each grid box independently for different tiles representing different land cover types (e.g. bare ground, high/low vegetation). At each grid cell, only the dominant types of high and low vegetation, respectively, are considered. The surface energy balance is coupled to the underlying soil (or snow) via the skin conductivity, which is currently a single parameter depending on land cover. This is a simplified approach to represent very complex processes such as within-canopy energy exchanges, while it is crucial for the LST computation.

The ECMWF ensemble prediction system (Vitart et al., 2008, 2014) is used daily for global forecasts up to the monthly range and it allows us to characterize the uncertainty in the meteorological forecast expressed by the spread of the ensemble members (51 forecast realizations in the operational configuration). The spread of the ensemble varies with the difficulty of predicting a given meteorological event, due to the complex evolution of the atmospheric flow and the local climate and seasonal conditions, and it is highly valuable information on the likelihood of the forecast being accurate. In this study we use 15-member ensemble forecasts.

The use of HTESSEL in an uncoupled, standalone setting is computationally inexpensive and allows us to perform long-term simulations across the entire European continent. We analyse 50 simulations of HTESSEL with default and perturbed parameters (see Sect. 2.2.3). The simulations are computed from 1983 to 2014, and forced with observed meteorological information such as that used in the computation of the ERA-Interim/Land dataset (Balsamo et al., 2015). The first 6 years are used to spin up the model, and are therefore not considered in the analysis.

Coupled forecasts with the ECMWF's ensemble prediction system were computed to assess the response of weather forecast skills to different (parameter) configurations of the land surface model. We employ 11 sets of global forecasts with default and perturbed land surface model configurations. The analysis of the forecasts focuses on the European domain used for the uncoupled HTESSEL simulations, and on northern hemispheric summer. This allows us to study impacts of the land–atmosphere coupling in Europe as this is strongest at that time, and to exclude confounding effects from snow and ice on our analysis.

Correspondingly, the forecasts are initialized on eight start dates (1 May, 15 May, 1 June, 15 June, 1 July, 15 July, 1 August, and 15 August) during 2001–2010 and computed until 45 days lead time. Even though weather predictability is low at such long lead times, the land surface may play a role for forecast skills at these lead times given its profound persistence characteristics (Orth and Seneviratne, 2012). Each forecast constitutes an ensemble of 15 members which enables us to perform deterministic and probabilistic skill evaluations. Note that as the forecast sets differ with respect to the land surface model configuration, the initial land conditions for the forecasts may also be different. They are taken from the uncoupled HTESSEL simulations with the respective configuration. Consequently, the forecast skill is not only impacted by the altered HTESSEL configurations during the forecasting period, but also by correspondingly different initial land conditions. All further required initial conditions are taken from the ERA-Interim dataset (Dee et al., 2011) and from the ECMWF ocean reanalysis (Balmaseda et al., 2013). This forecast initialization methodology is also used for the ECWMF operational sub-seasonal forecasts.

O16 perturbed a set of six poorly constrained parameters which are deemed important for the performance of the HTESSEL model. They are listed in Table 2.

All selected parameters are perturbed at once (Saltelli et al., 2008). For this purpose, multiplicative factors between 0.25 and 4 (between 0.5 and 2 in the case of the soil depth) were applied to the default values of each of the chosen parameters. This range is chosen to still yield meaningful parameter values while allowing some variation in order to study the sensitivity of model performance to the perturbed parameters. The multiplicative factors were determined with a quasi-random sampling approach (Sobol, 1967) which allows us to efficiently sample the entire parameter space without introducing correlations between the perturbations of the considered parameters. In this way, a large sample of perturbed parameter sets was generated, of which 50 parameter sets were chosen by O16 to limit the computation effort for the uncoupled HTESSEL simulations covering Europe. We use the same parameter sets in this study. Out of these 50 parameter sets, 25 were chosen randomly while ensuring that the resulting multiplication factors applied to particular parameters are not correlated. The remaining 25 parameter sets were selected from corresponding HTESSEL simulations that agreed best with a suite of Earth observations at six locations across Europe. For this purpose the parameter sets were ranked in terms of each reference dataset, and then for each particular parameter set the sum of all ranks was computed. The resulting best-ranked 25 parameter sets include the default configuration of the model.

As coupled global forecasts are computationally demanding, they were only computed for a subset of 11 out of the 50 sets of perturbed parameters. This subset includes the default configuration, 5 configurations of the randomly chosen parameter sets, and 5 configurations of the best-performing parameter sets. For the selection of 5 (out of 25, or 24 in the case of the best-performing parameter sets as the default parameter set is already considered for computing the forecasts), all possible sets of 5 configurations were tested to choose the configurations with the lowest correlations between the multiplicative factors of the particular parameters.

In addition to comprehensively evaluating the LST performance of HTESSEL, it is a main objective of this study to analyse and illustrate the value of using an array of Earth observation datasets instead of single datasets to calibrate a land surface model. For this purpose we consider several reference datasets.

Soil moisture

We use data from 11 stations across Europe, which are displayed in Fig. S1. Stations are located in Finland (4), Switzerland (5), and Italy (2), and therefore provide data from all relevant European climate regimes. Data are available from different soil depths, and during different time periods, which both vary with respect to the station (see Table S1 in the Supplement). For every station, however, there are at least 4 years of data (Fig. S1). Aggregating the data from the different depths, we derive a weighted average (with respect to observed depths) to represent soil moisture within the top metre of the soil. The same is done with the HTESSEL data, using the three uppermost soil layers.

All these reference datasets are complementary in terms of spatial coverage and temporal availability. For example, whereas the soil moisture stations represent particular locations, the GRACE and LST data fully cover the European continent (except for cloudy regions in the latter case), but at lower spatial resolution. And while the ET data help to validate HTESSEL's energy balance during the early years (1989–2005), the LST data cover the recent years (2007–2014).

Note that for the in situ soil moisture and GRACE we only consider anomaly correlations to compute the agreement between the reference data and the HTESSEL output. For ET, streamflow, mean daily LST, and daily LST range we additionally consider the bias. This results in a total of 10 validation metrics which we use in our analysis.

We determine the skill of the coupled forecasts against gridded temperature and precipitation observations from the E-OBS dataset version 12 (Haylock et al., 2008). It is based on corresponding station observations from across Europe which are filtered and interpolated to a regular grid using state-of-the-art methods. See Sect. 2.3 for the employed measures of agreement between forecasted and observed data.

In this section we analyse whether HTESSEL configurations performing well against particular reference datasets also perform well against other reference datasets, i.e. whether a parameter set that yields for example good soil moisture performance also yields realistic LSTs. For this purpose we assess the performance of parameter perturbations performing best against particular reference datasets with respect to all other reference datasets in Fig. 3. White colours mean that parameter perturbations which perform well against particular reference datasets (x-axis) also perform well against other reference datasets (y-axis). Vice versa, black colours indicate that they do not also perform well against other reference datasets. Note that in the case of a perfect model and perfect observations this plot would be completely white. The many dark coloured fields in Fig. 3 indicate that the parameter perturbations performing best against particular reference datasets are different, i.e. there is no parameter perturbation that performs best in all respects. This can be explained by (1) equifinality (i.e. many different parameter sets leading to equally well-performing model simulations) as there are 25 pre-selected well-performing parameter sets among all 50 considered parameter sets, and by (2) inconsistencies within HTESSEL, especially between hydrological and skin-temperature-related processes. This is apparent as for example HTESSEL configurations performing well in terms of LSTs yield particularly poor performance in terms of hydrology, and vice versa. These inconsistencies might be partly associated with missing processes in HTESSEL, for example the over-simplification that a single parameter represents the complex energy transfers between the top of the canopy and the underlying soil.

Investigating the role of equifinality, we also perform this analysis with the 25 randomly chosen HTESSEL configurations only as displayed in Fig. S3. In general, results are robust with respect to the employed set of parameter perturbations, as indicated by the comparable patterns in the plots. This indicates a higher importance of inconsistent process representations in the HTESSEL model than of equifinality. Consequently, all the reference datasets considered in this study are needed to constrain HTESSEL, whereas for a perfect model, one dataset would be sufficient.

An analysis of this kind can moreover be used to assess overall model performance which can be measured with the mean rank (greyness) across all tested combinations. Note, however, that the result is influenced by the selection and the quality of reference datasets.

In this section we assess the relative performance of the 50 HTESSEL configurations against multiple metrics, i.e. against several reference datasets using different measures of agreement between the reference and modelled data. In this context, we compute the ranks of all simulations against all considered metrics. Thereafter we calculate for each simulation the sum of the individual ranks obtained against the considered metrics. This sum of ranks is then a measure of the overall performance of each simulation, and can be used to rank the overall performance against multiple metrics.

In Fig. 4, we test how the best- and worst-ranked parameter sets rank in the case that the validation metric(s) is/are replaced by the same number of other validation metric(s). For a perfect model and perfect observations we would find that the best- and worst-ranked parameter sets in terms of particular validation metrics are also best- and worst-ranked, respectively, when compared against other validation metrics. For HTESSEL we find that for an increasing number of employed metrics, worst-performing parameter sets tend to also perform worse in the case of replaced validation metrics. This is a main result of this study: it means that poorly performing model configurations can be more robustly identified when assessing model performance against multiple validation metrics.

However, this behaviour is not found for the best-performing parameter sets. There are two main reasons for this.

The poor correspondence of model performance against the considered single validation metrics as shown in Fig. 1. The impact of the underlying partly inconsistent process representations within HTESSEL on the results in Fig. 4 increases when using more validation metrics. When computing Fig. 4 without the two validation metrics for which we find the highest average ranks in Fig. 1 (runoff bias and LST range bias), the best-performing parameter sets are more robustly identified with increasing number of employed validation metrics (green dashed line). This can be explained by the performance of HTESSEL assessed against the two omitted metrics being most inconsistent with its performance against the remaining metrics. The opposite is found when re-computing Fig. 4 without the two validation metrics which correspond best to the remaining metrics, i.e. for which we find the lowest average ranks (soil moisture anomaly correlation and GRACE anomaly correlation).

This implies that the better the model performance rankings against individual metrics correspond to each other (i.e. the whiter colour there is in Fig. 3), the fewer metrics are required to robustly identify best- and worst-performing parameter sets. This supports the previously discussed importance of the mean rank in Fig. 3; it furthermore provides an indication of the required number of metrics to calibrate the considered land surface model.

Adding to the above analyses, we finally investigate whether a more robust calibration of HTESSEL against multiple datasets yields more accurate weather forecasts. For this purpose we perform a similar analysis to that in Fig. 4. We test how the best- and worst-ranked parameter sets rank if the (uncoupled) HTESSEL validation metrics are replaced by (coupled) weather forecast skill metrics. The results are displayed in Fig. 5.

Also in this analysis we find benefits of using multiple validation metrics. The parameter sets ranked best (worst) yield better (worse) forecast performance for an increasing number of employed validation metrics. It is another main finding of this study that land surface model calibration against multiple reference datasets instead of a single reference dataset can lead to better weather forecast performance. This result is found at all considered forecast lead times (1–15, 16–30 and 31–45 days). However, the differences between few and many considered metrics are smaller compared with Fig. 4. This can be explained by (1) fewer tested parameter sets (lower signal-to-noise ratio) and by (2) low predictability of temperature and especially precipitation at long lead times (e.g. 31–45 days) such that there is poor forecast skill overall independent of the HTESSEL configuration. Interestingly, forecast skills expressed as anomaly correlations or biases are more related to land surface model calibration than the reliability skill, as can be seen from the large distance between the dashed red and green lines in Fig. 5 compared with that of the dotted lines.

In summary, these results underline the importance of the land surface (model calibration) for coupled weather forecast skill (Koster et al., 2011), especially in terms of anomaly correlations and bias, and to a weaker extent also in terms of reliability.