Evaluating an Earth system model from a water manager perspective

The large spatial scale of global Earth system models (ESMs) is often cited as an obstacle to using the output by water resource managers in localized decisions. Recent advances in computing have improved the fidelity of hydrological responses in ESMs through increased connectivity between model components. However, the models are seldom evaluated for their ability to reproduce metrics that are important for and resonate with practitioners or that allow practitioners to situate higher-resolution model outputs within a cascade of uncertainty stemming from different models and scenarios. We draw on the combined experience of the author team and water manager workshop participants to identify salient water management metrics and evaluate whether they are credibly reproduced over the conterminous USA by the Community Earth System Model v2 (CESM2) Large Ensemble. We find that, while the exact values may not match the observations, aspects such as interannual variability can be reproduced by CESM2 for the mean wet day precipitation and length of dry spells. CESM2 also captures the proportion of total annual precipitation that derives from the heaviest rain days in watersheds that are not snow-dominated. Aggregating the 7 d mean daily runoff to two-digit Hydrological Unit Code (HUC2) watersheds also shows that rain-dominated regions capture the timing and interannual variability of annual maximum and minimum flows. We conclude that there is potential for far greater use of large-ensemble ESMs, such as CESM2, in long-range water management decisions to supplement high-resolution regional projections.

1 Introduction

Water availability and water quality for human consumption, ecosystems, and agriculture are fundamental requirements, making pertinent assessments of future change crucial for adaptation planning (IPCC, 2022). Climate-related changes in the hydrological cycle will affect substantial portions of the world's population, most directly through changes in water availability at or near the surface (Mankin et al., 2020; Sedláček and Knutti, 2014). The information required by water managers for decision-making is not readily available in a relevant format or at sufficient spatial or temporal resolutions from global Earth system models (ESMs; e.g., Ekström et al., 2018). We explore how the Community Earth System Model (CESM) represents the climatology of water availability, focusing on metrics that are familiar to decision-makers in planning investment-scale decisions.

The inability of ESMs to explicitly resolve subgrid-scale (∼ 100 km) processes is often cited as the limitation preventing direct model use in decision-making. The literature from large organizations making infrastructure decisions (e.g., Brekke, 2011; Brekke et al., 2009; Reclamation, 2016, 2014; Lukas et al., 2020) emphasizes downscaling of climate model data closer to the scale of the watersheds they manage. These additional modeling steps add complexity and may increase statistical errors (Clark et al., 2015; Ekström et al., 2018). Extracting useful and robust information directly from ESMs would reduce such errors if metrics most important to decision-makers, such as the timing of peak flow, are known to be robustly represented.

There are many comprehensive examples of metrics used to evaluate climate and hydrological models (e.g., Ekström et al., 2018; Mizukami et al., 2019; Wagener et al., 2022), communicate the impacts of climate change (e.g., Reed et al., 2022), or identify decision-relevant metrics (e.g., Bremer et al., 2020; Mach et al., 2020; Underwood et al., 2018; Vano et al., 2014). However, very few have examined whether user-defined metrics can be reproduced reliably by ESMs (Mankin et al., 2020) and whether further model development and scale reduction are warranted instead of improved communication (Pacchetti et al., 2021). Better communication may also reduce the temptation of some users to calculate “standard hydroclimate metrics” that are not supported by the climate model data (Ekström et al., 2018).

In contrast, climate model output can be rejected unnecessarily when simulated annual minima from freely running simulations do not “match” the sequence of observed low flows (Ekström et al., 2018; Moise et al., 2015). Similarly, the benefits of a range of projected outcomes from different climate models are not widely appreciated beyond the climate model community (Tebaldi and Knutti, 2007). Large ensembles from a single climate model initialized with a range of atmospheric and ocean conditions, such as the CESM v2 (CESM2) Large Ensemble (LENS2; Rodgers et al., 2021), help to bound the uncertainty that derives from a naturally chaotic system. Averaged over the full ensemble, they give a better estimate of the model's response to internal and external forcing (Deser et al., 2012) and enable assessments of the rarity of projected extremes. The additional analysis for identifying structural (i.e., model formulation) and internal variability within regional climate models means that there are fewer large ensembles at a high resolution (Deser et al., 2020).

Since different decision-makers have different priorities and timescales of interest, Shepherd et al. (2018) recommended the development of climate storylines to communicate with those using climate data to make decisions. Informed by previous surveys of water managers (e.g., Brekke, 2011; Brekke et al., 2009; Cantor et al., 2018; Raff et al., 2013; Wood et al., 2021), Fig. 1 aims to map the different types of water decisions (e.g., Raff et al., 2013; Fig. 3) to the different scales of model resolution (Meehl et al., 2009; Fig. 2). Water managers make daily operational decisions (e.g., to control instantaneous river flow) with the aid of fine-scale weather and flood models (< 4 km) that reliably represent convective and local weather processes, even though their predictability is relatively short-lived (Yuan et al., 2019; far left of Fig. 1). Larger watershed operations (such as reservoir management or groundwater recharge; Regional Water Authority, 2019) depend on seasonal outlooks (center left of Fig. 1). Smaller adaptation and mitigation projects take place at the typical policy or decadal prediction scale (i.e., 4–10 years; center right of Fig. 1). Finally, major public investments and interbasin agreements are made at the same timescales as climate projections (30–100 years; far right of Fig. 1), where persistent and relatively predictable synoptic and planetary-scale processes are represented well in lower-resolution (∼ 100 km) climate models (Phillips et al., 2020). While forecasts (seasonal or decadal) are re-initialized from specific atmosphere, ocean, or land states at regular time intervals, climate projections are run freely from a variety of atmospheric and oceanic conditions that take several decades to converge to a mean climatology. In considering the utility and usability of information directly from ESMs, we focus on decisions made over decadal to climate scales at larger spatial scales.

Figure 1 Mapping the temporal and spatial scales of models to the time frames for water management decisions.

Given that ESMs have advanced immeasurably in the last decade, it is time to re-evaluate whether their direct output can support decision-makers. Such an evaluation needs to focus on how well the models can reproduce metrics used by decision-makers and whether the results are credible (Briley et al., 2020; Jagannathan et al., 2021). Here we evaluate the credibility of one ESM in generating metrics known to be salient for water management decisions, specifically decisions for water management infrastructure project investments.

The motivation for this paper is to identify

• a set of water availability metrics that resonates with decision-makers and supports their investment-scale decisions;

• how well CESM2 represents the climatology and recently observed behaviors of those metrics; and

• the range of CESM2 structural uncertainty and internal variability for these metrics.

This paper builds off a decade of collaboration between scientists at the National Center for Atmospheric Research (NCAR) and US water agencies that led to a virtual workshop (Tye, 2023), and it presents a test case for improved communication with water management decision-makers. The focus is on the conterminous United States (CONUS) in order to match the interests of workshop participants.

2 Climate information needs from prior research

Information needs vary greatly, from 5 min rainfall totals at a point (ASCE, 2006) to basin-wide measures of annual minimum and maximum total runoff. Water management decision metrics can be grouped into similar types such as timing, frequency, magnitude, extreme values, variability, and duration of events (Ekström et al., 2018). While some aspects of timing, magnitude, or variability can be reproduced reliably by ESMs (e.g., Deser et al., 2020; Tebaldi and Knutti, 2007), others such as short-duration extremes are less reliable.

Methods of evaluation and data use also differ. For instance, Clifford et al. (2020) reported that predicting general changes in the frequency of extreme precipitation events is more useful for future planning than precise prediction of mean values evaluated by model developers. Lehner et al. (2019) emphasized that models need to be evaluated for their ability to reproduce sensitivities (e.g., streamflow changes in response to temperature and precipitation changes), in addition to mean states. However, metrics that are meaningful for evaluating a model's capabilities (e.g., the ratio of precipitation to runoff) are less valuable for management decisions (Lehner et al., 2019; McMillan, 2021; Mizukami et al., 2019). When reporting results, water managers are more familiar with the water year than the calendar year when capturing the full annual hydrological cycle (Ekström et al., 2018). While the use of water years is a nuance that does not add substantial value to climate model assessments, communication with decision-makers is improved by presenting data in a familiar format (Briley et al., 2020).

There is a need for information at the local scale that is unlikely to be met directly by raw outputs from the current generation of ESMs. However, better communication of the variability in future daily precipitation and the associated runoff can add value to the detailed models by bringing in the added statistical context and perspective of the large ensembles. Thus, we believe that ESMs can produce useful information about hydrometeorological extremes when presented at different spatial or temporal scales and offer the benefits of large climate model ensembles to constrain future impact uncertainty.

Appendix A summarizes potential hydrological metrics used in water management decisions (Jagannathan et al., 2021), statistical assessments of extremes (Zhang et al., 2011), and model evaluations (Phillips et al., 2020). The metrics in bold are presented in this paper. We only considered a simplistic measure of meteorological drought (absence of rain) in the current work, as drought is sensitive to the definition (Bachmair et al., 2016) and local conditions (Mukherjee et al., 2018) and so is not suited to a generalized assessment. Similarly, snow measures are not included in this assessment, due in part to the limited availability of high-quality, long-duration, quality-controlled, and observational data (McCrary et al., 2017) and the biases in snow distribution arising from the smoothed topography in ESMs (McCrary et al., 2022).

3 Data and methods

3.1 Climate model data

CESM2 (Danabasoglu et al., 2020) is a fully coupled global model that simulates Earth's climate system through interactive models for the atmosphere, ocean, land, sea ice, river runoff, and land ice. The variables considered in this project are taken from the Community Atmosphere Model version 6 (CAM6) and the Community Land Model version 5.0 (CLM5; Lawrence et al., 2019) and are part of the default model outputs. A schematic of the model components is included in Appendix B. This project uses daily to annual values (e.g., annual maximum daily precipitation) on a grid with ∼ 1° resolution. Data were extracted over the CONUS from 10 ensemble members of LENS2 (Rodgers et al., 2021) for model validation in the current era (1981–2010).

3.2 Observations

Gridded daily observations of precipitation at $\mathrm{1}/\mathrm{16}$° horizontal resolution (∼ 6 km) were obtained from the Livneh et al. (2013) dataset covering the CONUS and southern Canada for the control period (1981–2010), hereafter referred to as “Livneh”. Pierce et al. (2021) provided an update to the Livneh dataset to address time adjustments that result in an underestimation of the most extreme daily precipitation totals and the resultant runoff and flood potential (Pierce et al., 2021). However, as we are also interested in other measures of precipitation and in runoff minima, we did not employ the updated gridded observations.

Livneh daily temperature maxima and minima as well as precipitation were used to force the Variable Infiltration Capacity Model (VIC; Liang et al., 1994) version 4.1.2 to obtain runoff estimates for the years 1980–2005 as evaluated in Livneh et al. (2013). Hereafter this model is referred to as “Livneh-VIC”.

3.3 Methods

All analyses were carried out using the North American water year (1 October to 30 September) to facilitate later communication.

3.3.1 Remapping

For ease of comparison, model outputs were regridded using a conservative second-order remapping (Jones, 1999) to place both datasets on the same grid and assess anomalies. Data were also calculated as areal averages or totals over the two-digit Hydrological Unit Code (HUC2) regions (Seaber et al., 1987). HUC2 basins represent 18 watersheds, covering areas ranging from 41 000 mi² (∼ 105 000 km²; Tennessee) to 520 960 mi² (1 350 000 km²; Missouri), as shown in Fig. 2. While the scale of HUC2 regions may be large for some local decision-makers, it is also a more appropriate and conservative scale for comparison with ESMs, as demonstrated by Lehner et al. (2019).

Figure 2 HUC2 regions used in data validation and analysis. Regions defined by Seaber et al. (1987): Region 01 New England (NE), Region 02 Mid-Atlantic (MA), Region 03 South Atlantic-Gulf (SA), Region 04 Great Lakes (GL), Region 05 Ohio (OH), Region 06 Tennessee (TN), Region 07 Upper Mississippi (UM), Region 08 Lower Mississippi (LM), Region 09 Souris-Red-Rainy (RR), Region 10 Missouri (MR), Region 11 Arkansas-White-Red (ARK), Region 12 Texas-Gulf (GUL), Region 13 Rio Grande (RIO), Region 14 Upper Colorado (UC), Region 15 Lower Colorado (LCO), Region 16 Great Basin (GB), Region 17 Pacific Northwest (PN), and Region 18 California (CA).

3.3.2 Percentile-based thresholds

The threshold for very heavy rain days (Q95) was calculated at each individual grid cell using only days with ≥ 1 mm of rain (“wet days”). Thresholds were derived empirically for each model ensemble member, with the ensemble mean threshold (Q95) used to identify the days per year exceeding the threshold (N95) and the total annual rainfall from those days (P95).

Runoff was aggregated over each HUC2 watershed and multiplied by the respective area to generate the total volume per day. The volume per day was then converted to measurements more familiar to users, such as acre feet per day or cubic meters per second. Daily time series of total volumetric runoff had a 7 d running mean smoother applied, and then the annual maximum, minimum, and mean values were extracted. The highest and lowest 7 d average runoffs expected once per decade (7Q90 and 7Q10) were estimated empirically from the 25 ranked values of annual maxima and minima per watershed. Stationarity was assumed over the climatological period for the purposes of these analyses, acknowledging that changes may have already occurred in the frequency of these events.

4 Model evaluation

The metrics used to evaluate CESM2's ability to reproduce large-scale features and physical behaviors (e.g., Danabasoglu and Lamarque, 2021, and the associated special issue) are not necessarily those employed by decision-makers. ESMs are designed to represent large-scale atmospheric processes and fluxes not specific to local responses (Gettelman and Rood, 2016), but this design assumption may not be sufficiently well communicated to decision-makers. The purpose of our evaluation is to establish whether CESM2 output is also fit for local decision purposes or whether the breadth of information from ESM ensembles remains unsuitable for immediate use in targeted water management decisions.

4.1 Rainfall metrics

While broad spatial patterns of seasonal mean daily rainfall are reproduced well (Danabasoglu et al., 2020; Feng et al., 2020; Simpson et al., 2022), CESM2 fails to capture details over high topography and overestimates summer precipitation where convective extremes dominate summer rainfall (Appendix C). The seasonal mean precipitation also fails to capture some important watershed-level processes, such as the seasonal variability in the number of days with precipitation and the associated intensity.

Figure 3 Average number of wet days per month (a, b) and interannual variability in mean annual precipitation on wet days for the Livneh climatological mean (red) with interannual spread (pink) and the CESM2 mean (cyan) with interannual and ensemble spread (gray). (c, d) Between 1981 and 2010 for observations derived from Livneh (red), an individual CESM2 ensemble member (blue), and the ensemble spread (gray) in (a, c) CA and (b, d) PN.

Estimates of mean annual rainfall on wet days, or wet day volume, are in broad agreement with Livneh and CESM2 output. Figure 3 shows examples of the mean number of wet days per month (NWD) and the mean wet day volume (WDV) averaged over California and the Pacific Northwest. While CESM2 represents the NWD annual cycle very well in regions such as California (Fig. 3a, c) and the Pacific Northwest (Fig. 3b, d), it does not capture NWD in many central and snow-dominated regions (Figs. S1 and S2 in the Supplement). This is likely due to the smoother topography of CESM2 missing the influence of orographic uplift as well as large-spatial-scale missing subgrid-scale convective systems (e.g., over the Central Plains). The figures also highlight the scale of model (structural and internal variability) uncertainty present in the ensemble. As noted in the previous sections, water management decision-makers are aware of the potential scale of uncertainty and have expressed a desire for the full ensemble range to be presented to them instead of ensemble means.

The annual variability in WDV, both year-to-year variations and the overall range of minima and maxima, is captured well by each of the model members for the different HUC2 regions, even if the absolute values do not match (Fig. 3c, d). As expected, the specifics of which years have high or low values of WDV are not the same for each ensemble member (i.e., demonstrating internal variability). As a result, the ensemble mean value of WDV (cyan) does not reflect the same year-to-year variability as the observations for individual ensemble members (blue). Decision-makers stated that the interannual variability demonstrated by each model member is more valuable for demonstrating the credibility of the data than the ensemble mean (Tye, 2023). We recommend that the full range of values of each metric (i.e., after individual computation for each ensemble member) be communicated in addition to the climatological means to help bound uncertainty around decisions (Wilby et al., 2021).

The magnitude of interannual variability in WDV (i.e., the absolute differences between the maximum and minimum values in each member time series) is typically within 10 % of observations in all the regions, as illustrated for two regions in Fig. 3. Exceptions are the Lower Colorado, South Atlantic-Gulf, and Upper Mississippi regions, where the simulated distributions are too narrow. Many different sources of error may contribute to this discrepancy, such as the inability to resolve convective precipitation (Chen et al., 2021) and elevation changes not captured by the coarse model resolution or the “drizzle effect” that is common in GCMs (Chen et al., 1996; Dai, 2006).

CESM2 captures the longest spells of consecutive dry days per year (CDD; Fig. 4a) and consecutive wet days per year (CWD; Fig. 4b), together with their variability. Many regions capture both the interannual variability and the climatological mean duration of CWD, particularly in those regions that are subject to large-scale synoptic systems (e.g., Pacific Northwest, Mid Atlantic-Gulf, and California). Several regions either overestimate (South Atlantic-Gulf) or underestimate (Great Lakes and Souris-Red-Rainy) the absolute durations of the longest wet spells but do reflect the magnitude of interannual variability. The exception is Tennessee, where both interannual variability and mean CWD are overestimated. At the grid scale, the broad spatial patterns of CWD are correct, but the finer atmospheric processes arising from topographic features are incorrect, as expected from the coarse model resolution. A similar pattern is present in CDD, except that some drier regions with CDD > 30 d do not capture the full range of interannual variability (Souris-Red-Rainy, Missouri, and Rio Grande). As GCMs have a tendency to produce drizzle, adjusting for a higher wet day threshold (e.g., 2 mm) might improve dry spell representation in those regions. It is also important to communicate such model sensitivities to users more effectively.

Figure 4 (a) Longest duration per year of consecutive days with < 1 mm of rain (longest dry spell) for Livneh over all the years (green) and the CESM2 ensemble range over all the years (blue) for all the HUC2 regions. (b) Longest duration per year of consecutive days with ≥ 1 mm of rain (longest wet spell). The regional abbreviations are given in Fig. 2.

The thresholds for heavy and very heavy rain days (P95 and P99) are defined with respect to the wet days, are calculated individually, and are compared for Livneh and CESM2 in order to understand whether the intensity of more extreme rainfall is captured and to evaluate the models' behavior. A comparison of the thresholds reflects the considerable improvements in modeling capabilities in recent years (Gettelman et al., 2022). For instance, earlier versions of CESM underestimated extreme precipitation intensity by 10–30 mm d⁻¹ east of the Rockies and overestimated intensity by 5–10 mm d⁻¹ to the west (Gervais et al., 2014). We found that CESM2 still underestimates the most extreme rainfall but that errors have approximately halved. As these differences are still inadequate for many engineering and major infrastructure decisions (Wright et al., 2019), we focus on CESM2's ability to capture the relative contributions of P95 and P99 to the annual total and the interannual variability in their frequency. A result with considerable useability is the proportion of total annual precipitation derived from the heaviest rain days, or “proportional contribution of extreme days” (P95Tot). This proportion and its interannual variability are well represented by CESM2 at the HUC2 scale and have proven skillful in other models (Tebaldi et al., 2021).

The interannual variability in the frequency (N95) and intensity of extreme rainfall, as represented by P95Tot, is illustrated in Figs. 5 and 6. In several HUC2 regions the simulations report more frequent events and proportionally higher totals (e.g., Rio Grande, Missouri, Upper Colorado, and Lower Colorado; Fig. S3). Overall, there is good subjective agreement between Livneh and CESM2, identifying an opportunity to inform local decisions from large-scale ESMs.

Figure 5 (a, c, e, g) Number of very heavy rain days per year and (b, d, f, h) total rain from very heavy rain days as a proportion of the annual total for the (a, b) TN, (c, d) GL, (e, f) RR, and (g, h) CA HUC2 regions. Observations are in red, the CESM2 ensemble spread is in gray, and single randomly selected ensemble members are in blue.

Figure 6 Boxplots of the interannual range of contributions to total annual rainfall from very heavy days (P95Tot), shown as percentages for observations (light blue) and the ensemble range for CESM2 (green) for all HUC2 regions. Boxes are bound by the interquartile range, black lines indicate the median, notches indicate the degree of spread from the median, and bars extend to the full data range.

4.2 Runoff metrics

Runoff estimates are taken from the individual components of surface and subsurface runoff generated within CLM5 (Lawrence et al., 2019) and compared to the Livneh-forced VIC runoff (“Livneh-VIC”).

Assessing the skill of runoff in large-scale models is complicated by many factors, including the mismatch of scales between in-channel flow (∼ 1–10² m) and the grid scale (∼ 10⁵ m). Thus, metrics of climate model runoff should be selected carefully, and the runoff should be aggregated or combined with other metrics rather than used directly (Lehner et al., 2019). Appendix D demonstrates the discrepancies between the grid-scale representation of runoff from Livneh-VIC and CESM2. The large discrepancies arise from different processes that are not captured adequately, such as groundwater, topography, and the associated snow ablation and snowmelt, in addition to meteorological biases.

Figure 7 Interannual variability in runoff in the RR region for (a) the mean seasonal cycle and (b) the cumulative watershed runoff over the water year. The Livneh-VIC climatological mean is in red, the range of all the years is in pink, the CESM2 ensemble mean is in blue, and the ensemble range is in gray. The figure highlights the underestimation of the lowest runoff volume by CESM2 by a factor of 10.

However, water management decisions are made over watersheds in units such as acre feet¹ or cubic meters, while model data are output as a depth of runoff over each grid cell (e.g., mm d⁻¹ km⁻²). We aggregated the 7 d running mean daily runoff (Q7) within each HUC2 region to generate Q7 time series in each basin. Figure 7a illustrates the 25-year mean seasonal cycle for Livneh-VIC in red and CESM2 in blue, together with the full range of values over all years and ensemble members for the RR basin (HUC Region 9). Additional basins are included in Fig. S4. Data are presented in millions of acre feet in order to align with decision-maker needs. The minimum simulated Q7 in any year considerably underestimates the lowest flows in this region compared to Livneh-VIC. In contrast, the largest total runoff volume is overestimated and peaks too early in the water year. Figure 7b plots the same information as the cumulative runoff volume from the start of the water year, highlighting that the lowest runoff volume is underestimated by a factor of 10 for this region; the other regions are illustrated in Fig. S5. Low runoff volumes were typically underestimated in smaller regions (e.g., NE and TN). High runoff volumes were only underestimated in three regions (LM, ARK, and GUL) and considerably overestimated in seven regions. Snow-dominated regions perform particularly poorly for both QMax and QMin, as snowpacks and the timing of the associated runoff are not simulated well. Transitional regions that straddle both snow- and rain-dominated hydrologies also fail to capture QMax but better estimate Qmin (not shown). Only the South Atlantic region reproduces both QMax and QMin.

We explored the relationships between the highest and total annual runoff (QMax $/$ QTot) and the lowest and total annual runoff (QMin $/$ QTot). Some regions performed well for QMax $/$ QTot and others performed better for QMin $/$ QTot, but there was no consistent relationship that could be utilized by decision-makers.

Participants in the NSF NCAR workshop (Tye, 2023) emphasized that the exact numbers produced by climate models are not very important for future decisions. Others have also emphasized the importance of well-represented processes in the model (Reed et al., 2022) and correlations with known experiences (Mach et al., 2020; Shepherd et al., 2018). Focusing on fidelity to the historical climate exaggerates the importance of model performance instead of robustness to different conditions without ensuring that model predictions are useful or reliable (Brunner et al., 2021; Wagener et al., 2022). Runoff estimates in transitional catchments may be inadequate in the current climate but plausible in the future if the model reproduces rain-dominated hydrological processes (McMillan, 2021).

Climatological mean runoff cycles are estimated from Pardé coefficients – calculated as Q7 $/$ QTot on each calendar day – a dimensionless value that enables comparison across regions. Figure 8 depicts the mean seasonal cycle for representative snow-dominated (Upper Colorado), transitional (Missouri), and rain-dominated (Tennessee) regions, demonstrating how an imperfect representation of snow in the Upper Colorado region results in CESM2 peak runoff occurring 2 months earlier than Livneh-VIC (Fig. 8a). The runoff regimes display very different seasonal characteristics, with CESM2 having a “mid to late spring” runoff regime rather than Livneh-VIC's “extreme early summer” regime (Fig. 8a; Haines et al., 1988). Peak runoff is also too early in the transitional regions but is closer to Livneh-VIC than in snow-dominated regions (Fig. 8b). Rain-dominated regions capture both the timing of QMax and the overall seasonal hydrograph shape (Fig. 8c).

Figure 8 Seasonal patterns of runoff for HUC2 regions: (a) UC, (b) TN, and (c) MR, constructed from normalized series of the ratio of 7 d mean runoff to the mean annual total. Livneh-VIC runoff climatological mean (red), climatological range (pink), CESM2 ensemble mean (blue), and ensemble range (gray with a dashed border). Vertical lines indicate the mean date of peak runoff with the number of days since the start of the water year.

Table 1 Very low (7Q10) and very high (7Q90) regional runoff, together with the standard deviation in the regional annual minima (σ QMin) and maxima (σ QMax) for Livneh and CESM2. Values in bold indicate where CESM2 and Livneh-VIC regional runoff is statistically similar according to a χ² test.

7Q10 and 7Q90 are estimated empirically from annual minima and maxima as occurring once per decade. Projected changes in the frequency of very low or very high runoff volumes are deemed credible where CESM2 replicates the standard deviation of annual minima and maxima according to a χ² test at the 5 % significance level. Table 1 reports CESM2 and Livneh-VIC regional estimates of 7Q10 and 7Q90 and standard deviations of the annual maxima and minima; values in bold indicate where estimates are statistically similar. It should be noted that the values in Table 1 have ≤ 10 % occurring in any year and so represent the tails of the runoff distribution.

Grid-scale estimates such as mean daily runoff readily highlight why decision-makers have low confidence in CESM2 output: the metrics are not salient and appear to have no skill. After aggregating the 7 d mean daily runoff to watershed scales, some skill emerges in the annual minima and maxima as well as the seasonal cycles. Snow-dominated watersheds perform poorly with regard to the peak runoff volume and the timings of the peaks and lows, as expected (McCrary et al., 2022). Rain-dominated watersheds capture the interannual variability and magnitudes of the peak and low flows as well as the seasonal hydrographs. While CESM2 at this coarse scale does not represent the local topography and cannot represent finer-scale snow, our analysis indicates that the land surface model correctly simulates the overall bulk water budget for most watersheds, as illustrated in Figs. 7 and 8. However, the tail behavior of the highest and lowest total watershed runoffs is only captured by a few basins, and so caution needs to be exercised in the interpretation and use of model results, as biases may propagate into the future. This is premised on the understanding of why the model can produce accurate results and whether the accuracy can be reproduced reliably for the future climate (Wagener et al., 2022).

While participants in the NSF NCAR workshop stated that precise estimates are not necessary, they also emphasized their desire for high confidence in the projected scale and direction of any changes. We note that “confidence” is derived from a combination of (1) credible process representation; (2) agreement with historical trends, given internal variability; and (3) agreement across multiple models. It is worth noting that trends in extremes may be important without being statistically significant, as a limited sample of points (e.g., one per year) from a stochastic series is inherently noisy. However, some of these trends may emerge from the noise in the distribution and so are important to monitor.

5 Discussion

As decision-makers have become more immersed in developing water management adaptation plans, the role of “climate services” in developing salient climate information has increased (Briley et al., 2020; Brugger et al., 2016; Dilling et al., 2019). We tested our hypothesis that recent improvements in ESMs can allow decision-relevant metrics to be produced directly by leveraging the combined experience of the author team, results from the NCAR workshop, and the wealth of literature on actionable knowledge (Bremer et al., 2020; Jagannathan et al., 2021; Mach et al., 2020; Vano et al., 2014). Given that no model can perfectly address all decision needs, we identified and evaluated multiple metrics that can frame specific water management decisions within the known constraints of the data (Lempert, 2021) or within the decision-makers' experiences (Austin, 2023; Clifford et al., 2020; Reed et al., 2022; Shepherd et al., 2018).

It is important to communicate the original purpose of the model and the associated weaknesses, so that decision-makers fully understand which information is appropriate to use in other applications (Fisher and Koven, 2020; Gettelman and Rood, 2016; Wagener et al., 2022). Given the balance between model fidelity and model complexity (Clark et al., 2015) and the absence of detailed global-scale observation data (e.g., Gleason and Smith, 2014; Reba et al., 2011), CESM2 provides a plausible representation of Earth system processes and moisture fluxes but may not capture basin-scale specifics (Ek, 2018; Lehner et al., 2019). That said, there are continued efforts to improve the simulation of land surface processes, and analyses such as those presented in this article can flag weaknesses for future improvement (Lawrence et al., 2019).

Establishing model fidelity also requires one to distinguish between an accurate representation of the climate processes from serendipitous correlations and observations. Whether the model has good process representation overall or exactitude in one simulation can be established through internal variability analyses using large ensembles (e.g., Deser et al., 2020; Tebaldi et al., 2021). Repeating the analyses with several different ESMs to establish the degree of agreement (Mankin et al., 2020) would further strengthen the usability of the metrics presented in this article. It is also worth noting that the analysis presented here only used one reference dataset. As different reanalysis and observational datasets can have large discrepancies, a thorough model evaluation would also benefit from comparison to several products (Kim et al., 2020; Newman et al., 2015), including an assessment of how removing temporal adjustments in observations affects the statistics of extremes (Pierce et al., 2021).

While the precise details of precipitation and runoff may not be simulated well by CESM2, we found some aspects to be credible. The frequency of wet days highlighted regions where the current seasonal behavior is captured well and may support planning around flood and drought control or wildfire risk when used in combination with other models or data sources (Austin, 2023; Clifford et al., 2020; Jagannathan et al., 2021; Reclamation, 2016).

6 Conclusions

This paper presented an assessment of whether a standard-resolution (∼ 100 km grid) Earth system model is capable of producing information that water users typically employ in their decisions. Our motivation was to explore whether it is possible to reduce the need for intermediate downscaling and to extend the use of large model ensembles to quantification of the influence of internal variability on localized decisions. We drew on the combined experience of the project team and workshop participants to identify potential metrics and familiar modes of visualization. This project only used CESM2 over the conterminous United States to develop example metrics that may be explored within other models and over other regions. CESM2 is unable to reproduce some metrics given the lack of topographical detail. A companion paper by Rugg et al. (2023) examines potential improvements to the subgrid-scale simulation of land processes in order to improve the representation of the hydrological cycle in mountainous regions.

We encourage others working in the decision space between climate data producers and users to be forthcoming about specific regions and reasons where model data are not credible or where the model has particular weaknesses (such as the drizzle effect) that may be overcome with a different analysis approach.

For future model assessors, the following metrics were found to be salient for water users and were skillfully reproduced in many regions.

Rainfall:

• Number of wet days (≥ 1 mm of rain) per year or season

• Mean precipitation on wet days

• Duration of the longest wet and dry spells per year

• Number of days with rain > 95th percentile of the current climate's wet day totals

• Proportion of the annual total derived from days > 95th percentile of the wet day totals

Runoff (aggregated up to the basin level as a volume for 3 and 7 d averages):

• Annual maxima and minima

• Frequency of very high or very low flows (< 10 % annual chance of occurring in the current climate)

• Proportion of averaged daily runoff in the annual total

The work presented in this paper is a small step towards establishing greater usability of climate model output by decision-makers. The present evaluation is also only the first step in evaluating ESM performance. Additional research is needed to support water managers, placing these results and their uncertainty in the context of additional observational data (such as remote sensing) that may already be available to them. Continued collaboration is essential for improving the transfer of knowledge (e.g., data requirements, model assumptions, or decision constraints) between communities.

Appendix A

Table A1 Hydrometeorological responses used in water management decisions and the specific metrics that have the potential for representation in ESMs. The metrics in bold are presented in this article.

Appendix B

Figure B1 Schematic of the Community Earth System Model version 2 (CESM2) model components, reproduced from Danabasoglu et al. (2020, Fig. 1).

Appendix C

Figure C1 Seasonal mean precipitation for winter (top row), spring (row 2), summer (row 3), and fall (bottom row) as shown in Livneh (left column) and CESM2 (middle column), together with the difference CESM2 − Livneh (right column).

Appendix D

Figure D1 Comparison of the maximum daily runoff between 1981 and 2005 in (a) Livneh, (b) CESM2, and (c) the difference CESM2 − Livneh.