The SMAP mission provides SM estimates for the top 5 cm of the soil column, with an average revisit time of 2–3 d, a resolution of 36 km, and a 50 h data latency. Both ascending (afternoon; PM) and descending (morning; AM) retrievals from the SMAP L3 Passive product data Version 4 (O'Neill et al., 2016) from 31 March 2015 to 31 December 2017 were used in this study. A few SMAP pixels with obvious quality flaws (i.e., near-constant retrieval values) were manually masked out. The internal quality flags provided by the SMAP mission were not applied in this study to preserve the measurements in the eastern half of the domain, where the data quality of the entire region is flagged as unrecommended due to relatively heavy vegetation cover. The native 36 km SMAP retrievals were used throughout the study without spatial remapping or temporal aggregation.

The Integrated Multi-satellitE Retrievals for GPM (IMERG) Level 3 Version 05 Early Run precipitation data were used in this study (Huffman et al., 2018). IMERG merges multiple satellite observations and provides a near-global precipitation product with a spatial resolution of 0.1∘ (Huffman et al., 2015). We used the “Early Run” version of this product since its short latency (4 h) makes it suitable for near-real-time data assimilation applications. However, this short latency also prevents correction of the IMERG Early Run product using ground-based rain gauge observations. We aggregated the original 30 min IMERG precipitation product to our every 3 h modeling time step and remapped it onto our 1/8∘ model spatial resolution.

Other than precipitation, the VIC model requires air temperature, shortwave and longwave radiation, air pressure, vapor pressure, and wind speed as forcing inputs. These variables were taken from the 1/8∘ gridded North American Land Data Assimilation System Phase 2 (NLDAS-2) meteorological forcing data product (Xia et al., 2009). We aggregated the original hourly NLDAS-2 meteorological variables to the every 3 h modeling time step.

Daily streamflow data at eight USGS streamflow sites in the study domain (USGS, 2018) were used to evaluate the streamflow time series from the dual-correction system (Fig. 2 and Table 1). These eight sites were selected for their lack of human regulation and their dense rain gauge coverage (Crow et al., 2017). We separately evaluated the rainfall correction scheme, in which the NLDAS-2 precipitation data were treated as the benchmark. The NLDAS-2 precipitation data were based on daily gauge-based rainfall measurements that were disaggregated into hourly intervals using ground-based weather radar (Xia et al., 2012). The reliance of NLDAS-2 on gauge observations (to obtain daily rainfall accumulations) ensures that it is more reliable (in a relative sense) than the remote-sensing-only Early Run IMERG products used in this study. Consequently, it provides an adequate evaluation benchmark for subsequent attempts to correction IMERG.

The Soil Moisture Analysis Rainfall Tool (SMART) rainfall correction algorithm (Crow et al., 2009, 2011; Chen et al., 2012) is based on sequential assimilation of SM measurements into an Antecedent Precipitation Index (API) model. 1APIt=γAPIt-1+Pt, where t is a time step index, P is the original IMERG precipitation observation (mm), and γ is a unitless loss coefficient. We implemented an every 3 h version of SMART (instead of the daily version in past studies) to receive the every 3 h IMERG rainfall input and both the ascending (PM) and descending (AM) SMAP retrievals at the correct time of day. We also extended the ensemble Kalman filter (EnKF) version of SMART introduced by Crow et al. (2011) to an ensemble Kalman smoother (EnKS), in which the API state is not only updated at time steps when SMAP is available but also updated during measurement gaps (see Sect. S1 in the Supplement for mathematical details underlying the SMART EnKS approach). We set γ to 0.98 such that the uncorrected API time series approximately captures the dynamics of SMAP retrievals (i.e., with high correlation; see Sect. S3 for a sensitivity analysis on γ). SMAP was rescaled to the API regime through cumulative distribution function (CDF) matching over the 2.5-year simulation period prior to assimilation. CDF matching was performed separately for SMAP AM and PM retrievals to account for their mutual systematic differences.

The SMART algorithm then uses the API increment, δt, to estimate the rainfall correction amount via a simple linear relation. We implemented an ensemble rainfall correction rather than the single deterministic rainfall correction used in past SMART applications. 2Pcorr,t(j)=Ppert,t(j)+λδt(j), where the superscript (j) denotes the jth ensemble member (ensemble size M=32), Pcorr,t is the corrected precipitation for time t, Ppert,t is the perturbed IMERG precipitation, and λ is a scaling factor that linearly relates the API increment to rainfall correction, which was set to a domain-constant of 0.1 (–) (see Sect. S4 for discussion on the choice of λ). We applied the rainfall correction only at time steps when the original IMERG rainfall observation was nonzero, taking advantage of the enhanced rain/no rain detection accuracy of IMERG (Gebregiorgis et al., 2018). This tactic mitigates the spurious introduction of low-intensity rainfall events by SMART (see also Sect. 3.1). Finally, following Crow et al. (2009, 2011), negative Pcorr,t values were set to zero, and the final corrected precipitation time series was multiplicatively rescaled to be unbiased over the entire simulation period against the original IMERG estimates (so that the long-term mean of the IMERG rainfall time series was preserved).

In this study, the SMART algorithm was run at each of the 36 km SMAP pixels individually. The original 0.1∘ IMERG product was remapped to the coarser 36 km resolution prior to SMART, and the corrected 36 km rainfall was then downscaled to the VIC 1/8∘ model resolution. In our implementation of an EnKS-based SMART system, the original IMERG precipitation was multiplicatively perturbed by log-normally distributed noise with the mean and standard deviation equal to one. SMAP measurement error ranges from 0.03 to 0.045 m3 m-3 across the domain, which was estimated from the SMAP ground validation studies (e.g., Colliander et al., 2017; Chan et al., 2017), and its spatial distribution was set to be proportional to the leaf area index (LAI) (denser vegetation cover corresponds to larger SMAP error). The API state was directly perturbed by zero-mean Gaussian noise to represent API model error. The perturbation variance was set to 0.3 mm2 over the entire domain such that the normalized filter innovation has variance of approximately one (which is a necessary condition for the proper parameterization of a Kalman filter; see Mehra, 1971, and Crow and Bolten, 2007). The SMAP measurement error and the state perturbation variance are the two primary variables impacting innovation statistics. Since we had a relatively good estimate of the measurement error, the state perturbation level can be uniquely determined via an analysis of normalized innovation variances (Crow and van den Berg, 2010).

As illustrated in Fig. 1 (the red box on the left), the SMAP SM retrievals were also assimilated into the VIC model to update model states using an EnKF. The EnKF implementation in this study generally follows Mao et al. (2019). Specifically, a 1-D filter was implemented for each 36 km SMAP pixel separately and at each pixel SMAP was assimilated to update the SM states of multiple underlying finer 1/8∘ VIC grid cells. Resolution differences between the coarser assimilation observations and finer modeling grid were accounted for via the inclusion of a spatial averaging step within the observation operator (Mao et al., 2019). Following Lievens et al. (2015, 2016) and Mao et al. (2019), only the upper two layers of SM states in VIC were updated by the EnKF, although the bottom-layer SM does respond to the update of the upper two layers through drainage (see Sect. S2 for mathematical details of the EnKF implemented here). An ensemble of 32 Monte Carlo model run ensembles was used for the EnKF.

The SMAP retrievals were rescaled (separately for AM and PM retrievals) to match the 2.5-year mean and standard deviation of the VIC-simulated surface-layer SM time series prior to assimilation. The error statistics of IMERG precipitation and unscaled SMAP retrievals were assumed to be the same as those applied in SMART (Sect. 2.4.1). Following Mao et al. (2019), VIC SM states were directly perturbed during the EnKF forecast step by zero-mean, additive Gaussian noise with a standard deviation of 0.5 mm over the entire study domain. This noise represents uncertainty in VIC's ability to propagate states estimates forward in time (note that the bottom-layer SM was perturbed, even though not directly updated by EnKF, to create a realistic ensemble spread for probabilistic estimates of baseflow and, thus, streamflow).

Although VIC modeling errors are likely spatially auto-correlated, we tested whether accounting for spatial correlation improved filter performance. Since it did not significantly improve the results, we did not account for spatial correlation in our EnKF implementation. This finding is consistent with Gruber et al. (2015) who described the limited benefit of 2-D filtering, versus a 1-D baseline, when assimilating distributed SM retrievals into a land surface model. We will further discuss this point in Sect. 4.

The ensemble of updated model states and the corrected rainfall forcing were combined to produce final streamflow estimates (black box in the bottom of Fig. 1). We first randomly paired ensemble members of corrected rainfall and updated VIC states and selected 32 such pairs to balance competing considerations of computational cost and statistical stability. For each pair, the VIC model was rerun with the updated states inserted sequentially over time and forced by the corrected rainfall. Other meteorological forcings were kept unchanged. The runoff output from VIC for each pair was then routed to the gauge locations, resulting in an ensemble of basin-outlet streamflow time series. To further separate the relative contribution of the state update and the rainfall correction schemes to overall streamflow improvement, two additional streamflow simulations were performed. The first was the “state-updated streamflow” case, where VIC was rerun with the updated states and forced by the original IMERG precipitation. The resulting streamflow reflects only the impact of state updating on streamflow simulations. The second was the “rainfall-corrected streamflow” case, where VIC was forced by the SMART-corrected rainfall ensemble but without inserting the updated states. The resulting streamflow reflects only the effect of SMART rainfall correction.

The EnKF state update and SMART rainfall correction schemes were executed independently to minimize the risk of cross-correlated error (Crow et al., 2009). In particular, note that VIC state estimates created using SMART forcing – see the black “Hydrologic prediction” box in Fig. 1 – were not fed back into the EnKF state update analysis. Nevertheless, cross-correlated error in (EnKF) state and (SMART) rainfall estimates potentially may still be present since the two schemes are informed by the same SM measurement time series. Such cross-correlated error could, in turn, degrade the quality of probabilistic streamflow estimates. In fact, due to this concern, Massari et al. (2018) intentionally avoided combining the state and rainfall correction schemes. To further investigate this risk, we performed a set of synthetic experiments where we compared probabilistic streamflow estimates obtained via the following two scenarios: (1) a single set of synthetically generated SM measurements assimilated into the state and rainfall correction schemes, mimicking the original dual-correction system and (2) two separate sets of SM measurements with mutually independent errors assimilated separately into the two schemes, thereby explicitly avoiding error cross-correlation in the system. Results show that the two scenarios achieve very similar streamflow correction performance and, therefore, minimal risk of degraded streamflow estimates (see Sect. S5).

Figure 3 shows the rainfall improvement in terms of correlation coefficient r based on both an EnKS- (the left column) and EnKF-based (the right column) implementation of SMART. For EnKF results, both δ and P in Eq. (2) were aggregated to windows prior to correction every 3 d to ensure SM data availability in every correction window (and the every 3 d correction was subsequently downscaled to every 3 h time steps uniformly). Overall, the EnKF implementation results in less r improvement than the EnKS implementation, which confirms the benefit of applying SMART using a smoothing approach.

The impact of our (previous choice) to update rainfall only at nonzero IMERG time steps is examined via domain-median categorical metrics (Fig. 4). When we correct rainfall every time step (Fig. 4, Column 1), FAR is largely degraded (by 0.1–0.4) at low rainfall event thresholds especially with shorter accumulation periods (every 3 h and every 1 d; see Fig. 4a). This is likely due to SMART misinterpreting SM retrieval noise as small rainfall events (Chen et al., 2014). POD is improved at these low thresholds (Fig. 4b), but it is not enough to compensate for the large FAR degradation. Therefore, TS, which accounts for both false alarms and missed events, is also degraded at low thresholds (by as large as 0.2 at every 3 h). In contrast, when we only correct rainfall at nonzero IMERG time steps (Fig. 4, Column 2), the FAR degradation is much less (note the different y axes in the two columns in Fig. 4). While this approach does sacrifice POD at low thresholds (Fig. 4e), the overall TS for every 1 and 3 d aggregation is improved for most event thresholds, especially the higher ones. As mentioned in Sect. 2.4.1, one possible reason for the success of this SMART choice is the improved rain/no rain detection quality of the baseline IMERG precipitation product, which was found to have an improved miss-rain, false-rain, and hit rate relative to older TMPA-RT (Real-Time TRMM Multi-Satellite Precipitation Analysis) products over the contiguous US (CONUS; Gebregiorgis et al., 2018). It is thus beneficial to retain the IMERG rain/no rain detection skill and not subject it to SMART-based correction.

With regards to binary rain/no-rain determination, one strategy for mitigating FAR problems is to arbitrarily set a (greater-than-zero) minimum accumulation threshold that must be exceeded for an event to be registered. To this end we carried out a sensitivity analysis to examine the impact of using a nonzero rain/no rain threshold versus our baseline assumption of a zero threshold. However, this analysis was unable to isolate an optimized threshold value for distinguishing rain/no rain cases. Instead, a continuous trade-off exists between POD and FAR at different rainfall thresholds. However, a zero rain/no rain threshold does appear slightly beneficial for PER and the correlation coefficient improvement (see Sect. S6).

After rainfall correction at every 1 and 3 d accumulation periods, PER exhibits a domain-median error reduction of ∼8 % (Fig. 5, Column 1). The PER improvement is consistent with the improvement of the categorical metrics at high-event thresholds (Fig. 4, Column 2), since PER is more sensitive to high rainfall values. PER every 3 h shows little improvement (Fig. 5a), suggesting that the deterministic correction is more effective at an accumulation period that more closely matches the SMAP retrieval interval. The same finding can also be drawn from the correlation and categorical results (Fig. 3, Column 2 and Fig. 4, Column 2). Overall, the correlation coefficient improves more in the western part of the domain, which is likely attributable to the better quality of SMAP retrievals in the lightly vegetated western portion of the basin. However, RMSE is reduced more in the eastern part of the domain, which is likely due to the increased frequency of large rainfall events in this region and SMART's tendency to be more effective for the correction of moderate-to-large precipitation events. Note that SMART rainfall correction cannot be evaluated in terms of overall bias, since – like all SM data assimilation systems – the SMART algorithm rescales the corrected time series back to the uncorrected mean prior to its evaluation.

The probabilistic metric NENSK (Fig. 5, Column 2) is less than one for most of the domain at an every 3 h time step, indicating an overdispersed ensemble on average. However, when evaluating at every 1 and 3 d accumulation periods, NENSK is closer to one, indicating a better representation of the uncertainty of the rainfall estimates. As we aggregate over longer accumulation windows (e.g., every 3 d), NENSK becomes slightly greater than one (i.e., underdispersed ensemble), since the SMART algorithm assumes only a random rainfall error but no systematic bias. As a result, it slightly underestimates the uncertainty range over longer-term periods. Ensemble rainfall tends to be underdispersed on the west edge of the domain with low rainfall, indicating that we are underestimating rainfall uncertainty in this region.

In summary, SMART successfully uses SMAP SM retrievals to correct IMERG rainfall during relatively larger events, with slight-to-moderate deterministic improvement. However, SMART correction is less successful for small rainfall events and can even lead to slight degradation. The correction is more effective, and the ensemble representation is better, when rainfall estimates are temporally aggregated to periods consistent with SMAP retrieval intervals (i.e., every 1 to 3 d accumulation periods).

The final daily streamflow performance from the dual-correction system is listed in Table 2 (the “dual” columns) for each sub-basin. Overall, streamflow estimates are improved but with large variability across sub-basins. Specifically, PER ranges from approximately 6 % to 34 %, and KGE improvement ranges from slightly negative to +0.95 across all sub-basins. For sub-basins with better baseline streamflow performance (as measured by KGE, i.e., the Ninnescah, Walnut, and Chikaskia sub-basins), the relative improvement after the dual correction is generally smaller.

Table 2 also summarizes the streamflow improvement from each of the correction schemes alone (i.e., the “state update only” and “rainfall correction only” columns). For sub-basins with relatively better open-loop model performance, the contribution of state updating generally surpasses that of rainfall correction. Conversely, at sub-basins with relatively poorer open-loop model performance (i.e., the Bird, Spring, Illinois, and Deep sub-basins), streamflow improvement is primarily attributable to the SMART rainfall correction.

To further understand the relationship between open-loop simulation performance, rainfall forcing error, and correction performance, we forced the VIC model by the NLDAS-2 benchmark rainfall (without state update). The subsequent streamflow improvement level is assumed to approximate the maximum improvement achievable via rainfall correction alone (Table 2 “NLDAS-2 forced” columns). While almost all sub-basins show streamflow improvement simply by switching to NLDAS-2 rainfall forcing, the improvement is especially large for sub-basins with poorer open-loop streamflow estimates. In these basins, PER is over 65 %, and the negative KGE for the open loop case improves to near zero or positive values for the NLDAS-forced case. This suggests that, despite advances in the quality of remotely sensed rainfall data products, poor open-loop streamflow simulations at these sub-basins are still largely attributable to poor-quality IMERG rainfall forcing error. In these basins, the SM-based rainfall correction scheme can potentially play an important role in improving VIC streamflow estimates. Unfortunately, this potential is not always realized. Note how the SMART-based rainfall-correction-only case generally fails to match the NLDAS-forced case in the Spring, Illinois, and Deep sub-basins (Table 2). This is likely because these basins are located in relatively high-biomass areas where SMAP retrievals (and thus SMART corrections) are less accurate.

In contrast, the sub-basins with better open-loop streamflow results (i.e., the Ninnescah, Walnut, and Chikaskia sub-basins) demonstrate less streamflow improvement when switching to the NLDAS-2 rainfall forcing. The sub-basin with the best (IMERG-forced) open-loop streamflow results, Chikaskia, even experiences a small degradation when forced by the NLDAS-2 rainfall (Table 2). This suggests that the NLDAS-2 benchmark rainfall at this sub-basin is not obviously superior to the IMERG baseline. Nevertheless, SMART is still able to extract information from SMAP and slightly correct IMERG rainfall and subsequent streamflow estimates.

The dual-correction scheme presented in this study is designed to correct only the random error present in a hydrologic simulation system. It does not correct systematic error or overall bias. Figure 6 shows example time series of the open-loop, USGS-observed and dual-corrected streamflow at three sub-basins (the Chikaskia, Deep, and Illinois) with various levels of open-loop performance. Although the dual system often nudges the simulated streamflow in the correct direction (especially during high-flow periods) and results in overall improved evaluation statistics, systematic error (in the model process representation as well as rainfall forcing) clearly exists. This systematic error, although difficult to quantify, cannot be corrected by the data assimilation approach discussed here. The NENSK statistic partly reflects such systematic error. NENSK is significantly above one at most sub-basins, indicating an underdispersed ensemble on average. In other words, at most sub-basins the ensemble spread created by the dual system only represents the random uncertainty around the open-loop streamflow and neglects systematic error that accounts for a significant fraction of total streamflow error.

The level of systematic error is tied closely to the quality of the hydrologic model parameters often estimated through calibration. The VIC parameters used in this study were taken from Maurer et al. (2002) and derived based on streamflow at the outlets of large basins. To further examine the effect of systematic error on data assimilation, we calibrated the model parameters for the eight sub-basins separately using streamflow acquired from the USGS (Table 1). Specifically, VIC parameters that control infiltration, soil conductivity, and baseflow generation as well as the recession rate of the grid-cell-scale unit hydrograph in RVIC were calibrated using the multi-objective autocalibration method (MOCOM; Yapo et al., 1998). Basin-constant parameters were calibrated toward USGS streamflow time series during 2015 to 2017 (forced by the baseline IMERG precipitation) to optimize daily KGE and monthly bias. Only a subset of the eight sub-basins achieved better-than-open-loop streamflow results via this traditional calibration method, due mainly to the relatively large IMERG forcing error present in some sub-basins that prevents the calibration scheme from finding an improved parameterization. Figure 7 shows three example sub-basins (i.e., Chikaskia, Deep, and Illinois) with relatively good calibration outcomes. Comparing Fig. 7 to Fig. 6, we observe that the streamflow improvement achieved by parameter calibration (i.e., systematic error reduction) alone is as important as, or more important than, that achieved by data assimilation (via random-error reduction) in all three sub-basins. In both cases (i.e., the default and calibrated VIC parameters), NENSK is significantly above one, indicating that we underestimate the streamflow simulation uncertainty when only random errors are considered.