Over 80 % of Canadian farms are concentrated in the CPE – i.e., southern portions of Alberta (AB), Saskatchewan (SK), and Manitoba MB;. The CPE has some of the world's highest climate and weather variability. It is predominately continental with long, cold winters, short, hot summers, and relatively low precipitation amounts during the short growing season of May to September . The annual mean precipitation is around 478 mm, of which rainfall accounts for almost two-thirds of it during the growing season, and snowfall makes up another 30 % of it. Average winter and summer temperatures are -10 and 15 ∘C, respectively . Such variabilities significantly affect the CPE's agriculture, environment, economy, and culture yearly . For example, the drought of 2001–2002 cost approximately USD 3.6 billion in agricultural production losses . Between 2008 and 2012, federal–provincial disaster relief payouts for climate-related events totaled more than USD 785 million and more than USD 16.7 billion in crop insurance. The 100-year record-breaking drought in 2017 caused massive wildfires, reduced yields (particularly canola), heat stress, poor grain fill, livestock feed shortages, and the relocation of nearly 3000 cattle in Saskatchewan and Alberta . The vulnerability of the CPE to agricultural production risks and the future scenarios of climate, which show more severe and frequent droughts with declining precipitation trends and surface water resources during summer and fall, makes the region ideal for developing and testing robust crop NI methodologies.

A total of four study farms, on average 160 ha each, were selected within the provinces of SK and MB (Fig. ). These farms are sites for other soil moisture core validation networks, such as the Agriculture and Agri-Food Canada (AAFC) RISMA (Real-Time In-Situ Soil Monitoring for Agriculture) network and the Kenaston Network in Saskatchewan for NASA Soil Moisture Active Passive (SMAP) validation . All four farms are rainfed and have alternating crop years . Farmers use pasture, spring wheat, shrubland, and other cover crops to avoid farrow and water-logged conditions in spring. Depending on field and weather conditions, planting typically occurs in late April and early May. For this study, we consider a fixed 7-month window for the growing season from 1 April to 31 October. Table  shows that, between 2015 to 2019, at least seven different crops were planted on the four study farms. Most crops were canola and spring wheat, although there were also soybeans, oats, barley, peas, and lentils. These crops have low to medium sensitivity to drought, and their root depth at maximum growth is anywhere from 0.6 m (lentils and soybeans) to 1.5 m (canola, barley, and spring wheat). The average crop water needs through the growing season are 550 mm; much less was provided by rain, as shown in Table  . Table  shows the amount of precipitation during and outside the growing season. Precipitation outside the growing season is primarily snow. Wind plays a critical role in moving and blowing snow. Therefore, the contribution of melting snow toward meeting future crop water requirements is not substantial and not considered in the FarmCan model. However, establishing soil water reservoirs or having stubble fields can improve snow contribution to SM in the future. Comparing PET with the total annual precipitation, we expect to confirm that the amount of water supplied by precipitation is insufficient to meet optimum crop growth.

Each farm's growing season aridity index (P/PET) is shown in the last column of Table . This index is used across the globe to represent vegetation's biogeographical distribution and estimate crop yield . Based on the aridity index, Manitoba farms have a higher expected crop yield than Saskatchewan farms.

The two main requirements for the datasets to develop FarmCan are the (1) availability of at least 5 years' worth of NRT remotely sensed data and (2) accessibility of such data in real time. These two factors would make the FarmCan algorithm trainable and updatable daily. Various datasets were considered, such as leaf area index (LAI) and the ET from the NASA ECOSTRESS satellite ECOsystem Spaceborne Thermal Radiometer Experiment on Space Station;, but they did not meet one or both of the requirements. On the other hand, MODIS (Moderate Resolution Imaging Spectroradiometer) ET and PET products are available and accessible in NRT at 500 m pixel resolution. The Multi-Source Weighted-Ensemble Precipitation (MSWEP) can provide global P values with a 3 h 0.1∘ resolution covering the period 1979 to the near present as an NRT or forecasted product. PET, ET, and P are critical predictors of crop water stress that link the water–energy–carbon cycle . SMAP SM products (surface and root zone) are also available and accessible in NRT and provide a highly accurate descriptor of crop stress globally . SM is a direct measure of agricultural drought . RZSM becomes important during particular growth stages (mid season and late season) and affects crop growth at maturity stage and final crop yield . The inclusion of SM as a dynamic parameter within crop water stress numerical modeling has improved forecast capabilities . The datasets used in this study are listed in Table .

The SMAP satellite was launched in 2015, and the data are available from 31 March 2015 to the present. SMAP level 3 SM (0–5 cm; SPL3SMP) is a composite based on daily passive radiometer retrievals of global land SM in the top 5 cm of the soil that is resampled to a global, cylindrical ∼36 km Equal-Area Scalable Earth Grid, version 2.0 (EASE-Grid 2.0). For this study, we used version 4 of SPL3SMP retrievals from the morning overpasses to minimize uncertainties and bias from the in situ data .

The SMAP level 4 (SPL4SMAU) is a daily global RZSM product (0–1 m) obtained by assimilating low-frequency (L-band) microwave brightness temperature observations (for which SPL3SMP is the gridded version) into the GEOS-5 catchment land surface model CLSM;, which is driven by surface meteorological data from the NASA Goddard Earth Observation System (GEOS) weather analysis . Additional corrections using gauge- and satellite-based precipitation estimates downscale to the model's temporal and 9 km scale .

ET and PET data are derived from MODIS, a modified MOD16A2/A3 Terra version 6 ET/latent heat flux algorithm. The units are 0.1 kg m-2 per 8 d (i.e., 0.1 mm per 8 d), which is the summation of total daily ET through 8 d . The last acquisition period of each year is a 5 or 6 d composite period, depending on the year. The algorithm used for the MOD16 data product collection is based on the Penman–Monteith equation, which includes inputs of daily meteorological reanalysis data along with MODIS remotely sensed data products such as vegetation property dynamics, albedo, and land cover. Provided in the MOD16A2 v006 product are layers for composited ET and PET along with a quality control layer from 1 January 2001 to the present. MODIS data are available from 2010 to the present.

MSWEP version 1 (0.25∘ spatial resolution) was released in May 2016 and since then has been applied regionally and globally for modeling SM and ET , estimating plant rooting depth , evaluating root zone soil moisture patterns , evaluating climatic controls on vegetation , and analyzing diurnal variations in rainfall and various other applications . The product blends gauge-, satellite-, and (re)analysis-based P estimates to improve the accuracy of the estimates globally. MSWEP is a global P product with a 3 h 0.1∘ resolution covering the period 1979 to the present. It does not provide a forecast. However, MSWEP V280 is largely consistent with a newer product, MSWX, that offers medium- and longer-term forecasts. Here, we used past dates to build a forecasting tool, so using the MSWEP V280 product was sufficient. For future software development applications, we will use MSWEP combined with MSWX to provide real-time forecasts .

We used the data in Table  in the parsimonious NI model of the FAO as follows: 1NI≈∑PET-∑P-ΔSM, where NI is the volume of water needed to compensate for the deficit between PET as a demand factor, P, and change in soil moisture content (ΔSM or ΔRZSM) as supply factors. All units in Eq. () are in millimeters. To take care of the unseen delays among system components and to reduce errors, we use 8 d composite periods in Eq. () and throughout this study. The use of 8 d is also consistent with the MODIS output data format.

To convert ΔSM or ΔRZSM volumetric values to depth in Eq. (), we multiplied their values by the corresponding depth of the soil mm;. For example, a 0.2 m3 m-3 of the surface SM (in the first 50 mm of the topsoil) is equivalent to 0.2×50=10 mm d-1, whereas the same volumetric soil moisture for the root zone (with a consistent depth of 1000 mm) is equivalent to 0.2×1000=200 mm d-1, meaning that 200 mm of water can be drawn from 1 m deep soil. FarmCan uses 50 or 1000 mm depth, depending on the crop's development stage. When the crop is in stages 1 or 2, the algorithm uses the first 50 mm depth, and when the crop is in stages 3 or 4, 1000 mm depth is used.

RF  is an ML method that has shown high accuracy in the function estimation and nonparametric regression of geospatial hydroclimatic and spaceborne data . The RF algorithm aggregates the predictions made by multiple decision trees of varying subsets called the bagged or bootstrapped datasets. Showing trees different training sets is a way of de-correlating them . It also decreases the variance in the model without increasing the bias, ultimately leading to better model performance. Furthermore, while the predictions of a single tree are highly sensitive to noise in its training set, the average of many trees is not, as long as the trees are not correlated. The FarmCan model uses RF in two stages, i.e. (1) to fill in the gaps of missing ΔSM and ΔRZSM, based on 8 d P from 2010 to 2020, and (2) to predict ΔRZSM, ΔSM, 8 d ET, and 8 d PET up to 14 d in advance. We divided the datasets from 2010 to 2020 into training and testing in a 0.7 to 0.3 ratio. 5f^=1B∑b=1Bfb(x′). The first round of running RF uses 500 decision trees. The optimum number of trees is the one that minimizes the mean square error (MSE) between the training and testing datasets. The second round of running RF involves dictating the optimum number of trees. If a training set X=x1, … xn (n being the number of training samples) has responses Y=y1, … yn, then the algorithm selects random samples with the replacement of the training set for B times. In Eq. (), for b=1, … B training samples from X and Y, called Xb and Yb, we produce a regression tree fb. After training, predictions for unseen samples x′ can be made by averaging the predictions from all the individual regression trees.

Figure  shows the key variables' 20th, 50th, and 80th percentiles from 2015 to 2020 during the growing season (April to October). Comparing P with the ET and PET map shows that, region-wide, crops do not receive the water needed from rain to reach an optimal yield. The growing season's P is typical of sub-humid and semi-arid climates , i.e., the amount of rainfall is often not sufficient to satisfy the water needs of crops. Except for portions of the province of AB, most CPE farming relies on rainfall and, therefore, is vulnerable to agricultural drought .

Most of Saskatchewan is identified by the lowest amount of SM, P, and ET throughout the growing season. Surface SM is generally lower than RZSM across all three provinces. This is expected as soil at the surface is affected directly by transpiration and wind. In contrast, the soil at the root zone holds onto the water longer, especially as brown-black Chernozemic clay, a typical type of soil in the CPE.

We ran a two-by-two Pearson correlation analysis with a 99 % significance level for the four selected farms and during the 7-month growing seasons from 2015 to 2020 (Fig. ).

The four farms' results show that the correlation between P and ΔRZSM and P and ΔSM is quite similar in M1 and M2. However, the correlation between P and ΔRZSM is slightly higher in S1 and considerably higher in S2. Although it is generally expected that instantaneous surface soil moisture shows more variability with P, this study is based on the 8 d cumulative P and changes in 8 d SM and not a direct measure of P vs. SM. For example, if the total amount of P over 8 d is 20 mm, the RZSM can change from 0.2 to 0.9 m3 m-3, which gives a higher ΔRZSM than ΔSM, which might have fluctuated instantaneously but essentially changed from 0.3 to 0.5 over 8 d. However, more studies in different regions must confirm such correlations. We speculate that soil type plays a role in how soil maintains moisture at different depths.

There is also no evidence of significant feedback from ET to SM, and vice versa. This can be because the relationship between SM and ET, in terms of feedback, mainly depends on the climate . During the growing season, the condition in CPE is either too wet, which makes the total energy for ET independent of SM, or too dry, which makes ET show little impact on fluxes because it is little or no moisture available.

Generally, a significant impact of SM on ET should be more noticeable in a transitional regime where soil water supply is available and sufficient . More studies from different regions are required to understand such interactions in SM and ET fluxes.

All four farms show a significant negative correlation between soil moisture values with 8 d NI within 99 % confidence.

To study the relationship between water supply and demand in the CPE, we conducted a three-way comparison of changes in 8 d P supply with variability in ΔSM and ΔRZSM (supply). We also included changes in 8 d ET and 8 d PET (demand factors) in a correlation plot shown in Fig. . Each row represents a province with the supply variables on the XY axes. For each region, the two left-hand plots show the relationship between 8 d P and ΔSM. Color changes correspond with 8 d PET and 8 d ET. The two right-hand plots are the same, except that the Y axis represents ΔRZSM instead of ΔSM.

Manitoba shows the most robust linear relationship between 8 d P and ΔRZSM. In contrast, Alberta shows the weakest linear relationship between 8 d P and ΔRZSM, likely because most Alberta farms are artificially irrigated. ΔRZSM is more responsive to the amount of 8 d precipitation, meaning that, over an 8 d increase in P, RZSM increases. Such a linear relationship is weaker between 8 d P and ΔSM. This can be because surface SM is also affected by exposure to other physiological elements such as wind, elevation, transpiration, and land cover.

There are also visible linear relationships between the 8 d PET and 8 d P, especially in Manitoba and Saskatchewan. The 8 d PET (and less for 8 d ET) tend to increase with higher 8 d P. The 8 d ET and 8 d PET do not show a linear correlation to the ΔSM, although for periods for which 10 mm < 8 d P < 40 mm, 8 d PET values tend to have a positive trend when the ΔSM is decreasing (negative). When 8 d P>40 mm, Saskatchewan and Alberta showed more mid-range PET (20–60 mm per 8 d). This can mean that SK and AB are regularly in dry conditions when the water supply is less than optimum. MB is generally moist but can range from adequate crop water availability to extreme water stress periods. The atmospheric demand is typically low for periods with 8 d P less than 10 mm. In all plots, the average 8 d PET is higher than the 8 d ET, which shows that a higher-than-supplied atmospheric demand exists throughout the growing season at the CPE.

Figure  is the variability plot of Farm S2 from 2015 to 2020. Each year's 7-month agricultural period is shown with a pink background. A negative ΔSM or ΔRZSM means a decrease in SM or RZSM, respectively, over the 8 d intervals, and vice versa.

During every agricultural year, the SM reacts to P with much higher variability and sensitivity than ΔRZSM. Although instantaneous rain is more correlated with the SM, as previous results showed, 8 d P shows a higher correlation with Δ RZSM. In the CPE, ΔRZSM generally reverts to zero, indicating a weakly stationary behavior. However, the amount and timing of daily RZSM can still be insufficient to support effective crop growth. As for surface SM, the changes do not seem stationary. The 8 d PET is consistently higher than 8 d ET, confirming that crops receive less than the optimal amount of their water demand throughout the year. We plotted variability plots for the other three farms (not shown here), and the patterns were consistent with those from Farm S2.

To illustrate the FarmCan real-time forecast process, we describe an example in which 2 July 2020 is “today's date”, the crop type is barley, the planting date is 1 April 2020, and the whole growing season is 150 d. The FarmCan algorithm uses these inputs and the FAO guidelines to provide the expected dates of stages 1 to 4, as shown in Table .

The observed variables are plotted for the assumed date in Fig. a. The total period shown in the plot is 21 d, from 22 June to 12 July 2020. The green bars are the daily precipitation from MSWEP, including the forecast values. The hindcast NI, shown by the gray bars, is distributed by calculating wadju. Because 2 July 2020 corresponds to the third stage of crop development, FarmCan predicts ΔRZSM (instead of ΔSM) and 8 d PET using the RF algorithm. The algorithm then calculates 8 d NI (in mm) for the remaining days shown in Fig. b. Note that the information in Fig. a is repeated in Fig. b. Figure  shows only the predictions for Farms S2, M1, and M2.

For validation, we performed a spatial and temporal generalization test to understand FarmCan's ability to train and predict all the days of crop planting in 2020 and for all of the four study farms using R2, RMSE, and Kling–Gupta efficiency (KGE) parametric tests. The ability of the FarmCan model to generalize the spatial regions (farms) was assessed by comparing these values.

Figure  shows the R2 and RMSE values between the testing and predicted values of NI in all the study farms during the agricultural periods from 2015 to 2020. FarmCan showed the highest R2 between observed and predicted values of 8 d ET, 8 d PET, and 8 d NI and the lowest RMSE for ΔRZSM and ΔSM values. The high R2 and high RMSE for 8 d NI values suggest that the amount of NI might be underpredicted in FarmCan, although the model captured the temporal patterns of water deficiency well. Table  shows the KGE values of 8 d ET, 8 d PET, and 8 d NI for the four study farms.

KGE test is the goodness of fit. Generally, values higher than 0.41 are considered reasonable and with a satisfactory model performance, but there has not been a direct reason to choose this benchmark across all models . We consider 0.5 ≤ KGE satisfactory in this study. The model's goodness of fit is reasonable for ET, PET, and NI. The KGE values of ΔSM and ΔRZSM (not shown) have been zero or very close to zero. Here, the KGE negative values do not necessarily indicate a model that performed worse than the mean benchmark. The reason is that the range of Δ values of SM and RZSM was relatively small (approx. [-0.87, 0.03] m3 m-3), making the values very sensitive to the statistical tests. For the same reasons, it was expected that ΔSM and ΔRZSM did not show a good correlation, although they showed the lowest RMSE values (Fig. ). Given the satisfactory performance in final NI calculations, ΔSM and ΔRZSM predictions did not negatively affect the model and NI.

Generally, there is inherent uncertainty in FarmCan forecasts since we cannot know the actual value of the water deficiency and other controlling factors that maximize the crop yield. However, despite the unknowns, FarmCan showed an effective prediction capability to improve our understanding of NI and some of its main controlling factors in the CPE.

With the FarmCan model, we can select any number of CPE farms from airborne imagery, retrieve their spaceborne data, and forecast each farm's crop-specific water supply and demand to calculate water deficit. This tool is versatile enough to allow access to any farm's critical hydroclimatic information for the best water-related decision-making without stepping into the farm or setting up expensive monitoring equipment.