The Uganda Red Cross Society, with support from the German Red Cross and the Red Cross Red Crescent Climate Centre, is implementing a forecast-based financing pilot in the north-eastern part of the country. As part of this pilot, the German Red Cross established a novel Preparedness Fund that can be disbursed to take predefined preparedness actions when a triggering forecast is issued in this region. At the time of writing, there are more than a dozen such forecast-based financing projects operational globally.

The Teso region of north-eastern Uganda is a swampy region, prone to river flooding and waterlogging during the two rainy seasons centred in May and October. The Uganda Red Cross Society project areas are in the sub-districts of Magoro and Ngariam in Katakwi district on the Apapi River, and Kapelebyong in Amuria district on the Akokoro River (see Fig. 1). Unfortunately, there is no calibrated hydrological model available for these rivers. Both rivers drain into Lake Bisina and eventually into the Nile.

The Uganda Red Cross Society selected this pilot region based on vulnerability to floods. As regional conflict subsided in the 1990s and 2000s, this region was gradually re-settled, and nowadays many of the current residents practice farming and raise livestock. Since that time, several flood events have impacted the area. The floods typically cause impassable roads, loss of crops, outbreaks of waterborne diseases, and collapse of houses and latrines (OSSO and LA RED, 2009).

Whenever a flood is reported, the Uganda Red Cross Society has a mandate to assess the situation and respond. In past events such as the 2007 floods, they have provided post-disaster shelter and relief items to the affected population (Jongman et al., 2015). Both the flood losses and the disaster response expenses could be reduced if anticipatory measures were deployed before the flood, after unusual conditions are forecast. Based on the methodology articulated in this paper, forecast-based financing thresholds were operationalized in mid-2015, consisting of standard operating procedures for forecast-based action. In November 2015, a “triggering” forecast successfully initiated action (Red Cross Red Crescent Climate Centre, 2015). This was the first time the local branch had used a preparedness fund to take action before flood disaster reports were issued, and while the impacts are still being analysed, the region reported flooding after the trigger had been reached in one of the project areas.

To set up the forecast-based financing system to initiate early action, the Uganda Red Cross Society project team identified preparedness actions that could be taken prior to a flood event, through consultations with people living in the affected areas as well as internal discussions and two facilitated workshops. Participants in the workshops included disaster managers, volunteers, the Uganda Meteorological Authority and district officials (Jongman et al., 2015). In each of the workshops, the disaster managers from the Uganda Red Cross Society discussed the quantitative and qualitative costs and losses associated with three scenarios: (1) taking successful action, (2) failing to act before a flood, and (3) acting “in vain”. For each action, they first answered questions individually before discussing collectively. Lastly, disaster managers estimated their willingness to “act in vain”, expressed as a number of times out of 10.

Ultimately, the team selected a set of actions that were seen as both impactful and implementable by the Uganda Red Cross Society. One action (evacuation) was eliminated because about one-quarter of the respondents indicated that they would not be willing to act in vain at all. The political and reputational costs of evacuating in vain are considerable. The remaining selected actions are specified in Table 1. For these three actions, the disaster managers came to a consensus that they would be willing to act in vain approximately 50 % of the time. Here, we use this as the “tolerable” amount of acting in vain to establish the forecast-based financing system for this set of actions. Later in the paper, we estimate the probability that the GloFAS forecast triggers are an “intolerable” system, or one that causes disaster managers to act “in vain” more than 50 % of the time.

For each action, the Uganda Red Cross Society specified how many days would be needed to carry out the action, which should correspond to the forecast lead time (Jongman, 2015). The specified lead times are contingent on the assumption that several of the procurement and volunteer training steps would be carried out at the beginning of the flood season, to enable quick action based on a short-term forecast.

Secondly, they identified the “action lifetime”: the period of time after the action is completed during which it offers preparedness or protection from the extreme event. Traditional flood forecast evaluations are specific to the time period forecasted, evaluating whether a single forecasted day did indeed flood. Humanitarians would count this as a “hit” and, unlike forecasters, they would also consider it to be a “hit” if the flood instead occurred 5 days after the forecasted date and the action lifetime was 30 days. In such a case, the action would still be effective in reducing impacts, even though the flood occurred slightly later than the forecasted date.

Therefore, the methodology detailed in this paper avoids re-triggering an action if the “action lifetime” of a previous action is still ongoing. For example, after digging drainage, the team would not re-trigger digging of trenches until the first set of trenches could be assumed to have degraded, likely about 90 days after digging. While the end of the “lifetime” is not a strict transition from useful to non-useful, it is an estimation of the date at which the Uganda Red Cross Society would find it acceptable to re-trigger the action in the region. We posit this constraint throughout the paper; an action cannot be re-triggered until the action lifetime of the preceding action is over.

The selected actions are listed in Table 1 (note that this is a subset of all actions that were originally considered).

This paper proposes a method for identifying a forecast that could trigger one or more of these actions before a potential flood, given the constraint of acting in vain less than 50 % of the time. As mentioned earlier, there is no locally available flood forecasting system, and there is only one river gauge with recorded discharge in the pilot area. Unfortunately, the large upstream catchment area dictates that rainfall in a specific village is not a useful proxy for flood risk in that village. Given these constraints, we choose to examine whether river discharge forecasts from the Global Flood Awareness System (GloFAS) can be used to trigger action in this data-scarce location in ways that are compatible with stakeholder priorities. Probabilistic hydrometeorological forecasts have been evaluated globally, and have been shown to have limited skill (e.g. Alfieri et al., 2012; Li et al., 2008; Wu et al., 2014).

GloFAS is an operational global ensemble flood forecasting system developed in partnership between ECMWF, the European Commission Joint Research Centre, and the University of Reading (Alfieri et al., 2013). Currently in a pre-operational development phase, calibration of the model with river flow observations, where available, is being carried out in a research mode. The model version used here is not calibrated for the north-eastern Uganda catchments. GloFAS is run once a day to produce probabilistic discharge estimates over the entire globe at a resolution of 0.1∘ (approximately 11 km at the Equator). Here, we use daily historical GloFAS forecasts from 2009 to 2014, as well as gauge data from the (only) local Akokoro gauge from 2009 to 2013, which overlap for 2014 days. The gauge is located at approximately 1.86∘ N, 33.85∘ E.

GloFAS is driven by the ECMWF ensemble forecasting system, with 51 ensemble members at lead times of 0 to 45 days. The first 15 days include rainfall forecasts, and the following days are river routing only. The probabilistic flood forecasts are available free of charge on a password-protected website (http://www.globalfloods.eu/). GloFAS takes a “model climatology” approach, aiming to forecast extremes or anomalies in river flow relative to historical “climatology” runs of the model (Hirpa et al., 2016). This approach addresses the problems of the lack of representation of local-scale channel geometry and bias in the precipitation forcing. However, one of the major challenges is to link the model climatology to the real world, focusing on the percentiles rather than absolute values of the forecast.

While the relationship between water levels and flood risk will vary over time due to trends in vulnerability and exposure, here we define a percentile of discharge that is qualitatively associated with reported flood events of the past few years, when avoidable losses were observed. For flood reports, we use two sources of information: humanitarian records and media reports.

With regards to the humanitarian records, we combine records from the Desinventar database (UNISDR, 2011) with an internal record system of disasters that are reported to the Uganda Red Cross Society. Between the two humanitarian data sets, they contain eight distinct records of floods in the Magoro area from 2009 to mid-2014; these floods occurred in 2010, 2011, and 2012.

For the media analysis, we analysed two national Ugandan newspaper repositories: Daily Monitor and New Vision. We filtered each repository with 40 flood-related keywords

The keywords are flood, floods, flooding, inundation, inundations, landslide, dam break, dam burst, dam bursting, dam breached, dam fail, dam failed, dam failing, dam failure, dam broken, dam collapse, dyke break, dyke burst, dyke bursting, dyke breached, dyke fail, dyke failed, dyke failing, dyke failure, dyke broken, dyke collapse, embankment break, embankment burst, embankment bursting, embankment breached, embankment fail, embankment failed, embankment failing, embankment failure, embankment broken, embankment collapse, submerged, overflowed, breach, and water-logging.

Within the database total of 3726 news articles, we clustered the sentences in the articles using a K-means clustering algorithm (Hürriyetoglu, unpublished; Kaufman and Rousseeuw, 1990). Next we annotated the clusters using four classes: 1. Current flood event; 2. Past event or flood warning; 3. Mixed; and 4. Unrelated. After annotation we found that a total of 1721 news articles held relevant flood information (annotated as class 1 or 2). To obtain geographical information, we filtered the sentences for any “marker” terms that are often used when the writer specifies a location

They are affected NOT not, hit NOT not, situation AND bad, situation AND worse, situation AND worst, cut off, displaced, destroyed, submerged, and collapsed

With these results from the algorithm, we validated the result manually for the districts of our interest by reading the articles. For 85 % of the events we had found an actual flood event described in the text, meaning that the flood event was automatically detected for the correct month/year in the correct location(s). Conversely, 15 % were false positives, meaning the text was describing a non-flood event. The result of this data mining of the news repositories is a historical flood overview with dates of flood occurrences in Katakwi district (it can be accessed here: https://www.floodtags.com/historic-floodmap-uganda). There are 13 newspaper reports of flooding within our time series.

While this accounts for many events, not all disasters are included in these databases, and some of those included may have had less impact than others. The effect of this under-representation is an overestimation of acting in vain, which renders our trigger selection conservative. In addition, impact is not perfectly correlated with flood magnitude, given that vulnerability can change over time. Therefore, we only attempt a qualitative comparison of discharge and reported flood events, which adds additional (unquantified) uncertainty to the calculation of false alarms in the following section.

As we do not have a gauge for the Apapai River, where Magoro and Ngariam sub-counties are located, we use the daily ensemble median of GloFAS forecasts at a lead time of 0 as a proxy for actual discharge and compare this with the above data sets of reported disasters in those two locations. We qualitatively select a threshold percentile of discharge to be considered the “danger level” or “flood” for this region, rather than an absolute value. The exact percentile is a subjective selection to approximate the base rate of reported floods, ideally including the maximum number of exceedances that were indeed followed by a reported flood event.

Using this selected “danger level” percentile as a proxy for the amount of discharge that causes a flood, we calculate what probability of exceeding the danger level should trigger action. Here, we calculate probabilities using the forecast ensemble, evaluating them against the gauge discharge.

The forecast verification score of interest to humanitarian actors is the false alarm ratio (FAR) (Hogan and Mason, 2012; Lopez et al., unpublished), defined as the number of forecast-based actions that were not followed by a flood, divided by the total number of actions that were triggered by the system. It thus represents the proportion of actions that are taken “in vain”. Here, we take into account the action lifetime, so any action that was followed by a flood during its lifetime is considered a “hit”, and only actions that have no flooding at any point during their lifetime are “in vain”. Similarly, a second action is never triggered during the lifetime of another action.

To estimate the FAR, we compare the nearest grid box of the GloFAS forecast with the river gauge on the Akokoro River, for which we have an overlapping time period of daily data from 10 January 2009 to 31 December 2014. The correlation of these two data sets is 0.52. In the context of a forecast-based financing system, the Red Cross or other humanitarian actors will take action when the forecast meets or exceeds a triggering probability of flooding. The FAR is therefore calculated as follows: (1) any forecast meeting or exceeding the trigger probability is considered an action; (2) any action followed by a flood within the action lifetime of 30 days is counted as a “hit”, otherwise an action “in vain”.

Our first goal is to estimate whether a forecast indicating a 50 % chance of flooding would indeed correspond to a 50 % chance of flooding in the real world. We plot reliability diagrams (Broecker, 2012) for the forecast at 4- and 7-day lead times at the gauge location, as well as the GloFAS forecasts in the two non-gauge project locations, comparing 4-day lead-time forecasts with the 0-day forecasts to approximate actual discharge. However, in such a small sample, the incidence of forecasts of rare events is low, and therefore the confidence intervals in these reliability diagrams are very wide.

Given such a small number of years to calculate the performance of forecasts with regards to extreme events, however, we cannot be sure that the estimate of FAR in the sample is representative of the true value. For example, if the estimate from our sample yields a FAR of 30 %, it is still possible that the real value is actually greater than 50 %. This means that, in reality, the selected trigger level for our forecast-based financing system would cause the Uganda Red Cross Society to act “in vain” more than 50 % of the time, which is not considered “tolerable”. To estimate the risk of setting up an “intolerable” system, we calculate confidence intervals around the FAR by using bootstrap resampling. To account for the autocorrelation of the discharge time series, we use a 60-day fixed block bootstrap to generate 10 000 samples by resampling with replacement the time series (n=2014) of forecast–observation pairs. Given a trigger forecast probability, for each sample we calculate the FAR and generate a distribution of all the sample FARs. This is repeated for each trigger probability, and we demonstrate results for three triggers: forecast probabilities of 30, 50, and 70 %. Based on these results, we estimate the likelihood that taking action when one of these forecast probabilities is exceeded will lead to a FAR above 50 %, which would fail to satisfy the decision-maker requirements for action “in vain”.

To estimate the percentile of discharge that is associated with flooding in the project region, we plot the historical median water levels forecasted at a lead time of 0 by the GloFAS model. Here, we focus on the Apapi River, where two project districts are located and several disaster records exist. Because Ngariam sub-county is directly upstream of Magoro sub-county, we plot simulated discharge at Magoro and reported flood events in both sub-counties (Fig. 2). Comparing this with historical floods (dark blue lines), and media reports from the district (light blue lines), we qualitatively select the 95th percentile (horizontal red line) as a proxy for disaster.

In the 6 years of 2009–2014, this danger level would have been exceeded in 2010, 2011, and 2012 (see Fig. 2). In April 2010, reports indicate that 12 secondary schools and 7000 people were affected by flooding in the area, followed by crop losses due to waterlogging in May 2010. Flooding continued to be reported through to September and October of that year, affecting several regional roads. This corresponds well to the simulated discharge for those years.

In 2011, simulated discharge again accords well with reported flooding that affected both people and infrastructure in the area. In 2012, waterlogging reports to the Uganda Red Cross Society arrived in August, which is substantially after the peak modelled discharge, and the newspaper reports are concentrated in October and November. It is possible that the peak discharge did correspond to the model data and was not reported, or was reported at a later time. Our threshold was not crossed in 2013 and 2014, which accords well with the lack of reported floods in those years.

We begin to see other years (with no disasters) counted as “floods” if we lower the danger level below 93 %, and if we raise it above the 99th percentile very few years exceed the threshold. Therefore, we assume from the limited data available that discharge above the 95th percentile is likely indicative of flood conditions in this location, and in the following analysis, anything above the 95th percentile is defined as a “flood”. In Kapelebyong sub-county, the other project location in this region, the only recorded disasters are from the devastating floods of 2007, which are not available in GloFAS reforecasts. Therefore, we also assume this percentile applies to Kapelebyong, as the infrastructure and vulnerability are similar in the two areas.

We consider forecasts of 4-day and 7-day lead times, aiming to identify a trigger that corresponds to a FAR of 0.5 or less. If the forecast probability of exceeding the flood danger level is 50 %, then the observed frequency of exceeding the flood threshold should be 50 % for a reliable forecast.

In Fig. 3, we plot the reliability of the forecast at both lead times when compared to gauge discharge on the Akokoro River. In the project locations with no gauge, we also examined the ability of GloFAS to forecast itself 4 days in advance (Fig. 3, right-hand reliability diagram). In both cases, we are unable to establish the reliability with confidence given the small sample size.

If we set the trigger given these limited data, how likely is it that we developed a system that is “intolerable” to the Uganda Red Cross Society, actually leading disaster managers to act “in vain” more than 50 % of the time? Figure 4 shows the FAR from 10 000 resamples as a probability distribution function. This assumes that action is triggered when the forecast probability of flooding reaches or exceeds 30, 50, or 70 %, and that there is a 30-day action lifetime.

The bootstrapped results indicate a high chance of a “tolerable” system, especially at higher forecast triggers. Only 24, 19, and 18 % of all the bootstrapped samples returned an “intolerable” system (grey bars) for a threshold of 30, 50, and 70 %, respectively. This is true for a sample size of 2014 days. This represents the chance that the system does not pass the required specifications, and would cause humanitarian actors to act “in vain” more than 50 % of the time in the long run. While increasing the forecast trigger does reduce this risk, the effect is not substantial given the small data set available.