Impact of cry wolf effects on social preparedness and the efficiency of flood early warning systems

Yohei Sawada, Rin Kanai, and Hitomu Kotani

Download

Final revised paper (published on 18 Aug 2022)
Supplement to the final revised paper
Preprint (discussion started on 27 Oct 2021)

Interactive discussion

Status: closed

RC1:
'Comment on hess-2021-497', Anonymous Referee #1, 07 Dec 2021
This paper presents the improvement of an existing socio-hydrological model on the interactions between flood forecasting and flood loss by including social collective trust. The manuscript uses the model to investigate the cry wolf effect (where individuals may be less likely to implement protective measures if they have experienced false alarms). I believe including trust and investigating its role is a relevant contribution to the socio-hydrological literature. The manuscript shows an interesting analysis of the (potential) role of social collective trust and its implications for early warning systems.

However, I believe a major limitation of the work is the lack of comparison between model results and data or empirical evidence. I appreciate that there may not be enough data available to actually compare the model results to data, but given this limitation I believe the model equations and parameter choices should be much better substantiated with evidence from the literature. In addition, one could, in a descriptive way, compare the results with findings in the literature related to the cry wolf effect rather than only compare the results to the results of another model. In the current state, the manuscript does not provide enough evidence for the model assumptions and their relevance. This means that it is impossible to draw any useful conclusions from the results of the analysis, since it is unclear how well the model represents reality.

Some other remarks:

The authors use socio-meteorology in their title and in the final paragraph of the discussion and conclusion they call for a new field called socio-meteorology. However, it is not clear to me why this work is so different that it does not fit within the field of socio-hydrology (the authors are only using discharge and forecasts of discharge in their model, to me this is hydrology, not meteorology). I would suggest to choose a different title, and stick to using socio-hydrology, as the authors do throughout the entire manuscript (the socio-meteorology is in fact only mentioned as an afterthought in the final paragraph of the manuscript).

Introduction, lines 43 to 80: after reading the introduction for the first time I had the impression that there was actually no evidence for the cry wolf effect and for a relationship between the false alarm ratio and the implementation of measures. This made me wonder what the relevance of the presented model and manuscript is. However, after re-reading I see that I misinterpreted and there are studies that do find evidence in support of the cry wolf effect, but also some that do not. I would suggest the authors rewrite this part of the introduction to better present the argument for why their study is important.

In the model description in line 148 (and after) the authors talk about preparedness actions (and mitigation and protection actions), please elaborate and explain what these actions are. There are many preparedness actions that do not depend on a flood warning to be implemented, what about those actions? These kind of measures may actually be implemented when experience of damage is high and trust in flood warning is low (which is the opposite of the cry wolf effect).

Equation 6 models the cost of mitigation and protection actions, why is this relevant? Please discuss why you calculate this. Later, in section 3.1, I see that the total loss is calculated as D + C. I suggest to move this to section 2, since it is quite important and now it is a bit hidden away, which means the importance of C is unclear. Also how are the costs of protection actions determined? What is this based on? Also, why is the loss calculated as D+C, please explain this.

In lines 177 to 179, the authors state that it is reasonable to assume that trust in FEWS increases (decreases) when prediction succeeds (fails). Please elaborate, this is the main contribution of the manuscript and this claim should be substantiated more. (The authors reference Wachinger et al. (2013), but Wachiger et al. (2013) actually hypothesise that the cry wolf effect may be an explanation for the risk perception paradox and do not provide the evidence to support this hypothesis.)

In lines 200 to 202 the authors state: “In our proposed model, high social collective trust in FEWS can maintain the high level of social preparedness even if a community completely loses past flood experiences (equation (7)).” To me it seems unlikely that preparedness stays high solely based on trust while people have forgotten about floods. Is there any evidence from the literature that supports this assumption?

For all variables and parameters: what are the units?

For all equations and values the authors choose: please provide more evidence from the literature as to why this is a good representation of reality. This is especially important given the lack of data for comparison with model results, as mentioned in my main point.

Table 2 and lines 207- 208: why are those parameters fixed and why do they have those values? Are they based on anything?

For the parameters that are varied, why those values?

Figure 1: what does half of social collective trust and social collective memory mean? Why half?

In line 289 it is stated that figure 2 shows predefined warning threshold, but the figure axis title is predefined probability threshold. Same for figures 3 and 4.
Citation: https://doi.org/10.5194/hess-2021-497-RC1
- AC1: 'Reply on RC1', Yohei Sawada, 27 Jan 2022
  
  Please see the attached supplement.
  
  Citation: https://doi.org/10.5194/hess-2021-497-AC1
RC2:
'Comment on hess-2021-497', Anonymous Referee #2, 19 Dec 2021

This paper presents an advanced modelling approach to studying the relationship between society and natural extremes. The authors developed an older model by Girons Lopez et al. (2017) by adding the "cry wolf effect" phenomenon. I find this a stimulating extension.

As mathematical modelling is not my field of expertise, I cannot assess the modelling section (which is the core of the paper). Therefore, my review is more focused on the context and overall results.

The paper is well-written and readable. However, some aspects should be explained better (see below). I also pose several questions for the discussion.

Questions and comments:

Title: Please consider changing the title to more reflect the paper's topic. E.g. "Possibilities of mathematical modelling in socio-meteorology: flood prediction etc." or "Mathematical modelling of the cry wolf effect".

Line 68: You mention that some studies claim that the cry wolf effect does not exist. How do your results affect this debate? Why do you think some authors have found the cry wolf effect problematic? Please discuss these questions.

Lines 78-80: I agree with this sentence. However, it would be nice to give a practical recommendation – what does your research imply? How should we consider the social aspects?

Line 139: You mention that the "trust in flood warnings" is based on the accuracy of warnings. Are there any other possibilities how to increase public trust? E.g. by social activities, education etc.? I understand that the mathematical model must be simple, but please discuss this.

Lines 163-164: Please define the terms "social collective memory" and "social collective trust".

Line 239: Why did you set gamma = 0.5? Why exactly 0.5? What does it mean?

Line 257: Please explain the terms "green society" and "technological society".

Tables 2, 3, 4, 5, 6: Please explain how you got the values of the parameters. Are these values based on empirical knowledge or literature review? Or are they just selected arbitrarily? What do these values mean?

Figure 1: Why do you show precisely the time range you show? Is it a random selection? What does mean the height of the colour bars? Is it the flood intensity or damage level? Would you please add a description of the y-axis?

Discussion and conclusion: According to your findings, it is possible to give a practical recommendation to FEWS strategy? I.e. do you suggest issuing fewer warnings (to reduce the cry wolf effect but risk the damages of flooding) or more warnings (to be safer but risk the cry wolf effect)? Please discuss.

Discussion and conclusion: Please also discuss your findings in the context of papers on the cry wolf effect you mentioned in the Introduction section.

Final remarks: Your paper is based on the modelling approach only. Would you please suggest how it would be possible to validate your findings on real data?

Citation: https://doi.org/10.5194/hess-2021-497-RC2
- AC2: 'Reply on RC2', Yohei Sawada, 27 Jan 2022
  
  Please see the attached supplement file.
  
  Citation: https://doi.org/10.5194/hess-2021-497-AC2

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision

ED: Reconsider after major revisions (further review by editor and referees) (07 Feb 2022) by Roberto Greco

AR by Yohei Sawada on behalf of the Authors (09 Mar 2022) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (30 Mar 2022) by Roberto Greco

RR by Anonymous Referee #1 (11 May 2022)

Suggestions for revision or reasons for rejection

I am not entirely satisfied with the revision of the paper in reply to both reviewers’ comments. Specifically:
- Comment 1.6: the authors’ response here is that the their assumption about trust is explained with the sentence: “Previous studies pointed out that the recent forecast accuracy and false alarm ratio affected the performance of preparedness actions (Simmons and Sutter 2009; Trainor et al. 2015; Ripberger et al. 2015; Jauernic and van den Broeke 2017).” This talks about the performance of preparedness actions, not about an increase or decrease in trust, or even about the implementation of measures. It does not explain why it is reasonable to assume that trust in FEWS increases (decreases) when prediction succeeds (fails). This needs to be better substantiated with evidence from the literature.
- Comment 1.7: in response to my point that it is not realistic that preparedness increases only because of trust while there is no memory of an event, the authors’ decided to remove this discussion from the text, rather than update the model so that it is more realistic. Even if this does not happen often, this behaviour is not realistic and therefore implies a flaw in the model structure. This should be addressed or at the very least discussed as a limitation.
- Comment 2.2: I believe the authors should more thoroughly discuss what their model results add to the discussion of whether the cry wolf effect exists or not. In my opinion, simply stating “the current evidence suggests the importance to understand the effect of false alarms on behavioral responses to warning in order to design efficient flood early warning systems.” is not the same as discussing your results in the context of the current debate in the scientific literature.
- Comment 2.6: in reply to the reviewers’ question about the choice of value for γ the authors reply there is no evidence from the literature on which to base this value. The justification for choosing 0.5 (i.e. not having to consider asymmetric contributions of E and T) is not good enough. The addition of trust in the model is the main contribution, if there is a parameter that determines the influence of trust on your model outcome and there is no evidence that suggests a certain value, one should at least investigate the effect of the parameter’s value on the end results of the study. I would suggest to add a sensitivity analysis of the study’s results to this parameter.
- Comment 2.9: while the chosen time range may clearly show the difference between the two models, it would also be good to include a figure that shows the behaviour of the model over the entire time range.
- Comment 1.1, 2.11, 2.12: The authors do now discuss their results in relation to the literature that they discuss in the introduction, but only in relation to the literature that supports the existence of the cry wolf effect. Why do the other papers find that this effect does not exist? And how do the results of this study relate to that? How do they support the evidence that the effect does exist?

In addition, after reading both reviewer’s comments and the author’s replies I tried to reproduce the results presented in the paper and was unable to do this with the information provided in the paper. Therefore, I have some further points that I believe should be addressed before the paper can be published:
- Using a gaussian distribution for the variance of the forecast implies you get negative values, which is not possible, how is this solved? Is a truncated distribution used? If so, this should be mentioned.
- Similarly, in general (even when assuming the above mentioned variance is set to be >0), the forecast distribution yields negative forecasts for the discharge. This does not make sense, the discharge cannot be negative.
- In addition, the parameter settings used for the discharge imply a discharge of 0.2 corresponds to the 2.4 year flood and a discharge of 0.45 corresponds to a 21 year flood. This last value, 0.45, is used as the damage threshold for the technological society, but a protection level protecting against a 21 year flood is rather low for a technological society. I would expect it to protect at least against a 100 year flood (i.e. a damage threshold between 0.6 and 0.61)
- The initial values for E and T are not reported. And related to that: do they remain the same across all experiments or do you vary them? The initial values may have a significant effect on the model results and therefore the conclusions.
- Similarly to the initial values, the order of the floods will influence the model results. Running the model for several different floods time series and then averaging the outcome metric (e.g. the relative loss) over those runs yields more robust results.
- In the form as it is written in the paper, the trust T explodes in a negative or positive direction, rather than being constrained between 0 and 1. Similarly the experience E is not constrained to be below 1. Therefore the preparedness can become very large or negative, giving negative damage.
- For most experiments conducted and reported in the paper, π is varied from 0 to 1. But for the first comparison between the two models, a fixed value is used. What is this value?

Hide

RR by Anonymous Referee #2 (13 May 2022)

ED: Reconsider after major revisions (further review by editor and referees) (27 May 2022) by Roberto Greco

AR by Yohei Sawada on behalf of the Authors (07 Jul 2022) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (07 Jul 2022) by Roberto Greco

RR by Anonymous Referee #1 (18 Jul 2022)

ED: Publish as is (22 Jul 2022) by Roberto Greco

Short summary

Although flood early warning systems (FEWS) are promising, they inevitably issue false alarms. Many false alarms undermine the credibility of FEWS, which we call a cry wolf effect. Here, we present a simple model that can simulate the cry wolf effect. Our model implies that the cry wolf effect is important if a community is heavily protected by infrastructure and few floods occur. The cry wolf effects get more important as the natural scientific skill to predict flood events is improved.