Moving beyond the cost–loss ratio: economic assessment of streamflow forecasts for a risk-averse decision maker

Matte, Simon; Boucher, Marie-Amélie; Boucher, Vincent; Fortier Filion, Thomas-Charles

doi:https://doi.org/10.5194/hess-21-2967-2017

Articles | Volume 21, issue 6

https://doi.org/10.5194/hess-21-2967-2017

© Author(s) 2017. This work is distributed under
the Creative Commons Attribution 3.0 License.

https://doi.org/10.5194/hess-21-2967-2017

© Author(s) 2017. This work is distributed under
the Creative Commons Attribution 3.0 License.

Articles | Volume 21, issue 6

Research article

|

19 Jun 2017

Research article |

| 19 Jun 2017

Moving beyond the cost–loss ratio: economic assessment of streamflow forecasts for a risk-averse decision maker

Simon Matte, Marie-Amélie Boucher, Vincent Boucher, and Thomas-Charles Fortier Filion

Download

Final revised paper (published on 19 Jun 2017)
Supplement to the final revised paper
Preprint (discussion started on 27 Sep 2016)
Supplement to the preprint

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC1: 'Referee comments', Anonymous Referee #1, 02 Nov 2016
- AC1: 'Answer to Reviewer 1', Marie-Amélie Boucher, 22 Dec 2016
RC2: 'Review of the paper "Beyond the cost-loss ratio..." by Matte et al.', Anonymous Referee #2, 31 Dec 2016
- AC2: 'Answer to Reviewer 2', Marie-Amélie Boucher, 16 Jan 2017

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

ED: Reconsider after major revisions (further review by Editor and Referees) (19 Jan 2017) by Uwe Ehret

AR by Marie-Amélie Boucher on behalf of the Authors (06 Mar 2017) Manuscript

ED: Referee Nomination & Report Request started (09 Mar 2017) by Uwe Ehret

RR by Anonymous Referee #1 (04 Apr 2017)

Suggestions for revision or reasons for rejection

This is my second review of the article entitled "Moving the cost-loss ratio: Economic assessment of streamflow forecasts for a risk-averse decision maker" submitted to Hydrology and Earth System Sciences by Matte et al.

This article offers a 'new' approach for the economic assessment of flood forecasts, which takes into account the risk aversion of many end users, which is not the case in the "classical" cost-loss ratio methodology ('new' in the hydrological field). Then, this paper addresses an economic issue which would interest many hydrologists, in the operational services as well as academics. The theoretical framework for the assessment is experimented in a practical case study: the 2004 floods of the Montmorency River (Québec, Canada).

First, I have to thank the authors for their very clear, precise and honest answers to my first comments (in the discussion). They helped me to understand much better their approach. I also congratulate them for the very serious work which led to this new version of the manuscript. Many - if not all - comments of the 2 reviewers have been taken into account. It is much clearer (in my opinion) and which is now almost ready for publication in HESS. Thanks to the added explanations and references, it is now much more adapted for non economist readers (such as many hydrologists). The results and discussion sections are particularly clear. However, one main difficulty and a few minor ones remain (see below). I therefore recommend a minor revision to allow the authors to add some final improvements to their paper (in my opinion, it does not require another revision round any more).

MAIN COMMENTS

Before discussing the main issue ("weakness") remaining in the article, I would like to continue the discussion about the cost-loss ratio approach and the notion of risk aversion. One point I still not understand is the possibility of a link between 2 ideas :
a) risk aversion: how much an individual is eager to pay to be insured against an event ;
b) the optimal ratio of false alarms and missed events for an individual, that is the number of false alarms (which cost a little amount of money c1) in order to avoid more than 1 missed event (which costs a large amount of money c2); this question can be treated by the cost-loss ratio.
Since both notions are related to the amount of money an individual is eager to pay to avoid the consequences of any (respectively: more than 1) event, it is not clear for me whether we can make a link between these 2 approaches rather than opposing them.

As pointed out by the authors in the discussion and in their revised manuscript, the results of the experiment on the case study are significant only if the data is representative of the catchment behaviour (see the notion of expectation, page 13). The case study here is driven over 4 years (2011-2014), thus over a (very) limited number of (significant) events. The sufficiency of data (to infer any conclusion) should be explicitly treated. It may be useful to:
a) display the observations and the forecasts (at least in the supplementary materials);
b) sum up all the events (e.g. discharge over a chosen threshold), and to pay a particular attention to missed events and successful forecasts: I expect that their "ratio" would strongly influence the results of this study.

DETAILED COMMENTS

Page 2, lines 5 - 6: if available, some reference would be useful.

Page 2, line 14 ("the latter gain in importance"): check grammar ("gains"?)

Page 2, line 15 ("However, AS there exist many sources of uncertainty in hydrological processes, there also exist many means [...]"): I did not get the logical implication between these 2 ideas.

Page 2 line 15 ("there also exists many means"): check grammar ("exist"?)

Page 2, line 22: since the "value" word is in italic 2 lines further, I suggest to use italic for the "quality" word as well.

Page 2, line 25: isn't a point sign missing before the words "In particular"?

Page 2, line 26 ("In particular, the usefulness of a forecast is inherently linked to the decision maker's ability to adapt their behaviour to the information provided"): I totally agree with this sentence but why is this important in the introduction? (why to state this issue here?) In a more general way, it is the ability to use the information which allows a forecast to be useful.

Page 4, line 29 ("the decision maker may only distinguish between a finite set of implied damages"). I am not sure of what the reader should understand here; is this a practical observation or a more theoretical affirmation?

Page 5, lines 11-12: some references (such as those given in appendix B) could usefully be provided here.

Page 6, line 15: since the month of the floods of 2012 and 2014 are given below, to precise in which month the 1964 flood occurred could be useful and consistent.

Page 6, lines 21-22 ("The greatest concern of public authorities occurs when people refuse to evacuate [...]"): this is unfortunately a too common behaviour. However, why is this information useful here to demonstrate the pertinence of the proposed economic assessment?

Page 6, lines 25 and following: very little is said about the spatial resolution of the model (how many RHHU? ...). It is not clear how well it is adapted to the data inputs and to the catchment (in particular, to state that it is a physics-based model).

Page 7, lines 3-5. The model description has been well improved (compared to the first submission). However, I still don't understand why details such as those about the vertical water budget scheme (BV3C) are relevant for this publication.

Page 8, line 19 ("In a study involving 20 catchments in Quebec"). I acknowledge the fact that the interested reader can easily find Thiboult et al. (2016). However, it could be worth providing a few details about these catchments (can we compare them to the Montmorency River catchment used in this case study?).

Page 8, line 20 ("the uncertainty for initial conditions dominate the other sources of uncertainty for short term (1-day to 3-day ahead)"): 3 days seem to be a very long period compared to the response time (12h, according to page 6, line 6). Is this correct?

Page 9. The description of the "rudimentary" data assimilation scheme is much clearer in this new version. However, I am still not convinced that it can be called an Ensemble Kalman filter (there is no sequential approach here). At least, the absence of a sequential scheme could usefully be pointed out.

Page 9 ("The inclusion of additive perturbations for precipitation is due to the fact that strong under-captation is suspected for this catchment."): I don't understand this. Do the authors mean "additive positive perturbations"?

Page 10, lines 1-3: the 'm' subscript is used for 2 different items: the ensemble members and the damage categories. This could infer some confusion.

Page 13, line 22 ("On the other hand, there can also be various sources of non-stationarity [...]"): this argument is rather specious. This may indeed occur but there are means to detect non-stationarity. The conclusion (lines 23-24) can not (should not?) be inferred from this argument. The representation of the expectation of the utility by its average value could be better discussed (see the main comments).

Page 17, line 28 ("We find that risk-averse end-users mainly consider the less favourable scenarios"): is this really a finding of this study? If so, it should be stated "with/in this model, we find [...]".

Page 18, lines 23-24 ("in this paper, we did not address the issue of potential cognitive biases and training issues for end-users"): I agree and thank the authors to point out these issues. In addition, it could be worth clearly stating that risk aversion is not a cognitive bias (see the answers of the authors in the discussion, page C10), e.g. page 10 after lines 21-22.

Page 27, Tab. 2: I don't understand the horizontal line under "No limit for a 1-day forecast". Should other lines (above and under "No limit for a 5-day forecast") be added?

Page 28, Fig. 1: this figure is a "schematic" representation of the CARA utility function for A > 0. Why not showing a real CARA utility function for A > 0? It could help the reader keeping in mind that the CARA function values are all negatives in such a case?

Page 28, Fig. 1 caption: is this equation coherent with the equation 2 (page 5)? Furthermore, I am also uncomfortable with this equation. Since C is the difference between 2 utility values, it is therefore not a money amount (see page 14, line 7: "the actual value of the decision maker's utility has no interpretation). Then, if the equation in the caption is correct, why and how could C be the amount that the individual would be willing to pay up to?

Hide

ED: Publish subject to minor revisions (further review by Editor) (11 Apr 2017) by Uwe Ehret

AR by Marie-Amélie Boucher on behalf of the Authors (20 Apr 2017) Author's response Manuscript

ED: Publish as is (24 Apr 2017) by Uwe Ehret

AR by Marie-Amélie Boucher on behalf of the Authors (01 May 2017) Manuscript

Short summary

In this study we set the basis of an alternative framework to replace the popular cost-loss ratio for the economic assessment of flood forecasting systems. The C-L ratio implicitly considers the decision maker to be risk-neutral, whereas it is rarely the case in real-life emergency situations. Instead of the cost-loss ratio, we propose using a utility function. We show that the decision-maker’s level of risk aversion is a crucial factor in the assessment of the economic value of flood forecasts.