Technical note: Diagnostic efficiency –  specific evaluation of model performance

Schwemmle, Robin; Demand, Dominic; Weiler, Markus

doi:https://doi.org/10.5194/hess-25-2187-2021

Articles | Volume 25, issue 4

https://doi.org/10.5194/hess-25-2187-2021

© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/hess-25-2187-2021

© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 25, issue 4

Technical note

|

22 Apr 2021

Technical note |

| 22 Apr 2021

Technical note: Diagnostic efficiency – specific evaluation of model performance

Robin Schwemmle, Dominic Demand, and Markus Weiler

Download

Final revised paper (published on 22 Apr 2021)
Supplement to the final revised paper
Preprint (discussion started on 12 Jun 2020)
Supplement to the preprint

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC1: 'Referee comments', Anonymous Referee #1, 08 Jul 2020
- AC1: 'Reply to Referee #1', Robin Schwemmle, 28 Aug 2020
RC2: 'Nice study, but assumptions require stronger supporting evidence', Anonymous Referee #2, 01 Aug 2020
- AC2: 'Reply to Referee #2', Robin Schwemmle, 28 Aug 2020

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

ED: Publish subject to revisions (further review by editor and referees) (09 Sep 2020) by Genevieve Ali

AR by Robin Schwemmle on behalf of the Authors (24 Sep 2020) Author's response Manuscript

ED: Referee Nomination & Report Request started (08 Oct 2020) by Genevieve Ali

RR by Anonymous Referee #1 (26 Oct 2020)

Suggestions for revision or reasons for rejection

General comments

The manuscript presents a potential useful new metric and method for evaluating hydrologic model inadequacy, which is definitely in the journal scope. The technical note has been somewhat improved by the revisions, and most comments have been addressed to some degree. The methods are valid and fully outlined, the results are adequately discussed and sufficient to support the conclusions. The overall structure of the technical note is fine, but there are some issues with the text and figures which detract from the comprehensibility of the paper. Some minor revisions should be completed prior to publication.

Specific comments

53-62: This section is not clear. The text states that observations should be checked for accuracy, but then uncertainties in observations remain, therefore we will not focus on uncertainties in observations. This is not helpful in clarifying why some sources of uncertainty were the focus of the new metric.

65-70: This may be a minor point, but what good is it three defined error types to be linked to potential error sources when, in your example, any of the three types can be linked to the same source (precipitation input)? If expert knowledge is need, how is the novel metric helping?

98: Polar plots are not very much like clocks, given both angle and distance from the origin matter in polar plots. This comparison is more confusing than helpful.

Figure 1: The y-axis has no title.

Figure 2: The y-axis has no title.

Figure 4: Why are the FDC y-axes in log space? It initially appears as though the simulation only over estimates low flows, when the magnitude of high flow under estimation is in fact similar.

Figure 5: One benefit of the novel metric and method which may not be highlighted in the manuscript is its ability to facilitate simulation comparisons. While the FDC curves contain more information than the polar plot, a single diagnostic polar plot can clearly summarize the errors in dozens or hundreds of simulations (which would be unreadable in a single FDC plot).

Technical corrections:

30: ‘Measure’ twice
191: ‘Interdependently’ is either being used incorrectly, or was not the word intended
256: ‘Illogical’ is not a good descriptor for the quirks and irregularities of the NSE and KGE

The authors may also wish to review the erratic comma usage, although this is not impeding the manuscript’s comprehensibility.

Hide

RR by Wouter Knoben (12 Nov 2020)

ED: Publish subject to revisions (further review by editor and referees) (25 Nov 2020) by Genevieve Ali

AR by Robin Schwemmle on behalf of the Authors (22 Jan 2021) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (03 Feb 2021) by Genevieve Ali

RR by Wouter Knoben (09 Feb 2021)

ED: Publish subject to minor revisions (review by editor) (17 Feb 2021) by Genevieve Ali

AR by Robin Schwemmle on behalf of the Authors (10 Mar 2021) Author's response Author's tracked changes Manuscript

ED: Publish as is (23 Mar 2021) by Genevieve Ali

AR by Robin Schwemmle on behalf of the Authors (24 Mar 2021)

Short summary

A better understanding of the reasons why model performance is unsatisfying represents a crucial part for meaningful model evaluation. We propose the novel diagnostic efficiency (DE) measure and diagnostic polar plots. The proposed evaluation approach provides a diagnostic tool for model developers and model users and facilitates interpretation of model performance.