Preprints
https://doi.org/10.5194/hess-2020-237
https://doi.org/10.5194/hess-2020-237

  12 Jun 2020

12 Jun 2020

Review status: this preprint is currently under review for the journal HESS.

Technical note: Diagnostic efficiency – specific evaluation of model performance

Robin Schwemmle, Dominic Demand, and Markus Weiler Robin Schwemmle et al.
  • University of Freiburg, Faculty of Environment and Natural Resources, Chair of Hydrology, Freiburg, Germany

Abstract. Better understanding of the reasons why hydrological model performance is good or poor represents a crucial part for meaningful model evaluation. However, current evaluation efforts are mostly based on aggregated efficiency measures such as Kling-Gupta Efficiency (KGE) or Nash-Sutcliffe Efficiency (NSE). These aggregated measures only distinguish between good and poor model performance. Especially in the case of a poor model performance it is important to identify the different errors which may have caused such unsatisfying predictions. These errors may origin from the model parameters, the model structure, and/or the input data. In order to provide more insight, we define three types of errors which may be related to their origin: constant error (e.g. caused by consistent input data error such as precipitation), dynamic error (e.g. structural model errors such as a deficient storage routine) and timing error (e.g. caused by input data errors or deficient model routines/parameters). Based on these types of errors, we propose the novel Diagnostic Efficiency (DE) measure, which accounts for the three error types. The disaggregation of DE into its three metric terms can be visualized in a plain radial space using diagnostic polar plots. A major advantage of this visualization technique is that error contributions can be clearly differentiated. In order to provide a proof of concept, we first generated errors systematically by mimicking the three error types (i.e. simulations are surrogated by manipulating observations). By computing DE and the related diagnostic polar plots for the mimicked errors, we could then supply evidence for the concept. Finally, we tested the applicability of our approach for a modelling example. For a particular catchment, we compared streamflow simulations realized with different parameter sets to the observed streamflow. For this modelling example, the diagnostic polar plot suggests, that dynamic errors explain the model performance to a large extent. The proposed evaluation approach provides a diagnostic tool for model developers and model users and the diagnostic polar plot facilitates interpretation of the proposed performance measure.

Robin Schwemmle et al.

 
Status: final response (author comments only)
Status: final response (author comments only)
AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment
[Login for authors/editors] [Subscribe to comment alert] Printer-friendly Version - Printer-friendly version Supplement - Supplement

Robin Schwemmle et al.

Model code and software

diag-eff 0.1: Diagnostic efficiency – specific evaluation of model performance R. Schwemmle, D. Demand, and M. Weiler https://doi.org/10.5281/zenodo.3769562

Robin Schwemmle et al.

Viewed

Total article views: 581 (including HTML, PDF, and XML)
HTML PDF XML Total Supplement BibTeX EndNote
392 178 11 581 46 16 15
  • HTML: 392
  • PDF: 178
  • XML: 11
  • Total: 581
  • Supplement: 46
  • BibTeX: 16
  • EndNote: 15
Views and downloads (calculated since 12 Jun 2020)
Cumulative views and downloads (calculated since 12 Jun 2020)

Viewed (geographical distribution)

Total article views: 489 (including HTML, PDF, and XML) Thereof 489 with geography defined and 0 with unknown origin.
Country # Views %
  • 1
1
 
 
 
 
Latest update: 05 Mar 2021
Download
Short summary
Better understanding of the reasons why model performance is good or poor represents a crucial part for meaningful model evaluation. We propose the novel Diagnostic Efficiency (DE) measure and diagnostic polar plots. The proposed evaluation approach provides a diagnostic tool for model developers and model users and facilitates interpretation of model performance.