Technical note: Diagnostic efficiency – specific evaluation of model performance

Abstract. Better understanding of the reasons why hydrological model performance is good or poor represents a crucial part for meaningful model evaluation. However, current evaluation efforts are mostly based on aggregated efficiency measures such as Kling-Gupta Efficiency (KGE) or Nash-Sutcliffe Efficiency (NSE). These aggregated measures only distinguish between good and poor model performance. Especially in the case of a poor model performance it is important to identify the different errors which may have caused such unsatisfying predictions. These errors may origin from the model parameters, the model structure, and/or the input data. In order to provide more insight, we define three types of errors which may be related to their origin: constant error (e.g. caused by consistent input data error such as precipitation), dynamic error (e.g. structural model errors such as a deficient storage routine) and timing error (e.g. caused by input data errors or deficient model routines/parameters). Based on these types of errors, we propose the novel Diagnostic Efficiency (DE) measure, which accounts for the three error types. The disaggregation of DE into its three metric terms can be visualized in a plain radial space using diagnostic polar plots. A major advantage of this visualization technique is that error contributions can be clearly differentiated. In order to provide a proof of concept, we first generated errors systematically by mimicking the three error types (i.e. simulations are surrogated by manipulating observations). By computing DE and the related diagnostic polar plots for the mimicked errors, we could then supply evidence for the concept. Finally, we tested the applicability of our approach for a modelling example. For a particular catchment, we compared streamflow simulations realized with different parameter sets to the observed streamflow. For this modelling example, the diagnostic polar plot suggests, that dynamic errors explain the model performance to a large extent. The proposed evaluation approach provides a diagnostic tool for model developers and model users and the diagnostic polar plot facilitates interpretation of the proposed performance measure.