Incremental model breakdown to assess the multi-hypotheses problem

Jehn, Florian U.; Breuer, Lutz; Houska, Tobias; Bestian, Konrad; Kraft, Philipp

doi:https://doi.org/10.5194/hess-22-4565-2018

Articles | Volume 22, issue 8

https://doi.org/10.5194/hess-22-4565-2018

© Author(s) 2018. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/hess-22-4565-2018

© Author(s) 2018. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 22, issue 8

Research article

|

29 Aug 2018

Research article |

| 29 Aug 2018

Incremental model breakdown to assess the multi-hypotheses problem

Florian U. Jehn, Lutz Breuer, Tobias Houska, Konrad Bestian, and Philipp Kraft

Download

Final revised paper (published on 29 Aug 2018)
Preprint (discussion started on 29 Nov 2017)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC1: 'Review of manuscript "Incremental model breakdown to assess the multi-hypotheses problem"', Lieke Melsen, 27 Dec 2017
- AC1: 'Response letter to Reviewer #1 (L.A. Melsen)', Florian Ulrich Jehn, 21 Feb 2018
RC2: 'Interactive comment on “Incremental model breakdown to assess the multi-hypotheses problem” by Florian U. Jehn et al.', François Anctil, 05 Jan 2018
- AC2: 'Response letter to Reviewer #2 (F. Anctil)', Florian Ulrich Jehn, 21 Feb 2018
RC3: 'Review comment on the article by Jehn et al.', Anonymous Referee #3, 12 Jan 2018
- AC3: 'Response letter to Reviewer #3 (Anonymus)', Florian Ulrich Jehn, 21 Feb 2018

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

ED: Reconsider after major revisions (further review by editor and referees) (23 Mar 2018) by Albrecht Weerts

AR by Florian Ulrich Jehn on behalf of the Authors (24 Apr 2018) Author's response Manuscript

ED: Referee Nomination & Report Request started (21 May 2018) by Albrecht Weerts

RR by Anonymous Referee #3 (12 Jun 2018)

Suggestions for revision or reasons for rejection

The authors provided a detailed reply to the review comments and made in-depth modifications of the article following the reviewers’ suggestions.

I think this clarified the main points raised by the reviewers and that the article now
deserves publication after considering the minor suggestions detailed below.

Detailed comments: (page and line numbers refer to the pdf version with track changes – “hess-2017-691-author_response-version1.pdf”)

1. Abstract: I think the last sentence could more clearly state that the Model 15 did not reach better performance than the benchmark, though it has some merits also (if it is the case, see comments 8 and 9).

2. Figure 2: Remind in the caption the starting and ending years of each period.

3. P8, L28: “too important”

4. Figs. 5 and 6: Explain in the caption which quantiles of the distribution are shown. To ease visual comparison of performance between calibration and validation, the same Y scale should be used on the two plots of each criterion. Besides, it does not really make sense to show distributions when the sample size (n) is very small (let say below ten). In these cases, I suggest not drawing a distribution but simply showing the individual performances by dots (maybe still highlighting the median dot in yellow).

5. P14, L18: The lower bound of the range for PBIAS should be –100% if I understood well the criterion (?)

6. P14, L19: “constraint”

7. P19, L21: “behave differently”

8. P20, L6-8: Is this comment not erroneous? When looking at the graphs, it seems that HBV-Light has the best median performance on logNSE. Is this comment based on mean performance over all behavioral parameters? In that case, this should be made clear in the text and I suggest adding the mean performance for each model version on top of the graph, above each boxplot.

9. P24, L4: Not sure why Model 1 is said to simulate low flows best, since Model 15 and HBV have better logNSE median performance. (see previous comment)

10. Section 4.3: I find this comparison with the benchmark is interesting since it maybe shows the difficulty of the proposed approach which consists in making a complex model more simple. It reminded me the discussion of Sten Bergström (1991),on model complexity and its role in model development (e.g. p.25 “Going from complex to simpler model structures requires an open mind, because it is frustrating to have to abandon seemingly elegant concepts and theories. It is normally much more stimulating, from an academic point of view, to show significant improvement of the model performance by increasing complexity. Nash and Sutcliffe (1970) presented a strategy for model development from the simple to the complex which may help to avoid this frustration…). Maybe the discussion of the proposed approach could also be put in that perspective.

Cited reference: Bergström, S. (1991) Principles and confidence in hydrological modelling. Nordic Hydrology 22, 123–136.

Hide

RR by Lieke Melsen (15 Jun 2018)

Suggestions for revision or reasons for rejection

Jehn et al. resubmitted their study on incremental model breakdown. The manuscript has significantly improved compared to the previous version, all the main comments from the reviewers seem to be addressed. I have a few (mainly minor) comments, on interpretation and textual.

Major:

On p8 it is stated that the importance of the process is determined based on behavioural runs in the calibration. As stated further in the manuscript and in my previous review, it would be better to determine this based on the validation period given the risk over overfitting. As is written further in the manuscript, the models mostly improves in the validation, so why not make the validation period decisive in the identification of important processes?

On p15 and p17, the results refer to the median objective-function values. Although I indeed suggested median rather than medium values, I still doubt whether looking at the median of behavioural runs is providing insights in model behaviour per se. Why not just look at the highest values? Because, as discussed in the previous review, the models with fewer parameters are sampled finer because the total parameter sample remained the same over all models. Theoretically, as also written in your reply, “Still, our procedure might allow models with less parameters to get more behavioural runs”, therefore more behavioural runs can also (negatively) influence the median model results while this does not imply that your model got worse. Later on in the manuscript, (p17, l15) you also discuss the max values for the objective function.

p.22 l. 10 “The improved performance of Model 15 shows that a priori model selection is not useful, as the models with different process implementations deliver very different results.”.
Although it is true that different models will lead to different results, this sentence is quite a strong statement, as it disqualifies the majority of hydrologic modelling studies in which a single (or even a few) off-the-shelf hydrologic models are used. Although I have the opinion that more attention should be paid to model selection (and therefore I am in favour of modular frameworks), saying that all these other studies are basically ‘not useful’ would be too strong.

Minor:

p.2 l. 25;
The structure does not need to be static in SUMMA, but is also flexible, as with CMF. I now realize that the methodology in this study can be seen as a methodology / application of dealing with modular modelling frameworks in order to identify the optimal model structure.

Please check again the units of Eq. 1, it seems that the units of Q0 differ from the units of Q given that V0 is a flux.

p.8 l. 27 “to[o] important to ignore”

I was expecting to read the calibration and validation period in section 2.2, but they were only mentioned in section 2.5. Perhaps shortly refer to it in section 2.2.

Nice Figure 8! It is indeed, as written, against expectation that the parameters are harder to identify for model 15 than for model 1, as model 15 has fewer parameters. Both models have a substantially different number of behavioural runs, which seems to indicate equifinality has not really decreased. What could be possible explanations? Perhaps, some parameters in Model 15 are responsible for several processes at once?

p.20 l. 19, the sentence on canopy seems contradictive (is important, but models without it behave well).

p.22. l. 14, the sentence over over-simplified lumped models does not really seem to support your method of simplifying models. Or you could use it as an argument: ‘therefore, starting from a complex model, simplifying it only where justified, could resolve this issue’ (or something like that).

Thanks for putting the reviewers in the acknowledgements, but my name is misspelled ;)

Hide

ED: Publish subject to minor revisions (review by editor) (25 Jun 2018) by Albrecht Weerts

AR by Florian Ulrich Jehn on behalf of the Authors (04 Jul 2018) Author's response Manuscript

ED: Publish subject to minor revisions (review by editor) (07 Aug 2018) by Albrecht Weerts

AR by Florian Ulrich Jehn on behalf of the Authors (07 Aug 2018) Author's response Manuscript

ED: Publish subject to minor revisions (review by editor) (10 Aug 2018) by Albrecht Weerts

AR by Florian Ulrich Jehn on behalf of the Authors (13 Aug 2018) Author's response Manuscript

ED: Publish as is (14 Aug 2018) by Albrecht Weerts

AR by Florian Ulrich Jehn on behalf of the Authors (14 Aug 2018)

Short summary

By realizing that hydrological models are not one single hypothesis, but an assemblage of many hypotheses, new ways to scrutinize hydrological models are needed. Up until now, studies concentrate on comparing existing models or built models incrementally. This approach here tries to tackle the problem the other way around. We construct a complex model, containing all processes important for the catchment, and deconstruct it step by step to understand the influence of single processes.