Identifying the connective strength between model parameters and performance criteria

Guse, Björn; Pfannerstill, Matthias; Gafurov, Abror; Kiesel, Jens; Lehr, Christian; Fohrer, Nicola

doi:https://doi.org/10.5194/hess-21-5663-2017

Articles | Volume 21, issue 11

https://doi.org/10.5194/hess-21-5663-2017

© Author(s) 2017. This work is distributed under
the Creative Commons Attribution 3.0 License.

https://doi.org/10.5194/hess-21-5663-2017

© Author(s) 2017. This work is distributed under
the Creative Commons Attribution 3.0 License.

Articles | Volume 21, issue 11

Research article

|

15 Nov 2017

Research article |

| 15 Nov 2017

Identifying the connective strength between model parameters and performance criteria

Björn Guse, Matthias Pfannerstill, Abror Gafurov, Jens Kiesel, Christian Lehr, and Nicola Fohrer

Download

Final revised paper (published on 15 Nov 2017)
Preprint (discussion started on 01 Feb 2017)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC1: 'Review of 'Identifying the connective strength between model parameters and performance criteria By Guse et al.'', Anonymous Referee #1, 12 Mar 2017
- RC2: 'Review of hess-2017-28', Anonymous Referee #2, 22 Mar 2017
  - AC1: 'Reply to the comments of reviewer#2', Björn Guse, 27 Mar 2017
- AC2: 'Reply to the comments of Reviewer#1', Björn Guse, 12 May 2017
EC1: 'Editor's (intermediate) comment', Dimitri Solomatine, 28 Mar 2017
- AC4: 'Reply to the intermediate comments from the Editor', Björn Guse, 12 May 2017
RC3: 'Review of "Identifying the connective strength between model parameters and performance criteria" by Guse et al.', Richard Arsenault, 01 Apr 2017
- AC3: 'Reply to the comments of reviewer#3 (Richard Arsenault)', Björn Guse, 12 May 2017
EC2: 'Editor's final comment', Dimitri Solomatine, 12 May 2017

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

ED: Reconsider after major revisions (further review by Editor and Referees) (12 May 2017) by Dimitri Solomatine

AR by Björn Guse on behalf of the Authors (26 Jun 2017) Author's response Manuscript

ED: Referee Nomination & Report Request started (04 Jul 2017) by Dimitri Solomatine

RR by Richard Arsenault (14 Jul 2017)

RR by Anonymous Referee #1 (01 Aug 2017)

Suggestions for revision or reasons for rejection

In the revised version of the manuscript, the authors have provided clarifications for study area and methodological choices. In addition, introduction has been edited to account for comments by the referees and editor. The manuscript has overall improved, but the following minor issues still remain unaddressed:
1. Unclear reasoning in the introduction: Some claims in the introduction are not substantiated by past literature. For instance:
a.Lines 10-13, Page 2 (‘In parameter identification ………. Identification’): It is not true that parameter identification implicitly assumes that model parameters are precisely identified by selected performance criteria. Most sensitivity analysis studies have already shown that this is not true, and that parameters can affect performance in interaction with other parameters within the model. In fact in some cases, the interaction effects dominate the sensitivity indices. In fact, the assumption that an accurate parameter identification can be done is questionable given the interaction effects and uncertainties.
b. Line 25-28, Page 3 (‘To our …. Task.’): There is a misinterpretation of sensitivity analysis here. If one applies sensitivity analysis with multiple performance measures, a matrix similar to Figure 4 can be obtained, where gray scales can be used to plot the first or higher order contributions of parameters to variation in each measure. A comparison across performance measures can then highlight which parameters uniquely identify which parameter measures and vice-versa. In fact, applying sensitivity analysis in this manner will give more information than the framework proposed in this study as they can also highlight higher order interaction effects. I think this study is not an advancement on sensitivity analysis but an alternative. Both methods can be used to attain the overall goals of the analysis. The only advantage of using CART over SA is that CART identifies the values of parameters that lead to good performance while SA does not. If used only for parameter/performance ranking, both are equally acceptable methods.

2. Choice of 2000 parameter sets: this choice is still poorly defended. If one has 12 parameters and each parameter range is divided into 10 equally spaced values, one still needs 10^12 values to represent the parameter space. 2000/10^12 is a very sparse set and cannot be claimed to represent the parameter space accurately. Therefore, a convergence analysis is the only realistic way to defend this choice. It is hard to say whether 2000 is far better than 1000. It is unclear why the results from a randomly chosen subset of 1000 (or 1500, or 1800) cannot be checked against the results presented in the main analysis, as a proof of concept.

Hide

RR by Anonymous Referee #2 (03 Aug 2017)

Suggestions for revision or reasons for rejection

After reading the revision and the responses to comments of all three reviewers. I'm kind of disappointed for several reasons. Therefore, I would rather suggest reject from publication at this point.
1. Authors mentioned in the response that the "focus of the manuscript is to investigate the relationship between model parameters and performance measures. Thus, we show which model parameters impact which performance measure and which performance measures are influenced by the different model parameters". I agreed, that's what I thought too. In my definition, that's also a part of parameter uncertainty, sensitivity analysis, and model calibration. One cannot investigate parameter uncertainty, sensitivity analysis by not doing what you did right (maybe not use LHS but other methods)? And, in most cases, that's a part of the work for model calibration. Then, scientists can use the calibrated model for further purposes, such as ANSWERING SCIENTIFIC QUESTIONS. That's why, I mentioned previously that the novelty of this manuscript is questionable.
2. In this study, some parameters were identified to be more influential that others. If you change the targeted watersheds, you'll definitely have different sensitive parameter sets and values. It is for sure technically-oriented, and I don't see the solid scientific values here.
3. By the way, I just don't understand why you just don't add a map for both watersheds? How difficult is that?
4. For what you did, I also think that's a failure. The reason is simple, and I didn't catch that in the first round but the reviewer#1 did. It was mentioned by reviewer#1 that "It is surprising that no appropriate performance criteria is found to relate to CN2, which is generally a sensitive parameter in SWAT". You justified that "CN2 was varied in these HRUs from 40 to 60 which is assumed to be sufficient to maintain the landscape heterogeneity". It seems the range of CN2 has been allocated in values that's already been validated in your previous work. In this case, of course the value of CN2 is not sensitive (it's in close to calibrated values anyway). Then, why are we even doing the proposed work in the first place? Are we supposed to explore the potential ranges (and the relationships between performance measures and parameters) of the parameters to validate the following steps? How can we know if other parameters are not being handled the same way?

There are still several other issues, but I would like to stop it here. Hope you understand that I love your work, but the issues of novelty presented in this study cannot be resolved in easy ways.

Hide

ED: Publish subject to minor revisions (further review by Editor) (09 Aug 2017) by Dimitri Solomatine

AR by Björn Guse on behalf of the Authors (15 Sep 2017) Author's response Manuscript

ED: Publish as is (19 Sep 2017) by Dimitri Solomatine

Short summary

Performance measures are used to evaluate the representation of hydrological processes in parameters of hydrological models. In this study, we investigated how strongly model parameters and performance measures are connected. It was found that relationships are different for varying flow conditions, indicating that precise parameter identification requires multiple performance measures. The suggested approach contributes to a better handling of parameters in hydrological modelling.