the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Technical Note: The CREDIBLE Uncertainty Estimation (CURE) toolbox: facilitating the communication of epistemic uncertainty
Trevor Page
Paul Smith
Francesca Pianosi
Fanny Sarrazin
Susana Almeida
Liz Holcombe
Jim Freer
Thorsten Wagener
Abstract. There is a general trend for increasing inclusion of uncertainty estimation in the environmental modelling domain. We present the CREDIBLE Uncertainty Estimation (CURE) Toolbox, an open source MATLABTM toolbox for uncertainty estimation aimed at scientists and practitioners that are not necessarily experts in uncertainty estimation. The toolbox focusses on environmental simulation models and hence employs a range of different Monte Carlo methods for forward and conditioned uncertainty estimation. The methods included span both formal statistical and informal approaches, which are demonstrated using a range of modelling applications set up as workflow scripts. The workflow scripts provide examples of how to utilise toolbox functions for a variety of modelling applications and hence aid the user in defining their own workflow: additional help is provided by extensively commented code. The toolbox implementation aims to increase the uptake of uncertainty estimation methods within a framework designed to be open and explicit, in a way that tries to represent best practice in applying the methods included. Best practice in the evaluation of modelling assumptions and choices, specifically including epistemic uncertainties, is also included by the incorporation of a condition tree that allows users to record assumptions and choices made as an audit trail log.
- Preprint
(2686 KB) -
Supplement
(6700 KB) - BibTeX
- EndNote
Trevor Page et al.
Status: final response (author comments only)
-
RC1: 'Comment on hess-2022-349', Tobias Krueger, 03 Jan 2023
This is a short technical note on the first release of an uncertainty estimation toolbox for Matlab. The toolbox itself has very useful features, most notably from my perspective: advanced multivariate distributional assumptions through copulas; an audit trail log of assumptions made in the process; and 12 workflow scripts that will help the analyst apply the toolbox.
In my opinion it would increase the value of the paper (as an addition to the documentation of the toolbox) if the authors could give a little more detail on the nature of the uncertainty problem in each workflow. Now, if the paper is the first port of call for those wanting to use the toolbox, they have to do a lot of additional background reading before they can decide which method to use or which workflow to follow. If the authors could give a little more detail on the nature of uncertainties considered in each workflow and why this led them to use a certain method and not another. This would then give the reader a sense of which method is appropriate for what.
The description of the toolbox is otherwise clear, with the exception of the handling of input data uncertainty in the “conditioned” case. See also my comment to Fig6 below: Is input data uncertainty handled the same way as parameter priors? There should be added complexity due to the timeseries nature of inputs. What are the choices available to the analyst here? I’m thinking of rainfall multipliers and various other approaches that have been suggested. And are these prior choices of input data uncertainty updated conditional on the observations of model output? I think here we need more information in the paper.
Specific comments:
There is a font size change from L75 onwards. Is this intentional?
When reviewing the other toolboxes from L75 onwards, could the authors flag those toolboxes that are still maintained? With some I had the feeling they might not be (certainly DUE) and then they are of limited value in my opinion.
On the CURE website there is still what seems like an older version of the paper submitted to EMS. This should be updated.
Fig3-4: The font sizes and tick labels (overlap) could be improved in some instances.
Fig3: I find the evolution plot of the Rhat statistic not so useful since we can’t see the area around 1 very well at that scale where we want Rhat to end up. I encourage the authors to consider removing this plot or finding a way to scale it better (I guess the final Rhat value is reported in any case, which would be important).
Fig6: I’m unclear about the two branches coming from “Conditioned Uncertainty Analysis”. Is the analyst meant to go down both branches? If so, this could be made clear. This way the analyst seems to end up with prior parameter distributions at the end of the left branch that then feed into the sampling strategies emerging from the right branch. But what about the multivariate cases – what do they lead to? The same could be said about the middle arrow on the very left (forward uncertainty analysis). More importantly, what about input data uncertainty? Is this handled the same way as parameter priors? There should be added complexity due to the timeseries nature of inputs. What are the choices available to the analyst here? I’m thinking of rainfall multipliers and various other approaches that have been suggested. And are these prior choices of input data uncertainty updated conditional on the observations of model output?
Comments for general discussion:
I’m here noting a few general discussion points that I invite that the authors to engage with, though they are not of central importance to the paper.
First, I want to encourage the authors to eventually publish their toolbox for an open-source software environment like R as well. Or at least comment on the compatibility of the toolbox with clones like GNU Octave.
Second, I’m increasingly wondering whether the distinction between formal and informal uncertainty methods is really productive (I say this having used this distinction myself). This terminology once served to circumvent accusations of incoherence of methods like GLUE just by introducing a different label (informal), but now distracts from engaging with what really matters: the assumptions made about various sources of uncertainty when using certain methods (a problem that the authors summarise well by the way and tackle through their audit trail). With the formal/informal terminology one is led to believe one has a choice between formal and informal methods, while in reality in both cases one has a much more difficult choice of how exactly to model uncertainties and how to aggregate individual (e.g. timestep-based) performance measures into an overall metric (e.g. multiplicative or additive) and what understanding of uncertainties and model performance this entails (often implicitly). Such understanding goes down to foundational axioms as discussed by Nearing et al. (2016), which the authors cite. Any formal/informal distinction would also be increasingly blurred by methods such as ABC – here I’m missing reference to work by Lucy Marshall and co-workers discussing the similarities between ABC and GLUE, by the way. Maybe the authors can re-evaluate their use of these terms and engage in a broader discussion.
Third, I find the comment about rigorous statistical treatment of aleatory but not epistemic uncertainty in L59-61 misleading. This seems to be relating more to differences between frequentist and Bayesian statistical methods than anything else. The Bayesian framework deals expressly with epistemic uncertainties. The question is just whether or not the assumptions one makes are justified – but this is the case with any uncertainty method (which the authors emphasise well in this paper). I encourage the authors to remove this reference to epistemic versus aleatory uncertainty and focus on the importance of choices (which they already do) – in any method.
Citation: https://doi.org/10.5194/hess-2022-349-RC1 - AC1: 'Reply on RC1', Keith Beven, 20 Feb 2023
-
AC3: 'Reply on RC1', Keith Beven, 20 Feb 2023
Publisher’s note: this comment is a copy of AC1 and its content was therefore removed.
Citation: https://doi.org/10.5194/hess-2022-349-AC3
-
RC2: 'Comment on hess-2022-349', Anonymous Referee #2, 24 Jan 2023
The comment was uploaded in the form of a supplement: https://hess.copernicus.org/preprints/hess-2022-349/hess-2022-349-RC2-supplement.pdf
- AC2: 'Reply on RC2', Keith Beven, 20 Feb 2023
Trevor Page et al.
Model code and software
CURE Uncertainty Estimation Toolbox Trevor Page, Paul Smith, Keith Beven, Francesca Pianosa, Fanny Sarrazin, Susana Almeida https://www.lancaster.ac.uk/lec/sites/qnfm/credible/default.htm
Trevor Page et al.
Viewed
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 454 | 161 | 18 | 633 | 56 | 5 | 5 |
- HTML: 454
- PDF: 161
- XML: 18
- Total: 633
- Supplement: 56
- BibTeX: 5
- EndNote: 5
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1