the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Making a case for power-sensitive water modelling: a literature review
Rozemarijn ter Horst
Rossella Alba
Jeroen Vos
Maria Rusca
Jonatan Godinez-Madrigal
Lucie V. Babel
Gert Jan Veldwisch
Jean-Philippe Venot
Bruno Bonté
David W. Walker
Tobias Krueger
Abstract. Hydrological models are widely used to research hydrological change and risk. Yet, the power embedded in the modelling process and outcomes are often concealed by claiming its neutrality. Our systematic review shows that in scientific literature relatively little attention is given to the power of models to influence development processes and outcomes in water governance. The review also shows that there is much to learn from those who are willing to be openly reflexive on the influence of models. In agreement with this emerging body of work, we call for power-sensitive modelling, which means that people are critical about how models are made and with what implications, taking into account that: i) The choice for and use of models for water management happens in a political context and has political consequences; ii) Models are the result of choices made by modellers and – since they have political consequences – these need to be made as explicit as possible as opposed to being “blackboxed”; iii) To consider the ethical implications of the choices of modellers, commissioners, and users, and to improve accountability, models and their power need to be understood by connecting the inner workings of a model with a contextual understanding of its development and use, iv) Action is taken upon these implications by democratising modelling processes. Our call should not be understood as a suggestion to do away with modelling altogether, but rather as an invitation to interrogate how quantitative models may help to foster transformative pathways towards more just and equitable water distributions.
- Preprint
(805 KB) - Metadata XML
- BibTeX
- EndNote
Rozemarijn ter Horst et al.
Status: open (until 25 Oct 2023)
-
RC1: 'Comment on hess-2023-164', Keith Beven, 31 Aug 2023
reply
Review of HESS-2023-164
Horst et al., Making a case for power-sensitive water modelling: a literature review
This is a really interesting paper that covers a wide range of ground but which ends up being rather vague in its outcomes and recommendations and I would suggest rather wrong in it analysis.
Because the authors seem to be falling into the trap of talking about models as if they were responsible for outcomes – and as I have written several times before, decision makers are only too happy to transfer responsibility to the models. But that is not how (in general) it works. With the exception of some academic studies (such as many of those cited here) models are conditioned for a purpose. The power relationship here is not at all in the model, the model is only a consequence of making assumptions, so the important factor is who decides on the assumptions. This will be constrained initially by how a project is framed (the political as used by the authors, which may well reflect the vested interests of whoever is doing the commissioning or decision making) and then by how the modeller interprets the brief (including how much time is costed when bids are competitive). Of course, in a sociohydrological model, then the relative importance of different stakeholder groups might be a factor that needs to be included – but that again comes down to whether the assumptions made are appropriate for the purpose.
The confounding of this distinction starts in the section on Defining Models which suggests that a model is anything that takes an input to produce an output. There is nothing here on purpose, fitness-for-purpose, responsibility or explicitly defining the condition tree of assumptions (including considerations of uncertainty) in any model application. Indeed, the study cited by Ramsey (2009) (L165ff) is an example of where the assumptions used made the model not fit-for-purpose (see the discussion of fit-for-purpose in Beven and Lane, 2022). The later example of Andersson et al (2004) is also an example of an application where the model was not fit for purpose. So is not part of the issue here the question of how to get consensus on what might be fit-for-purpose (depending on the purpose). It could be useful here to bring in the idea of the condition tree of assumptions as a way of improving this process – both to get participatory agreement and understanding (and an audit trail for later reconsideration). See for example, Beven et al. CIRIA 2014; or the Page et al., HESS 2023 toolbox). This is a way of getting the more open/explicit approach suggested in Section 4.1.3 and in the conclusions.
I would argue that the political in the sense used by the authors is not really the “broad influence of the models” that they suggest but rather the whole framing of the decision making process, within which the model is only a tool. This can often be seen in how historical legal constraints on water resources management dominate any attempts to achieve sustainability of use (for either human use or biodiversity) regardless of whatever model might be used. I accept, of course, that models are not necessarily neutral in this process, even if most modellers that I know of will try to do the best they can given the data they might have available). The assumptions can certainly reflect the power of requiring a certain outcome (a nice example, in the inquiry into the Sellafield Rock Experiment Facility as the first stage in the UK nuclear waste disposal strategy, where the two opposing sides used quite different assumptions in assessing groundwater flow pathways).
The idea of purpose also interacts with framing – this is evident in the examples of the Rhone and Seine in Section 4.1.2. These projects were tasked with different purposes when the projects were commissioned – it is too simple to say that this was the result of contrasting world-views of geographers and engineers. It was actually the result of upstream decisions about commissioning the projects before any model or modeller was even involved (actually that might not be entirely correct, as I believe there was initially a commission project at least on the Seine to consider what might be technologically possible - but more background detail would be needed to make that point). It is again the wider political framework that is determining outcomes here – not the model (though, yes, as above for the Seine, in some cases the available model technology might feedback to how projects are defined, but that is not what you are saying here. It would be an interesting study in itself – many of the commercial SHE model projects were of that type, for example)
L265 – you put ‘optimal’ in commas but without further comment when there has long been discussion in hydrological modelling critical of the concept of optimality, either in model outputs or in decision making (e.g. my 2006 equifinality paper and earlier). So yes, modeller choices matter and make a difference but these are old examples now – better if multiple model outputs are considered (model ensembles are now widely used) in a context of assessing fitness-for-purpose within the limitations of uncertainty.
L297. 20%. – yes, but there was a history to that in a simplified interpretation of the results of a CEH study of the impacts of climate change on the Severn and Thames catchments. It was conditional on the climate scenarios chosen and available at that time (and so should have been subject to revision). It is not what the models or modellers of that study said but, for simplicity, (and cheapness in low budget applications) it has persisted. Another example of the dominance of the wider political framework over the model.
Section 4.3.1. OK, so models can be used in ways to support vested interests but that again is a political issue, not a model-specific issue (as in the Sellafield example above where the opposing side effectively won the case by demonstrating the huge potential uncertainty in model results. Similar model, different assumptions. So why is the comment about models being value neutral just thrown in at the end without further discussion? That is a claim made for political ends, not a feature of the model.
L335. In respect of groundwater models there are also examples of studies showing that different experts come up with different conceptual models (e.g. Refsgaard conceptual groundwater model paper), but, more important here, are the post-audit studies of Bredehoeft and Konikow (1992) and Anderson and Woesnner (1992) in a special issue of AWR who showed that nearly all groundwater models did not prove to be correct with hindsight, but mostly because the projected boundary conditions had not proven correct (ie. the assumptions rather than the model again). That has put people off doing post-audit analyses for both hydrological and groundwater models ever since….. (even though that could be an invaluable learning process).
Section 4.3.2. “When modelling is presented as a neutral scientific process …. legitimacy given by external consultants.” These are surely not the same thing as is being implied here. The consultants may well have been trying to do the best job possible given the data available, independently of any vested interest (you do not provide any evidence to the contrary). The model, given better data might well have been a better representation of the system. The model can be neutral in that respect; even if the way it is used might not be. The question again is whether that implementation should have been considered fit-for-purpose for the decisions being made. “Framing their actions as illegal…” also surely has nothing to do with the model?
L363 “The decision over water allocation was eventually enforced through influence at the highest political level, the President of Mexico. Jensen (2020) also confirmed that the power of high-level decision makers plays a key role. In the case of the Mekong, the author showed there is a certain saturation in knowledge developed by models, and there is a clear limitation in their impact as governments were unwilling to build on these insights.” But exactly! That is not the same as you then go on to say “The previous examples show how models can work exclusively…” when their outputs are rejected or ignored????? The models are not working exclusively in support of the chosen solution surely? And what you discuss in the remainder of this section is how a model might be useful as a tool in framing good practice (as was also the case of Pickering). But it is again not the model as such (which might still be too much of a simplification or lacking in data), but the way the model is used.
Section 4.4.1 You mention the importance of scale here but not the importance of visualisations in how local people can interact with model outputs (as in the models of everywhere concepts of Beven, HESS 2007, Beven et al., JRBM 2014; and Blair et al., EMS 2019).
Avoid models that are overly complex? Over-complex with respect to what? To the problem at hand or to the understanding of stakeholders. The second should surely not override the adequate complexity of the first?
Having said that, the power relationships of this type of co-evolutionary modelling are definitely an issue, but as some of your example studies have shown not necessarily insurmountable if the will is there. But many of the problems you have identified so far are a result of the imposition of power structures, regardless of the model or its results.
Section 4.4.2. Consideration of assumptions about uncertainty (including epistemic uncertainties) in modelling is critical to model evaluation and fitness-for-purpose but this is only really mentioned in this Section in relation to an application of SWAT (widely applied – it is free to use - but which elsewhere has been shown to be not fit-for-purpose for the type of application described here, even allowing for uncertainty – see Hollaway et al., JH, 2018).
What do the authors mean here by “As modelling inhibits more uncertainty than measurement” (L431)? The model here is being used to predict years ahead so should surely be more uncertain than any available measurements (that cannot in any case be made in the future).
L447 Pickering – “ultimately played a key role in shaping flood management strategy in the area.” Well yes – but you should then perhaps finish the story. It showed that the NFM strategy preferred by local stakeholders would not protect the properties at risk. The fact that the EA had also been involved in the process then meant that they invested £1.5m in a concrete flood detention basin, despite the cost-benefit for the scheme being considered too low compared with other sites. That was then a political decision.
L459. “they conclude that models assuming that residents are well informed and have shared understandings of the water supply system might lead to an oversimplification of sociohydrological dynamics in a given location, and that more local involvement could mitigate this”. OK (though not clear what type of quantitative model you might mean here) but is that just not another example of poor assumptions/poor practice/not fit-for-purpose/design for vested interest – ie. there is again surely a need to distinguish between the model and the way in which it is used.
L477 “which raises questions about the responsibility and accountability of those making and using models” Well yes, and that is the problem I have with most of your discussion since you are placing responsibility on the “model” and not on those who use and misuse them (not necessarily even the modeller, but even more so those who commission studies and/or use the results for decision making as you have demonstrated).
L606 “how quantitative models may help to foster transformative pathways towards more just and equitable water distributions.” But why put the focus on the model here? What is needed is the political will for more just and equitable water distribution. Given that will, everything else would probably follow, but I do not see how you expect models (or modellers come to that) to influence the neoliberal capitalism or centralised communist systems that prevail in most countries where sustainable and equitable water issues are important. Look at the UK – we do not have the problems of degrading water quality because of any modelling or modeller issues. And announced this week is a relaxation of the rules on new housing developments in respect of water quality. A relaxation of rules in designating water quality categories is also expected now that we do not have to conform to the EU WFD standards. Underlying many of the examples you provide is exactly that political framework, outside the control of modellers and their models.
So while I appreciate the sentiments that lie behind the paper, I think you have got the framing wrong. Most of the power issues in moving towards better water management have very little to do with models or modellers – they are political (in the sense of the authors’ use). So to say that models need to become power sensitive is not correct. It might be better put that modellers need to be more critical or sensitive to the context within which they use their models – both in terms of the framing of a project and whether a model is fit-for-purpose within such a project, but it is often the case that there is much that is outside their control (the legal basis of water rights in a country, the details of a project commission, etc). You give examples of where model outputs were rejected because they were in conflict with a desired outcome – that is surely a bigger problem of power.
So my apologies for these extended comments but it is something I have thought about over a long period of time. I think there are ways ahead (I have suggested some partial solutions in the past such as the condition tree / audit trail of assumptions, models of everywhere as ways of facilitating interaction and criticism, better evaluation of fitness-for-purpose and consideration of model uncertainties) but I think this paper needs to be reformulated much more about power in the political framing of the modelling process than assigning the responsibility to power sensitive models.
Some other points
The choice of papers analysed seems incomplete. For the once case study that I know something about (Pickering) some papers are included (the papers by Lane et al. and Landström et al), but others are not (particularly Lane et al. 2011, Doing flood risk science differently in TIBG, and the papers on Pickering by Sarah Whatmore). Were these considered as having too much overlap or is it an indication that the methodology could have retrieved other relevant papers?
The Morgan and Morrison 1999 paper does not appear in the references.
The Pielke Jr. reference appears twice.
Keith Beven
References
Beven, K J, 2006, A manifesto for the equifinality thesis, J. Hydrology, 320, 18-36.
Beven, K J, 2007, Working towards integrated environmental models of everywhere: uncertainty, data, and modelling as a learning process. Hydrology and Earth System Science, 11(1), 460-467.
Beven, K J, Lamb, R, Leedal, D T, and Hunter, N, 2014, Communicating uncertainty in flood risk mapping: a case study, Int. J. River Basin Manag., 13(3):285-296, DOI:10.1080/15715124.2014.917318.
Beven, K. J., Leedal, D. T., McCarthy, S., 2014, Framework for assessing uncertainty in fluvial flood risk mapping, CIRIA report C721, available at http://www.ciria.org/Resources/Free_publications/fluvial_flood_risk_mapping.aspx
Beven, K. J. and Lane, S., 2022. On (in)validating environmental models. 1. Principles for formulating a Turing-like Test for determining when a model is fit-for purpose. Hydrological Processes, 36(10), e14704, https://doi.org/10.1002/hyp.14704.
Blair, G.S., Beven, K.J., Lamb, R., Bassett, R., Cauwenberghs, K., Hankin, B., Dean, G., Hunter, N., Edwards, E., Nundloll, V., Samreen, F., Simm, W., Towe, R., 2019, Models of Everywhere Revisited: A Technological Perspective, Environmental Modelling and Software,https://doi.org/10.1016/j.envsoft.2019.104521
Hollaway, M.J., Beven, K.J., Benskin, C.McW.H., Collins, A.L., Evans, R., Falloon, P.D., Forber, K.J., Hiscock, K.M., Kahana, R., Macleod, C.J.A., Ockenden, M.C., Villamizar, M.L., Wearing, C., Withers, P.J.A., Zhou, J.G., Haygarth, P.M., 2018, Evaluating a processed based water quality model on a UK headwater catchment: what can we learn from a ‘limits of acceptability’ uncertainty framework?, J. Hydrology. 558: 607-624. Doi:10.1016/j.jhydrol.2018.01.063
Page, T., Smith, P., Beven, K.J., Pianosi3, F., Sarrazin, F., Almeida, S., Holcombe, E., Freer, J., Chappell, N.A., 2023, Technical Note: The CREDIBLE Uncertainty Estimation (CURE) toolbox: facilitating the communication of epistemic uncertainty, Hydrology and Earth Systems Science,
Citation: https://doi.org/10.5194/hess-2023-164-RC1 -
CC1: 'Comment on hess-2023-164', Bich Ngoc Tran, 26 Sep 2023
reply
Congratulations and thanks to the authors for a very interesting literature synthesis.
I'm abit confused with your choice of words. You used the passive form of verbs when describing the modeling process: "models are developed", "models are made", and "models are represented/understood" which indicates that these actions are done by some people with ethical responsibilities. But when discussing the influence or power of models, you sometimes used the active form of verbs as if a model (a simplification of the world) could act on its own, thus, hiding the people who gave power to models. Is there a reason for this choice of words? This makes it sound like you are "placing responsibility on the “model” and not on those who use and misuse them" (K. Beven). For example:
L319 "the models ended up legitimising the worldviews of industry and state...". Why not "models are used by ... to legitimise..."?
L322 "models also contributed to ground the debates on scientific terminology and concepts, thereby forcing groups contesting these worldviews to draw on the same language and knowledge claims." Why not "models were used by .... to ground the debates ..."? Who forces the contesting groups to draw on the same language and knowledge claims?
L352 "the model facilitated the implementation of a policy...". Why not "water resources agency used the model to facilitate the implementation of a policy"?
L361 "models supported top-down management of water-scarcity issues...". Why not "models are used by .... to support..."?
Would these models have the same power if they were used by people with less political power? Would these models have the same influence on the decision-making if they were funded/commissioned by people with the same political power but with a stronger political will for equitable water governance?
I'm curious how "models are commissioned/funded" in the selected studies in Section 4.3. Before the modeling process was even started, how does the decision to develop a model was made? Who made that decision and provided funding for model development? Maybe these models had some real-world impacts because they were commissioned to support decision and make an impact in the first place, unlike models that are developed to advance scientific knowledge.
I don't think all hydrological models are political or equally political. I think it's important to identify when, where, and how "a simplification of the world" starts to have an influence on the world. For example, we can list certain conditions that make a model more or less political (e.g., diagnostic/explanatory vs. prognostic/predictive models, open-source vs. close-source, commissioned vs. original research ideas).
I think section 5 would be clearer if we could distinguish the technical and ethical responsibilities of the different actors: modelers, model-users (do you mean the ones who use model results to make decisions?), and funders in each of the four mentioned considerations.
Some questions on the methodology:
L134: what do you mean by "stood out"? It's not clear to me how you identify these aspects.
Figure 1: "absence of elements of reflexivity". what are those elements? how did you identify those elements in the literature?
"pre-screened articles from other sources". what are the sources? how did you pre-screen those?
"studies included in qualitative synthesis - X not done". it seems you still need to update X.The fact that you have more articles by pre-screening from other sources than articles from searching using keywords suggests that the choice of keywords was not as efficient as your "pre-screening" method. Did you find the pre-screened articles in your search results?
Some articles may be interesting:
Horton, P., Schaefli, B. and Kauzlaric, M., 2022. Why do we have so many different hydrological models? A review based on the case of Switzerland. Wiley Interdisciplinary Reviews: Water, 9(1), p.e1574.
White, D.D., Wutich, A.Y., Larson, K.L. and Lant, T., 2015. Water management decision makers' evaluations of uncertainty in a decision support system: the case of WaterSim in the Decision Theater. Journal of Environmental Planning and Management, 58(4), pp.616-630.
Bijlsma, R.M., Bots, P.W., Wolters, H.A. and Hoekstra, A.Y., 2011. An empirical analysis of stakeholders’ influence on policy development: the role of uncertainty handling. Ecology and society, 16(1).
Jackson, S., 2006, May. Water models and water politics: design, deliberation, and virtual accountability. In Proceedings of the 2006 international conference on Digital government research (pp. 95-104).
Citation: https://doi.org/10.5194/hess-2023-164-CC1
Rozemarijn ter Horst et al.
Rozemarijn ter Horst et al.
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 331 | 72 | 6 | 409 | 3 | 2 |
- HTML: 331
- PDF: 72
- XML: 6
- Total: 409
- BibTeX: 3
- EndNote: 2
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1