Review of HESS-2023-164

Horst et al., Making a case for power-sensitive water modelling: a literature review

This is a really interesting paper that covers a wide range of ground but which ends up being rather vague in its outcomes and recommendations and I would suggest rather wrong in it analysis.  

Because the authors seem to be falling into the trap of talking about models as if they were responsible for outcomes – and as I have written several times before, decision makers are only too happy to transfer responsibility to the models.  But that is not how (in general) it works.   With the exception of some academic studies (such as many of those cited here) models are conditioned for a purpose.  The power relationship here is not at all in the model, the model is only a consequence of making assumptions, so the important factor is who decides on the assumptions.   This will be constrained initially by how a project is framed (the political as used by the authors, which may well reflect the vested interests of whoever is doing the commissioning or decision making) and then by how the modeller interprets the brief (including how much time is costed when bids are competitive).   Of course, in a sociohydrological model, then the relative importance of different stakeholder groups might be a factor that needs to be included – but that again comes down to whether the assumptions made are appropriate for the purpose.

The confounding of this distinction starts in the section on Defining Models which suggests that a model is anything that takes an input to produce an output.   There is nothing here on purpose, fitness-for-purpose, responsibility or explicitly defining the condition tree of assumptions (including considerations of uncertainty) in any model application.   Indeed, the study cited by Ramsey (2009) (L165ff) is an example of where the assumptions used made the model not fit-for-purpose (see the discussion of fit-for-purpose in Beven and Lane, 2022).   The later example of Andersson et al (2004) is also an example of an application where the model was not fit for purpose.  So is not part of the issue here the question of how to get consensus on what might be fit-for-purpose (depending on the purpose). It could be useful here to bring in the idea of the condition tree of assumptions as a way of improving this process – both to get participatory agreement and understanding (and an audit trail for later reconsideration).   See for example, Beven et al. CIRIA 2014; or the Page et al., HESS 2023 toolbox).   This is a way of getting the more open/explicit approach suggested in Section 4.1.3 and in the conclusions.

I would argue that the political in the sense used by the authors is not really the “broad influence of the models” that they suggest but rather the whole framing of the decision making process, within which the model is only a tool.   This can often be seen in how historical legal constraints on water resources management dominate any attempts to achieve sustainability of use (for either human use or biodiversity) regardless of whatever model might be used.   I accept, of course, that models are not necessarily neutral in this process, even if most modellers that I know of will try to do the best they can given the data they might have available).   The assumptions can certainly reflect the power of requiring a certain outcome (a nice example, in the inquiry into the Sellafield Rock Experiment Facility as the first stage in the UK nuclear waste disposal strategy, where the two opposing sides used quite different assumptions in assessing groundwater flow pathways).

The idea of purpose also interacts with framing – this is evident in the examples of the Rhone and Seine in Section 4.1.2.   These projects were tasked with different purposes when the projects were commissioned – it is too simple to say that this was the result of contrasting world-views of geographers and engineers.  It was actually the result of upstream decisions about commissioning the projects before any model or modeller was even involved (actually that might not be entirely correct, as I believe there was initially a commission project at least on the Seine to consider what might be technologically possible - but more background detail would be needed to make that point).  It is again the wider political framework that is determining outcomes here – not the model (though, yes, as above for the Seine, in some cases the available model technology might feedback to how projects are defined, but that is not what you are saying here.  It would be an interesting study in itself – many of the commercial SHE model projects were of that type, for example)

L265 – you put ‘optimal’ in commas but without further comment when there has long been discussion in hydrological modelling critical of the concept of optimality, either in model outputs or in decision making (e.g. my 2006 equifinality paper and earlier).   So yes, modeller choices matter and make a difference but these are old examples now – better if multiple model outputs are considered  (model ensembles are now widely used) in a context of assessing fitness-for-purpose within the limitations of uncertainty.

L297. 20%. – yes, but there was a history to that in a simplified interpretation of the results of a CEH study of the impacts of climate change on the Severn and Thames catchments.  It was conditional on the climate scenarios chosen and available at that time (and so should have been subject to revision).   It is not what the models or modellers of that study said  but, for simplicity, (and cheapness in low budget applications) it has persisted.   Another example of the dominance of the wider political framework over the model.

Section 4.3.1.  OK, so models can be used in ways to support vested interests but that again is a political issue, not a model-specific issue (as in the Sellafield example above where the opposing side effectively won the case by demonstrating the huge potential uncertainty in model results.  Similar model, different assumptions.   So why is the comment about models being value neutral just thrown in at the end without further discussion?   That is a claim made for political ends, not a feature of the model.

L335.  In respect of groundwater models there are also examples of studies showing that different experts come up with different conceptual models (e.g. Refsgaard conceptual groundwater model paper), but, more important here, are the post-audit studies of Bredehoeft and Konikow (1992) and Anderson and Woesnner (1992) in a special issue of AWR who showed that nearly all groundwater models did not prove to be correct with hindsight, but mostly because the projected boundary conditions had not proven correct (ie. the assumptions rather than the model again).   That has put people off doing post-audit analyses for both hydrological and groundwater models ever since….. (even though that could be an invaluable learning process).

Section 4.3.2. “When modelling is presented as a neutral scientific process …. legitimacy given by external consultants.”    These are surely not the same thing as is being implied here.   The consultants may well have been trying to do the best job possible given the data available, independently of any vested interest (you do not provide any evidence to the contrary).   The model, given better data might well have been a better representation of the system.   The model can be neutral in that respect; even if the way it is used might not be.  The question again is whether that implementation should have been considered fit-for-purpose for the decisions being made.      “Framing their actions as illegal…” also surely has nothing to do with the model?

L363 “The decision over water allocation was eventually enforced through influence at the highest political level, the President of Mexico. Jensen (2020) also confirmed that the power of high-level decision makers plays a key role. In the case of the Mekong, the author showed there is a certain saturation in knowledge developed by models, and there is a clear limitation in their impact as governments were unwilling to build on these insights.”   But exactly!  That is not the same as you then go on to say “The previous examples show how models can work exclusively…” when their outputs are rejected or ignored?????  The models are not working exclusively in support of the chosen solution surely?   And what you discuss in the remainder of this section is how a model might be useful as a tool in framing good practice (as was also the case of Pickering).    But it is again not the model as such (which might still be too much of a simplification or lacking in data), but the way the model is used.

Section 4.4.1 You mention the importance of scale here but not the importance of visualisations in how local people can interact with model outputs (as in the models of everywhere concepts of Beven, HESS 2007, Beven et al., JRBM 2014;  and Blair et al., EMS 2019).

Avoid models that are overly complex?  Over-complex with respect to what?   To the problem at hand or to the understanding of stakeholders.   The second should surely not override the adequate complexity of the first?

Having said that, the power relationships of this type of co-evolutionary modelling are definitely an issue, but as some of your example studies have shown not necessarily insurmountable if the will is there.   But many of the problems you have identified so far are a result of the imposition of power structures, regardless of the model or its results.

Section 4.4.2. Consideration of assumptions about uncertainty (including epistemic uncertainties)  in modelling is critical to model evaluation and fitness-for-purpose but this is only really mentioned in this Section in relation to an application of SWAT (widely applied – it is free to use - but which elsewhere has been shown to be not fit-for-purpose for the type of application described here, even allowing for uncertainty – see Hollaway et al., JH, 2018).    

What do the authors mean here by “As modelling inhibits more uncertainty than measurement” (L431)?  The model here is being used to predict years ahead so should surely be more uncertain than any available measurements (that cannot in any case be made in the future).

L447 Pickering – “ultimately played a key role in shaping flood management strategy in the area.”  Well yes – but you should then perhaps finish the story.   It showed that the NFM strategy preferred by local stakeholders would not protect the properties at risk.  The fact that the EA had also been involved in the process then meant that they invested £1.5m in a concrete flood detention basin, despite the cost-benefit for the scheme being considered too low compared with other sites.  That was then a political decision.

L459. “they conclude that models assuming that residents are well informed and have shared understandings of the water supply system might lead to an oversimplification of sociohydrological dynamics in a given location, and that more local involvement could mitigate this”.   OK (though not clear what type of quantitative model you might mean here) but is that just not another example of poor assumptions/poor practice/not fit-for-purpose/design for vested interest – ie. there is again surely a need to distinguish between the model and the way in which it is used.  

L477 “which raises questions about the responsibility and accountability of those making and using models”   Well yes, and that is the problem I have with most of your discussion since you are placing responsibility on the “model” and not on those who use and misuse them (not necessarily even the modeller, but even more so those who commission studies and/or use the results for decision making as you have demonstrated). 

L606 “how quantitative models may help to foster transformative pathways towards more just and equitable water distributions.”    But why put the focus on the model here?   What is needed is the political will for more just and equitable water distribution.  Given that will, everything else would probably follow, but I do not see how you expect models (or modellers come to that) to influence the neoliberal capitalism or centralised communist systems that prevail in most countries where sustainable and equitable water issues are important.    Look at the UK – we do not have the problems of degrading water quality because of any modelling or modeller issues.   And announced this week is a relaxation of the rules on new housing developments in respect of water quality.   A relaxation of rules in designating water quality categories is also expected now that we do not have to conform to the EU WFD standards.   Underlying many of the examples you provide is exactly that political framework, outside the control of modellers and their models.

So while I appreciate the sentiments that lie behind the paper, I think you have got the framing wrong.   Most of the power issues in moving towards better water management have very little to do with models or modellers – they are political (in the sense of the authors’ use).   So to say that models need to become power sensitive is not correct.   It might be better put that modellers need to be more critical or sensitive to the context within which they use their models – both in terms of the framing of a project and whether a model is fit-for-purpose within such a project, but it is often the case that there is much that is outside their control (the legal basis of water rights in a country, the details of a project commission, etc).  You give examples of where model outputs were rejected because they were in conflict with a desired outcome – that is surely a bigger problem of power.

So my apologies for these extended comments but it is something I have thought about over a long period of time.    I think there are ways ahead (I have suggested some partial solutions in the past such as the condition tree / audit trail of assumptions, models of everywhere as ways of facilitating interaction and criticism, better evaluation of fitness-for-purpose and consideration of model uncertainties) but I think this paper needs to be reformulated much more about power in the political framing of the modelling process than assigning the responsibility to power sensitive models.

Some other points

The choice of papers analysed seems incomplete.  For the once case study that I know something about (Pickering) some papers are included (the papers by Lane et al. and Landström et al), but others are not (particularly Lane et al. 2011, Doing flood risk science differently in TIBG, and the papers on Pickering by Sarah Whatmore).   Were these considered as having too much overlap or is it an indication that the methodology could have retrieved other relevant papers?

The Morgan and Morrison 1999 paper does not appear in the references.

The Pielke Jr. reference appears twice.

Keith Beven

References

Beven, K J, 2006, A manifesto for the equifinality thesis, J. Hydrology, 320, 18-36.

Beven, K J, 2007, Working towards integrated environmental models of everywhere: uncertainty, data, and modelling as a learning process. Hydrology and Earth System Science, 11(1), 460-467.

Beven, K J, Lamb, R, Leedal, D T, and Hunter, N, 2014, Communicating uncertainty in flood risk mapping: a case study, Int. J. River Basin Manag., 13(3):285-296, DOI:10.1080/15715124.2014.917318.

Beven, K. J., Leedal, D. T., McCarthy, S., 2014, Framework for assessing uncertainty in fluvial flood risk mapping, CIRIA report C721, available at http://www.ciria.org/Resources/Free_publications/fluvial_flood_risk_mapping.aspx

Beven, K. J. and Lane, S., 2022. On (in)validating environmental models. 1.  Principles for formulating a Turing-like Test for determining when a model is fit-for purpose. Hydrological Processes, 36(10), e14704, https://doi.org/10.1002/hyp.14704.

Blair, G.S.,  Beven, K.J., Lamb, R.,  Bassett, R.,  Cauwenberghs, K.,  Hankin, B.,  Dean, G., Hunter, N.,  Edwards, E., Nundloll, V.,  Samreen, F., Simm, W., Towe, R., 2019, Models of Everywhere Revisited: A Technological Perspective, Environmental Modelling and Software,https://doi.org/10.1016/j.envsoft.2019.104521

Hollaway, M.J., Beven, K.J., Benskin, C.McW.H., Collins, A.L., Evans, R., Falloon, P.D., Forber, K.J., Hiscock, K.M., Kahana, R., Macleod, C.J.A., Ockenden, M.C., Villamizar, M.L., Wearing, C., Withers, P.J.A., Zhou, J.G., Haygarth, P.M., 2018, Evaluating a processed based water quality model on a UK headwater catchment: what can we learn from a ‘limits of acceptability’ uncertainty framework?,  J. Hydrology.  558: 607-624. Doi:10.1016/j.jhydrol.2018.01.063

Page, T.,  Smith, P., Beven, K.J., Pianosi3, F., Sarrazin, F., Almeida, S., Holcombe, E., Freer, J.,  Chappell, N.A., 2023, Technical Note: The CREDIBLE Uncertainty Estimation (CURE) toolbox: facilitating the communication of epistemic uncertainty, Hydrology and Earth Systems Science,

Review by Stuart Lane, University Lausanne

My apologies that this review is a little late. This paper needed quite some thought. I find its goals useful and there is some very useful material therein that the hydrological modelling community should be aware of. I should be honest about a few of my own positions with respect to this work. I don’t particularly like this kind of methodology – finding literature this way often means that very interesting papers that perhaps frame themselves differently get lost – and because they are not found other kinds of framing that exist are left hidden. Second, I apologise that there is quite some reference to my own work in my report but I have been working on this topic for almost 20 years. Some of the ideas the authors have could and have been developed a lot more (and the work I refer to, notably Lane 2012, might help them access the work of others besides me). In especially the Discussion, I felt that the authors did not really show as much awareness as might be ideal of some of the dimensions that they address (mental images, framing) and which have already been discussed in relation to hydrology. There could be a lot more useful Discussion and crucially, perhaps my key recommendation, some much deeper thinking regarding what a power-sensitive modelling approach might look like and how it might come about (which itself would have to be a political act).

The detailed comments identify some of these issues that could be developed. As major comments though, and in addition, I think the following need some treatment in the paper.

1. Models are themselves political objects – they contain intellectual property that can come to have commercial value – the history of how flood inundation modelling went from 1D to 2D in the UK was about primarily the protection of vested interests in intellectual property (for example, see one of the sections in Lane, S.N., November, V., Landström, C. and Whatmore, S.J., 2013. Explaining rapid transitions in the practice of flood risk management. Annals of the Association of American Geographers, 103, 330-42) – this is similarly reflected in James Porter’s fascinating work about how flood maps get made (which is essentially based upon flood inundation modelling) and the negotiation between modellers/mappers and regulatory agencies (Porter, J. and Demeritt, D., 2012. Flood Risk Management, Mapping and Planning: The Institutional Politics of Decision-Support in England. Environment and Planning A). There is a tendency to reduce politically sensitive modelling to the interface between models, modellers and society – and to overlook the politics of modelling itself and how this also needs to become more politics sensitive. In turn this would sit this paper also in wider Science-Technology-Studies perspectives on power within scientific practice. Politics exists within modelling communities and this can have a profound impact on how modelling is done. Politics can constrain what is modelled and how. This needs more consideration and discussion.
2. If we accept the hypothesis that power is imbricated with and within models of the water environment (which I do), the question becomes what to do. The paper has very little on what this means for practice. A starting point may be the obvious point that we should not make recourse to some kind of naïve goal that we should exclude power and politics from modelling such that models give us access to some kind of higher, truer knowledge and models can then reign supreme ; that would simply be an internal contradiction in that it would imply the transfer of power to models and modellers (that is itself be a political statement). So, if power and politics cannot be excluded what would a more democratic form of modelling mean in practice. What do we mean by “democratic”? How might different kinds of politics (e.g. majoritarian versus minoritarian) lead to models being used, power-sensitively, in different ways? What are the ways of using models that allow those excluded from decision making to become included in decision-making through the use of models. When I got to the end of the discussion I felt short-changed that after all the examples this hadn’t been thought through.
3. When I wrote the 2014 paper published in HESS I should confess that this was motivated by an extreme skepticism (that I still hold strongly) of socio-hydrology and the focus of the Panta Rhea decade; this included the appearance of diagrams coupling models of people to models of the hydrological cycle that reminded me of those the Dick Chorley and others were advocating in the 1960s and 1970s. Yet again, we as hydrologists seemed to be bolting people onto the natural environment with very little clue as to actually what the “socio-“ really means. This was ironic because at the same time there was much interest from social scientists, notably Linton, Budd etc., in the Hydrosocial cycle which had, in my view, a much deeper understanding of how people and water are truly coupled. Now, I regret leaving in Lane (2014) this argument a little too subtle. But, I think the distinction between socio-hydrology and the Hydrosocial cycle does actually matter to this article because the need for political sensitivity reflects (for ethical reasons, at the very least) the need for a much more nuanced understanding of how hydrological models fit into the world. This is lost at the moment in this article. It probably does not need much discussion added but the notion of a hydrosocial cycle and the mutual influences between people and water strengthen the argument that we should also take a political perspective for how hydrological modelling is done.

Minor comments

L36 – but also by the situated nature of the modeller themselves – and the work they do to come to trust their own models – see Lane, S.N., 2012. Making mathematical models perform in geographical space(s). Chapter 17 in Agnew, J. and Livingstone, D. Handbook of Geographical Knowledge. Sage, London

L36-7 – your references here, unless I am mistaken, largely support the counter-factual – do you have any better references of modellers themselves presenting their models as neutral tools? Without this there is a risk that you are setting up a “straw doll”.

L37-8 – I don’t agree with this as it is written – we may think models can travel easily between places but much modelling in practice is about making them work in particular settings. There are studies of hydrological modelling in practice – see for instance Landstrom, C., Whatmore, S.J. and Lane, S.N., 2011. Virtual Engineering: computer simulation modelling for UK flood risk management. Science Studies, 24, 3-22

L63 – the notion of power sensitive modelling is interesting – but I think the paper is missing a wider link here into science technology studies that argues that power is an inevitable (and sometimes malign) component of any piece of scientific investigation. Isabelle Stengers has treated this generally (her book “Une autre science est possible” is probably the simplest entry point); there is then work around any one of a number of disciplines exploring this as well as some very specific to modelling of the environment (Demeritt, 2001 is a classic in this sense). I appreciate that the paper is about water but it is not well situated with respect to the wider STS work on the demonstrated influence of power upon scientific practice (and the very obvious point that the moment you advocate a model as neutral and therefore the valid basis for decision-making, you immediately transfer power to the model and the modeler, that is you make a political decision). Some of these examples also clould be used to develop the argument around L90 a bit more – where it is under-developed in terms of what we already know about the relationship between power and modelling.

Shackley, S., Risbey, J., Stone, P., and Wynne, B.: Adjusting to policy expectations in climate change modeling – An interdisciplinary study of flux adjustments in coupled atmosphere-ocean general circulation models. Climatic Change, 43, 413–454, 1999

Demeritt, D.: The Construction of Global Warming and the Politics of Science, Annals of the Association of American Geographer, 91, 307–337, 2001

Lahsen, M.: Seductive simulations? Uncertainty distribution around climate models, Social Studies of Science, 35, 895–922, 200B.

Sundberg, M.: The Everyday World of Simulation Modeling: The Development of Parameterizations in Meteorology, Science Technology and Human Values, 34, 162–181, 2009

Brysse, K., Oreskes, N., O’Reilly, J., and Oppenheimer, M.: Climate change prediction: Erring on the side of least drama?, Global Environmental Change, 23, 327–337, 2013.

L84-88 See Lane (2012) op cit.

L108 – water is not just about hydrology but also hydraulics – is there a reason you did not also look at hydraulics. Note also that this way of searching likely misses more specific model applications or where models are not really referred to as “models” even if modelling is an integral part of what is being done.

L115 – but isn’t this also an important result in the context of this paper?

L124 – clarify what you mean by “narrative style”

L145-150 – see in particular Beck (1999) who talks about “mental images” - Beck, M.B., 1999. Coping with ever larger problems, models, and data bases. Water Science and Technology, 39, 1-11 – but there is also a wider literature on framing in hydrological modelling and its critical role in shaping what is modelled – see Odoni, N. and Lane, S.N., 2010. Knowledge-theoretic models in hydrology. Progress in Physical Geography, 34, 151-71 – this applies also from L160 onwards

L172-4 see for this argument made for the case of flood risk modelling in relation to Slow Science and the need for scientists to put themselves where, in the spirit of Isabelle Stengers, you escape the constraints that allow you to develop a different understanding of the world - Lane, S.N., 2017. Slow science, the geographical expedition and critical physical geography. The Canadian Geographer, 61, 84-101

L185 – in this section you have also missed communities of practice within modelling communities – see Lane 2012 op cit

L268 – this is in my view a bit narrow and through the examples used misses some wider thinking about what makes a modeler choose a model – and how this is a very political process – see for example Lane et al., 2013 op cit. ; but also the role of power within the academic system in constraining what is deemed acceptable modelling – see for example Lane 2012, op cit. – and this prompts me to note that the notion of power is largely explored in terms of examples at the modeler-society interface and that the article does not really consider enough power within the academy and how this influences modelling as a practice

L284 – see also Lane 2012 for a discussion of this around how modellers come to trust their models and how this rarely conforms with how we present the modelling process; this includes notions of performativity in modelling, the role of academic norms in controlling trust, the difference between trust in the model and trust in the modeler, the importance of communities of practice for trust in the modeler, and other kinds of trustess (e.g. those forced to live with model prediction)  

L375 – I would largely agree that models can be highly exclusionary – but it is also important to think about their role as a means of supporting political interventions – you do not pick here with reference to Landstrom et al (2011) that you do cite but also Lane et al (2011 (Transactions of the Institute of British Geographers) where a community made a hydrological model that in turn allowed them to make a political intervention – which then shifted a stalled flood risk management project. They used their model to show that the models used by the UK Environment Agency had not addressed everything it could do. This modelling built a “new public” capable of making an intervention and so unsettling the process. Modelling was used as a means of unsettling the dominance of a particular kind of modelling. The public then got what they wanted – upstream storage – achieved because they pointed out to the EA that this worked if you embarked upon local flood protection measures for houses flooded frequently but designed the storage to stop more extreme and wider impact floods – this then became the Pickering scheme but it did then have to make recourse to an engineered solution and not simply more natural storage.

L446 – I don’t recall this division – we had one model which was part of showing the potential for upstream flood storage and which was written and used with the community through the competency group meetings – and a second which used an off the shelf model, HEC-RAS, to look at how river management influenced flooding of farmland. The latter was fascinating as the competency group ended up concluding that vegetation management (or lack of) did indeed cause flooding – but that was what the local EA wanted (move away from protecting farmland from flooding). However, the EA was mobilizing the argument that vegetation doesn’t impact flooding as this was easier to make (a “scientific one”) than the political one (allow flooding of farmland). Thus, this model then allowed the local members of the group to make interventions of a political nature.

L460 – this criticism was also the philosophical basis of the Pickering and Uckfield work – the need to build knowledge controversies and to use them to effect change. That is, progress is made when we find ourselves put in a position where what we hold to in terms of beliefs and knowledge is challenged – what Isabelle Stengers argues in 2005 should be the essence of being scientific (putting yourself in a slightly different position such that you end up coming wp with a different understanding of the world around you) – see Stengers, I., 2005. The cosmopolitical proposal. In Making things public, eds Latour and P. Weibel. Cambridge, MA: MIT Press, 994––1003.

L496-7 – no – these papers were part of an inter-University collaboration under a UK government initiative to support interdisciplinarity research for rural environments following the 2001 Foot and Mouth Disease outbreak

L519 – section ending here – this is quite under-developed and there is a lot more you could say (see comments above) about how modellers are reflexive during the modelling process

L561 – I think you should reflect on the scope of your suggestions here – following Nowotny et al.’s ideas – there are models developed for models’ sake (to improve the model being used) and there are models developed as part of practical applications (more “Type 2 science”) – and ones in between. The modellers and the relationship with the modelled – will change according to where one is on this spectrum – and hence the way politics is mobilized will change. I think there is a risk that the former is not appreciated enough (model development is situated within an academy that is itself highly political) is perhaps overlooked and ultimately a get out clause for a reader of this paper (who might mistakenly think that the political sensitivity of modelling only appears when a model is used to effect something in society). This is a nuance that is a bit lost at the moment.

L570 – I have to say I am a bit disappointed here with these four recommendations – with such a rich paper they are very thin indeed and miss a whole set of other possibilities. A good example is what does “democratising modelling processes” mean – democracy can be done in different ways that will cause the modelling processes also to change in different ways. The notion of democracy itself needs thought – we all too often see it as being about some kind of majority representing process – but what about the role of modelling as part of a minoritarian politics – allowing minorities to get the power needed to make a political intervention in a system from which they have been excluded? All of this is left tantalizingly undeveloped. The how under (i) through (iv) is also undeveloped.

First of all, I have to apologize for being incredibly late with my review.

Then to the content of this review itself: I am afraid that my conclusion is that I think this paper is not ready yet for further review after revision. I find the topic of the paper crucial for the hydrological community, if only – but not restricted to – because the hydrological community has recently discovered that human agents matter in how water flows are shaped and that hydrological models have a role to play in the world. The socio-hydrological ideas and the HELPING initiative are signs of this, respectively, and each with some rather major issues on what they actually entail in theoretical and practical terms, if you ask me. Having said that, this paper is not the input I think we need to enhance the debate on such issues.

I have three main reasons to defend this (harsh) claim about the paper, each of which I will detail below:

1. The method is arbitrary and not transparently displayed
2. The content discussion tends to the superficial and does not engage with crucial issues
3. The conclusions are not supported by the review itself and include some strange aspects

Method

What (some of) the authors did was search for material with key words in a database and write a text. One could use specific names for this, but I do not see anything structurally different from my simple positioning of the method. That method is in itself obviously the basis for most literature reviews, but I would have liked to see much more detail about the choices – as the search itself is shaping the data (the resulting texts). The choices are described, not explained and not detailed.

The step to reduce the number of texts from about 300 to about 20 suggests to me that the keyword search was actually not very useful. How somehow magically afterwards some 30 other papers were found to be added remains obscure as well. The search did not find them – other type of text perhaps, but see below on that? I am quite certain that I would come up with a different selection of texts if I would follow the method as the authors describe – assuming the method description would provide the details I would need to replicate anything.

A search strategy with the keywords runs the risk that modelling applications that may have been of interest to the review – for example how stakeholders are involved or how models are used in practices – without using the specific terms of interest of the authors of this text are not found. One can never be certain in a review to have found everything relevant, but one could do a better job to stretch the limits of available texts using approaches like citation analysis. Do these texts/authors cite each other, do they refer to similar texts – or not? These authors bring the texts together, but do the texts agree?

It would have been useful to specify much more about the papers that were finally used. Regional coverage of cases and authors, years of publication, journals, key words, type of water that is modelled, type of model used, perhaps even which equations were used – all these seemingly administrative items can already be revealing about what the community of authors does represent.

The texts that are finally used include both papers and PhD theses. Is that actually allowed – or perhaps less strict “advisable”? It does include some issues of how much papers can include in their word count compared to a thesis, right?

How the narrative aspects were defined (on their content see below) is obscure. The reader must trust the authors that what they have selected makes sense, including which text is used for which aspect. I happen to know a few of the texts, and my selection of themes and the link to texts would have been different. A sample of n=1 is weak (yes, I know) but I would have used something like text analysis in combination with the admin-indicators I mentioned earlier to make my selection process of themes transparent.

Content

The Introduction is very general and surprisingly low on references. Section 2 does include a few more, and might be integrated into the Intro. Are some of the references in 1 and 2 perhaps also feasible inputs for the review process itself?

The definition of what a model is to the authors is very broad. I would argue that the definition holds for almost any theory. I would actually agree that hydrological models are indeed theoretical claims, but that does not mean I would use such a generous definition. A broad use of terminology might actually obscure that the quantification aspect of models might be quite important – including differences between types of models, the actual equations that are used, the temporal and spatial coverage and steps, etcetera. This would also allow including the notion that many hydrological signals can be mimicked in many models – up to the point that rainfall can be estimated from a groundwater model, something that was obviously not the idea of that model when designed. The realization within the hydrological community that anything can be modelled has actually created a debate within that same community, perhaps not yet in the terms of the authors, but possibly of relevance for them.

The broad definition and some of the contents mentioned (see below) would have actually allowed the authors to include discussions within the hydrological community on building links with other fields, on models as I already mentioned, and comparable more interdisciplinary awareness in the hydrological community to position their model debate. Work by the group of Dr. Van Loon comes to my mind, to name just one example. I do agree that models are worth discussing, but let’s not argue that they are special in terms of supporting the powerful, offer limited world views and the like. Research efforts have the tendency to do the latter, with only a few having an answer to the former.

This observation of the relevance of other (connected) topics is also clear from the content descriptions. We read about data collection as specific issue, we read about decision on project focus, and – granted – we read about modelling decisions. But why these different aspects can be connected remains unclear. The discussion of the topics does not try to connect the topics, even when several of the texts seem to cover several of the authors’ themes. The topics read like an unevenly distributed shopping list, with an unequal number of texts per topics, with quite superficial summaries of the texts, and again without any cross references between texts and topics.

In the discussion, suddenly the texts that were rejected for the actual review are used to make a claim about what they discuss or even represent. I am sorry, but that is simply not allowed without a proper review of these texts in itself and providing the reader with the identity of these texts.  

The conclusion

The conclusions do include quite a number of references. That is at least unusual, but I would say suggests that either the authors do not have a conclusion (we read an extended discussion) or the conclusion does not yet come strongly enough out of the review.

The call for improvement if using models reads like a wish list, without the wished being confirmed as practically possible. The ideas are also rather general and surprisingly de-linked from the aspects that the authors have highlighted in section 4. Again, here the effect of missing possible texts that do discuss interesting uses of models without the terms used by the authors may be seen. Furthermore, a text like Junier (2017) shows quite clearly how relatively good intentions shift within the modelling process – in the sense that the intentions are kept, but the model does no longer align with them. The wish list is nice, but meaningless without much more discussion about implementation in actual practices.

One quite strong suggestion at the end is the need to move out of disciplines. This might neglect the less silo-ish nature of the hydrological discipline than the suggestion suggests, but the suggestion also drops out of the sky. The review itself does not clearly prefer this interdisciplinary aspect, so why is it so crucial? The suggestion that the issues with models come from disciplinary focuses is not even mentioned in the review. This suggests that the authors already knew the conclusion before doing the review. In itself the call for interdisciplinarity is not strange, and even done by hydrologists, but it would still need to be related to the review.

Furthermore, I find the claim that involving social sciences would solve the issues strange and in need of much more refinement. Are all social scientists equipped for and/or interested in the same issues as the authors? I would argue this is not the case. Why can the observation that many hydrological modellers do things that may be less useful be combined with the claim that collaboration between the general communities of hydrology and social sciences will solve this? The original modellers that did un-useful things will still be member of the hydro-community, right? Why are social scientists in general in the position to teach the hydrologists in general?

Final remarks

The topic of the text is important, but I think that the evidence that the text brings is not convincingly presented, as I have tried to argue. I would have liked to think that revisions would have been possible to continue the process, but the combination of a weak (description of the) method and a rather unbalanced analysis makes me strongly suggest that the text as is should be rejected.

Please find my handwritten notes on the text in the pdf attached. If there are any questions about remarks (including my handwriting) I am obviously available for further exchange. Again, the topic of this paper deserves further elaboration – but the paper itself needs that elaboration first even more…