Technical note: “Bit by bit”: a practical and general approach for evaluating model computational complexity vs. model performance

Azmi, Elnaz; Ehret, Uwe; Weijs, Steven V.; Ruddell, Benjamin L.; Perdigão, Rui A. P.

doi:https://doi.org/10.5194/hess-25-1103-2021

Articles | Volume 25, issue 2

https://doi.org/10.5194/hess-25-1103-2021

© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/hess-25-1103-2021

© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 25, issue 2

Technical note

|

03 Mar 2021

Technical note |

| 03 Mar 2021

Technical note: “Bit by bit”: a practical and general approach for evaluating model computational complexity vs. model performance

Elnaz Azmi, Uwe Ehret, Steven V. Weijs, Benjamin L. Ruddell, and Rui A. P. Perdigão

Download

Final revised paper (published on 03 Mar 2021)
Preprint (discussion started on 06 Apr 2020)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

SC1: 'Suggestions to expand the list of candidate models', John Ding, 10 Apr 2020
AC1: 'Replies to SC 1', Uwe Ehret, 17 Apr 2020
- SC3: 'On new Table 1', John Ding, 25 Apr 2020
  - SC4: 'Correction to the term MAE', John Ding, 25 Apr 2020
    - AC4: 'Replies to SC4', Uwe Ehret, 26 Apr 2020
  - AC3: 'Replies to SC3', Uwe Ehret, 26 Apr 2020
SC2: 'Suggestions to clarify and expand the discussion', Robert Reinecke, 21 Apr 2020
AC2: 'Replies to SC 2', Uwe Ehret, 23 Apr 2020
RC1: 'Referee's review', Elena Toth, 29 May 2020
- AC5: 'Replies to RC1', Uwe Ehret, 09 Jul 2020
RC2: 'Review of "bit by bit"', Anonymous Referee #2, 12 Jun 2020
- AC6: 'Replies to RC2', Uwe Ehret, 09 Jul 2020

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

ED: Reconsider after major revisions (further review by editor and referees) (23 Jul 2020) by Roberto Greco

AR by Uwe Ehret on behalf of the Authors (31 Jul 2020) Author's response Manuscript

ED: Referee Nomination & Report Request started (20 Aug 2020) by Roberto Greco

RR by Anonymous Referee #2 (22 Sep 2020)

Suggestions for revision or reasons for rejection

I have gone through the revised version of the manuscript. Unfortunately the authors either have evaded almost all my comments (except putting the paper in the landscape of select complexity studies) or have decided to respond to comments of their own. They have accentuated their reliance on Weijs and Rudell without providing either theoretical or empirical support for their claims. Following are some of my comments (only on the added text in red by the authors)

Please also discuss the work of hydrological complexity by Sivapalan, Wagener and colleagues. Where/how does your paper compare and why if not?

“Weijs and Rudell (2020), we express model performance in terms of information losses” is limited by how p(.) is estimated (it being prone to misspecification) in Equation 2. This in itself is a model, subject to complexity associated challenges. I would therefore suggest caution in suggesting Weijs and Rudell (2020) at the same level of rigour as some other fanstastic work done on Occam’s razor in Mathematical Physics and Applied Probability. For example, no underlying arguments for uniform convergence, asymptotic consistency etc are provided other than authors guiding us to literature on information theory.

What are the implications of measuring information gains compared to a lower benchmark – the entropy of a uniform distribution, in terms of quantifying model performance. Is it an approximation? Or just some measure unrelated to approximation error (which I think is the case)?

“Choice of 𝑛 is typically guided by the objective to balance resolution and sufficiently populated bins” – exactly and this somehow sounds like Occam’s razor again. See my comment above.

Not clear what validation set approach actually doing. I would like to see how model chosen based on the principle presented here performs on the validation set (if it somehow identifies a generally good performing models) in comparison with other ‘demoted’ models. Otherwise its just a trivial exercise, sure you can have very simple but really bad models. We do not need to understand it bit by bit.

“We continued by describing several paradigms to guide model development:.. Weijs and Ruddell (2020) express both model performance and descriptive complexity in bit, and by adding the two obtain a single measure for what they call 'strong parsimony';..” that ok, but the authors are not the only ones attempting to describe parsimony. Don’t even think Weijs and Ruddell (2020) is a paradigm. Please delete or list many others who have dealt with concepts of parsimony.

“Occam's razor puts an emphasis on descriptive complexity, considers performance as a side condition, but it ignores computational complexity;..” this is incorrect interpretation (on performance being side condition). Please delete in order not to misguide readers.

“i) measuring computational complexity by 'Strace' is general in the sense that it can be applied to any model that can be run on a digital computer; ii) 'Strace' is sensitive to all aspects of a model, such as the size of the model itself, the input data it reads, its numerical scheme and time-stepping; iii) the 'bit by bit' approach is general in the sense that it measures two key aspects of a model in the single unit of bit, such that they can be used together to guide model analysis and optimization in a pareto trade-off manner in the general setting of incremental learning. It can be useful..” In general I agree with i). I find their iii) statement misguided – I don’t think strace can be used for incremental learning, since computational complexity doesnot make us unlearn something. “Descriptive complexity’” may do so (not based on bit by bit idea however) if interpreted appropriately in how it affects learning. Therefore computational complexity is a redundant concept in context of learning.

Over all, I think another revision is at least needed where Weijs and Rudell (2020) is not glamorized (and reference to it kept to a minimum) since all arguments based on it are rather weak. The authors should clearly show how computational complexity (or even Weijs and Ruddell description complexity) is affecting learning based on performance (only) of all the models on the validation set. My contention here is that computational complexity in this regard (of generalized learning) is rather of little value and that the paper is merely another demonstration of an interesting concept that does not add much to learning from data. I would advise humble response in the next iteration, where limitations and purpose of their manuscript is clearly documented.

Hide

RR by Elena Toth (02 Oct 2020)

Suggestions for revision or reasons for rejection

Dear Authors,
I do appreciate that you are now presenting the results of a calibration/validation split-sample test and you improved the manuscript in many points.
I am also glad that you have better explained the contribution of the paper and highlighted that “It is important to note that the purpose of the model comparison here is not primarily to identify the best among the different modelling approaches”, since actually I keep thinking that for a fair comparison of the performance of the models you cannot use different input.
When you write (in the discussion) “It is therefore neither required nor useful to establish models that all rely on the same input data, because the bit by bit approach might be used for exactly the purpose of comparing two models relying on different input data”, I think that this holds true for the complexity measure but not for the performance one. In fact, the correct input strongly influence the ‘side information’ that is needed to reduce the information loss, since the input information is crucial to the information content embedded in the model prediction.
As written in Campolo (WRR, 1999), the last observed discharges are generally included as inputs to ANN runoff-prediction models, since they provide information on the state of saturation of the basin, which is a function of the history of the meteorological input in the period preceding the streamflow generation: the capability of the system to respond to rainfall perturbation is represented by the ongoing streamflow measured in real-time at the closing section. And in fact you use just such input for the best performing model (#7). Such model is by far the best performing one since it is the only one that uses such a precious information (and actually also a fair comparison of the performances of the conceptual models would require, see Toth and Brath 2007, a real-time updating procedure making use of such data).
For this reason, the new ANN you applied, still fed by rainfall only, still provides a relatively poor performance in comparison to Model 7 (and the complexity seems to be even worse than in the first version).
If, instead of using your complex LSTM fed by the rainfall values, you had used a simple multi-layer ANN (again see what I did in Toth and Brath, WRR, 2007) with the same input (past Q values) that you use with the autoregressive model, you would have got performances similar (or probably better) than Model 7, even if the complexity would be, I guess, still much higher than that of Model 7. Such comparison would be not only fairer, but also more significant than what you are presenting now.
Such a simple ANN model implementation would take very little energy for you to set up, and, respectfully, I would insist for you to do so, since it would provide sounder results, supporting the scientific relevance of your interesting approach.
Elena Toth

Hide

ED: Reconsider after major revisions (further review by editor and referees) (16 Oct 2020) by Roberto Greco

AR by Uwe Ehret on behalf of the Authors (11 Dec 2020) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (17 Dec 2020) by Roberto Greco

RR by Anonymous Referee #2 (29 Jan 2021)

ED: Publish as is (29 Jan 2021) by Roberto Greco

Short summary

Computer models should be as simple as possible but not simpler. Simplicity refers to the length of the model and the effort it takes the model to generate its output. Here we present a practical technique for measuring the latter by the number of memory visits during model execution by Strace, a troubleshooting and monitoring program. The advantage of this approach is that it can be applied to any computer-based model, which facilitates model intercomparison.