On the visual detection of non-natural records in streamflow time series: challenges and impacts

Strohmenger, Laurent; Sauquet, Eric; Bernard, Claire; Bonneau, Jérémie; Branger, Flora; Bresson, Amélie; Brigode, Pierre; Buzier, Rémy; Delaigue, Olivier; Devers, Alexandre; Evin, Guillaume; Fournier, Maïté; Hsu, Shu-Chen; Lanini, Sandra; de Lavenne, Alban; Lemaitre-Basset, Thibault; Magand, Claire; Mendoza Guimarães, Guilherme; Mentha, Max; Munier, Simon; Perrin, Charles; Podechard, Tristan; Rouchy, Léo; Sadki, Malak; Soutif-Bellenger, Myriam; Tilmant, François; Tramblay, Yves; Véron, Anne-Lise; Vidal, Jean-Philippe; Thirel, Guillaume

doi:https://doi.org/10.5194/hess-27-3375-2023

Articles | Volume 27, issue 18

https://doi.org/10.5194/hess-27-3375-2023

© Author(s) 2023. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/hess-27-3375-2023

© Author(s) 2023. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 27, issue 18

Research article

|

22 Sep 2023

Research article |

| 22 Sep 2023

On the visual detection of non-natural records in streamflow time series: challenges and impacts

Laurent Strohmenger, Eric Sauquet, Claire Bernard, Jérémie Bonneau, Flora Branger, Amélie Bresson, Pierre Brigode, Rémy Buzier, Olivier Delaigue, Alexandre Devers, Guillaume Evin, Maïté Fournier, Shu-Chen Hsu, Sandra Lanini, Alban de Lavenne, Thibault Lemaitre-Basset, Claire Magand, Guilherme Mendoza Guimarães, Max Mentha, Simon Munier, Charles Perrin, Tristan Podechard, Léo Rouchy, Malak Sadki, Myriam Soutif-Bellenger, François Tilmant, Yves Tramblay, Anne-Lise Véron, Jean-Philippe Vidal, and Guillaume Thirel

Download

Final revised paper (published on 22 Sep 2023)
Preprint (discussion started on 12 Apr 2023)

Interactive discussion

Status: closed

RC1:
'Comment on hess-2023-58', Martin Gauch, 16 May 2023
Summary

The paper describes the outcomes of an effort to have human annotators detect non-natural/incorrect streamflow measurements. The authors find remarkable inconsistency in different annotators' responses.

Overall, I think this paper is well-written and to the point. I appreciated the discussion and suggestions for future related studies.

I think that a few changes in the analysis of results could further improve the paper. Please find my detailed comments below.

Major Comments

I would guess that it is not always clear which exact timesteps around an anomaly should be annotated. I think it might therefore be interesting to see if the consistency increases noticeably when we consider a windowed approach: e.g., consider two annotations to agree when they fall within a 1-week window.

I appreciate the effort to measure the impact of removing annotations through the "change rate", but it is hard to put the values into perspective. For that, I would propose the following baseline: report the change rate that you get when randomly removing the same fraction of time steps (ideally, do this a few times and report the average change rate).

It might be interesting to see if there is a noticeable difference in the responses/consistency of novice vs. advanced vs. senior annotators.

Reproducibility/data availability: For purposes of reproducibility, I would appreciate the download to include:
the streamflow data (and ideally also the supplemental information like precipitation, temperature, GR5J simulation, etc.), such that others can reproduce and build upon the analyses. The French-only HydroPortail link is not helpful and it's unclear what data one would need to collect from there.

the code used in the study (e.g., to calculate the various statistics)

information on the evaluators (background academic/operational, level of experience)

It would be really interesting to see how removing bad data impacts the quality of hydrological models. Is a model that is calibrated/trained only on "good" timesteps better than one that is calibrated on all timesteps (on a validation period)? This might be out of scope for this paper, but could at least be mentioned as future work.

Minor Comments

L90 states >30 years of available records as a condition, but then L96 reports that some rivers have only 26 years of streamflow. Am I missing something?

L224 "little impact on the change rate". If I understand correctly, the change rate is exactly what measures the magnitude of impact. I.e., the change rate only exists after removing the anomalies (and not before). So shouldn't this rather be reworded to something like "did not result in large change rates"?

L242f Regarding the first reason you provide for the stabilized variability: I don't think it is the variability that lowers as the number of stations increases. I think it is the quality of the estimate that improves as more samples are available. Both station quality and human randomness have some variance, which will be better estimated from larger sample sizes.

L266: This notion of annotators getting "bored" could be verified by checking whether the number of annotations also decreases across stations (those rated first vs. those rated last)

L281: Do you know how this "good-quality" label was assigned? Does this mean another human had visually deemed these records as good? Or via automatic checks?

L290: The logarithmic scale is an important detail to me, because I think it makes linear interpolation much harder to notice. This might be something to add to L349f.

L328ff: I would move this brief literature review to the related work section and remove it from the discussion.

Out of curiosity: would a map of France show any spatial patterns in the annotations?

Typos
L125 rainfall--runoff
Citation: https://doi.org/10.5194/hess-2023-58-RC1
- AC1: 'Reply on RC1', Laurent Strohmenger, 27 Jun 2023
  
  Please find our answer in the attached pdf file.
  
  Citation: https://doi.org/10.5194/hess-2023-58-AC1
RC2:
'Comment on hess-2023-58', Frederik Kratzert, 25 May 2023
The manuscript, by Laurent Strohmenger et al., presents results of a survey, in which 42 hydrologists were asked to annotate anomalies in streamflow time series. In my eyes, the main results of this manuscript:
A dataset of anomalies in streamflow time series annotated by human experts.

Another proof of the subjectivity and inconsistency of human experts, when tasked to rate/compare/annotate hydrographs.

Recommendations for future studies of this kind.

Overall, the manuscript is very well written, easy to follow. Given the already published comments by Martin Gauch, I have very little (see below) to add and recommend the publication of this manuscript after considering these minor comments.

Data sharing.
The authors state in L 354ff. The following:
An automatic detection of anomalies could avoid these issues of subjectivity and weariness. Using the bias between model simulations and measured time series could be a starting point for identifying potential anomalies. Unfortunately, to our knowledge, these techniques still require improvements. Such an algorithm should be flexible regarding the types of anomaly to identify, and might be trained for each study to avoid the risk of removing data of interest (e.g., using a visual inspection such as the one reported in this study). Ideally, hydrologists should share a common library of anomaly types such as suggested by Wilby et al. (2017). “
I 100% agree with this statement and e.g. the automatic quality control functions in the GSIM paper yield very suboptimal results. I therefore value that the authors opted for publishing the annotations from their study, which could be beneficial when testing future approaches for automatic detection algorithms. In this regard though, I agree with the review comment by Martin Gauch, that it would be very helpful if you would also include the streamflow time series in the published data, and maybe the GR5J simulations. As Martin Gauch mentioned, the linked homepage is in French, which e.g. I do not speak. I tried to use a translation tool but after 10 minutes of trying to figure out how to download data, I gave up. It also seems like there is no API access for downloading the data, which makes the effort to get the ~600 station time series quite cumbersome. If this data already exists in the hands of the authors and there are no constraints from the data provider that prohibits the publication of their data, then why not include the streamflow time series as well. Otherwise I see limited use in the published annotations, which would be a shame.
Citation: https://doi.org/10.5194/hess-2023-58-RC2
- AC2: 'Reply on RC2', Laurent Strohmenger, 27 Jun 2023
  
  Please find our answer in the attached pdf file.
  
  Citation: https://doi.org/10.5194/hess-2023-58-AC2
CC1: 'Referee Comment on hess-2023-58', Alexander Gelfan, 03 Jun 2023

Publisher’s note: this comment is a copy of RC3 and its content was therefore removed.

Citation: https://doi.org/10.5194/hess-2023-58-CC1
RC3:
'Comment on hess-2023-58', Alexander Gelfan, 03 Jun 2023

The comment was uploaded in the form of a supplement: https://hess.copernicus.org/preprints/hess-2023-58/hess-2023-58-RC3-supplement.pdf

Citation: https://doi.org/10.5194/hess-2023-58-RC3
- AC3: 'Reply on RC3', Laurent Strohmenger, 27 Jun 2023
  
  Please find our answer in the attached pdf file.
  
  Citation: https://doi.org/10.5194/hess-2023-58-AC3

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision | EF: Editorial file upload

ED: Publish subject to revisions (further review by editor and referees) (29 Jun 2023) by Jan Seibert

AR by Laurent Strohmenger on behalf of the Authors (03 Jul 2023) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (15 Jul 2023) by Jan Seibert

RR by Martin Gauch (24 Jul 2023)

RR by Alexander Gelfan (10 Aug 2023)

ED: Publish as is (23 Aug 2023) by Jan Seibert

AR by Laurent Strohmenger on behalf of the Authors (23 Aug 2023)

Short summary

We present the results of a large visual inspection campaign of 674 streamflow time series in France. The objective was to detect non-natural records resulting from instrument failure or anthropogenic influences, such as hydroelectric power generation or reservoir management. We conclude that the identification of flaws in flow time series is highly dependent on the objectives and skills of individual evaluators, and we raise the need for better practices for data cleaning.

On the visual detection of non-natural records in streamflow time series: challenges and impacts

Download

Interactive discussion

Peer review completion

Suggestions for revision or reasons for rejection