Deep learning for the estimation of water-levels using river cameras

Vandaele, Remy; Dance, Sarah L.; Ojha, Varun

doi:https://doi.org/10.5194/hess-2021-20

Preprints

https://doi.org/10.5194/hess-2021-20

Preprints

12 Feb 2021

Review status: a revised version of this preprint was accepted for the journal HESS and is expected to appear here in due course.

Deep learning for the estimation of water-levels using river cameras

Remy Vandaele^1,3, Sarah L. Dance^1,2, and Varun Ojha³ Remy Vandaele et al.

Show author details

¹Department of Meteorology, University of Reading, U.K
²Department of Mathematics, University of Reading, U.K
³Department of Computer Sciences, University of Reading, U.K

Received: 12 Jan 2021 – Accepted for review: 07 Feb 2021 – Discussion started: 12 Feb 2021

Abstract. River level estimation is a critical task required for the understanding of flood events, and is often complicated by the scarcity of available data. Recent studies have proposed to take advantage of large networks of river camera images to estimate the river levels, but currently, the utility of this approach remains limited as it requires a large amount of manual intervention (ground topographic surveys and water image annotation). We develop an approach using an automated water semantic segmentation method to ease the process of river level estimation from river camera images. Our method is based on the application of a transfer learning methodology to deep semantic neural networks designed for water segmentation. Using datasets of image series extracted from four river cameras and manually annotated for the observation of a flood event on the Severn and Avon rivers, UK (21 November–5 December 2012), we show that our algorithm is able to automate the annotation process with an accuracy greater than 91 %. Then, we apply our approach to year-long image series from the same cameras observing the Severn and Avon (from 1 June 2019 to 31 May 2020) and compare our results with nearby river-gauge measurements. Given the high correlation (Pearson's Correlation Coefficient > 0.94) between our results and the river-gauge measurements, it is clear that our approach to automation of the water segmentation on river camera images could allow for straightforward, inexpensive observation of flood events, especially at ungauged locations.

Remy Vandaele et al.

Status: closed

RC1:
'Comment on hess-2021-20', Kenneth Chapman, 19 Feb 2021

This paper addresses the annotation problem in machine learning with images which can be a daunting task while, at the same time, delivering. Starting with CNNs for semantic learning, inductive transfer learning is used with much smaller training sets to build accurate models that “ease the process of river level estimation from river camera images.” Below we provide some over-arching, general notes about methodology that could use some clarification, followed by some minor comments for consideration.

General comments:

Two challenges for the data selected for the study seem to be 1) the use of ground surveys that "measure the topographic height of several landmarksâ¯within the field-of-view of the camera?" and 2) the fact that "the number and spread of measured landmarksâ¯over the camera’s field-of-view was constrained to locations that were accessible during the ground survey". These challenges might be hard to overcome in real world scenarios where access is not available and the measurement of the landmark heights is a large task. Might there be alternative ways to get that information? Maybe worth aâ¯very short discussion.â¯

There was one brief note about the cameras being fixed in the images used in the paper. With a camera that is tens of meters from a scene it is common (even under the best of conditions) to get movement of the camera relative to the objects of interest in the scene. The reason for mentioning this is that it is relatively easy to test for camera movement. You might have tested for camera movement and ruled it out as a source of variation. In that case, it might be worth mentioning that in the paper. If not, it might be good to discuss how your already very good predictions might improve if that motion were taken into consideration by using image processing techniques to find the landmarks in the scene and make adjustments for camera movement.

A brief discussion of landmark vs. area feature analysis might add to the paper. Readers will likely benefit from understanding how you weighed out the pros/cons of those approaches. Did you decide to use landmarks based on the need to ease the (already reduced) annotation task? Or did you choose it because (1) of the desire to use the “by-pixel” output of the CNN’s without post processing, (2) it was just beyond the scope of this analysis because you wanted to focus on the transfer learning, and/or (3) for some other reason? Is it possible that the analysis of the classified pixels in a landmark region might yield a better predictor than just the landmarks themselves? A good deal of effort was directed at finding a heuristic to ignore outlier flood predictions. Ignoring outliers is a good objective, but analysis of landmark regions could have improved the specificity and sensitivity of your predictions (i.e., provided a more robust assessment of prediction quality). Again, this might have been beyond the scope of performing annotation of a large number of images with the transfer learning methodology, but more discussion around strengths and challenges of your approach would strengthen the paper.

The scope of the literature review in the Introduction and Background sections is narrow and could be expanded to better connect with the broader literature on image-based stream gauging and/or extreme event detection/measurement literature. For example, the idea that non-contact image-based stream gauging techniques can also be used to flag extreme events is not mentioned. The following (and/or literature cited within) could be integrated without excessively lengthening the paper. Here are some examples from a quick Google Scholar search:

Muste, M., Fujita, I., & Hauet, A. (2008). Large-scale particle image velocimetry for measurements in riverine environments. , (4). https://doi.org/10.1029/2008WR006950

Boursicaud, R. L., Pénard, L., Hauet, A., Thollet, F., & Coz, J. L. (2016). Gauging extreme floods on YouTube: application of LSPIV to home movies for the post-event determination of stream discharges. , (1), 90–105. https://doi.org/10.1002/hyp.10532

Eltner, A., Elias, M., Sardemann, H., & Spieler, D. (2018). Automatic Image-Based Water Stage Measurement for Long-Term Observations in Ungauged Catchments. , (12), 10,362-10,371. https://doi.org/10.1029/2018WR023913

Gilmore, T. E., Birgand, F., & Chapman, K. W. (2013). Source and magnitude of error in an inexpensive image-based water level measurement system. Journal of Hydrology, 496, 178–186. https://doi.org/10.1016/j.jhydrol.2013.05.011

Creutin, J. D., Muste, M., Bradley, A. A., Kim, S. C., & Kruger, A. (2003). River gauging using PIV techniques: a proof of concept experiment on the Iowa River. , (3), 182–194. https://doi.org/10.1016/S0022-1694(03)00081-7

A. A. Royem, C. K. Mui, D. R. Fuka, & M. T. Walter. (2012). Technical Note: Proposing a Low-Tech, Affordable, Accurate Stream Stage Monitoring System. , (6), 2237–2242. https://doi.org/10.13031/2013.42512

Schoener, G. (2018). Time-Lapse Photography: Low-Cost, Low-Tech Alternative for Monitoring Flow Depth. , (2), 06017007. https://doi.org/10.1061/(ASCE)HE.1943-5584.0001616

Citation: https://doi.org/10.5194/hess-2021-20-RC1
- AC1: 'Reply on RC1', Remy Vandaele, 08 Mar 2021
  
  We would like to thank the reviewer for his constructive review and comments. We respond to these comments below.
  Comment #1 regarding the use of alternative ways to obtain topographic data
  We agree that obtaining ground surveys for the field-of-view for each camera is a significant challenge. An alternative could be to use the camera images in conjunction with a high resolution digital surface model (DSM). An open access lidar DSM is available across the UK. Over other parts of the world, it might be necessary to use a DSM that is commercially available (for example the 12m WorldDEM), but the accuracy of the results using a low resolution DSM would need to be carefully evaluated.
  If we are invited to revise our paper, we would add the following sentence in the conclusion section:
  “Future work will focus on the merging of the water segmentation results with lidar digital surface model (DSM) data available at 1m resolution over the UK (Environment Agency, 2017). This would allow the water segmentation algorithms to provide a direct estimate of the water levels in the areas that are studied, without requiring any ground-surveys.”
  Environment Agency. LIDAR Composite DSM 2017 - 1m. https://data.gov.uk/dataset/80c522cc-e0bf-4466-8409-57a04c456197/lidar-composite-dsm-2017-1m (2017). Last accessed 26 February 2021.
  Comment #2 regarding camera movement
  From our inspection of the datasets, we found that the camera movement was negligible (a maximum of 2-3 pixels, even on objects far away from the camera). We were especially careful to choose landmark pixel locations that were not too close to the edge of the landmarked object in order to avoid confusions. We would expect typical intensity-based image registration algorithms to deteriorate the results due to changes in image illumination and movement within the image (of boats, wildlife and debris), although we have not tested this.
  If we are invited to revise our paper, we would like to add the following sentence at the end of section 4.1.1 (Dataset presentation):
  
  “An inspection of our datasets and results showed us that the impact of camera movement was negligible. Machine-Learning based landmark detection algorithms (e.g, Vandaele et al., 2018) could have been used otherwise, but they are unnecessary in the context of this study.”
  Vandaele, R., Aceto, J., Muller, M. et al. Landmark detection in 2D bioimages for geometric morphometrics: a multi-resolution tree-based approach. Sci Rep 8, 538 (2018). https://doi.org/10.1038/s41598-017-18993-5
  
  Comment #3 regarding the use of area features instead of landmarks
  The starting point of this study was to show how well we were able to automate the annotation process in comparison with the manual landmark annotation approach used in our former study (Vetra-Carvalho et al. 2020). We agree that the use of landmarked areas as opposed to landmarked pixels could strengthen the robustness of our results, but it would have required a new and more complex ground survey that was not in the scope of this study.
  To address this comment, we would like to add the following sentence at the end of section 4.1.1, after the modification of Comment #2:
  
  “Also note that we focus on a simple process relying on single pixel landmark locations annotated during a former study (Vetra-Carvalho et al., 2020). The use of landmarked areas of multiple pixels sharing the same height could likely help to increase the detection performance and should be considered for an optimal use of this landmark-based approach.”
  Comment #4: Literature
  We would like to add citations to the literature mentioned by the reviewer at line 41 of the introduction section as follows:
  “There have been a number of citizen science projects that investigated the use of crowdsourced observations of river level (e.g. Royem et al., 2012; Lanfranchi et al., 2014; Etter et al., 2020; Lowry et al., 2019; Walker et al., 2019; Baruch, 2018).”
  In our introduction section, we would like to add the following paragraph regarding the use of river cameras:
  “Several studies have already attempted to use videos and still camera images in order to observe flood events. Surface velocity fields can be computed using videos (e.g., Muste et al., 2008; Le Boursicaud et al.,2016; Creutin et al. 2003; Perks et al., 2020). Several studies also considered the use of river camera images to observe the water levels, either manually (e.g., Royem et al, 2012; Schoener, 2018; Etter et al, 2020) or automatically, for example by considering image processing edge detection techniques (Eltner et al. 2018). Under the right conditions, these automated water level estimation techniques can provide good accuracy with uncertainties of only a few mms (Gilmore et al., 2013; Eltner et al. 2018). However, the performance of these approaches lacks portability (Eltner et al.,2018.). ”
  
  Citation: https://doi.org/10.5194/hess-2021-20-AC1
RC2:
'Comment on hess-2021-20', Anonymous Referee #2, 01 Mar 2021

Dear Author,

Thanks for your contribution to the research combing computer scienece and hydrology. Some revisions are suggested to made. My suggestions are listed in the supplement file.

Citation: https://doi.org/10.5194/hess-2021-20-RC2
- AC2: 'Reply on RC2', Remy Vandaele, 23 Mar 2021
  
  Dear Reviewer,
  Thank you for your comments and suggestions. You can find our answer and amendment propositions in the supplement file.
  
  Citation: https://doi.org/10.5194/hess-2021-20-AC2

Status: closed

RC1:
'Comment on hess-2021-20', Kenneth Chapman, 19 Feb 2021

This paper addresses the annotation problem in machine learning with images which can be a daunting task while, at the same time, delivering. Starting with CNNs for semantic learning, inductive transfer learning is used with much smaller training sets to build accurate models that “ease the process of river level estimation from river camera images.” Below we provide some over-arching, general notes about methodology that could use some clarification, followed by some minor comments for consideration.

General comments:

Two challenges for the data selected for the study seem to be 1) the use of ground surveys that "measure the topographic height of several landmarksâ¯within the field-of-view of the camera?" and 2) the fact that "the number and spread of measured landmarksâ¯over the camera’s field-of-view was constrained to locations that were accessible during the ground survey". These challenges might be hard to overcome in real world scenarios where access is not available and the measurement of the landmark heights is a large task. Might there be alternative ways to get that information? Maybe worth aâ¯very short discussion.â¯

There was one brief note about the cameras being fixed in the images used in the paper. With a camera that is tens of meters from a scene it is common (even under the best of conditions) to get movement of the camera relative to the objects of interest in the scene. The reason for mentioning this is that it is relatively easy to test for camera movement. You might have tested for camera movement and ruled it out as a source of variation. In that case, it might be worth mentioning that in the paper. If not, it might be good to discuss how your already very good predictions might improve if that motion were taken into consideration by using image processing techniques to find the landmarks in the scene and make adjustments for camera movement.

A brief discussion of landmark vs. area feature analysis might add to the paper. Readers will likely benefit from understanding how you weighed out the pros/cons of those approaches. Did you decide to use landmarks based on the need to ease the (already reduced) annotation task? Or did you choose it because (1) of the desire to use the “by-pixel” output of the CNN’s without post processing, (2) it was just beyond the scope of this analysis because you wanted to focus on the transfer learning, and/or (3) for some other reason? Is it possible that the analysis of the classified pixels in a landmark region might yield a better predictor than just the landmarks themselves? A good deal of effort was directed at finding a heuristic to ignore outlier flood predictions. Ignoring outliers is a good objective, but analysis of landmark regions could have improved the specificity and sensitivity of your predictions (i.e., provided a more robust assessment of prediction quality). Again, this might have been beyond the scope of performing annotation of a large number of images with the transfer learning methodology, but more discussion around strengths and challenges of your approach would strengthen the paper.

The scope of the literature review in the Introduction and Background sections is narrow and could be expanded to better connect with the broader literature on image-based stream gauging and/or extreme event detection/measurement literature. For example, the idea that non-contact image-based stream gauging techniques can also be used to flag extreme events is not mentioned. The following (and/or literature cited within) could be integrated without excessively lengthening the paper. Here are some examples from a quick Google Scholar search:

Muste, M., Fujita, I., & Hauet, A. (2008). Large-scale particle image velocimetry for measurements in riverine environments. , (4). https://doi.org/10.1029/2008WR006950

Boursicaud, R. L., Pénard, L., Hauet, A., Thollet, F., & Coz, J. L. (2016). Gauging extreme floods on YouTube: application of LSPIV to home movies for the post-event determination of stream discharges. , (1), 90–105. https://doi.org/10.1002/hyp.10532

Eltner, A., Elias, M., Sardemann, H., & Spieler, D. (2018). Automatic Image-Based Water Stage Measurement for Long-Term Observations in Ungauged Catchments. , (12), 10,362-10,371. https://doi.org/10.1029/2018WR023913

Gilmore, T. E., Birgand, F., & Chapman, K. W. (2013). Source and magnitude of error in an inexpensive image-based water level measurement system. Journal of Hydrology, 496, 178–186. https://doi.org/10.1016/j.jhydrol.2013.05.011

Creutin, J. D., Muste, M., Bradley, A. A., Kim, S. C., & Kruger, A. (2003). River gauging using PIV techniques: a proof of concept experiment on the Iowa River. , (3), 182–194. https://doi.org/10.1016/S0022-1694(03)00081-7

A. A. Royem, C. K. Mui, D. R. Fuka, & M. T. Walter. (2012). Technical Note: Proposing a Low-Tech, Affordable, Accurate Stream Stage Monitoring System. , (6), 2237–2242. https://doi.org/10.13031/2013.42512

Schoener, G. (2018). Time-Lapse Photography: Low-Cost, Low-Tech Alternative for Monitoring Flow Depth. , (2), 06017007. https://doi.org/10.1061/(ASCE)HE.1943-5584.0001616

Citation: https://doi.org/10.5194/hess-2021-20-RC1
- AC1: 'Reply on RC1', Remy Vandaele, 08 Mar 2021
  
  We would like to thank the reviewer for his constructive review and comments. We respond to these comments below.
  Comment #1 regarding the use of alternative ways to obtain topographic data
  We agree that obtaining ground surveys for the field-of-view for each camera is a significant challenge. An alternative could be to use the camera images in conjunction with a high resolution digital surface model (DSM). An open access lidar DSM is available across the UK. Over other parts of the world, it might be necessary to use a DSM that is commercially available (for example the 12m WorldDEM), but the accuracy of the results using a low resolution DSM would need to be carefully evaluated.
  If we are invited to revise our paper, we would add the following sentence in the conclusion section:
  “Future work will focus on the merging of the water segmentation results with lidar digital surface model (DSM) data available at 1m resolution over the UK (Environment Agency, 2017). This would allow the water segmentation algorithms to provide a direct estimate of the water levels in the areas that are studied, without requiring any ground-surveys.”
  Environment Agency. LIDAR Composite DSM 2017 - 1m. https://data.gov.uk/dataset/80c522cc-e0bf-4466-8409-57a04c456197/lidar-composite-dsm-2017-1m (2017). Last accessed 26 February 2021.
  Comment #2 regarding camera movement
  From our inspection of the datasets, we found that the camera movement was negligible (a maximum of 2-3 pixels, even on objects far away from the camera). We were especially careful to choose landmark pixel locations that were not too close to the edge of the landmarked object in order to avoid confusions. We would expect typical intensity-based image registration algorithms to deteriorate the results due to changes in image illumination and movement within the image (of boats, wildlife and debris), although we have not tested this.
  If we are invited to revise our paper, we would like to add the following sentence at the end of section 4.1.1 (Dataset presentation):
  
  “An inspection of our datasets and results showed us that the impact of camera movement was negligible. Machine-Learning based landmark detection algorithms (e.g, Vandaele et al., 2018) could have been used otherwise, but they are unnecessary in the context of this study.”
  Vandaele, R., Aceto, J., Muller, M. et al. Landmark detection in 2D bioimages for geometric morphometrics: a multi-resolution tree-based approach. Sci Rep 8, 538 (2018). https://doi.org/10.1038/s41598-017-18993-5
  
  Comment #3 regarding the use of area features instead of landmarks
  The starting point of this study was to show how well we were able to automate the annotation process in comparison with the manual landmark annotation approach used in our former study (Vetra-Carvalho et al. 2020). We agree that the use of landmarked areas as opposed to landmarked pixels could strengthen the robustness of our results, but it would have required a new and more complex ground survey that was not in the scope of this study.
  To address this comment, we would like to add the following sentence at the end of section 4.1.1, after the modification of Comment #2:
  
  “Also note that we focus on a simple process relying on single pixel landmark locations annotated during a former study (Vetra-Carvalho et al., 2020). The use of landmarked areas of multiple pixels sharing the same height could likely help to increase the detection performance and should be considered for an optimal use of this landmark-based approach.”
  Comment #4: Literature
  We would like to add citations to the literature mentioned by the reviewer at line 41 of the introduction section as follows:
  “There have been a number of citizen science projects that investigated the use of crowdsourced observations of river level (e.g. Royem et al., 2012; Lanfranchi et al., 2014; Etter et al., 2020; Lowry et al., 2019; Walker et al., 2019; Baruch, 2018).”
  In our introduction section, we would like to add the following paragraph regarding the use of river cameras:
  “Several studies have already attempted to use videos and still camera images in order to observe flood events. Surface velocity fields can be computed using videos (e.g., Muste et al., 2008; Le Boursicaud et al.,2016; Creutin et al. 2003; Perks et al., 2020). Several studies also considered the use of river camera images to observe the water levels, either manually (e.g., Royem et al, 2012; Schoener, 2018; Etter et al, 2020) or automatically, for example by considering image processing edge detection techniques (Eltner et al. 2018). Under the right conditions, these automated water level estimation techniques can provide good accuracy with uncertainties of only a few mms (Gilmore et al., 2013; Eltner et al. 2018). However, the performance of these approaches lacks portability (Eltner et al.,2018.). ”
  
  Citation: https://doi.org/10.5194/hess-2021-20-AC1
RC2:
'Comment on hess-2021-20', Anonymous Referee #2, 01 Mar 2021

Dear Author,

Thanks for your contribution to the research combing computer scienece and hydrology. Some revisions are suggested to made. My suggestions are listed in the supplement file.

Citation: https://doi.org/10.5194/hess-2021-20-RC2
- AC2: 'Reply on RC2', Remy Vandaele, 23 Mar 2021
  
  Dear Reviewer,
  Thank you for your comments and suggestions. You can find our answer and amendment propositions in the supplement file.
  
  Citation: https://doi.org/10.5194/hess-2021-20-AC2

Remy Vandaele et al.

Data sets

Deep learning for the estimation of water-levels using river cameras: networks and datasets Remy Vandaele, Sarah L. Dance, and Varun Ojha https://doi.org/10.17864/1947.282

Model code and software

Deep learning for the estimation of water-levels using river cameras: networks and datasets Remy Vandaele, Sarah L. Dance, and Varun Ojha https://doi.org/10.17864/1947.282

Remy Vandaele et al.

Viewed

Total article views: 834 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	BibTeX	EndNote
659	161	14	834	5	6

HTML: 659
PDF: 161
XML: 14
Total: 834
BibTeX: 5
EndNote: 6

Views and downloads (calculated since 12 Feb 2021)

Month	HTML	PDF	XML	Total
Feb 2021	380	69	3	452
Mar 2021	139	39	9	187
Apr 2021	51	23	0	74
May 2021	28	8	0	36
Jun 2021	35	10	2	47
Jul 2021	26	12	0	38

Cumulative views and downloads (calculated since 12 Feb 2021)

Month	HTML	PDF	XML	Total
Feb 2021	380	69	3	452
Mar 2021	139	39	9	187
Apr 2021	51	23	0	74
May 2021	28	8	0	36
Jun 2021	35	10	2	47
Jul 2021	26	12	0	38

Viewed (geographical distribution)

Total article views: 618 (including HTML, PDF, and XML) Thereof 618 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 24 Jul 2021

Short summary

The acquisition of river level data is a critical task for the understanding of flood events, but is often complicated by the difficulty to install and maintain gauges able to provide such information. This study proposes to apply deep learning techniques on river camera images in order to automatically extract the corresponding water levels. This approach could allow for a new flexible way to observe flood events, especially at ungauged locations.


Total:	0
HTML:	0
PDF:	0
XML:	0