A fundamental and highly non-trivial question in many applied sciences is how to make reliable predictions about the dynamics of a complex system. In hydrological modelling in particular, the ability of predicting extreme events like floods is obviously of paramount importance. Conceptual rainfall-runoff models that incorporate only a few state variables and a few system parameters often represent a very practical and efficient solution for making probabilistic predictions. The basic idea is to describe slow processes occurring at our observation scale by phenomenological differential equations and include all other processes as noise. Incorporating the noise in the model, where it arises, naturally leads to stochastic differential equation (SDE) models. Model parameters then need to be calibrated on observed data, usually provided in the form of noisy time series. The goal of the calibration process is to determine the parameters that allow the model to reproduce the observed data and quantify their uncertainties, expressed as probability distributions. For this purpose, Bayesian statistics is a consistent framework where our knowledge about model parameters is described by probability distributions and learning as a data-driven updating process of prior beliefs. Bayesian inference methods bear the great advantage over traditional optimization algorithms of providing an uncertainty estimation for the calibrated parameters in the form of a probability distribution. The knowledge of such uncertainty is important for making probabilistic predictions, which can in turn be a useful tool for decision-makers. Hydrology could potentially take great advantage of more realistic stochastic models and a fast and reliable method for their calibration. However, Bayesian inference turns out to be computationally very expensive for non-trivial stochastic models.

Uncertainty in rainfall-runoff hydrological modelling arises mostly from input errors associated with an inaccurate estimation of the integrated rainfall over a catchment . These input errors, typically due to a combination of heterogeneous rainfall, sparse rain-gauge measurements and insufficient temporal resolution , can seriously deteriorate the quality of model calibration results for heavily input-driven hydrological systems . A variety of stochastic weather generators have been proposed to simulate precipitation with its uncertainty. Although such weather generators can provide uncertain inputs to rainfall-runoff models and therefore reproduce the effect of rainfall errors on runoff predictive uncertainties, such input uncertainties have been largely neglected in studies focusing on model parameter inference , probably because of the computational difficulty of including them in a likelihood function . Input uncertainties should be included directly in the input as a stochastic contribution and then transported through the model and thereby naturally incorporated in the likelihood function describing the probability distribution of observations given model parameters. In the Bayesian framework, the sought posterior probability distribution for the model parameters is proportional to the product of the likelihood function and the prior probability distribution describing our prior knowledge about model parameters. However, it is still common practice in hydrology, as well as in other applied disciplines, to consider an over-simplified error model based on likelihood functions defined as uncorrelated normal distributions centred on the outputs of a deterministic model . This inevitably leads to biased parameters and unreliable predictions . Although so-called rainfall multipliers can mitigate this problem , they fail in assessing input uncertainties when a rainfall event is not detected by the available rain gauges .

proposed a method based on an input uncertainty model describing the rainfall as a continuous stochastic process. The method is called SIP (acronym for stochastic input process). The idea of describing selected model parameters or inputs as stochastic time-dependent processes in order to take intrinsic uncertainties realistically into account in hydrological modelling has gained momentum in recent years . The SIP technique uses (i) possibly inaccurate rain-gauge precipitation data, (ii) runoff data from a flowmeter at the catchment outlet, (iii) a hydrological runoff model, (iv) a rainfall model in the form of a transformed stochastic Ornstein–Uhlenbeck (OU) process, (v) models for rainfall and runoff observation errors, and (vi) prior distributions, to infer both the marginal posterior distributions for the parameters of interest and a “true” spatially integrated average rainfall over the catchment in a Bayesian manner. The SIP method uses the catchment as an additional rain gauge to gather information about a catchment-averaged real rain, which is inferred from both prior knowledge and observations of rainfall and runoff. Both the hydrological model describing the catchment dynamics and runoff observations are assumed to be accurate in comparison to the rainfall data. The inferred rainfall pattern can then be used to calibrate the model and significantly reduce the bias in the estimated parameters, despite the inaccuracy of the rainfall data. However, as described in detail in , the likelihood function turns out to be a high-dimensional discretized version of an infinite-dimensional path integral that makes this approach computationally demanding.

In we presented an efficient Hamiltonian Monte Carlo (HMC)-based algorithm for the calibration of SDE models on noisy time series. The proposed HMC method, combining molecular dynamics principles with the well-known Metropolis algorithm, relies on the reinterpretation of the Bayesian posterior probability distribution as the partition function of a statistical mechanics system. This interpretation reduces the parameter inference problem to the task of simulating the dynamics of a fictitious statistical mechanics system, which can be solved in a computationally efficient way. The dynamics of the statistical mechanics system may occur on very different timescales, which can be exploited by a multiple timescale integration approach. In , the method was demonstrated using a simple, albeit general, rainfall-runoff toy model, and we claimed that the HMC algorithm, combined with the multiple timescale integration, would be applicable to a wide range of inference problems, making many SDE models amenable to a consistent Bayesian parameter inference.

Here we combine the HMC and SIP methods to perform Bayesian inference with a stochastic input model. We show that the HMC method can be extended from the toy model and the smooth synthetic data used in to a real-world hydrological case study with real noisy rainfall and runoff time series, albeit at the cost of a somewhat substantial analytical effort. We intentionally use sparse and inaccurate precipitation data, provided by a rain gauge located far away from the catchment area, to demonstrate that a “true” average rainfall pattern can be reconstructed from the corresponding runoff data, and we compare the inferred rain to the more accurate observations obtained from rain gauges within or very close to the catchment.

The HMC method bears valuable advantages with regard to both generality and efficiency. Indeed, it is by no means limited to an OU process, unlike the original SIP approach of , which strictly requires a linear stochastic process as a rain generator. Although in this study we also opt for an OU process for the sake of simplicity, it should be clear that such a process could be arbitrarily replaced by any other stochastic process, thus giving the method significantly more flexibility in reproducing the statistical properties of real rainfall. The HMC method is also not at all limited to urban hydrology and could be applied for instance in natural catchment hydrology as well. The specific case study presented here was chosen to put emphasis on input uncertainties, which often represent the biggest source of uncertainty in hydrological modelling in general. More information on possible hydrological applications can be found in . Moreover, the HMC algorithm allows us to sample both model parameters and a true averaged rainfall pattern that are simultaneously compatible with data, models and prior distributions from the posterior probability density. This yields very high acceptance rates, even in the context of expensive high-dimensional problems, with great benefits in terms of performance and efficiency of the algorithm. Another interesting method from the family of particle filters to tackle high-dimensional inference problems for stochastic model calibration is particle Markov chain Monte Carlo (PMCMC) , which combines piece-wise forward simulations of the stochastic model with data-based importance sampling. Like HMC, PMCMC methods can be applied to any stochastic process, including (unlike HMC) processes with discrete states (e.g. numbers of organisms in ecological models). A generally low implementation effort could be an additional argument in favour of PMCMC algorithms, which do not require the differentiation of the posterior distribution like HMC methods. However, PMCMC methods may suffer significantly in terms of efficiency when compared to an HMC-based approach. A detailed comparison, based on the same case study presented here, of different methods for Bayesian inference with stochastic models, can be found in . The HMC approach is very general and suitable for a broad range of applications requiring Bayesian inference with stochastic models. It should be noted that the method presented here is meant for offline-calibration of stochastic models, not for real-time updating, which might be needed in a model-based control setting. For the latter problem, filtering algorithms might be more appropriate, but we do not discuss this topic here. Moreover, HMC is inherently a MCMC method, and it is therefore embarrassingly parallelizable by breaking it up into an arbitrary number of independent Markov chains. This makes HMC very well suited for applications in the big-data regime or with expensive models.

The SIP method of describes the rain input to a hydrological system based on an unobserved and continuous stochastic process, the rainfall potential ξ(t). The latter should not be interpreted as a potential in the physical sense but rather as a rain generator based on a latent, i.e. “potential”, stochastic process that can be transformed into real precipitation P by a suitable empirical transformation P(t)=r(ξ(t)), which will be addressed below in Sect. .

The inference process allows us to learn from noisy rainfall and runoff time series, Pobs and Qobs, respectively; the parameters θ of the hydrological system; the uncertainties σξ and σz of both rainfall and runoff observation models, respectively; and the unobserved true rainfall over the catchment expressed as a discretized rainfall potential ξ={ξi}, to be interpreted as an evaluation of the stochastic process ξ(t) at a discrete set of time points ti, with i=1, …, N.

For this purpose, we subdivide each interval between consecutive rain observations into jP bins, yielding a total of nPjP+1=N discretization points, where nP+1 is the number of rain observations, that is, the length of the precipitation time series Pobs. Analogously, the system output dimension is discretized by partitioning the intervals between consecutive runoff observations into jQ sub-intervals, such that the total number of discretization points is nQjQ+1=N, where nQ+1 is the number of data points in Qobs. The number of discretization points is thus the same (N) in both rainfall and runoff dimensions, and it defines the discretization time step dt=T/(N-1), where T is the total time interval covered by observations. We will provide numerical values for all these constants later in Sect. . Here, suffice it to say that the only important requirement of the method is that the total number N of discretization points is large enough compared to the number of measurement points, nP and nQ, in order to accurately probe the fine dynamics occurring on short timescales between observations. Other features, such as for instance having the same number N of discretization points for rainfall and runoffs, are just arbitrary choices to ease the practical implementation of the method, which could be removed without altering the results and conclusions of this work. The same holds if the observations were irregularly spaced in time, in which case one could use observation windows with more/less intermediate discretization points. Note that a large number of discretization points N, with a fixed number of observation points, does only moderately increase the computational effort of the algorithm, since the part of the Hamiltonian dynamics that scales with N can be calculated analytically, as shown below.

The inference goal is to sample both parameter combinations (θ, σξ, σz) and realizations ξ of the stochastic process ξ(t) from a posterior distribution obeying Bayes' equation, which reads in its discretized form as 1fξ,θ,σξ,σz∣Pobs,Qobs∝fQobs∣ξ,θ,σzfPobs∣ξ,σξfξfθ,σξ,σz, where f(θ, σξ, σz) is the joint prior distribution for the model parameters, and f(Qobs∣ξ, θ, σz)f(Pobs∣ξ, σξ) is the likelihood expressing the probability of observed data (Pobs, Qobs) given model parameters (θ, σξ, σy) and a system realization ξ, weighted by the prior probability density f(ξ).

The stochastic process realization ξ in Eq. () is a time series of length N≫1. The high dimensionality of the problem renders the likelihood computationally expensive, thus making the inference problem intractable with traditional Bayesian inference algorithms, such as random walk Metropolis algorithms, which require a large number of likelihood evaluations.

On the other hand, when the inference target is only the posterior distribution for the model parameters (θ, σξ, σz) and model simulations are fast, one may resort to approximate Bayesian computation (ABC) algorithms (e.g. ), which approximate the parameters' posterior through repeated comparisons of model simulations with observations in terms of a low-dimensional set of summary statistics. However, here we are interested in the joint inference of model parameters and the real rainfall ξ, which makes ABC an inefficient approach.

To tackle this problem, we apply a Hamiltonian Monte Carlo (HMC) algorithm , which exploits the principles of Hamiltonian dynamics to attain very high acceptance rates and low auto-correlation, even on high-dimensional sampling spaces. This makes it possible to explore such spaces with large steps, without compromising the acceptance rate and thus making the algorithm very efficient. The inherent high efficiency of HMC is further boosted here by a timescale separation analogous to the one described in .

The HMC algorithm allows us to sample simultaneously from the posterior of Eq. () both model parameters (θ, σξ, σz) and realizations of the stochastic process ξ. In particular, the rainfall potential ξ is inferred indirectly using prior knowledge, the observed runoff Qobs and the possibly inaccurate observed precipitation Pobs. The discharge data are used as an indirect source of knowledge about the rainfall, which complements the unreliable information due to the sparse rain-gauge measurements.

The method described here requires a stochastic input process (i.e. a rainfall model), a hydrological rainfall-runoff model to describe the observed discharges Qobs, observation models for both rainfall and runoff, and prior probability distributions, which are all outlined below.

The HMC algorithm interprets the negative logarithm of the posterior density as a potential energy driving the dynamics of a fictitious statistical mechanics system whose configurations, namely, the system's degrees of freedom, are described by combinations of both parameters (θ, σξ, σz) and realizations ξ of the stochastic process ξ(t). The degrees of freedom of the system are coupled to conjugate variables, that is, to momenta π, paired with the parameters (θ, σξ, σz), and p, paired with the realizations ξ. The posterior density of Eq. () can be rewritten in the following discretized form: 22fξ,θ,σξ,σz∣Pobs,Qobs∝∫exp⁡-HHMCξ,θ,σξ,σz;π,pdπdp, with the Hamiltonian HHMC defined as 23HHMCξ,θ,σξ,σz;π,p=-log⁡fξ,θ,σξ,σz∣Pobs,Qobs+∑α=1Npπα22mα+∑i=1Npi22mi, where Np is the number of parameters to be inferred (eight in our case). We have introduced two auxiliary terms on the right-hand side using the vectors of momenta π and p, akin to the kinetic energies associated with the degrees of freedom of the fictitious statistical mechanics system.

The masses mα and mi are tuning parameters of the algorithm. Since we want the coordinates ξi corresponding to actual observations to be more tightly constrained than those corresponding to the intermediate discretization points, we set msjP+1=M with s=0, …, nP, for the “heavy” measurement points, and msjP+k=m with s=0, …, nP-1 and k=2, …, jP, for the “lighter” intermediate discretization points, with M≫m.

The potential energy, proportional to -log⁡[f(ξ, θ, σξ, σz∣Pobs, Qobs)], guarantees that each state of the system is compatible with the observations and constrained within their measurement uncertainties, as well as with the prior distributions for both model parameters and rainfall potential realizations ξ.

The HMC algorithm iterates the following steps. First, vectors of momenta π and p are drawn from the zero-mean normal distributions defined by the kinetic terms in Eq. (). Then, the system is left to evolve for a fictitious time interval dτ in the (ξ, θ, σξ, σz; π, p) phase space according to Hamilton's equations. This Hamiltonian dynamics is controlled by tuning the fictitious masses in Eq. (), which represent the variances of the normal distributions for the corresponding momenta. Finally, the discretization error on the energy conservation introduced with the integration of Hamilton's equations is corrected by a Metropolis accept/reject step. The resulting marginal distributions of the Markov chains of the system configurations (ξ, θ, σξ, σz) represent a sample of the sought posterior density. The method is described in detail in .

Note that the presence of pronounced local minima in a high-dimensional phase space might represent an insurmountable obstacle, even for more refined implementations of the HMC method, for example, with automatic tuning of the algorithm hyper-parameters (see Sect.  for more details). In that case one would have to resort to further enhancements such as metadynamics or parallel tempering . However, this is not an issue in the specific application discussed here.

Using Eq. () and the definitions given in Sect. –, we write the HMC Hamiltonian as 24HHMC=HN+Hn+H1, where the three components describe dynamics occurring on different timescales. Let us consider them individually. The first component, 25HN=∑s=1nP∑k=2jPp(s-1)jP+k22m+kτ4(k-1)dtu(s-1)jP+k2, contains the harmonic term for the intermediate discretization points from the action of Eq. () and scales as the total number of discretization points N. The second component, 26Hn=∑s=1nP+1p(s-1)jP+122M+∑s=1nPτ4jPdtu(s-1)jP+1-usjP+12+∑s=1nQ+1HQobs,s-HQM,(s-1)jQ+1(u,θ)22σz2+nQ+1log⁡σz+∑i,P≠0r-1Pobs,i-u(i-1)jP+122σξ2+log⁡Ji+nP≠0log⁡σξ-∑i,P=0log⁡121+erfξr-u(i-1)jP+1σξ2, contains a harmonic term for the measurement points from Eq. () and the observation error models f(Qobs∣u, θ, σz) and f(Pobs∣u, σξ) from Sect.  and , respectively. All the components of Eq. () scale as the number of observations nP or nQ. Note that both the observation models for runoff and rainfall are rewritten in the space of u-coordinates. While the coordinate transformation ξ→u is straightforward for the rainfall observation model, which only depends on measurement points, it is somewhat more cumbersome for the runoff observation model, which requires the ξ-dependent component QM,i(ξ) (see Eqs.  and ) to be rewritten in the u space as QM,i(u). Such transformation is described in Sect. S3. The third component of the Hamiltonian HHMC is 27H1=∑α=1Npπα22mα+uN24+u124+dt4τ∑s=1nPusjP+12+∑k=2jP∑l=kjP+1k-1l-1u(s-1)jP+l+jP-k+1jPu(s-1)jP+12+∑θ∈θNθ-μθ22σθ2+∑θ∈θLNlog⁡θ+log⁡θ-μLN,θ22σLN,θ2, which includes the remaining terms from the rainfall potential prior (Eq. ) and the model parameter priors. The latter, i.e. the last two terms in Eq. (), are sums over the parameters θN=(S1, ξr) and θLN=(K,Qgw,λ,γ,σξ,σz), whose prior densities are normal and log-normal distributions, respectively. Note the sign change in front of the term u12/4 with respect to Eq. () due to the initial condition for u1 (see Eq. ). The component H1 does not grow unbounded with either N or nP or nQ (notice that dt∝1/N).

Let us now exploit the different timescales characterizing the three components of the Hamiltonian HHMC. We assume the regime HN≫Hn≈H1, implying that the number of data points is not too large and that the total number of discretization points is instead large compared to the number of data points. Under these conditions, the dynamics of the system occur on very different timescales. In particular, the dynamics described by HN are much faster than the other components of the Hamiltonian and therefore impose the most stringent limitations on the step size for the numerical integration of Hamilton equations. However, we use Trotter's formula to construct a reversible molecular dynamics integrator to take the different timescales into account as described in . In particular, we exploit the fact that the much faster dynamics of the intermediate discretization points described by HN is analogous to a system of uncoupled harmonic oscillators that can be solved analytically. This analytical solution gives a significant boosting contribution to the intrinsic efficiency of the HMC algorithm. The “slow” remaining components of the Hamiltonian, Hn and H1, can be integrated using a much larger integration step, which allows us to explore the high-dimensional parameter space of the system with remarkable efficiency.

As explained in , the decoupling of the different dynamics and the analytical solution of the expensive fast component is always possible for one-dimensional SDE models. It is also possible in the case of multiple independent variables, where the decoupling procedure can be applied to each of them individually. In this way, our HMC approach covers a significant range of possible hydrological modelling scenarios. However, in this work we focus only on 1D models, leaving the exploration of higher-dimensional models to future studies.

In this work we apply a HMC method with a stochastic input model (SIP) following to investigate the rainfall-runoff dynamics of an urban catchment based on real rainfall and runoff observations. The catchment is a small and fast-reacting sewer network in Adliswil, near Zurich, Switzerland. Two typical scenarios of possible rainfall data are considered here; that is, we use the following: (1) high-resolution data, with a time resolution of 1 min, recorded by two pluviometers (denoted P1a and P1b) located in the immediate vicinity of the catchment area, and (2) low-resolution data, with a time resolution of 10 min, recorded by a pluviometer (denoted P2) located further away from the catchment, at a distance of about 6 km. We shall refer to the two scenarios as scenario 1 (Sc1) and scenario 2 (Sc2), respectively. More information can be found in .

The precipitation data Pobs used in this study contain nP+1=241 observations in Sc1 and nP+1=25 observations in Sc2, covering a total observation time T=240 min. The initial and final observation time points are the same for both scenarios and include a storm event that took place in the evening of 10 June 2013, approximately between 18:00 and 20:00 LT. We should mention here a substantial difference with respect to . In their work, the authors base the inference process on three independent time series covering a total observation time of 1710 min distributed over a time span of 48 d. In the work presented here, instead, we consider only the first of the three time series, which also happens to be the shortest. Although the HMC algorithm could deal with the multiple independent time series used by , our simplification finds its justification in that it reduces the computational requirements of the problem, without compromising the goal of our work, that is, showing the feasibility of Bayesian inference of both model parameters and high-dimensional system states (i.e. the ξ's) under the considerable hardship of a stochastic input process.

The discharge flow at the outlet of the catchment was measured with a time resolution of 4 min, and the output observations Qobs, containing nQ+1=61 measurements, are the same for both scenarios and considered accurate compared to the precipitation data. The initial and final observation times are the same as for the input time series. The time series for the observed outputs (Qobs) as well as the observed precipitation (Pobs) for both Sc1 and Sc2 are shown in Fig. .

Scenario 1 represents a best-case scenario of input data availability, and we shall therefore classify it as the accurate input scenario, while scenario 2 is a typical example of inaccurate and unreliable input data due to both its sparsity and the distance of the P2 rain gauge from the area of interest. The runoff observations Qobs exhibit an important rainfall event (the storm) represented by an evident discharge peak. While this event is properly recorded by the rain gauges of Sc1, the inaccurate input data of Sc2 misleadingly recorded the event at an earlier time period, presumably when the storm passed over the location of the distant pluviometer P2.

We are particularly interested in the performance of the combined SIP-HMC method in the latter case, characterized by faulty precipitation data, which clearly represents the most challenging scenario and therefore the hardest test for the HMC method. Therefore, our work consists of three main steps. First, we use the combined SIP-HMC approach described above, with the inaccurate precipitation data Pobs of Sc2, to calibrate the model and infer the unknown “true” average rainfall pattern over the catchment. Then, we use the accurate rainfall observations from Sc1 as a reference to assess the quality of the simulated “true” rain. Finally, we repeat the calibration process using the accurate data of Sc1 as a further validation for the method.

The goal of this work is to demonstrate that HMC algorithms employing a timescale separation can solve hard inference problems with stochastic hydrological models. In we proposed a novel implementation of an HMC algorithm combined with a multiple timescale integration for Bayesian parameter inference with nonlinear SDE models, and we demonstrated the performance of the method using a rainfall-runoff toy model with synthetic data time series. This work is the first application to a real-world case study, with real time series of measured rainfall and outflows. We show that the HMC algorithm is a powerful inference method for the data-driven calibration of hydrological model parameters, especially well suited for both computationally expensive stochastic models and cases, far from rare in hydrology, where the precipitation data are inaccurate and unreliable.

The combined SIP-HMC method presented here allows us also to estimate the “true” average rain input to a hydrological system in the case of highly inaccurate precipitation data probabilistically and with great accuracy, using only prior knowledge and the observed outflow. Runoff data are used by the algorithm as a first-hand information resource about the unknown precipitation over the catchment. This information can override the available, and possibly inaccurate, rainfall data. The reconstructed precipitation is then used to infer the hydrological model parameters, which are thus protected from the deteriorating effect of the uncertainty on the rainfall observations. This approach considerably reduces the bias in the inferred parameters and therefore leads to more reliable runoff predictions, which can in turn be very useful for decision-makers in planning and policy-making.

The use of AD makes the algorithm in principle applicable to more complex models. Indeed, the generalization of the algorithm from the toy model used in presented only one possibly challenging requirement, namely rewriting the Hamiltonian HHMC, while the rest of the implementation of the algorithm remained essentially unchanged. This application shows that ≈103 parameters can be inferred in less than 2 h. This leaves us with a considerable margin to tackle more complex problems and datasets of much larger sizes. However, it must be noted that the HMC algorithm in the form described in this work is limited to non-spatially resolved models depending only on a time variable. In principle, the method could cope with further spatial dimensions, for example, describing inundations, but this would require a non-trivial adaptation of the algorithm, which is not discussed here. More importantly, a significant limitation of the HMC method presented in this work is that it requires, unlike methods that only involve forward-solving the model, such as SIP , an explicitly discretized solution of the hydrological model (see Eq. ), which might not be always easily available. In those cases, one may need to resort to appropriate numerical solvers, often employing advanced implicit schemes, which in turn may make AD problematic.

The extension of the HMC method described here to further hydrological models and systems will be the focus of future works. Furthermore, the HMC algorithm presented here is not at all limited to hydrology. It is a very general, efficient, easily parallelizable and scalable algorithm that makes Bayesian inference with expensive stochastic models feasible in spite of its computational hardships, with a very broad range of potential applications in applied sciences that can benefit from stochastic modelling and a fast Bayesian inference method.