A classic choice in the statistical modelling of extremes is between (a) constructing a detailed model of an entire process, from which its extremal properties can be estimated, either analytically or more usually by numerical methods, or (b) direct modelling of the extremes themselves.  If adequate reliable data are available and the investigator has sufficient time, then approach (a) allows information from other sources (such as physical models) to be included at the modelling stage and has the benefit that estimates of all quantities, including extremes, stem from a single overall model and therefore are consistent.  However this approach is demanding of data and of time, and makes the implicit assumption that the details of the underlying process are relevant to the extremes.  Approach (b) is less demanding of data and avoids detailed modelling by applying the classical theorems of extreme-value theory (EVT) to block maxima or threshold exceedances for the phenomenon of interest.  Although originally developed for independent and identically distributed observations, these theorems have been shown to be robust to plausible types of dependence in the underlying data, and have been widely and generally successfully applied in environmental settings.  They can be regarded as semiparametric models, in the sense that they do not depend heavily on the underlying process.  A major concern is that they rely on limiting approximations (the GEV and GPD) that may fit data at observed levels satisfactorily but extrapolate poorly to unobserved levels.  Such models provide a simple and direct empirical approach to modelling extremes of the underlying phenomenon but it may be a struggle to incorporate physical constraints or other background knowledge into them. 

The paper under discussion can be viewed as a critique of a particular type (a) approach, namely metastatistical extreme-value (MEV)  modelling, from the viewpoint of a classical type (b) approach, namely the fitting of GEV and GPD models to block maxima and threshold exceedances.  There are two main criticisms:

• that papers proposing MEV have done so by application to and illustration on `real data’, in which the true data-generating mechanism is unknown, which implies that it is impossible to compare the behaviour of different approaches under ideal conditions (when the target of inference is known);
• that in any case the comparisons are incorrect, because of confusion over the target of inference (see Figure 7).  Here the point is more subtle, but it is summarised in equation (16) of the paper.  The point here is that if one is estimating a quantile function $Q_\theta(p)=F^{-1}(p;\theta)$ that depends on an unknown parameter $\theta$ and one will estimate $\theta$ from a single sample using an estimator $\hat\theta$, then the estimator of $Q_\theta(p)$ is $Q_{\hat\theta}(p)$, whose properties should be assessed over repeated sampling using independent replicates $Q_{\hat\theta_1}(p), \ldots, Q_{\hat\theta_S}(p)$ based on $S$ samples leading to estimates $\hat\theta_1,\ldots, \hat\theta_S$.   The average of these estimates would be $S^{-1} \sum_{s=1}^S Q_{\hat\theta_s}(p)$, i.e., the right-hand side of (16), rather than $Q_{\bar \hat\theta}(p)$ (the left-hand side of (16)), where $\bar\hat\theta$ is the average of the parameter estimates for the $S$ samples.  The paper under discussion illustrates the difference via the left- and right-hand panels of Figure 7.  (Though the discussion at lines 532-535 leaves it unclear how the `median GEV/Gumbel’ curves are computed — the median for each $p$, giving a result that would not corresponding to any single quantile function, or what?  And if the median, why not the mean?)

Both of these criticisms seem to me to be correct, and they should in my view embarrass the reviewers of the original MEV papers and the journals that published them.

I found the paper to be quite poorly written, to the point of unclarity in numerous places, including lines 405 (`the spreader …’?), 429 (` …, or better, …’?) or 524 (what is a predictive quantile function of a predictive quantile function?), and with many minor errors.  Examples of the latter are that (i) the Beirlant et al book cited at line 18 was published in 2004, not 2006, and (ii) stating on line 476 that the distribution of an order statistic is beta is incorrect — the beta distribution represents variables on a finite interval, and clearly this does not apply to order statistics from, say, a Gaussian sample (did the authors mean that the distribution of an order statistic can be represented _using_ that of a beta random variable?), (iii) equation (17), the left-hand side of which is a function of $z$, while the right-hand side is a number (as the expectation of $Z$ is a constant), and (iv) at line 461, where results from a simulation study are `eventually used to build confidence intervals’ — but in a simulation study the truth is known, so confidence intervals are not needed — as a confidence interval is based on a single sample, we have to guess that the authors mean that their $S$ return level estimates will be used to compute quantiles of a distribution.  The paper is full of inaccuracies of this sort, so the reader is continually wondering `is that correct?’ and concluding `not quite’; this does not give confidence in the main results.   It is the role of the authors to produce a well-crafted article, not that of a reviewer, so I will not give more examples (it would take many pages to list them all), but generally I found the writing to be unclear, long-winded, and in need of a careful review by a native English-speaker (see, e.g., line 514).  Reducing the paper radically by revising and trimming the text throughout would improve it.  I would also target much of the general text in the earlier sections for cuts, since it is mostly not germane to the criticism of the MEV work.  A 15-page paper in the current format would make the main points more clearly and should be more readable.  

Please refer to the attached pdf.