Improving object-oriented radar based nowcast by a nearest 1 neighbour approach 2

6 The nowcast of rainfall storms at fine temporal and spatial resolutions is quite challenging due to the erratic 7 nature of rainfall at such scales. Typically, rainfall storms are recognized by radar data, and extrapolated in the future by 8 the Lagrangian persistence. However, storm evolution is much more dynamic and complex than the Lagrangian 9 persistence, leading to short forecast horizons especially for convective events. Thus, the aim of this paper is to investigate 10 the improvement that past similar storms can introduce to the object-oriented radar based nowcast. Here we propose a 11 nearest neighbour approach that measures first the similarity between the “to-be-nowcasted” storm and past observed 12 storms, and later uses the behaviour of the past most similar storms to issue either a single nowcast (by averaging the 4 13 most similar storm-responses) or an ensemble nowcast (by considering 30 most similar storm-responses). Three questions 14 are tackled here: i) what features should be used to describe storms in order to check for similarity? ii) how to measure 15 similarity between past storms? and iii) is this similarity useful for storm oriented nowcast? For this purpose, individual 16 storms from 110 events in the period 2000-2018 recognized within the Hannover Radar Range (R~115km2), Germany, 17 were used as a basis for investigation. A “leave-one-event-out” cross-validation is employed to train and validate the 18 nearest neighbour approach for the prediction of the area, mean intensity, the x and y velocity components of the “to-be19 nowcasted” storm for lead times up to + 3 hours. Prior to the training, two importance analyses methods (Pearson 20 correlation and partial information correlation) are employed to identify the most important predictors. The results 21 indicate that most of storms behave similarly, and the knowledge obtained from such similar past storms can improve 22 considerably the storm nowcast compared to the Lagrangian persistence especially for convective events (storms shorter 23 than 3 hours) and longer lead times (from 1 to 3 hours). The nearest neighbour approach seems promising, nevertheless 24 there is still room for improvement by either increasing the sample size or employing more suitable methods for the 25 predictor identification. 26


29
Typically, radar based nowcasts are used for short-term rainfall nowcast. The rainfall is either considered as an 30 object (a set of radar grid cells with the intensity above a threshold that moves together as a unit and is regarded as a storm 31 (Dixon & Wiener, 1993;Johnson et al., 1998)) or as an intermittent field (intensity is moving from one pixel of the radar 32 image to the other (Ruzanski et al., 2011;Zahraei et al., 2012)). Whilst the field-based approach of rainfall nowcasting 33 has gained popularity recently, here the focus is only on the object-oriented forecast, thus on the nowcasting of storms. In 34 such forecast three mains steps are performed (illustrated in Figure 1): a) first the storm is identified -so a group of grid 35 cells with intensity higher than a threshold is recognized in the radar image at time t0, b) the storm identified is then 36 tracked for the time t0+Δt (where Δt is the temporal resolution of the radar data) and velocities are assigned to the 37 movement of the storm, and finally c) the storm as lastly observed at time t (when the forecast is issued) is extrapolated 38 at a specific lead time (the time in the future when the forecast is needed) t+TL, with the last observed velocity vector. This

64
The errors due to the Lagrangian persistence are particularly high for convective events at longer lead times (past 65 1 hour) as the majority of convective storms last no longer than 60 minutes (Goudenhoofdt & Delobbe, 2013; Wilson et 66 al., 1998). At these lead times, the persistence fails to predict mainly the death of these storm cells, while for shorter lead 67 times it fails to represent the growing/decaying rate and the changing movement of a storm cell (Germann et al., 2006).

68
For stratiform events, since they are more persistent in nature, Lagrangian persistence can potentially give reliable results 69 up to 2 or 3 hours lead time (Krämer, 2008). Nevertheless studies have found that for fine spatial (1km 2 ) and temporal

72
For object-oriented radar based nowcast, this predictability limit can be extended up to 1 hour for stratiform events and 73 up to 30-45min for convective events if a better radar product (merged with rain gauge data) is fed into the nowcast model 74 (Shehu & Haberlandt, 2021). Past these lead times, the errors due to the growth/decay and death of the storm govern.

75
Nevertheless, one can estimate roughly these processes by acknowledging previously observed storm cells. For 76 instance, if it is known that a storm is of convective nature, most probably it will die from 20 min to two hours of the 77 storm birth, the peak intensities happen mainly in the afternoon or evening, and that they dissipate fast after the peak 78 intensity has been reached. Such characteristics of storm behaviour can be analysed from the past observation 79 (Goudenhoofdt & Delobbe, 2013;Zawadzki, 1973   by 20 %. These studies suggest that past observed storms may be useful in extending the predictability limit of the storms 87 at hand. Thus, the aim of this study is to investigate if non-linear relationships learned from past observed storms can 88 surpass the Lagrangian persistence and extend the predictability limit of different storms. For this purpose, a nearest so an ensemble of responses. Ensemble nowcasts are more preferred for rainfall nowcasts due to the high uncertainty 104 associated with rainfall predictions at such fine scales (Germann & Zawadzki, 2004).

105
Before applying a k-NN for the storm nowcast, question that arise are I) what features are more important when 106 describing a storm, II) how to evaluate similarity between storms and III) how to use their information for the nowcasting 107 of the storm at hand. To answer these questions and to achieve the aim of this study, the paper is organized as follows.

108
First in Section 2 the study area is described, following with the structure of the k-NN method in Section 3.

115
The study area is located in northern Germany, and lies within the Hannover Radar Range as illustrated in Figure   116 2. The radar station is situated at the Hannover Airport, and it covers an area with a radius of 115 km. The Hannover radar were extracted (see Shehu & Haberlandt (2021) or Shehu (2020)). These events were selected for urban flood purposes, 125 thus contain mainly convective events and few stratiform events.
bigger than 16 radar grid cells has an intensity higher than 25 dBz, and as stratiform -if a group bigger than 128 radar 133 grid cells has an intensity higher than 20 dBz. The tracking of individual storms in consecutive images is done by the 134 optimization of the cross-correlation between the last 2 images (t=0 and t-5 min), and local displacement vectors for each 135 storm are calculated. In case a storm is just recognized, then global displacement vectors based on cross-correlation of 136 the entire radar image are assigned to them.

137
Thus, a dataset with several types of storms is built and saved. The storms are saved with an ID based on the 138 starting time and location, and for each time step of the storm evolution the spatial information is saved. Here the spatial 139 rainfall intensities of a storm at a particular time step (in 5min) of the storms' life, is referred to as the "state" of the storm.

140
A storm that has been observed for 15 minutes, consists of three "states" each occurring at a 5 min time step. For each of 141 the storm states an ellipsoid is fitted to the intensities in order to calculate the major and minor axis and the orientation 142 angle of the major axis. This storm database is the basis for developing the k-NN method and for investigating the 143 similarity between storms. Some characteristics of the identified storms like duration, mean area, maximum intensity,

144
number of splits/merges, local velocity components, and ellipsoidal features, are shown in the Figure 3.

145
As seen from the number of storms for each duration in Figure 3, the unmatched storm cells make the majority 146 of the storms recognized. These are storms that last just 5 min (one-time step) as the algorithm fails to track them at 147 consecutive time steps. These "storms" can either be dynamic clutters from the radar measurement, as they are 148 characterized by small area, circular shapes (small ratio of minor and major axis) and by very high velocities, or artefacts 149 created by low intensity thresholds used for the storm identification, or finally produced by the unrepresentativeness of 150 the volume captured by the radar station. Apart from the unmatched storms, the majority of the remaining storms are of 151 convective nature: storms with short duration (shorter than 6 hours), high intensity and low areal coverage.

152
Here two types of convective storms are distinguished: local convective with very low coverage and low intensity,

156
(respectively circa 20 and 50 storms). Therefore, it is to be expected that the k-NN approach may not yield very good can be seen that for stratiform storms that live longer than twelve hours the variance of the characteristics is quite low

159
(when compared to the rest of the storms) which can either be attributed to the persistence of such storms or to the low 160 representativeness in the database. Thus, even though the data size for stratiform is quite small, the k-NN may still deliver 161 good results as characteristics of such storms are more similar.

174
where Pi is the predictors value at time i, and P30 the average value of the predictor over last 30min. The selected features 175 (both present and past) that are used here to describe storms as objects and hence tested as predictors are shown in Table   176 1. The present features help to recognize storms that are similar at the given state when the nowcast is issued (blue storm 177 in Figure 4) and the past ones give additional information about the past evolution of the storm (average of grey storms 178 in Figure 4).

179
The aim of these features is to recognize the states of previously observed storms that are most similar to the 180 current one (shown in blue in Figure 4) of the "to-be-nowcasted" storm. Once the most similar past storm states are 181 recognized, their respective future states at different lead times can be assigned as the future behaviour (shown in green 182 in Figure 4) of the current state of the "to-be-nowcasted" storms. Since the storms are regarded as objects with specific 183 features, future behaviours at different lead times are determined by four target variables: area (A+LT), mean intensity (I+LT) 184 and velocity in X (Vx+LT) and Y (Vy+LT) direction. Additionally, the total lifetime of the storm is considered as a fifth target 185 (Ltot). Theoretically, the total lifetime is predicted indirectly when any of the first four targets is set to zero, however here 186 it is considered as an independent variable in order to investigate if similar storms have similar lifetime durations. Figure 4 The features describing the past (grey) and present (blue) states of the storm used as predictors to nowcast the future states of the storm (green) at a specific lead time (T+LT) that is described by 4 target variables (in red). The nowcast is issued time t0.

199
Thus, to avoid the influence of these outliers, the given range is employed.

212
The method is based on a metric called the Partial Information Correlation and is computed from the Partial Information 213 as:

215
where PIC is the Partial Information Correlation and the PI is the Partial Information. The Partial Information itself is a 216 modification of the Mutual Information in order to measure partial dependency between the predictors and the target 217 variable, by adding predictors one at a time (step-wise procedure). The evaluation of PIC needs a pre-existing identified 218 predictor from which the computation can start. If the pre-defined predictor is correctly selected, then through the Equation

219
(3), the method is able to recognize and leave out the new predictors which are not related to the response and which don't 220 bring additional value to the existing relationship between the current predictors and target variable. Relative weights for 221 the k-NN regression application can be derived for each predictor, as a relationship between the PIC metric and the 222 associated partial correlation:

224
where X is the predictor, Y the target response, SY|X(-j) the scaled conditional standard deviations between the first predictor αj the predictors weight. The R package NPRED was used for the investigation of the PIC derived importance weights 227 (Sharma et al., 2016).
228 Table 1 List of all the past and present features of the storm object that are investigated for their importance as predictors, and the respective target variables calculated for different lead times.

237
The structure of the proposed k-NN approach at the storm scale is illustrated at Figure 5 -left) the current "to-238 be-nowcasted" storm is shown, while at -right) the past observed storms. First in Step 1, the Euclidean distance between 239 the most important predictors (either present or past predictors), of past storm states and the current one is calculated to 240 identify the most-similar states of the past storms (distance between the blue shapes at left and right side of Figure 5):

242
where w is the weight of the respective i th predictor, X the predictor of the "to-be-nowcasted" storm, Y the predictor of a 243 past observed storm, N the total number of predictors used and Ed the Euclidian distance between the "to-be-nowcasted" 244 and a past observed storm. The assumption made here is that the smaller the distance, the higher the similarity of future 245 behaviour between the selected storms and the "to-be-nowcasted" storm. Therefore, in Step 2 these distances are ranked

253
where k is the number of neighbours obtained from optimization, R and Pr are respectively the response and weight of 254 the i th neighbour and the Rnew the response of the "to-be-nowcasted" storm as averaged from k neighbours. Contrary, if a 255 probabilistic nowcast is selected, 30-ensembles are issued independently; to each neighbour a probability is assigned 256 according to their rank with the "to-be-nowcasted" storm:

258
where k is the selected number of neighbours and R and Pr are respectively the rank and the probability weights of the i th 259 neighbour. The probability weights calculated here are as well used for computation of the single nowcast in Equation

260
(6). Only neighbours that display a distance lower than 0.5 are selected for both single and ensemble nowcast in order to 261 minimize the influence of non-similar storms.

262
Since the performance of the single k-NN nowcast is highly dependent on the number of k -neighbours used for 3.2 Training of the k-NN and performance assessment

311
As stated earlier the results depend on the time of nowcast and also storm duration (in regard to available storms).

312
Therefore, the performance criteria for both single and ensemble nowcast were computed separately for different storm 313 durations and time of nowcasts as illustrated in Table 2. It is important to mention as well, that since one event may 314 contain many storms of similar nature, when leaving one event out for the cross-validation, the number of available storms 315 is actually lower than the numbers given in Table 2. This is particularly affecting the performance of the storms longer 316 than 6 hours, as the "leave-one-event-out" cross-validation causes fewer available storms for the similarity computation.  On the other hand, are the velocity components, which seem to be highly dependent on the 330 autocorrelation and slightly correlated to area and ellipsoidal axes. It has to be mentioned that apart for the standard 331 deviation intensities also the mean, median, and maximum spatial intensities were investigated. Nevertheless, it was found 332 that the Isd1 and Isd2 had the higher correlation weights, and since there is a high collinearity between these intensity 333 predictors, they were left out of the predictor's importance analysis.

334
The application of the PIC analyses requires that the most important predictors should be introduced to the

354
as seen in the PIC. Therefore, the predictors estimated from the correlation with the given weights in Figure 6 are used 355 as input to the k-NN application. In order to make sure that the predictor set from the Pearson correlation was the right

379
Overall, it seems that averaging between 2 to 10 neighbours give the best results depending on the lead time, and 380 there is a clear decreasing trend of neighbours with increasing lead time. The best achieved k-numbers from the two k-

381
NN applications are different from one another at some lead times, nevertheless they seem to converge around k=3 or

387
-for storms that lived 30min, ii) middle row -for storms that lived up to 3 hours and iii) lower row -for storms that 388 lived longer than 3 hours, and are averaged per nowcast times given in Table 2. To getter a better overview of the majority

401
For the storms living up to three hours, the same behaviour is, more or less, observed.   storm-based 4-NN produces circa 10% lower errors than the target-based one for the nowcast times lower than 30min, 409 while for later nowcast times the errors are clearly higher than the target based one (up to 100% higher). As the sample 410 size is the same for both approaches, it seems like storm-based may be more appropriate at the beginning of the storm's 411 life and that these storms behave more similarly at the first 30 minutes of their evolution rather than in their later life.

412
For the storms that live longer than 3 hours (under 100 storms available) the same problem, as in the nowcast

429
For the 30min storms, the 4-NN approach (both target-and storm-based) are considerably better than the 430 Lagrangian persistence: improvement is higher than 50% from the LT+15min and up to 100% from LT+30min. The

445
For storms living longer than 3 hours, the improvements are present for lead times higher than 2 hours. Since the 446 features of the long storms (mostly of stratiform nature) are persistence in time, is understandable for the Lagrangian

447
Persistence to deliver better nowcast up to LT+2h. Past this lead time non-linear transformations should be considered.

448
Here, even though the storm database is small, the non-linear predictions based on the 4-NN capture better these 449 transformations than the persistence. The improvement introduces by the storm-based are generally from 20-100% lower 450 than the improvements introduced from the target based.

451
To conclude, the 4-NN single nowcast brings up to 100% improvements for lead times higher than the 452 predictability limit of the Lagrangian persistence and are dependent mainly on the storm type and the size of database.

453
Overall for all of the storms the improvement is mainly at the high lead times and later times of nowcast, as the k-NN is 454 capturing particularly well the death of the storms. The results from the long events are suffering the most from the small 455 size of the database. This was anticipated, as the events were mainly selected from convective and mesoscale convective 456 events that have the potential to cause urban floods. A bigger database, with more stratiform events included, will 457 introduce a higher improvement to the Lagrangian persistence. These improvements are expected to be higher for lead 458 times longer than 2 hours, but is yet to be seen if a larger database can as well behave better than the persistence even for 459 lead times shorter than the predictability limit. Regarding the two different 4-NN approaches, the storm-based performs 460 around 20% worse than the target-based nowcast, introducing generally 40% lower improvements to the Lagrangian all errors are zero. This suggests once again that storms in this duration behave similarly and their response can be 475 predicted adequately by similar neighbours.

476
For storms that live shorter than 3 hours, the error of the best ensemble member is decreasing with increasing is not able to capture the storm evolution. For short storms (duration shorter than 30min) the number of ensembles is low 515 for lead times up to 30 min and in this range the ensembles are worse for the early times of nowcasts. However, past this 516 lead time, the number of better ensembles is more than 80 % (24 ensembles) with no clear difference between different 517 times of nowcast. This coincides with the predictability limit of the Lagrangian persistence at such scales. Thus, it makes 518 sense that the ensemble nowcasts behave better after the predictability limit of the persistence is reached. Moreover, for 519 these storms the difference between the two types of 30-NN is insignificant (less than 1% for all target variables and times 520 of nowcasts).

521
For storms that live shorter than 3 hours, the results are slightly worse than the very short storms. Here as well 522 the number of better ensembles increases drastically for all the target variables between LT+15min to LT+30min.

523
Interesting in this storm group are the results from the nowcast at the 3 hours of storm existence that exhibit different The percent of time steps is calculated for storms that lived shorter than 30 min (upper row), shorter than 3 hours (middle row) and longer than 3 hours (lower row), and for the respective nowcast times.

527
For the longer storms the percent of better ensembles is increasing with the time of nowcast and are increasing 528 mainly for LT+45min to LT+60min, but still not as high as in the other storm groups. The worse performance is at 529 nowcasts at the 1 st time step of the storm where the percent of better ensembles is quite low (between 1 and 0 ensembles)

530
for the LT+180min for all of the target variables. What is interesting from these storms, is that the percent of better 531 ensembles is higher at the Velocity components than in the Area and Intensity predictions. This suggest the velocity 532 components are more persistent (see Figure 3) and easier to be predicted from similar storms. Still it is worth mentioning 533 that the percent of better ensembles is almost never zero. Even with a small database for the long storms, the 30-NN can 534 recognize 1-5 similar past storms that can give useful information in improving the nowcast when compared to the 535 Lagrangian persistence.

541
The focus of this paper was the improvement of the storm-oriented radar based nowcasts by considering other non-linear 542 behaviour for future extrapolation instead of the Lagrangian persistence. For this purpose, a nearest neighbour approach 543 was proposed that predicts future behaviours based on past observed behaviours of similar storms. The method was

550
First an importance analysis was performed in order to recognize the most important predictors for each of the 551 target variable. Two different approaches were employed for this purpose: Pearson correlation, and Partial Information 552 Correlation (PIC). A comparison of these two methods revealed that for the application at hand the Pearson Correlation 553 is more reliable at determining important predictors, and delivers 5%-30% better results than the PIC method. However, 554 the PIC seems promising mainly for determining the most important predictors of the Area and Total Lifetime for storms 555 longer than 3 hours, and is still recommended to investigate for further works. The Area, Intensity and Total Lifetime of 556 the storms seem to be co-dependent on one another and on the features that describe their evolution. In particularly the 557 variance of the spatial intensity is an important predictor for the three of them. On the other hand, the velocity components 558 are dependent as well more on features that describe their evolution. Nevertheless, there is still a dependency of the area 559 and velocity components, and should be included when predicting each other.

560
The weights derived from the Person correlation were used for the similarity estimation of different storms based 561 on the Euclidian distance. Two k-NN approaches were developed on two measurement of similarity: a) target-based 562 approach -similarity was computed for each target independently and indicates the best performance possible by the 563 given predictors and weights, and b) storm-based approach -similarity was computed for each storm keeping the 564 relationship between the target variables. For the two approaches a single (averaging the 4 closest neighbours) and an 565 ensemble (with 30 nearest neighbours) nowcast were issued for all of the storms in "leave-one-event-out" cross-validation 566 mode. In the single nowcast the difference between the two lied mainly at short lead times (up to 30 min) with the event-567 based results yielding 10-30% higher errors than the target-based ones. Exception was the Total Lifetime where the storm-568 based prediction was almost the same as the target-based approach. However, at higher lead times the difference between 569 the two became insignificant, as the death processes was captured well for the majority of the storms. The same behaviours 570 were observed as well in the ensemble nowcast, with target-based ensembles being slightly better than the storm-based 571 nowcast.

572
To investigate what value each of the two k-NN approaches introduces to the nowcast, their errors (for both 573 single and ensemble nowcast) were compared to the errors produced by the Lagrangian persistence. For both of the 574 approaches the improvement was up to 100% for convective storms for lead times higher than 15 min, and up to 50% for 575 mesoscale storms for lead times higher than 2 hours. The results were particularly good for the small convective storms 576 due to the high number of storms available in the database. For the mesoscale storms (with duration longer than 3 hours) 577 the improvements were not satisfactory due to the small sample size of such long storms. An increment in the sample size 578 is expected to improve the performance of the k-NN for these storms as well. However, when consulting the ensemble k-579 NN application it seems that, even for these storms and the given database, there are at least 5-10 ensemble members that 580 are better than the Lagrangian persistence. This emphasizes not only the importance of the ensemble nowcast in 581 comparison to the single one, but also the importance of nearest neighbour method in its potential to improve the nowcast.

582
Overall the results suggest that if the database is big enough, storms that behave similarly can be recognized by