Spatio-Temporal Characteristics of Heavy Precipitation Forecasts from ECMWF in Eastern China

Conventional verification methods rely on basic grid overlays, where the forecast grid aligns with the observation grid. A standard contingency table is then created, categorizing the forecast and observation grid points into four scenarios. Skill scores can be derived from the counts in this table (Doswell et al.^[1]). While these statistics offer some insight into the overall forecast performance, they do not provide specific details on the areas where the forecast succeeded or failed. Therefore, alternative approaches are necessary (Gilleland et al.^[2]).

Over the past two decades, several new spatial verification approaches have emerged to assess forecasts (Casati et al.^[3]; Ebert and McBride^[4]; Marzban and Sandgathe^[5]; Davis et al.^[6]; Davis et al.^[7]; Ebert^[8]; Ebert^[9]; Casati et al.^[10]; Wernli et al.^[11]; Lack et al.^[12]). Among these, the Method for Object-Based Diagnostic Evaluation (MODE) stands out as a novel method that deviates from traditional verification techniques by involving processes such as object identification, matching, and comparison of forecast and observed object attributes (Davis et al.^[13]; Gilleland et al.^[14]; Wolff et al.^[15]). This approach was advanced by Bullock, who developed a time-domain version of MODE (MODE-TD) and extended the method into three dimensions. Here, "three dimensions" refers to the combination of two dimensions of horizontal space and the dimension of time. While the MODE method provides only two-dimensional spatial error information, the MODE-TD method offers not only two-dimensional spatial error information but also dynamic three-dimensional error information.

At present, there are some forecast-verification studies on precipitation and other elements using the MODE-TD method. The MODE-TD was applied to ensemble precipitation prediction with different microphysics parameterizations (Clark et al.^[16]). Mittermaier and Bullock used the method to verify 36-h two-dimensional total cloud forecasts from one convection-permitting and one near-convection-resolving Met Office Unified Model (Mittermaier and Bullock^[17]). The method was also used to comprehensively analyze the precipitation features in western Canada based on multiple observations, reanalysis, and model data (Li et al.^[18]).

Rainstorms are one of the most severe weather events in China (Tian et al.^[19]). Due to the influence of the East Asian summer monsoon, heavy rainfall frequently occurs in China (Gao et al.^[20]; Lu et al.^[21]). With the northward shift of the East Asian summer monsoon, the rain belt in eastern China moves from the south of the Yangtze River valley to the Yellow River valley (Zhu et al.^[22]). The sufficient water vapor brought by summer monsoon from the sea may easily cause a flood disaster (Fu et al.^[23]). In recent years, rainstorm events have become more frequent. There were 30 large-scale heavy rainfall events in China during the main rainy season of 2020 (Ding et al.^[24]). The middle and lower reaches of the Yangtze River had a super-long Meiyu period of 62 days, which was the longest since 1961 (Cui et al.^[25]). In terms of tropical cyclones, the heavy rainfall caused by Typhoon "Meranti" brought huge economic losses to Xiamen (Huang et al.^[26]; Tang et al.^[27]). An extreme rainstorm occurred in the central and northern regions of Henan Province on 20 July 2021 (Liu et al.^[28]; Chyi et al.^[29]; Zhong et al.^[30]; Huang et al.^[31]). In this extreme event, the Zhengzhou automatic weather station broke the historical record since its establishment.

At present, the numerical weather prediction model is still an important reference index for heavy rainfall forecasts. It is necessary to strengthen research on how to evaluate the predictive ability of numerical weather models scientifically. The verification of high-impact weather systems by numerical models needs to pay attention not only to their ability to predict precipitation areas but also to their ability to predict the occurrence, development, movement speed, and history of the life of the whole weather system. General two-dimensional spatial verification approaches are difficult to achieve such functionality due to the lack of time dimension. Therefore, it is necessary to strengthen the application research of the MODE-TD method.

The European Centre for Medium-Range Weather Forecasts (ECMWF) has become an indispensable reference basis for operational daily forecasting in China (Xiao et al.^[32]). At present, there is relatively little research on the ability of the ECMWF model to predict heavy precipitation in eastern China, which is caused by variable strong synoptic forcing types. This study aims to explore the spatio-temporal deviation characteristics of ECMWF heavy rainfall forecasts under various strong synoptic forcing types in eastern China using the MODE-TD method. This study aims to provide more diagnostic information for researchers and forecasters on the ability of ECMWF to predict heavy precipitation in eastern China and more guidance for forecasters to apply the ECMWF model better. The remainder of this paper is organized as follows. Section 2 describes the ECMWF model, MODE-TD, and weather cases. The results of several case studies are presented and discussed in section 3. Finally, section 4 summarizes the whole text and presents the conclusions.

The ECMWF began to produce and release medium-range integrated forecasts in December 1992. The integrated forecasting system of ECMWF has developed into a relatively complete earth system model widely used in daily weather forecasting services (Xie et al.^[33]). The ECMWF data used in this study were obtained from the ECMWF IFS Cycle 45r1 version in 2018, the ECMWF IFS Cycle 46r1 version in 2019, and the ECMWF IFS Cycle 47r1 version released after 2020. The forecast starts at 08:00 BJT (Beijing time). The spatial range is 105°–122°E and 20°–42°N, and the horizontal resolution is 0.125° × 0.125°. To facilitate verification, the model forecast data are processed according to the specific time of weather cases to obtain the 3-h interval forecasts of the first 24 hours.

Hourly rainfall data from automatic weather stations are used as observation data for comparison in this study. The total number of automatic weather stations is about 26000. To compare with the forecasts, the hourly rainfall observation data are accumulated to obtain the 3-h rainfall observation data.

The MODE-TD is an extension of the MODE object-based approach to verification (Bullock^[34]). There are two spatial dimensions and one time dimension. The method of MODE-TD for preliminary image processing of forecast and observation data is similar to the two-dimensional MODE method, which includes smoothing processes and thresholds.

The process of decomposing precipitation objects from the original data field in MODE-TD is called convolution thresholding. The original data field is first convolved by a filter function, as shown below:

Formula (1) presents the original data field, ϕ presents the filter function, and C presents the convolved field obtained after processing. The variables (x, y) and (u, v) present grid point coordinates. Unlike the two-dimensional MODE method, the amount of data processed increases significantly after introducing the time dimension. A square convolution filter function is chosen instead of a circular filter function to improve data processing speed.

Among them, the parameter R and H are not independent of each other and satisfy the following relationship:

Therefore, the influence radius R is the only adjustable parameter in the convolution process. That means that once the radius is determined, the height (H) value is fixed. The threshold is set for the convolved field C to obtain the masked field M:

The object is the continuous region where M=1. Finally, the original data is restored to obtain the object field F from the initial object.

In this way, the two parameters (influence radius R and threshold T) control the entire process of decomposing the original field into objects.

Figure 1 shows how to incorporate the time dimension into MODE-TD, assuming we have two-dimensional data with continuous and equidistant spatio-temporal intervals. The blue dot in the figure represents the identified centroid position of the object. As time changes, the object moves to generate a movement trajectory. Apart from the two dimensions of space, time is the third dimension. The MODE-TD forms part of the Model Evaluation Tools (MET), a verification toolset available from the Developmental Testbed Center (DTC) of the National Center for Atmospheric Research (NCAR).

Figure 1.  The schematic diagram of MODE-TD incorporating time dimension.

The evaluation metrics of MODE-TD include two-dimensional and three-dimensional evaluation metrics. The two-dimensional evaluation metrics of MODE-TD are similar to those of the MODE method, mainly including object centroid position, object centroid longitude and latitude, object area, and axis angle difference. The three-dimensional evaluation metrics can be further divided into three-dimensional single-object metrics and three-dimensional paired object metrics. The three-dimensional single-object metrics mainly include object start and end time, object axis angle, object centroid longitude and latitude, and percentile intensity. The three-dimensional paired object metrics mainly include object initial time deviation, object end time deviation, angular deviation, speed_delta, direction deviation, volume ratio, and overall correlation.

In this study, 23 heavy rainfall cases in eastern China are selected during 2018–2021, which are mainly caused by low-level jets, the Meiyu front, typhoons, WPSH, and the Jianghuai cyclone (Table 1). In view of the fact that the forecast domain of the model is mainly in eastern China, so the heavy rainfall cases caused by southwest vortex and northeast cold vortex are not discussed.

In this section, MODE-TD is first applied to a representative weather case to demonstrate how it evaluates the precipitation object area and life history. Then, based on the identification of precipitation object characteristics in 23 weather cases, they are classified and analyzed from spatial and temporal features, respectively.

Typhoon "Rumbia" landed on the southern coast of Pudong in Shanghai at 2000 UTC (Universal Time Coordinated) on August 16, 2018. Within 24 hours after landing, it mainly caused heavy rainfall in Jiangsu and Anhui Provinces. Fig. 2 shows the actual 24-h cumulative precipitation from 08:00 on August 17, 2018 to 08:00 on August 18, 2018. The 24-h precipitation in central Anhui and southern Jiangsu exceeded 70 mm and even 140 mm in some areas.

Figure 2.  The observed 24 h accumulated precipitation from 0000 UTC on August 17 to 0000 UTC on August 18, 2018.

The 3-h observation and forecast rainfall objects recognized by MODE-TD from 0000 UTC on August 17 to 0000 UTC on August 18, 2018, are shown in Fig. 3. The ECMWF model simulations initialized at 0000 UTC on August 17, 2018, use a threshold of 10 mm (3h)⁻¹ and a smoothing radius of 36 km. The colored area indicates the observation object, and the outline is the forecast object. As shown in Fig. 3, the 10 mm typhoon precipitation forecast objects can be recognized in each forecast time and correspond well with the observation objects. The two-dimensional spatial verification indexes of 3 h precipitation objects are analyzed in Table 2. The average centroid distance of matched objects is 7.28 grid spacing. The smallest centroid distance occurs in 9–12 h, and the largest occurs in 21–24 h. The average angle difference is 23.14º. The smallest difference occurs in 21–24 h, and the largest occurs in 9–12 h. It can be seen from the area ratio index that the most matching time between the forecast object and observed object occurs in 9–12 h, 12–15 h, and 21–24 h. The total number of precipitation object areas predicted by the model is 5821, and the corresponding number of observations is 6461. Therefore, the precipitation influence area forecasted by the model in 24 hours is smaller than the observation. Fig. 4 shows the centroid trajectories of the 3 h precipitation object. As can be seen, the observation objects mainly move westward and northward (Fig. 4a). The simulation results (Fig. 4b) show that the centroid trajectory of the forecast precipitation object has a good correspondence with that of the observation object.

Figure 3.  The 3 h observed and forecasted rainfall objects of Typhoon "Rumbia" recognized by MODE-TD from 0000 UTC August 17 to 0000 UTC August 18, 2018. The ECMWF forecast is initialized at 0000 UTC on August 17. The threshold is 10 mm (3h)⁻¹. The colored area indicates the observation object, and the outline is the forecast object.

Figure 4.  Tracks of time-domain precipitation objects were defined by using a threshold of 10 mm (3h)⁻¹ and a smoothing radius of 36 km for the ECMWF simulation initialized at 0000 UTC on August 17, 2018. (a) observation; (b) forecast.

The forecasted and observed object tracks of the 23 heavy rainfall cases (Table 1) are shown in Fig. 5. Through object tracks, the movement trajectory and life history of precipitation objects are understood. To test the sensitivity of MODE-TD results to different verification parameter configurations, we repeat the tests twice with a threshold of 10 mm (3h)⁻¹ and smoothing radii of 9 km and 36 km. As shown in Figs. 5a and 5b, the total numbers of forecasted and observed objects identified with a radius of 9 km are 87 and 155, respectively. When the convolution radius increases to 36 km, the total numbers of forecasted and observed objects are reduced to 43 and 57, respectively (Figs. 5c–5d). Hence, the total number of tracks will decrease with increasing smoothing radius. The number of precipitation object tracks predicted by the model is generally less than that of observation, especially at a relatively smaller radius.

Figure 5.  Precipitation object tracks of (a, c) forecast and (b, d) observation with the rainfall threshold of 10 mm (3h)⁻¹ and the smoothing radii of 9 km and 36 km.

Figure 6 shows the centroid offsets of paired classified precipitation identified by the MODE-TD test for 23 intense precipitation cases. In this study, precipitation objects with an area larger than 50000 square kilometers are defined as large-area precipitation objects, while those with an area smaller than 5000 square kilometers are defined as small-area precipitation objects. Objects with a velocity faster than 10 km h^–1 are defined as fast-moving objects, while those with a velocity slower than 2 km h^–1 are defined as slow-moving objects. Objects with a lifespan shorter than 6 hours are defined as short-lifespan objects, while those with a lifespan longer than 18 hours are defined as long life-span objects. As shown in Fig. 6a, the forecast centroid offsets of large-area precipitation objects are mostly less than 2 degrees in both longitude and latitude, with larger offsets in longitude than latitude, and the forecast offsets are mainly westward. The forecast centroid offsets of small-area precipitation objects are significantly larger than those of large-area objects (Fig. 6b), with individual samples showing offsets of up to 5 degrees in longitude. As shown in Fig. 6c, the forecast centroid offsets of fast-moving precipitation objects have significant longitudinal offsets compared to latitudinal offsets. The latitudinal offsets are mostly within two degrees, while some samples in the longitude direction exceed two degrees and even four degrees. Similar to the forecast offsets of large-area objects, the overall forecast offsets are also westward. The forecast centroid offsets of slow-moving precipitation objects are significantly larger than those of fast-moving objects (Fig. 6d), mainly manifested by a significant increase in latitudinal deviations. As shown in Fig. 6e, the forecast centroid offsets of precipitation objects with long lifespans are mainly longitudinal deviations, and the overall bias is westward. The forecast centroid offsets of short-lived precipitation objects show a significant increase in latitudinal deviations compared to long-lived objects (Fig. 6f).

Figure 6.  Centroid offsets of the identified objects for (a) large area cases, (b) small area cases, (c) fast-moving speed cases, (d) slow-moving speed cases, (e) long life history cases, and (f) short life history cases with a threshold of 10mm (3h)⁻¹ and a smoothing radius of 9 km.

Figure 7 illustrates the distribution characteristics of the area ratio between different classified precipitation forecast objects and observation objects. The closer the area ratio is to 1, the more similar the forecast and observation are. As shown in Fig. 7a, for large-area precipitation objects, the area ratio mainly falls within the range of [1, 1.5), accounting for approximately 43.3% of the distribution frequency. Secondly, it is located in the interval less than 0.5, and the interval [1, 1.5) has the least distribution. For small-area precipitation objects, the area ratio is mainly distributed in the interval greater than 1.5, accounting for approximately 35.8% of the distribution frequency. The intermediate intervals are less distributed (Fig. 7b). Therefore, the model has a relatively better area forecast for large-area precipitation objects. For rapid-moving precipitation objects (Fig. 7c), the distribution of the model's forecast area ratio is similar to that of large-area precipitation objects, with an area ratio mainly in the range of [1, 1.5) has the least distribution. For small-area prec, accounting for about 32.6%. Next are the intervals with less than 0.5 and those greater than 1.5. The distribution of area ratios for slow-moving precipitation objects is similar to that of small-area precipitation objects, with the main difference being that the area ratios are mainly located in intervals less than 0.5, accounting for about 42.2% (Fig. 7d). As shown in Fig. 7e, the forecast area ratios for long-lifespan precipitation objects by the model also mainly fall within the range of [1, 1.5), accounting for about 36.3%. In contrast, short-lifespan object area ratios are primarily located at intervals greater than 1.5 (Fig. 7f).

Figure 7.  The classification method is the same as that in Fig. 6, but it displays the percentage of forecast and observed target area ratios in different intervals. The bar chart shows the number of samples distributed in different intervals. The curve displays the percentage of interval samples to the total sample size.

Figure 8 shows the lifespan forecast errors of the model for different classified precipitation objects, with the starting time, lifespan, and ending time given in the graph. As shown in Fig. 8a, the model has a relatively accurate lifespan forecast for large-area precipitation objects, with eight object pairs showing that the predicted and observed lifespans are exactly the same. The forecast for small-area precipitation objects differs (Fig. 8b). Among the 18 matched objects, 8 pairs had the same lifespan as predicted and observed objects, accounting for about 44.4%. Among the remaining 10 matched objects, there are 7 pairs with a longer predicted lifespan than observed and 3 pairs with a shorter predicted lifespan than observed. As shown in Fig. 8c, there are a total of 10 pairs of fast-moving precipitation object pairs. Among the 10 pairs of fast-moving precipitation objects, there are 3 pairs with exactly the same lifespan between the predicted and observed objects, accounting for about 30%. Among the remaining 7 pairs, there are 3 pairs where the predicted lifespan is longer than the observed and 4 pairs where the predicted lifespan is shorter than the observed. For slow-moving precipitation objects, there are a total of 17 pairs of matched objects (Fig. 8d). Among them, there are 7 pairs where the predicted lifespan is exactly the same as the observed lifespan, accounting for about 41.1%. Among the remaining 10 pairs, there are 5 pairs where the predicted lifespan is longer than the observed and 5 pairs where the predicted lifespan is shorter than the observed. As shown in Fig. 8e, among the 5 long lifespan precipitation-matched objects, there are 3 pairs where the predicted and observed lifespan are exactly the same, accounting for about 60%. The remaining 2 pairs both show that the predicted lifespan is shorter than the observed. There are a total of 9 paired short-lifespan precipitation objects, among which 4 pairs have exactly the same predicted and observed lifespan, accounting for about 44.4% (Fig. 8f). The remaining 5 pairs both show that the predicted lifespan is longer than the observed.

Figure 8.  The same as Fig. 7, but for the lifetime of observed and forecasted objects.

The MODE-TD calculates several 3D attributes for a single object. Among them, the speed_delta attribute can be used to verify the prediction ability of the model on the moving speed of the precipitation system. The term "speed_delta" refers to the difference in velocity between the predicted and observed precipitation objects. As shown in Fig. 9a, the frequency distribution of forecast speed_delta for large area precipitation objects in the range of [–2.5, 0) and [0, 2.5) is 50%. The forecast speed_delta of the model for small-area precipitation objects is mainly located in the range of [0, 2.5), accounting for approximately 52.2% (Fig. 9b). About 39.1% of the samples are located in the range of [–2.5, 0), and 8.7% are located in the range of [2.5, 5). As shown in Fig. 9c, the forecast speed_delta for fast-moving precipitation objects is mainly located in the range of [–2.5, 0), accounting for approximately 60%. About 40% of the samples are located in the range of [0, 2.5). The slow-moving precipitation objects differ from the fast-moving objects (Fig. 9d). The forecast speed_delta is mainly located in the range of [0, 2.5). There are 5% and 10% of the samples located in the ranges of [–5, –2.5) and [2.5, 5) respectively. As shown in Fig. 9e, the forecast speed_delta for long life precipitation objects is mainly located in the range of [0, 2.5), accounting for approximately 80%. On the other hand, all the forecast speed_delta for short life precipitation objects are located in the range of [0, 2.5) (Fig. 9f).

Figure 9.  The same as Fig. 6, but for speed_delta frequency of matched 3D objects.

The frequency distributions of objects initiating and dissipating at different forecast lead time are shown in Fig. 10 and Fig. 11, respectively. It is necessary to pay attention to the number of generated and dissipated objects, as this is related to the number of objects at a specific forecast time. If the number of generated objects is greater than the number of dissipated objects, it will lead to an increase in the total number of precipitation objects and vice versa. As shown in Fig. 10a, most large-area precipitation objects initially appeared in the first 9 hours, with the highest number appearing in the first 3 hours. The model also better simulated this feature. For small-area precipitation objects (Fig. 10b), the model has a better simulation of the initial appearance of precipitation objects in the first 6 hours. However, there is a large difference in observations after 9 hours. As shown in Fig. 10c, the model has a similar simulation of the initial appearance of fast-moving precipitation objects as in Fig. 10a. The frequency forecast of the initial occurrence of observed precipitation objects in the first 9 hours is relatively accurate. Fig. 10d is similar to Fig. 10b. As depicted in Fig. 10e, with regards to long-life precipitation objects, all observed precipitation objects were initially observed within the first three hours. However, some forecasted precipitation objects were projected to appear at 6 and 9 hours. As illustrated in Fig. 10f, most short-lived precipitation objects initially appear within the first three hours. However, the remaining objects mainly appear at 24 hours. Nevertheless, there is a certain level of discrepancy in forecasting the initial appearance of precipitation objects at 24 hours. Specifically, the model predicts that some objects will initially appear at 21 hours.

Figure 10.  The number of precipitation objects initiated at different forecast lead time. The objects are identified by the precipitation threshold of 10 mm (3h)⁻¹ and the convolution radius of 9 km for each prediction time. The red histograms represent the observation, and the blue histograms represent the corresponding forecast.

Figure 11.  The same as Fig. 10, but for the number of precipitation objects dissipating at different forecast lead time.

As depicted in Fig. 11a, the dissipation time of precipitation objects observed over a large area mostly occurs at 24 hours, with a few instances occurring at 12 hours. The model also simulates this feature well, with most forecasted dissipation times of precipitation objects occurring at 24 hours and individual samples occurring at 6 hours. As illustrated in Fig. 11b, the model has a satisfactory simulation of the dissipation times observed at 6 hours and 15 hours for small-area precipitation objects. However, it tends to underestimate the dissipation times observed at 3 hours and 21 hours and overestimate the dissipation times observed at 9 hours, 12 hours, 18 hours, and 24 hours. The primary dissipation time for fast-moving observed precipitation objects is observed at 21 hours, followed by 9 hours and 18 hours (Fig. 11c). Similarly, the primary dissipation time for forecasted precipitation objects also occurs at 21 hours, with the remaining samples distributed between 3 hours and 12 hours. For slow-moving precipitation objects, the model has a good simulation of the dissipation times observed at 6 hours and 18 hours. However, it tends to overestimate the dissipation times observed at 3 hours, 15 hours, and 24 hours while underestimating the dissipation times observed at 9 hours, 12 hours, and 21 hours (Fig. 11d). As shown in Fig. 11e, the model exhibits excellent consistency in forecasting the dissipation times of long-life precipitation objects compared to observations. For precipitation objects with short lifespans, the primary dissipation times are observed at 3 hours and 24 hours (Fig. 11f). The model effectively simulates the dissipation time observed at 24 hours. However, it tends to underestimate the dissipation time observed at 3 hours.

The spatio-temporal characteristics of heavy precipitation forecasts from ECMWF in eastern China were studied by using the MODE-TD method in this study. A total of 23 heavy rainfall cases during 2018–2021 were investigated. These heavy rainfall cases were mainly caused by low-level jets, the Meiyu front, typhoons, WPSH, and the Jianghuai cyclone. The MODE-TD with smoothing radii of 9 km and 36 km was used to analyze all cases. The main findings are summarized below.

Through the MODE-TD example test of the individual case of Typhoon "Rumbia, " it can be seen that after introducing the time dimension, the MODE-TD method can not only evaluate the spatial metrics such as area and direction of precipitation forecast from the model like the two-dimensional MODE method but also evaluate the overall simulation ability of the model for the life history of the precipitation system.

The results of multiple tests with different parameter configurations indicate that as the smoothing radius increases, the number of identified trajectories using the MODE-TD method decreases. Additionally, the number of precipitation objects trajectories by the model is less than the observed values, particularly at smaller radii.

The results of the centroid offsets verification for different categories of paired precipitation objects and the analysis of the area ratio of paired precipitation objects show that the model performs better in predicting large-area precipitation objects than small-area objects, fast-moving precipitation objects than slow-moving objects, and long-lifespan precipitation objects than short-lifespan objects. The main difference is reflected in the forecast of latitudinal deviation.

The analysis of the model's forecast errors in predicting the lifespan of precipitation objects reveals that the model tends to have more accurate duration predictions for large-area or long-lifespan precipitation events (such as typhoon-related precipitation), while having relatively larger error in predicting the duration of small-area or short-lifespan precipitation events.

Overall, the model tends to exhibit an overestimation of forecasted translational velocity for precipitation objects characterized by small-area, slow movement, or both long and short lifespans. Conversely, the model shows an underestimation of forecasted translational velocity for precipitation objects with fast movement.

The model's simulation results regarding the generation and dissipation of precipitation objects show that it performs relatively well in simulating the generation of large-area or fast-moving precipitation objects. However, there are significant differences in the forecasted generation of small-area or slow-moving precipitation objects after 9 hours. The model also demonstrates better simulation skills in the dissipation of large-area or long-lifespan precipitation objects, while there are notable discrepancies in the forecasted dissipation of small-area or short-lifespan precipitation objects.