An Innovative Bias-Correction Approach to CMA-GD Hourly Quantitative Precipitation Forecasts

Numerical weather prediction (NWP) is not always accurate in rainfall prediction (Bauer et al.^[1]; Leutbecher et al.^[2]; Dias et al.^[3]; Xue and Liu^[4]; Subramanian et al.^[5]; Zhang et al.^[6]). One effective method of improving the accuracy of NWP is to correct the deviation of the forecast from the observation based on statistical corrective methods (Bentzien and Friederichs^[7]; Volosciuk et al.^[8]; Mendez et al.^[9]). For example, Zhu and Luo^[10] demonstrated that the frequency-matching (FM) method can significantly reduce the systematic bias of precipitation intensity. Hamill et al.^[11] pointed out that an analog post-processing method based on historically similar weather system can effectively correct the spatial distribution of precipitation. In addition, meteorologists also pay attention to extracting effective information from ensemble forecasts or multi-model forecasts and to generating deterministic quantitative precipitation forecasts (QPF) with higher accuracy (Woodcock and Engel^[12]; Tartaglione et al.^[13]; Ebert et al.^[14]; Kumar et al.^[15]; Sukovich et al.^[16]). For example, Dai et al.^[17] contended that the optimal percentage method could integrate effective information with different percentiles, while the probability-matching (PM) method could acquire a better spatial distribution field from multi-models (Ebert^[18]), especially the weighted PM (WPM) method, which could automatically give a very low weight to the less skillful model (Liu et al.^[19]).

Another approach to improving QPF is to develop better weather predicting models by interpreting high-resolution NWP outputs (Gagne et al. ^[20]; Rasp and Lerch ^[21]; Herman and Schumacher ^[22]; Zhou et al. ^[23]; Chattopadhyay et al. ^[24]; Xia et al. ^[25]; Han et al. ^[26]). More recently, deep learning (DL) algorithms have been explosively applied in weather forecasts and have proven successful in difficult rainfall classification tasks (Nagaselvi and Deepa ^[27]; Reichstein et al. ^[28]; Chen et al. ^[29]). For instance, Yuan et al. ^[30] pointed out that the artificial neural network (ANN) showed good performance in adjusting for high-resolution probabilistic QPF outputs. Scher and Messori ^[31] suggested that the convolutional neural network (CNN) could be useful for quantifying the uncertainty of NWP. Zhang et al. ^[32] proposed a prediction model combining k-means clustering and CNN to improve short-term rainfall forecasting.

However, due to the highly nonlinear and stochastic nature of the temporal-spatial distribution of precipitation, the above methods have some limitations. Statistical corrective methods cannot overcome major model errors (Maraun et al. ^[33]). Although the DL method has limited modeling capacity for small-sample events (e. g., short-term rainstorms), it can use big data relationships to create a better classification forecast model for big-sample events (e. g., weak rain). As a result, problems in integrating different bias-corrected technologies have several unique challenges, requiring methodologies converting precipitation classifications to QPF and combining the benefits of various methods.

This paper introduces a simple but powerful approach for overcoming these existing challenges based on CMA-GD model from the Guangzhou Institute of Tropical and Marine Meteorology of China Meteorological Administration (CMA). CMA officially approved the CMA-GD model for service operation in 2011 (Zhong and Chen. ^[34]; Zhong et al. ^[35]). The rest of this article is organized as follows. Section 2 provides an overview of data and methodology. Section 3 discusses the results of the new approach. Section 4 gives case studies of successes and failures. Section 5 provides concluding remarks by briefly discussing the effectiveness and limitations of this approach.

The high-resolution (3 km, hourly) products of CMA-GD model (Fig. 1) were used in this paper from January 2018 to March 2021, with the initial times at 00:00 and 12:00 UTC and a valid forecast period of 0-24 h. The grid data were the nearest neighbor interpolated to 420 stations in Hunan Province (24-31° N, 108-115 ° E) in order to be consistent with observations. Moreover, the observational data was provided by the China Integrated Meteorological Information Sharing System (CIMISS) (Sun et al. ^[36]).

Figure 1.  Domain of the CMA-GD model in Lambert map projection. The red line represents the location of Hunan Province.

Data division: All samples were divided into a training dataset, verification dataset, and test dataset (Table 1), with the verification set used to determine the integration solution and the test set used as an independent sample for forecasting. According to CMA's service requirements for short-time precipitation (T/CMSA 0013-2019), the hourly rainfall was divided into six grades at thresholds of 0.1, 2, 4, 8, and 20 mm h^-1. Here a clear day occurred when the per-hour rainfall was less than 0.1 mm h^-1, since 0.1 mm is the smallest detectable amount of rain gauge in China. And rainstorms occurred when the per-hour rainfall exceeded 8 mm h^-1. Since the number of smallscale samples (> 4 mm) in the training set was greatly imbalance from that of the large-scale samples, the small-scale samples were under-sampled at a ratio of 1: 10, and the number of samples at each scale is ten times that of the severe rainstorm samples to obtain balanced samples.

Three techniques were used in this step to build the QPF correction models (Table 2), including deep neural network (DNN), frequency-matching (FM) and optimal threat score (OTS). Among them, the DNN technique aims to reduce the location and magnitude biases of rainfall. While FM and OTS can only eliminate the deviation in magnitude between QPF and the observation, they are unable to correct for pattern biases (Zhu and Luo ^[10]; Wu et al. ^[37]; Wu and Chen^[38]). The thresholds of FM and OTS are 0.1, 2, 4, 8, and 20 mm.

For DNN, the training set was used to obtain the classification of precipitation grades consistent with Table 1, and verification was performed with the verification set. Throughout the whole training process, DNN maintained the model with highest accuracy of classification in the verification set.

Hyperparameters: The Adam optimizer (Kingma and Ba ^[39]), cross-entropy loss function, and StepLR scheduler were used. The learning rate was set at 0.001 initially and was updated with the optimizer and scheduler. Batch normalization was used (Ioffe and Szegedy ^[40]). The number of epochs was set at 200. All of the stations were used here for unified modeling.

Selection of variables: From different levels and types of physical variables in CMA-GD, 51 physical variables (Table 3) related to precipitation and severe convection were selected. All variables were normalized to eliminate the dimensional difference between different variables before training.

Mapping: To conduct a quantitative comparison with FM and OTS technologies, the DNN classification results should be transformed into QPF. The linearly mapping method from forecast probability to QPF was used here (Fig. 2). The rainfall is calculated as follows:

Figure 2.  Illustration of the mapping from forecast probability to QPF.

where x and y are the forecast probability and QPF for different precipitation grades, respectively; OBS_k is the kth precipitation threshold, which is consistent with the thresholds of FM and OTS (0.1, 2, 4, 8, and 20); and P is the forecast probability set of different magnitudes for the calculated training dataset and validation dataset.

The optimal integration (OPI) method acquires the optimal forecast by integrating the best members in weak and severe precipitation forecasts. The steps are shown in Fig. 3. Firstly, a study area was selected. Among all the multi-method calibrated members, members A and B were assumed to have the optimal performance for weak precipitation and severe precipitation, respectively. Next, all stations in the study area were divided into two groups. The stations where members A or B showed severe precipitation were classified into G1, and the remaining stations were classified into G2. For stations in G1, the OPI model integrated the values of member B. For stations in G2, the OPI integrated the values of member A to obtain the integrated QPF. Finally, a new rule was introduced to avoid the problem of continuous precipitation under long-term statistical conditions. That is, for all severe precipitation stations in the integrated QPF, whether their precipitation grades in members A and B spanned more than two orders was checked. Stations with more than two orders were replaced with the average of members A and B.

Figure 3.  Schematic diagrams of the principles of the OPI method. Assuming that members A and B have the best weak rainfall and severe rainfall among all the multi-method calibrated members, respectively. The black and red circles represent the severe rainfall stations of A and B, respectively, and the green part represents stations with weak rainfall. M_A and M_B indicate the precipitation magnitude of stations in A and B, respectively.

Then, a comparative experiment using a shorter sliding training window was conducted (Table 4). Specifically, the OPI experiment used the optimal severe rain member in the whole validation period as member B; the OPI-sliding experiment uses the sliding optimal rainstorm (≥8 mm) member in training window.

Two assessment indicators including threat score (TS) and clear-rainy TS were used to assess the QPF of different bias-corrected methods. Root-mean-square error (RMSE) and mean absolute error (MAE) were used to verify the performance of DNN. Table 5 shows the calculation formulas.

Figure 4a shows the classification accuracy of the DNN model on the training and validation datasets. On the training dataset, the accuracy increases steadily to relative equilibrium with the increase in training epochs. In the validation set, the model has the best effect in about the 30-50th epochs, with an accuracy of more than 76.5%, and the highest point appears in the 44th epoch, which is 77.2%. It is noted that the accuracy of the training set is lower than that of the verification set, which may be related to their sample balance and imbalance (Table 1).

Figure 4.  (a) Classification accuracy on the training and validation datasets. (b) Significance test of RMSE and MAE in each initial time in the verification datasets between CMA-GD and DNN.

Figure 4b compares the RMSE and MAE in each initial time between CMA-GD and DNN. DNN achieves almost the same performance as CMA-GD, with their difference failing to pass the Wilcoxon rank sum test with a confidence level of 95%, which also demonstrates that the DNN follows the basic physical laws of CMA-GD.

Comparing the results of the three techniques (DNN, FM and OTS) in the validation set revealed that DNN has an optimal performance for clear-rainy and light rain forecasts, but OTS has the best performances for the other magnitudes of precipitation (Fig. 5). Specifically, in clear-rainy forecasts, DNN (0.869) and FM (0.868) show the highest scores, with no noticeable difference. In light rain forecasts, DNN has the highest TS score of 0.374, followed by OTS (0.358) and FM (0.338). From moderate rain to rainstorms, OTS improved TS by 0.6%, 5.6%, and 1.4% compared with CMA-GD.

Figure 5.  Threat score of multi-method hourly QPF. Details of the method are shown in Table 2 and 4.

As a result, DNN and OTS are used as the overall optimal members of weak rain and severe rain in the sensitivity experiments (Table 4). Heavy rain (4 mm) is regarded as the 'weak' and 'severe' dividing line here in OPI (Fig. 2). According to the experimental results, both OPI and OPI-sliding show significant improvements over CMA-GD in weak forecasts. But in severe forecasts, only OPI can enhance the original forecast. Furthermore, OPI realizes improvements of 2.5%, 5.4%, 7.8%, 8.3%, and 6.1% compared with CMA-GD for each magnitude of precipitation and of 2.1%, 5.3%, 7.1%, 2.5%, and 4.7% compared with OTS.

Figure 6 displays the performance of the OPI method in the 0-24 h forecast periods. Compared with CMA-GD, OPI improves the forecast periods by 100%, 100%, 91.7%, 87.5% and 75% for predictions from clear-rainy to rainstorms, respectively.

Figure 6.  Comparison of hourly TS between CMA-GD and OPI in a lead time of 0-24 h on the validation datasets.

In conclusion, OPI can improve QPF for all magnitudes of precipitation simultaneously, and the improvement works at more than 75% of the 0-24 h lead time.

To verify the stability of OPI method, it is applied and evaluated on the test set. Overall, three methods can simultaneously improve QPF of CMA-GD in all magnitude: OTS, OPI, and OPI-sliding (Fig. 7a). Additionally, OPI has the best performance, with increases of 3.3%, 4.8%, 6.6%, 6.3%, and 9.9 % for prediction from clear-rainy to rainstorms. From the perspective of the 0-24 h lead time, the results in the test set are consistent with the results on the validation set, improving the forecast periods by 100%, 95.8%, 83.3%, 79.2%, and 75% for predictions from clearrainy to rainstorms (Fig. 7b). This indicates that OPI has good stability in different datasets.

Figure 7.  Percentage of TS improvement by different methods on test datasets. (a) Bar chart of overall TS and (b) line chart of hourly TS. Details of the method are shown in Table 2 and 4.

Advantages and disadvantages of the new approach are discussed here using four cases (Fig. 8).

Figure 8.  Overall TS (left panels) and 12-hour precipitation distribution of observation, CMA-GD, and OPI (right panels). Initial forecasting time was 00: 00 UTC May 29, 00: 00 UTC July 9, 12: 00 UTC July 29, and 12: 00 UTC September 9 in 2020, respectively.

Advantages: For weak precipitation, CMA-GD shows high false alarms while the OPI method can effectively eliminate false alarms and improve the clear-rainy forecast (Fig. 8c-h). For severe precipitation, OPI can decrease the number of missing rainstorms compared with CMA-GD (Fig. 8b) and the number of false alarms (Fig. 8d, f). For instance, in Case Ⅲ (Fig. 8f), redundant precipitation of CMA-GD in the northwest of Hunan Province are eliminated by OPI, and the false alarms for heavy rainstorms in the southeast are weakened (≥100 mm). As a result, OPI shows closer rainfall patterns and intensities to those of the observations.

Disadvantages: It is noted that the rainstorm pattern of OPI relies heavily on the original forecast. For example, in Case Ⅳ (Fig. 8g-h), CMA-GD mistakenly predicted the rainstorm in the north as being in the south, and OPI cannot correct this deviation in location although the TS increased.

Based on the high-resolution CMA-GD products (3km, hourly) from 2018 to 2021, this paper proposes a simple and powerful approach to improve the QPF of a single model. The following is a summary of the findings:

(1) The QPF of DNN follows the basic laws of CMA-GD. DNN and CMA-GD show almost the same performances as their difference failed to pass the significance test. In addition, DNN has the optimal performance compared with the other two techniques (FM and OTS) in clear-rainy and light rain forecasts, but OTS has the best performances for the other magnitudes of precipitation. As a result, DNN and OTS are used as the overall optimal members of weak rain and severe rain in the sensitivity experiments.

(2) According to the experiment results, OPI with the overall optimum member is better than OPI-sliding with the sliding optimal member in the verification set. Although both OTS and OPI can improve QPF from clear-rainy to rainstorms simultaneously, OPI is better than OTS. Furthermore, OPI realizes improvements of 2.5%, 5.4%, 7.8%, 8.3%, and 6.1% compared with CMA-GD for each magnitude of precipitation and the improvement works at more than 75% of the 0-24 h forecast period for all magnitudes.

(3) In test dataset, OPI shows good stability. Additionally, OPI still has the best performance for all magnitudes among all techniques, with increases of 3.3%, 4.8%, 6.6%, 6.3%, and 9.9% compared with CMA-GD from clear-rainy to rainstorms. The results are consistent with those in the validation set. In the case study, the new approach can effectively eliminate false alarms of weak precipitation and decrease the number of missing rainstorms and the false alarm.

According to the findings above, the new approach achieves state-of-the-art performances on a single model for all magnitudes of precipitation. In this study, the OPI method based on DNN and OTS can effectively improve the QPF of CMA-GD. DNN can create a predictive model by mining the relationships between physical variables to improve the QPF. However, such an improvement is ineffective for severe precipitation. OTS can help improve severe weather forecasting, and OPI combines their advantages. However, it is mentioned that the rainstorm pattern of OPI relies heavily on the original forecast and that OPI cannot correct for deviations in the location. More research into the severe precipitation issue will be conducted in the future.