Blacklist Design of AMDAR Temperature Data and Their Application in the CMA-GFS

Observations are a main part of the assimilation system of a numerical weather prediction model, which determines the estimated difference between the initial values (analytical fields) of the model and the true state of the atmosphere. The vertical profiles in the atmosphere are of particular operational value. Aircraft data represent one of the most important sources of meteorological observations (Cardinali ^[1]; Langland and Baker ^[2]; Mahajan et al. ^[3]; Liang et al. ^[4]). Aircraft Meteorological Data Relay (AMDAR) is an automatic weather report obtained from the observation instruments onboard civil aircraft, which are characterized by high temporal resolution. The reports are widely used in numerical weather predictions, playing an important role in their accuracy (Rukhovets et al. ^[5]; Pouponneau et al. ^[6]; Cardinali et al. ^[7]; Drüe et al. ^[8]; Benjamin et al. ^[9]; Chen et al. ^[10]). There are many studies on the application of AMDAR data (Schwartz and Benjamin ^[11]; Benjamin et al. ^[12]; Wang et al. ^[13]). Observations based on aircraft data represent nearly 60 percent of the conventional observations in the United States (Petersen ^[14]). In the Rapid Update Cycle (RUC) developed by the National Oceanic and Atmospheric Administration, improvements have been made to the system thanks to the availability of a large amount of space-time aircraft data (Benjamin et al.^[15]; Smirnova et al.^[16]; Moninger et al.^[17]; Benjamin^[18]). AMDAR data are among the most important meteorological data sources along with satellite remote sensing data, radar and radiosonde data for mesoscale and short-term weather forecasts.

In the data assimilation system of numerical prediction, data quality has a great influence on the analysis and forecast accuracy, but many observational data present systematic errors (Harris and Kelly^[19]; Dee^[20]; Bannister^[21]; Farchi et al.^[22]). Aircraft observations are affected by instrumental errors, system errors and processing fault errors. Therefore, in all the observation stages, including the recording stage of airborne instruments, the receiving stage and the transmission stage, there are quality controls including extremes checks, thinning checks, internal consistency checks, time consistency checks, and background checks, that help eliminate some of the incorrect or inconsistent data. Because observations and models have systematic biases, background checks alone do not completely remove inconsistent and poor quality data (Sakov and Sandery ^[23]). The Canadian Meteorological Centre (CMC) runs ongoing statistics on the number of aircraft observations received and lists aircraft with suspect data (Gilles ^[24]). The Japan Meteorological Agency developed a bias correction method to study the impact of aircraft temperature data assimilation in its global analysis and forecast system. When temperature data from an aircraft show biases (Observation-Background) larger than 2.5 K in one-month statistics, data from the aircraft are not used in the following month (Hiroshi ^[25]). The International Operations Center of the National Center for Environmental Prediction (NCEP) monitors the models and observations, counts a month's worth of Observation-Forecast results, and selects a list of aircraft with poor data quality. There is an aircraft data quality control named "reject list" in the Rapid Refresh numerical weather prediction (NWP) system (RAP) of the National Oceanic and Atmospheric Administration (NOAA), wherein aircraft with large departures from the model background are rejected from the data assimilation system (James ^[26]). The European Center for Medium-range Weather Forecasts (ECMWF) statistically monitors global routes to identify the worst ones. However, the analysis of aircraft temperature in the Northern Hemisphere and tropical regions in China' s independent Global/Regional Assimilation Prediction System Global Forecasting System (CMA-GFS) is not very effective and currently lacks an aircraft blacklisting method. It is expected that the assimilation of AMDAR temperature data can improve forecast skill. In this work, we created a blacklist of AMDAR temperature data and evaluated the impact of blacklist methods on NWP by assimilating the blacklist-filtered data in CMA-GFS 4D-Var.

This paper is organized as follows. In Section 1, the utilization of aircraft data in meteorology is introduced. In Section 2, the data, methods, and model are described. In Section 3, the results of AMDAR blacklist data, analysis and forecasts are illustrated. We conclude with a summary and discussion summary in Section 4.

The AMDAR data used in this paper are received by the National Meteorological Information Center through a Global Telecommunication System from the International Exchange Station via China Integrated Meteorological Information Service System (CIMISS)- national integrated meteorological information sharing platform. The amount of data collected from aircraft has been increasing in recent years. By the end of December 2019, the CIMISS database had recorded 1 million daily observations from aircraft. Fig. 1 shows the global distribution of AMDAR data at 12UTC on August 12, 2019.

Figure 1.  Global distribution of AMDAR data at 12UTC on August 12, 2019.

AMDAR data are mainly distributed on the routes to and from the United States, Western Europe, East Asia, Australia, and South Africa. 64% of the data are from routes with length of less than 8000 meters, 21% from routes with length of 8000~10000 meters and 15% from routes with length of 10000 meters or more. Most of the data on these routes are from altitudes above 10, 000 meters (Fig. 1), about 36% of the total data. There are no aircraft data south of 60° S, and the vertical distribution is from the ground to 200 hPa. In terms of observation frequency, the climbing stage is observed every 35 seconds, the cruise stage is observed every 3 minutes, and the descent stage is observed every 60 seconds.

The ERA5 reanalysis made available by the ECMWF is a new reanalysis product at a high resolution replacing ERA-Interim (EI) since September 1, 2019 (Hans et al. ^[27]). The ERA5 has a horizontal resolution of ~0.25° and higher temporal (hourly analysis) output resolution. Besides the higher temporal and spatial resolution compared to EI, the ERA5 has a higher number of vertical levels (137 versus 60 in EI). In this paper, ERA5 temperature reanalysis data are used as the reference field. ERA5 data are interpolated from 0.25 degrees to 1-degree spatial resolution.

The difference between the ERA5 and temperature observations is used to calculate the bias and standard deviation. The statistical period corresponded to August 1-31, 2019. The method of the AMDAR blacklist design used in this paper is based on: (a) NCEP data quality monitoring statistical report (https://www.nco.ncep.noaa.gov/pmb/qap/amdar/). For each aircraft, based on the difference between observed values of the AMDAR temperature data and ERA5 reanalysis, the bias and standard deviation are computed and compared with the given threshold of data quality monitoring. The NCEP quality monitoring evaluation indexes included gross error, bias, standard deviation and root mean square error (RMSE) (Table 1); (b) Pre-quality controls for the AMDAR temperature observations, mainly including internal consistency checks, extreme value checks, time consistency and space consistency checks (Table 2).

The bias is calculated as:

where Bias_i is the average difference between the temperature observations of the aircraft with the ith identification number and ERA5 in the statistical period, O_ij is the jth observation of the aircraft with the i identification number in the statistical period, and B_ij is the interpolation of ERA5 reanalysis in the same time window corresponding to the observation position of the ith identification number. N is the total number of observation records of the i th aircraft during the statistical period. Std_i is the standard deviation of the temperature of the ith aircraft, and RMSE_i is the RMSE between the observation and the reference field.

The standard deviation is calculated as:

The RMSE is calculated as:

When the sample number of observations for each aircraft is greater than the given number, the bias, standard deviation and RMSE are calculated.

AMDAR blacklist rules are formulated as follows:

(1) The ith aircraft will be blacklisted if its STD_i or RMSE_i are greater than the threshold in Table 1, or

(2) When the monthly count of failed inspections of the ith aircraft in any check (indicated by N_S) is greater than 30, it will enter the blacklist.

In this paper, we use the global CMA-GFS numerical weather forecast model independently developed and applied by the Chinese Meteorological Administration (Xue et al. ^[28]). This model has a horizontal resolution of 0.25°, 87 vertical layers, and the top layer reaches 0.1hPa. The assimilation system is the four-dimensional variational assimilation (Zhang et al. ^[29]), and the analysis variables are non-equilibrium dimensionless pressure, flow function, non-equilibrium velocity potential and specific humidity. The assimilation data include sounding, ground report, ship, AMDAR, and NOAA15, 18, 19, Metop-A, B and Fengyun series 3 and 4 (FY3, 4). Two groups of experiments are designed to research the influence of blacklisted data on the analysis and forecast. Experiment 1 (Exp1) is used as the control experiment to assimilate all the data described above including all the AMDAR data, and Experiment 2 (Exp2) is used as the influence experiment to assimilate the above data and the AMDAR data after blacklist screening. The experiment period is from September 1 to September 30, 2019. The forecast period is 00UTC per day, with a 10-day forecast. The Globe is divided into four study areas: the Northern Hemisphere (20-90 degrees north), the Southern Hemisphere (20-90 degrees south), the tropics (20 degrees south to 20 degrees north), and East Asia (70-145 degrees east and 15-65 degrees north).

The statistical sample is all the aircraft in the world in August 2019 and includes 8, 316 aircraft with different identification numbers, of which 1, 080 aircraft are on the blacklist of temperature variables after applying the above criteria. The Globe is divided into grids of 20° × 20°, and the number of blacklist data in each grid is counted. Fig. 2 shows the distribution of the global blacklist data in one month. As can be seen from the figure, the blacklist information is more widely distributed in North America (especially the United States), Western Europe, western South America and East Asia. The largest grid value in a month is about 700, 000. The routes in these areas are dense and stable, suggesting that some types of aircraft are on the blacklist. According the above criteria, Fig. 3 shows the horizontal distribution of the temperature bias from the blacklist. The temperature bias of the blacklist is significantly greater than that after the removal of the blacklist, and the impact of the bias of the blacklist is greater in places with larger data volumes (the United States, Western Europe and East Asia). In these regions, the temperature bias can be reduced by up to 0.5k. Some observations with large bias are eliminated. The global horizontal distribution of the temperature standard deviation is shown in Fig. 4. Overall, the standard deviation of temperature after removal of the blacklisted data is smaller than that of the blacklisted data, and there are fewer large values. In general, the type and flight state of aircraft on a usual route are stable. This suggests that many of aircraft on these routes have biases. The bias is mainly related to the type of aircraft and flight state (Ballish and Kumar ^[30]).

Figure 2.  Global number distribution of blacklisted AMDAR data in August 2019.

Figure 3.  Horizontal distribution of AMDAR temperature bias at the 250 hPa in August 2019. (a) Temperature bias distribution in the AMDAR blacklist. (b) Temperature bias distribution after removing the AMDAR blacklist.

Figure 4.  Horizontal distribution of the temperature standard deviation at the 250 hPa in August 2019. (a) Temperature standard deviation distribution in the blacklisted AMDAR data. (b) Temperature standard deviation distribution after removing the blacklisted AMDAR data.

The vertical bias and standard deviation of the temperature data in the blacklist and after the removal of the blacklist are shown in Fig. 5. ERA5 reanalysis data is used here as reference data. Fig. 5 shows that the aircraft temperature data are usually warmer than the ERA5 reanalysis data above 800 hPa, especially the data on the blacklist. In August 2019, the temperature bias after the removal of the blacklist is smaller than that of the blacklist. The aircraft temperature data below 900 hPa has a negative bias compared with the ERA5 reanalysis. The temperature data in the blacklist has a positive bias above 900 hPa. The standard deviation of the temperature data after removing the blacklist is also smaller than the standard deviation in the blacklist, and the bottom layer is at most 1.5 K smaller. The blacklist data in each layer accounts for about 25-30% of the layer of data.

Figure 5.  The vertical bias and standard deviation of blacklisting data (dashed lines) and after removing the blacklisting (solid lines), and the blacklist rate; (a) bias, and (b) standard deviation.

Figure 6 shows the scatter diagram of temperature observations and ERA5 reanalysis before and after blacklist removal. In the temperature observations screened for blacklisted data, most of the errors and outliers are filtered out, and the observed values are close to ERA5 reanalysis data.

Figure 6.  Scatterplot of temperature observations and ERA5 reanalysis for August 2019. The x-axis shows the observations, and the y-axis is ERA5 reanalysis. (a) Without blacklisting and quality control screening. (b) With blacklisting and quality control screening.

Figure 7 shows the probability density distribution of the difference between AMDAR temperature observations and ERA5 data (Obs-ERA5). The probability distribution of the blacklisted data has a positive bias and a large standard deviation, indicating that the data has a large dispersion and bias compared with ERA5 on the whole. The mean value of the OERA5 after removing the blacklisted data is around zero, and the standard deviation is smaller than using all the AMDAR temperature data, which conforms to the assumption of the unbiased normal distribution of the observation error in the assimilation system.

Figure 7.  Probability density distribution of the difference between AMDAR temperature observations and ERA5 reanalysis data. The green line is the probability distribution of all data, red line is the probability distribution of blacklisted data, and blue line is the probability distribution after removing the blacklisted data.

As shown in Fig. 8, compared with assimilating all the AMDAR temperature data, the bias and RMSE of the geopotential height analysis field with assimilating removing the blacklist data are both reduced in the region with the most aircraft data in the Northern Hemisphere. In the 200 hPa cruise phase with the most aircraft data, the bias is reduced by about 1.5 gpm at most, and the RMSE is reduced by 1 gpm at most. In East Asia and tropical regions, the bias and RMSE of the potential height analysis field of Exp2 decrease, and the RMSE decreases by about 1.5 gpm at 200 hPa in East Asia and tropical regions. In the Southern Hemisphere, where there are fewer data collected, blacklist screening causes a further decrease of aircraft data, which may have led to a greater bias of the results. However, the removal of data outliers through blacklist screening also results in a smaller RMSE. As shown in Fig. 9, compared with assimilating all the AMDAR temperature data, the bias and RMSE of the temperature analysis field with assimilating removing the blacklist data are both reduced, and the RMSE of the temperature analysis field without the blacklist data decreases by a maximum of ~ 0.15 K at 250 hPa. Fig. 10 displays the differences of temperature analysis with the ERA5 reanalysis at 250 hPa during the period from September 1 to September 30, 2019 for Experiment 1 and Experiment 2. Because the aircraft temperature data are usually too warm at 250 hPa, the application of blacklist methods should result in a cooler analysis in many areas. In those regions where there is a lot of blacklist data, after removing the blacklist data, the temperature analysis of Experiment 2 is cooler than that of Experiment 1. For instance, in the United States, East Asia and Western Europe, where blacklist data are mainly distributed, the maximum improvement in temperature analysis is about 0.2k.

Figure 8.  Bias and root mean square error (RMSE) of the analysis field of geopotential height. Red lines represent the analysis field after removing the blacklist data, and blue lines represent the analysis field without removing the blacklist data. (a) Northern Hemisphere, (b) Southern Hemisphere, (c) tropics, and (d) East Asia. Units: gpm.

Figure 9.  Bias and root mean square error (RMSE) of the temperature analysis field. Red lines represent the analysis field after removing the blacklist data, and blue lines represent the analysis field without removing the blacklist data. (a) Northern Hemisphere, (b) Southern Hemisphere, (c) tropics, and (d) East Asia. Units: K.

Figure 10.  The differences of temperature analysis with the ERA5 reanalysis at 250 hPa during the period from September 1 to September 30, 2019 for Experiment 1 (left) and Experiment 2 (right). Units: K.

Figure 11 shows the forecast comparison score-card between the influence experiment and the control experiment (Zhao et al. ^[31]). This card includes the anomaly correlation coefficient and RMSE with the ERA5 reference field of geopotential height, temperature and wind field. The comparative areas include East Asia, Northern Hemisphere, Southern Hemisphere, and Tropical region. The big red triangle indicates a greater forecast improvement in the influence experiment, and the small red triangle indicates a slight forecast improvement. Grey indicates no change and green indicates a decrease in forecast performance compared with the control experiment. The forecast has a more noticeable improvement in East Asia and the Southern Hemisphere. The RMSE of the impact experiment is smaller than that of the control experiment. The RMSE on the whole is neutral, suggesting that the aircraft temperature data has significant effects on short-term forecasts. The RMSE of the tropics is significantly decreased; the wind field improvement is even more apparent except for the geopotential height at 250 hPa. This may be related to the complexity of the circulation patterns in the tropical areas, or it may be strictly related to the setting of the blacklist, which may lead to the loss of key information of weather data due to the large amount of data removed. In the East Asian region, there is some decrease in the prediction score after 5 days compared with the control experiment. It is possible that the East Asian region is sensitive to aircraft data, and the variation of the observed data affects the forecast results.

Figure 11.  Forecasting field score-card for control experiment (Exp1) and influence experiment (Exp2). The sign of large red triangle, small red triangle, red square, and brown square represents far better improvement, better improvement, better but not significant improvement and no change, respectively. The sign of large green triangle, small green triangle and green square represents far worse change, worse change and worse but not significant change, respectively. NH: North Hemisphere; SH: South Hemisphere; EASI: East Asia; TRO: Tropical region.

In this paper, the ERA5 reanalysis data are used as the reference field, and the bias and standard deviation of AMDAR temperature data for August 2019 are calculated before and after blacklisting. The original data and the data after blacklist removal are input into the CMA-GFS assimilation system, and a 4D-VAR variational assimilation experiment is carried out for one month. The following main conclusions are drawn from the experiments:

(1) The aircraft blacklists of temperature data are mainly distributed in the United States, Western Europe, western South America and East Asia;

(2) The bias of the temperature data of the blacklist is about 0.5K, and the standard deviation is up to 1K larger than the temperature of the non-blacklist;

(3) The blacklisted aircraft temperature data account for about 25-30% of the total data of each layer;

(4) The error distribution of temperature data after deducting the blacklist is closer to the unbiased normal distribution; blacklisting removed virtually all outliers;

(5) Compared with the analysis assimilating all the AMDAR temperature data, the RMSE of the temperature analysis field without the blacklist data decreased by a maximum of ~ 0.15 K at 250 hPa, and the RMSE of geopotential height analysis field decreased by a maximum of ~ 1.5 gpm at 200 hPa. The forecasting effect of the influence experiment is improved overall, especially in the tropical and northern regions.

The bias of aircraft temperatures is mainly caused by differences among observation instruments on various aircraft types and flight conditions. Overall aircraft temperature data are found to be about 0.2 K higher than the ERA5 reanalysis data, and the aircraft data are mainly distributed in North America (especially in the United States), East Asia, Western Europe and Australia. The distribution of blacklist observations is mainly concentrated in regions with larger amounts of aircraft observations, especially in the United States. We found that the bias of blacklists did not satisfy the condition of unbiased normal distribution of variational assimilation system. The bias of the aircraft temperature data screened by the blacklist is smaller than that of the blacklist data. Statistics on the blacklists should be updated at least once a quarter to take into account global aircraft replacement and calibration of the aircraft instruments. In addition, the criteria for blacklisting should be adjusted constantly. Extremely strict criteria will lead to the loss of information, especially for the observation of some small-scale extreme weather phenomena, and loose criteria will lead to erroneous data entering the assimilation system, thus affecting the analysis and forecast. The analysis of this paper is limited to the formulation and application of blacklists for AMDAR temperature. The formulation and application of blacklists for wind field observations will be carried out in future work.