Monitoring Sea Fog over the Yellow Sea and Bohai Bay Based on Deep Convolutional Neural Network

Sea fog is a type of condensation phenomenon that occurs in the lower atmosphere over coastal areas, islands, or sea surface, where a large number of water droplets and ice crystals suspended in the atmospheric boundary layer make the atmospheric horizontal visibility less than 1 km (Gultepe and Milbrandt ^[1]; Zhang et al. ^[2]; Lee et al. ^[3]; He et al. ^[4]). Notably, sea fog poses a significant threat, often leading to numerous accidents at sea and in coastal regions (Xian et al. ^[5]; Gao and Jiang ^[6]; Dorman et al. ^[7]; Fu and Guo ^[8]; Du et al. ^[9]). The observations from shore-based stations often fail to reflect the condition of sea fog at long distances, making the detection of this phenomenon a crucial yet challenging aspect in enhancing the overall capabilities of sea fog forecasting (Heo et al. ^[10]; Wang et al. ^[11]; Miao et al. ^[12]; Guijo-Rubio et al. ^[13]; Park et al. ^[14]).

The experimental land and sea observation network, established by the United States Naval Air Systems Command during 1972–1982, incorporated radiosondes, ships, aircraft, balloons, and kites for detection (Leipper ^[15]). However, ground-based observations do not cover the entire sea area, and more resource-intensive space-based observations are needed. Ernst ^[16] pointed out that observations from the geostationary satellite GMS-1 can be used for cloud and fog observation. Based on improved spatial and temporal coherence methods, Ahn et al. ^[17] used the GMS-5 satellite observations with a high temporal resolution to develop a synthetic map of clear-sky radiation. They proposed an algorithm for monitoring sea fog using the infrared channel data from the GMS-5 satellite by comparing them with hourly infrared radiation observations. Eyre et al. ^[18] and Turner et al. ^[19] explored the dual channel difference method based on the difference in brightness temperature at wavelengths of 10.8 and 3.7 μm to delineate fog or stratus in terms of optical properties, and this method was later widely used (Derrien et al. ^[20]; Luis et al. ^[21]; Wang et al. ^[22]; Xie et al. ^[23]; Yang et al. ^[24]). In contrast to most passive-microwave observations, Wu et al. ^[25] used active cloud-aerosol lidar with orthogonal polarization observations to identify sea fog and evaluated the results by comparing them with the data from a moderate-resolution imaging spectroradiometer. Moreover, Li et al. ^[26] successfully extracted sea fog information from the multi-functional transport satellite-1R (MTSAT-1R) infrared channels using principal component analysis, texture analysis, and threshold detection (Jun et al. ^[27]). Sea fog detection using the optical properties of individual pixels cannot consider neighboring pixels and is prone to misidentify clouds as sea fog. Harun-Al-Rashid and Yang ^[28] estimated and predicted sea fog motions using the sea fog region of interest from the images of the geostationary ocean color imager and simulated wind data based on weather studies and forecasts. Overall, with the advantages of high temporal resolution and wide coverage, geostationary satellite observations can achieve continuous dynamic observations and become an effective means to monitor sea fog (Yang et al. ^[29]; Bai et al. ^[30]; Yi et al. ^[31]; Sim et al. ^[32]; Xu et al. ^[33]; Zhu et al. ^[34]; Li et al. ^[35]; Jeon et al. ^[36]).

Recently, deep learning methods, especially semantic segmentation methods, have been widely applied to remote sensing inversion and have achieved remarkable performance. For example, Huang et al. ^[37] proposed a two-stage superpixel-based fully convolutional network (SFCNet) to achieve sea fog detection by learning the dependencies between pixels. Furthermore, Zhu et al. ^[38] attempted to propose a weakly supervised semantic segmentation method to deal with sea fog recognition using both satellite images and observational data.

The remainder of this paper is organized as follows. Section 2 introduces the data and methods used in this research. Section 3 presents the AI-based sea fog monitoring model. Section 4 verifies model performance. Finally, the discussion and main conclusions are shown in Section 5.

In this research, we focused on sea fog over the Bohai Sea and the Yellow Sea, regions prone to frequent occurrences of such weather events. The data used in this study were primarily the Himawari-8 half-hourly standard data. For satellite observation, there were 16 channels, covering the region of 32°–42°N, 117°–127°E, as shown in Table 1.

This dataset contained two parts, i.e., images and labels. Each image measured 1024×1024 pixels and incorporated information from 16 distinct channels. To ensure consistency, multi-channel data were processed using linear interpolation, achieving a uniform resolution of 0.5 km across all channels with varying spatial resolutions. Color images can be generated by combining three channels. Similar to Xu et al. ^[33] and Huang et al. ^[37], we utilized the XGBoost algorithm to rank the correlation features of 16 channels related to sea fog, and selected the top three channels, namely channels 3, 4, and 14, to compose a false color image of daytime sea fog detection. Similarly, for nighttime sea fog, the algorithm selected channels 7, 13, and 15 to form a corresponding false color image.

Regarding the labels, each label shares the same dimensions as the corresponding images, measuring 1024×1024 pixels. The sea fog areas were distinctly labeled as white, while all other regions were designated as black. Two metrics were used to evaluate the results, i.e., the mean intersection over union (mIOU) and the observation test. When using the metric mIOU, we randomly divided 201 daytime images with pixel-level manual labels from 2017 and 2018 into 151 labeled training images and 50 labeled test images.

When using the other evaluation metric, we employed a satellite training dataset comprising 201 manually labeled pixel-level daytime satellite images for training, spanning the years 2017 and 2018. Additionally, we incorporated test data consisting of 22 daytime images per day, captured from March 11 to July 20, 2019, across a 132-day period. All images were acquired during the time frame of 07:00 to 17:30 Beijing Time (BJT), with only minor gaps in data, summing up to a total of 2756 images.

We selected the observations from 37 stations in the Yellow Sea and Bohai Sea regions, including island and buoy data. The data from sea stations were selected to be able to intersect with the satellite test data in time and space, with a time interval of three hours and four sets per day (08:00, 11:00, 14:00, and 17:00 BJT). The observations from three stations in Qingdao were complete, and their time interval was one hour. In the following paragraphs, the 40 stations in the Yellow Sea, the Bohai Sea, and Qingdao are collectively referred to as sea stations, and in total, there were 10, 559 sets of observational data, which were not used for modeling but only to verify the satellite image results.

Figure 1 is the schematic diagram of the model. Specifically, the first step was to acquire continuous geostationary satellite images (GSI) as source data. The second step was to arrange the above source data in chronological order to obtain temporal image data, and in this step, cloud and fog regions or cloud-fog mixture regions of each image in the source data were extracted by using color-based image segmentation method to obtain an initial mask corresponding to each image. In the third step, the Farneback optical flow method was adopted to generate an optical flow map of the temporal image data. In the fourth step, the average optical flow size of each pixel of the cloud area and fog area was calculated based on the optical flow map and the initial mask, and the cloud-fog classification threshold was determined based on the average optical flow size of the cloud and fog areas and the GSI data. The fifth step was to perform cloud-fog classification at each pixel point of the GSI based on the threshold to obtain automatic identification results of clouds and fog.

Figure 1.  Schematic diagram of the model.

Figure 2 is the schematic diagram of the deep neural network. In this neural network, the 16-channel data from the Himawari-8 satellite was first evaluated by using the extreme gradient boosting algorithm for sea fog detection. The results indicate that the top three most influential channels were, in descending order of significance, channels 3, 4, and 14. The images in these three channels were fused into a false color image. Then, the superpixel segmentation of the false color image was conducted based on the simple linear iterative clustering superpixel algorithm. According to superpixel blocks, manual sea-fog annotation was performed to obtain fine-grained annotation labels. The deep convolutional neural network D-LinkNet was built on the ResNet backbone, and in its central part, dilated convolutional layers with direct connections were added to form a string-and-combine structure with five branches having different depths and receptive fields.

Figure 2.  Schematic diagram of the deep neural network.

Accordingly, we also proposed an automatic cloud-fog recognition system based on GSI sequences. This system included an acquisition module for acquiring continuous-time source data of GSI. Also, the automatic cloud-fog recognition system included a data pre-processing module, which was used to arrange the GSI source data in chronological order to obtain temporal image data, and for color-based image segmentation to extract the cloud and fog areas from each image of geostationary satellite source data, in order to obtain the initial mask corresponding to each image. Moreover, this system included an optical flow algorithm module for generating an optical flow map of the temporal image data based on the Farneback optical flow method, as well as a threshold calculation module. The threshold calculation module was used to calculate the average optical flow of each pixel point of the cloud and fog regions based on the optical flow map and the initial mask to determine the cloud-fog classification threshold. Additionally, a classification module was also included in this system, which was used to classify clouds and fog according to the above cloud-fog classification threshold in each connected domain of the initial mask of GSI source data. In this way, the automatic cloud recognition results can be obtained.

The mIOU and the observation test were selected as the evaluation metrics of the semantic segmentation results.

The mIOU is a standard metric for measuring semantic segmentation, which can be described as the ratio of the intersection of two sets to the union. In the semantic segmentation, these two sets are the actual values and the predicted values. The mIOU can be expressed as Eq. 1:

where TP indicates the true and positive values, FN the false and negative values, and FP the false and positive values. The mIOU is the result of summing and then averaging the intersection over union (IOUs) for each set, where the IOU indicates the ratio of the agreement between the predicted values and the actual values. The mIOU can be rewritten as Eq. 2:

where i represents the true values, j the predicted values, and pij the prediction of i into j.

The observation test, which compares the obtained satellite test data with the observations of sea stations, uses the accuracy rate of fog area, the recognition rate of fog area and the detection accuracy rate as evaluation metrics, as expressed in Eqs. 3–5:

where mf indicates fog data observed by the offshore observatory, sf indicates fog data detected by our method, mn indicates no-fog data observed by the offshore observatory, sn indicates no-fog data detected by our method, cc indicates dense impermeable medium-high cloud cover, ntotal is the number of all observed samples, Pa indicates the accuracy rate of fog area, Pr indicates the recognition rate of fog area and Da indicates the detection accuracy rate. The recognition rate of fog area does not have statistical cloud coverage because it is based on images and is difficult to recognize.

A sea fog recognition model based on the D-LinkNet deep convolutional neural network was used to test the satellite data test set. The training and test sets were 151 and 50 daytime sea fog images from 2017 to 2018, respectively, and the mIOU was used as an evaluation metric for the semantic segmentation results. The mIOU on the test set was 0.9436. The test data results of the trained model were compared with the results of the sea observations, which were tested from March 11, 2019 to July 20, 2019, with a total of 10, 559 observations from 40 observation stations.

The satellite test data results and the offshore observational data results were counted. According to the formulae for the accuracy rate of fog area, the recognition rate of fog area and the detection accuracy rate in Section 3.1 of this paper, the accuracy rate of fog area (proportion of detected real fog to detected fog) was calculated to be 66.5%, the recognition rate of fog area (proportion of detected real fog to real fog or cloud cover) was 51.9%, and the detection accuracy rate (proportion of samples detected correctly to total samples) was 93.2%. The final data are recorded in Table 2.

We selected two cases of the artificial intelligence (AI) sea fog inversion recognition at 14:00 BJT on April 6, 2022 (daytime) and 04:00 BJT on April 7, 2022 (nighttime) for case test, as shown in Fig. 3 and Fig. 4. The false color images on the left were generated by fusing data from channels 3, 4, and 14.

Figure 3.  AI monitoring and identification of sea fog on April 6, 2022.

Figure 4.  AI monitoring and identification of sea fog on April 7, 2022.

As shown in Fig. 3, the left column is the Himawari-8 satellite cloud image, and the right column is the AI sea fog inversion recognition at the same time. As can be seen, the deep-learning-based image fusion segmentation algorithm can better and more accurately inverse the pattern of sea fog over the southern Yellow Sea. Fig. 4 shows that the algorithm also has a good inversion recognition effect for night fog, and also has a good recognition effect for the sea fog area in the central–eastern part of the Yellow Sea and the southern Yellow Sea on April 7, 2022.

In summary, whether for the overall test from March to July 2019 or for individual cases, the geostationary satellite multi-channel image fusion segmentation algorithm based on deep learning had better results for the recognition of sea fog in the Yellow and Bohai Sea by using the D-LinkNet deep convolutional neural network semantic segmentation algorithm model. This paper proposed an AI-based sea fog recognition system, which automatically recognized cloud and fog through acquisition module, data pre-processing module, optical flow algorithm module, threshold calculation module, and classification module. It was based on the optical flow algorithm, and the difference in timing information between fog and other meteorological phenomena can significantly reduce the interference of similar meteorological phenomena on cloud and fog differentiation, which effectively improves the accuracy of cloud and fog recognition. Overall, the algorithm does not need too much modification when recognizing clouds and fog in different time and location. The difference in motion of clouds and fog is universal, so it has high universality.

This paper proposed a deep learning-based multi-channel image fusion segmentation algorithm for geostationary meteorological satellites, using the D-LinkNet deep convolutional neural network semantic segmentation algorithm model for 16 channels of Himawari-8 satellite data with a spatial resolution of 0.5 km in the Yellow and Bohai Sea area (32°–42°N, 117°–127°E). The training and test sets were derived from 151 and 50 daytime sea fog images from 2017 to 2018, and the mIOU was used as an evaluation metric for the semantic segmentation results, with a mIOU of 0.9436 on the test set. The results of the trained model were compared with that of sea observations. The testing period was from March 11, 2019 to July 20, 2019, with a total of 10, 559 observations from 40 observation stations. The accuracy rate of fog area (proportion of detected real fog to detected fog) was calculated to be 66.5%, the recognition rate of fog area (proportion of detected real fog to real fog or cloud cover) was 51.9%, and the detection accuracy rate (proportion of samples detected correctly to total samples) was 93.2%.

The main conclusions of this paper are as follows. The system used an optical flow algorithm to recognize clouds and fog based on time-continuous GSI source data. It is found that the use of the difference between fog and other meteorological phenomena timing information can obviously reduce the interference of similar meteorological phenomena (especially stratus) on cloud and fog distinction.

However, the theory of interpretability of deep learning methods has not yet been well developed, and the recognition of sea fog did not correspond to the evolutionary mechanism of sea fog. If the interpretability of deep learning in sea fog monitoring can be enhanced and corresponded with the physical mechanism of sea fog, fostering a deeper integration between meteorology and artificial intelligence, we are confident that AI technology could yield superior recognition outcomes in sea fog monitoring and contribute to a deeper understanding of the mechanisms behind the formation and dissipation of sea fog.