Single image dehazing based on fusion of sky region segmentation

In order to solve the problem of image sharpening with sky region, a single image dehazing method combining sky region segmentation was proposed. Firstly, a fusion sky segmentation method combining OTSU algorithm with Canny operator edge detection is proposed to divide foggy image into sky region and non-sky region. Secondly, a linear index was constructed for the sky region using the dark channel prior sky transmittance to modify the sky region transmittance. For the non-sky region, a fast guided filter was used to optimize the dark channel prior non-sky transmittance. The two were combined to obtain the final transmittance, and the atmospheric light value was optimized by means of average estimation. Finally, combined with the atmospheric scattering model, the image is reconstructed dehaze. The experimental results show that the proposed method can segment the sky more accurately, estimate the atmospheric light value and transmittance more accurately and efficiently, and has the advantages of high contrast, natural color and clear details.


Introduction
In daily life, due to the fog and haze formed by a large number of small water droplets, smoke, dust and other substances suspended in the air, the quality of images captured by outdoor photography equipment and computer vision applications is seriously reduced, mainly shown as reduced contrast, blurred content, color offset and low visibility. Therefore, it is of great practical significance and application value in the field of digital image processing and computer vision to realize the clearness processing of degraded image, improve image quality and improve image visual effect [1].
For image defogging methods, algorithms are mainly divided into three categories. The first method is based on image enhancement. This method is based on non-physical model to achieve the purpose of improving image saturation and contrast by enhancing the global or local features of fog image. It mainly includes Retinex algorithm [2], the wavelet transform [3],histogram equalization [4].
The second is depends on the physical model for recovery of image restoration algorithms, analysis of the physical model of image degradation and prior knowledge or assumptions to estimate unknown parameters in the model to recover without fog image, He put forward main dark channel prior algorithm [5], the method based on statistical analysis of a large number of outdoor image of the dark channel prior information to solve atmospheric scattering model, Good image dehazing effect is obtained, but color distortion is easy to occur when large sky area exists in the target image. Zhu used the color attenuation algorithm [6] to model the scene depth of fuzzy images. The supervised learning method was used to learn the parameters of the model, which could well recover the depth information.

Atmospheric scattering model
In the field of computer vision, atmospheric scattering model [5]has been widely used in the study of fog removal technology. The expression is as follows:

Dark channel a priori and its defects
He et al [5] proposed a Dark Channel Prior (DCP) algorithm theory by counting a large number of outdoor images, i.e., the intensity values of some pixels in the R, G, and B color channels of certain pixels in the non-sky local area of a clear outdoor image are approximately zero in at least one color channel, which can be expressed as: is the dark channel of the image at pixel point x, is a neighborhood centered on pixel point x , c represents the three RGB color spaces, and ) ( y J c represents the three channel components of the image. Using the DCP theory and transforming Eq. (1), the transmittance estimation process of the foggy sky descending mass image based on the atmospheric scattering model can be expressed as: Eq. (4),  is to maintain the realism of the image and the introduction of the adjustment factor, and 1 0    , take 0.95. Substituting the transmittance and atmospheric light value into the atmospheric scattering model can restore the fog-free image: Eq. (5), in order to prevent noise amplification, the lower bound  Because the dark channel prior algorithm adopts the minimum filtering operation, the transmittance is underestimated at the depth-of-field mutation, so that the restored image will have obvious blocky effect at the depth-of-field mutation. He et al. took the Soft Mating algorithm further to optimize the transmittance to achieve better edge retention. But the Soft Mating algorithm is too computationally heavy to handle in real time. At the same time, the transmittance of bright areas such as the sky will be seriously underestimated, resulting in color distortion of the restored image.

Algorithm of this paper
Aiming at outdoor foggy sky image with sky part, a fusing sky segmentation method was proposed to solve the problem of sky part resolution processing. It includes four parts: sky region segmentation, scene transmittance estimation, global atmospheric light value estimation and foggy sky image restoration.

Fusion algorithm for sky segmentation
For foggy images, the OTSU algorithm, which is simple to calculate and can obtain ideal segmentation results even when the image contrast and brightness changes greatly, is used for the first segmentation operation. Since the OTSU algorithm cannot obtain the segmentation results very accurately, a second fusion segmentation operation with the canny edge extraction segmentation algorithm is needed to make the intersection of the non-sky region and the sky region more accurate and clear after the segmentation.
First, calculate the gradient of the foggy image and take the gradient map   x D , and then take the maximum value max D from the gradient map. After passing a parameter g , and , the control gradient enhancement is: And: The enhanced gradient maps are obtained according to Eqs. (6), (7) and (8). It has been verified by experiments that too much g value will affect the accuracy of edge detection, and too little will cause excessive sky noise. Generally, it is more accurate to set the value between 0.04 and 0.25.
After the edge enhancement, Gaussian filtering is performed to remove noise, and then the processed gradient map is subjected to a canny operator edge detection process. Firstly, Gaussian filtering is used to remove the image noise and enhance the image edge information; then the canny operator is used to detect the edge region of the image and inverse the detection map, and the inverse image is subjected to the open operation of first erosion and then expansion, mainly using the structural elements of the disk, whose radii are 4 and 3, respectively, and the maximum connected domain is obtained from the image after the open operation, and the OTSU sky segmentation and canny edge detection are The OTSU sky segmentation and canny edge detection are multiplied to obtain a more accurate sky and non-sky segmentation line. As shown in Fig. 2, by combining the advantages of the two algorithms, the fused sky segmentation algorithm can improve the segmentation accuracy and optimize the edge details to obtain more reasonable sky region segmentation results.

Transmittance correction
The accurate finding of transmittance in image defogging is one of the factors that directly affect the final effect of defogging. In this paper, the transmittance of the sky region and the non-sky region of the fogged image are optimized based on the dark channel a priori theory, respectively. For non-sky area, fast guided filtering [13] is used to replace image matting to filter the dark channel graph after maximum filtering so as to achieve optimization of non-sky transmittance. The expression is: In Eq. (9), I is the bootstrap image, q is the output image of the filter, k a and k b are a series of linear coefficients in window k  , k represents a local square search window with r as the radius. The linear regression method is used to obtain the window coefficient with the smallest window cost. The cost function is: In Eq. (10),  is introduced to prevent a too large adjustment factor. The window coefficient can be obtained by using the above formula: In Eqs. (11) and (12), k p is the mean value of the input image p in window k  ; k  and k  are the mean and variance of the guide image I in window k  ;  is the number of pixels in window k  .
The pixel i involved in i q is different in different windows k  . Therefore, all possible values of i q related to the pixel i must be averaged to obtain the final expression of the guided filtering: In Eq. (13), i a and i b are the average coefficients in the window centered on i .The input foggy image and the guided image are sampled and preprocessed by the fast guide filter, and the final result q is calculated. Then the transmittance of the obtained non-sky area is refined, the gray image In Eqs. (14) and (15) , which is 0.96 in this article. For the sky region, the transmissivity of the original sky region obtained from the dark channel is re-evaluated, and the sky transmittance is corrected by the linear relationship between the exponential function and the original sky transmittance.
First, find the maximum and minimum sky transmittance in the original fine transmittance map, and perform subtraction to obtain the transmittance difference, and subtract the sky area transmittance estimated by the dark channel a priori from the minimum sky area transmittance divide the obtained difference with the difference between the maximum and minimum values of the sky to obtain the original sky transmittance index k , and enter the Eq. (2) to obtain the corrected sky transmittance. As follows: )) ( min( )) ( max( )) ( min( ) ( In Eqs. (17) and (18) is the sky area transmittance estimated a priori by the dark channel.
The final transmittance of the combination of sky area and non-sky area is:

Obtaining atmospheric light value A
Another important indicator for image clarity and defogging is the global atmospheric light value A. In order to accurately solve the atmospheric light value, Kim et al. [14] proposed an atmospheric light value estimation method based on quadtree search. However, this method will be affected by sunlight and bright and smooth objects on the surface, which will cause deviations in the estimated position of the atmospheric light value. In this article, the acquisition of atmospheric light value A is improved, and the formula is as follows: In Eq. (20), 1 A is the maximum pixel value in the sky area, and 2 A is the atmospheric light value of the quadtree search. For images that include regions, the difference between the maximum value of the sky region and the quadtree search value will not be too large, and the average of the two is selected as 6 the atmospheric light value; for images that do not include the sky region, the sky region segmentation is invalid, resulting in The estimated atmospheric light value in this area may be quite different from the quadtree atmospheric light value, and the quadtree atmospheric light value after removing the noise interference may be more accurate, which improves the robustness of the atmospheric light value estimation.

Image enhancement after defogging
Substituting the estimated final transmittance and atmospheric light value into Eq. (5), a defogging image can be obtained. Experiments have found that even after the dark channel prior defogging processing has obvious enhancement effects in subjective perception, the final defogging image will be slightly darker in brightness, so further optimization of the image is needed. This article uses non-The linear overlay image enhancement algorithm optimizes the brightness to make the image details more prominent [15]: In Eq. (21), P represents the darker image to be processed, and S represents the brighter image after processing. It is also possible to introduce parameters  and to control the intensity of brightness. After many iterations, an ideal clear and bright defogging image is obtained.

Experimental results and comparative analysis
All experimental simulations in this article are performed on a computer configured with Intel(R)Core(TM)i5-5200U CPU@2.20GHz and running environment Windows10 using MATLAB (R2017b).
In order to illustrate the effectiveness and practicability of the proposed restoration algorithm, the image containing the sky area is mainly selected as the experimental image. As shown in Fig. 3(a), from top to bottom, there are foggy images containing bright areas, and the depth of field changes significantly. The qualitative and quantitative analysis of the experimental image and a variety of classic algorithms are compared with the foggy image of, and multiple foggy images containing sky areas. There are mainly the dark channel prior algorithm of He et al. [5], the color decay prior algorithm of Zhu et al. [6], the boundary constraint algorithm of Meng et al. [7], and the multi-scale neural network of Ren et al. [8] Algorithm, the end-to-end defogging algorithm of Cai et al. [9].

Subjective evaluation
Qualitative evaluation is based on subjective evaluation. Although subjective evaluation is subjective, it is also the most direct and most consistent evaluation method for human vision. This paper uses five sets of foggy images in different scenes to conduct experiments, and compares the algorithms of He, Zhu, Meng, Cai and Ren. and the experimental results are shown in Figure 3. From the results in Fig. 3(b), it can be seen that He's dark channel prior algorithm has significantly increased defogging details, and has a better defogging effect on darker scenes, but residual fog will appear at alternate depths of field. The color is too dark, and the color cast caused by the inaccurate estimation of atmospheric light value will appear in the picture containing the sky area; from the result of Fig. 3(c), it can be seen that Zhu's algorithm will also appear in the picture containing the sky area. Residual fog remains and appears whitening and distortion in sky pictures with white clouds, but certain dehazing effects are achieved in the color fidelity experimental pictures; as shown in Fig. 3

(d)
The results show that Meng's algorithm has a better processing effect in the sky area, but the image will appear color distortion in the darker sky area; from the result of Fig. 3(e), Cai's boundary constraint algorithm can solve the atmosphere by adding constraints, although the ill-conditioned problem of the model has obvious effects in details, color distortion will occur and there will be obvious residual fog in the scene containing the sky; the result of Fig. 3(f) shows that Ren's algorithm has a relatively good dehazing effect, however , color distortion may also occur in areas that include the sky; compared with the above algorithm, it can be seen from the results of Fig. 3(g) that the algorithm proposed in this paper has significantly increased image details and can reflect the depth information of the image, appropriate brightness, natural colors in the close-range area, rich details, and good brightness in the distant area. Processing the sky area can obtain higher clear contrast, and can effectively avoid the phenomenon of halo color distortion, even in non-sky areas with a good dehazing effect.

Objective comment
Quantitative evaluation is mainly based on objective evaluation, and the objective evaluation indicators in the literature [16] (that is, the ratio of newly added visible edges e , the mean value of visible edge gradient r , the visual contrast measurementVCM and the running time T ) are used to evaluate the dehazing image. The mathematical expression is: The data in the above table is the average of the results of multiple experiments, we can see that the lowest average value produced by Meng algorithm in the ratio of visible edges e coincides with the phenomenon of color bias that appears in this paper, the experimental effect of the proposed method is better and has obvious advantages; for the average contrast gain r , He algorithm and Cai algorithm have obvious advantages over Zhu algorithm, Meng algorithm and Ren algorithm, and it can be found from the VCM that this paper has obvious advantages over other algorithms, and the advantages of this paper's algorithm are greater, for the average running time, this paper's algorithm is also better than other defogging algorithms, with lower time complexity.

Conclusion
In this paper, based on the theory of traditional dark channel a priori, a fused sky segmentation method for sky image defogging is proposed, using Otsu algorithm and canny edge detection for fusion algorithm to segment the foggy images containing sky part in the region, which has more perfect segmentation accuracy compared with other segmentation algorithms; fast bootstrap filtering is used for transmittance optimization processing for non-sky region to avoid the sky The transmittance optimization process with fast bootstrap filtering for non-sky regions avoids color distortion in the sky part; at the same time, the global atmospheric light acquisition method is optimized to improve the accuracy of atmospheric light estimation. The experimental results show that the sky segmentation method in this paper has finer segmentation accuracy and algorithm adaptiveness compared with other methods. It can also eliminate the color bias problem caused by the influence of non-sky regions on the atmospheric light estimation and avoid color distortion of the sky. After objective evaluation, more scene information is restored and the defogging processing speed is accelerated.