Multi-exposure image fusion based on recursive filter

Multi-exposure image fusion technology is to fuse images of the same scene with different exposures into a high-quality image that contains more scene information. This paper proposes a multi-exposure image fusion method based on a recursive filter. The method first calculates three characteristic indicators of the multi-exposure image sequence: image contrast, entropy, and exposure information. And use three characteristic indicators to estimate the initial image fusion. In order to obtain a more accurate weight map for image fusion, a recursive filter is used to refine the weight map. Finally, the weighted sum method is used to construct the fused image. The experimental results show that the fusion results obtained by the algorithm in this paper have high image clarity, better subjective visual fusion effects and objective performance indicators, and maintain a high computational efficiency.


Introduction
The dynamic range of brightness in real life scenes can reach 8 to 10 orders of magnitude, but the dynamic range of ordinary digital imaging equipment is 2 to 3 orders of magnitude, and the actual image dynamic range exceeds all the information of the real scene that the equipment can capture [1]. Due to the limitation of dynamic range, the captured single image often has underexposed and overexposed areas, the edge information and texture details are seriously lost, the image quality is obviously reduced, and the visual perception is poor. The current methods for solving high dynamic range image (HDR) are mainly divided into the following two categories: tone mapping method based on inverse camera response curve and direct fusion method of low dynamic range image (LDR) image. This article mainly studies the second method.
The method of direct fusion of LDR images is to use LDR images of the same scene with different exposures to directly fusion generate HDR images according to certain rules. This method is simpler and more efficient than the first type of fusion technology. Burt and Adelson proposed the pyramid model for the first time, and applied it to the image fusion algorithm, and achieved good results [2]. Mertens et al. proposed a multi-exposure fusion method to generate HDR images based on the Laplacian pyramid model [3]. This method selects the weighted fusion of image contrast, saturation, and exposure information to generate HDR images. The result of the fusion is high in definition, but the image loses more information in the darker and lighter areas. Wu et al. proposed an HDR image generation method based on optimal block fusion [4]. First, the multi-exposure image is divided into blocks, and then the light gradient factor, saturation factor and appropriate exposure factor are used to select the optimal exposure block. Finally, the HDR image is generated according to certain fusion rules. The resulting image of the fusion has a blocky effect, which needs to be eliminated later, which increases the complexity and running time of the algorithm. Zhang [5]. This algorithm can eliminate the "ghost" of the image, but this algorithm is only suitable for dynamic images. Shen et al. proposed a multiexposure image fusion algorithm based on random walk [6], which calculates the image contrast and color constancy as probability, and obtains the fused image through the global optimal solution. This paper proposes a simple and effective method for multi-exposure image fusion. According to the three characteristic indicators of image contrast, entropy and good exposure, the initial fusion weight of the image is calculated, and then the recursive filter [7] is used to modify and refine the initial weight map to obtain an accurate fusion weight map, and finally, use the weighted sum method to fuse the image. This method is only suitable for image fusion of static scenes, and further research is needed for image fusion of dynamic scenes. Experiments show that this method is superior in both subjective and objective aspects.

Algorithm introduction
The algorithm in this paper is mainly divided into four steps: image feature index calculation, weight map estimation, weight map correction, and weighted fusion. The algorithm flow chart is shown in Figure 1:

Local contrast
Contrast is the ratio of black to white in the picture, which reflects the details of the image and the Grayscale and the degree of color performance. The better the color performance of the image with relatively large contrast, the clearer the image, the more natural the visual effect of the image. The local contrast of each pixel [8] is calculated as follows: In the formula: n Y represents the input image; n is the number of input images;  is the convolution operation., n Y is defined as follows: In the formula: I denote the intensity of R, G, and B channels respectively. h represents a high-pass filter, and the specific formula is as follows: Since local differences indicate contrast, the higher the contrast, the more details the corresponding area contains. If two pixels look very different and share the same grayscale value, the surrounding pixels are different, so they get different values through the high-pass filter. In the multi-exposure fusion, the difference between the multi-exposure images is caused by the exposure time and has a great influence on the grayscale value in the image. Therefore, in the same area, the pixel patch will not be different. The colors share the same grayscale value, and this measurement can detect good In the formula: N is the number of source images; ( , ) x y n C  is the generated local contrast information, whose purpose is to maintain image details.

Image entropy
In information theory, it can be known that the entropy value of information can reflect the size of its information volume, so the entropy value of the image is calculated as an indicator to evaluate the size of the image information volume. The calculation formula is as follows: In the formula, ( ) p z is the histogram data of the image; L is the gray level of the image.

Exposure information
The exposure factor is introduced based on the human visual characteristics and the spatial frequency characteristics of the image. A moderately exposed image has a more even distribution of pixels and a wider range of pixel values. In the case of over-exposure or under-exposure, the pixel points will be more distributed in extreme points. In order to get a better result, the image should be Choose wellexposed pixels for fusion. The calculation formula is as follows: In the formula, I (x,y) is the normalized pixel value of the image;  is the standard deviation of the distribution of the control weight mapping function, which 0.02 is taken in the experiment.

Weight map estimation
In order to retain more scene details and rich color information, and eliminate the effects of underexposed and overexposed pixels, the algorithm in this paper multiplies the three characteristic indicators of the exposure sequence: contrast, entropy, and good exposure information to obtain the initial weight map. The formula is as follows: The weight map is normalized to ensure that the sum of the weights of each pixel is 1. The formula is as follows:

Weight refinement and weighted fusion
Because the weight map is easy to introduce noise in the processing process, in order to eliminate the influence of noise, a recursive filter is used to filter the initial weight map to obtain an accurate weight map. Recursive filter is a real-time edge-preserving filter, which is widely used in image and video processing. The processing formula is as follows:

Experimental results and analysis
All experiments are done using MATLAB (R2018a) programming on a PC platform with Inter i7 processor (3.5GHz, 64-bit) and 16G memory. This article selects several sets of classic static multiexposure image fusion sequences for testing, and compare and analyze algorithm of this paper with the algorithm of Mertens et al and FMMR.
During the experiment, the relevant parameters of the algorithm were set, and the spatial parameters and smoothing coefficients of the recursive filter were taken as 100 In addition to subjective evaluation, this article uses three commonly used image quality evaluation indicators: image clarity, information entropy and standard deviation. Quantitatively evaluate algorithm performance through the above three indicators.

Average gradient
The definition of the image is described by the average gradient of the image. The larger the value, the richer the minute details contained in the image and the clearer the result image of the fusion. The calculation formula is: In the formula, , r c Z represents the pixel grayscale value, r and c represent the row and column coordinates of the pixel, respectively; R and C represent the number of pixel rows and columns of the test image.

Information entropy
Information entropy is a measurement method that reflects the richness of image information. The larger the entropy value, the richer the information contained in the image. Information entropy is closely related to the image quality perceived by human subjectively, and the influence of different types and distorted images can be predicted and found. The calculation formula is as follows:

Standard deviation
The standard deviation reflects the degree of dispersion between the image pixel value and the mean value. The larger the standard deviation, the better the image quality.

Experimental results
This part conducts an experimental analysis on the proposed method, and compares the performance of the three algorithms through average gradient, information entropy and standard deviation. This experiment uses six "garage" images with different exposures as the source images for fusion display.  Figure 2 shows a set of "garage" image sequences with different exposure levels and the resulting images of three different fusion methods. Table 1 shows the evaluation results of three evaluation indicators of different algorithms. From this group of images and table data, it can be concluded that the classic pyramid algorithm has a good effect on fusing images, but it is obvious from the figure that there is a problem of loss of detailed information in the underexposed areas, and the image is too bright. The result of FMMR algorithm fusion results in serious loss of image detail information, and many details of the car cannot be displayed well. Although the algorithm in this paper is not as good as the overall image fusion effect of the classic algorithm, the algorithm in this paper has better local contrast and rich color information, and the result of image fusion is more natural in terms of visual effects.

Conclusion
This paper proposes a multi-exposure HDR image fusion method based on a recursive filter, without HDR construction and tone mapping, and directly uses multiple LDR images with different exposures for fusion. The algorithm in this paper obtains the initial weight map of image fusion through three image feature indicators, uses recursive filters to modify the weight map to obtain a more accurate weight map, and finally uses the weighted sum fusion rule to obtain the final result image. It is proved that the method is simple and the fused image has rich details, natural color and high image quality.