An optimal algorithm for bilateral filtering

Noise is unavoidable in digital signal processing, therefore studying effective denoising method is very important. Bilateral filter is one of them as effective filter methods. But the parameters of conventional bilateral filter are often fixed, thus we can’t realize adaptive parameters according to contents of videos or pictures. A video denoising algorithm based on both spatial domain and time domain is proposed in this paper to improve the video quality based on bilateral filter. This algorithm can adaptively estimate noise intensity, edge intensity and apply motion detection, which will be employed to adaptively adjust the bilateral filter strength. Peak signal to noise ratio (PSNR) is elevated significantly by the proposed algorithm after experimental verification..


Introduction
Filtering is a technique for improving quality of images and videos.In digital photography, noise signal is inevitable when taking images or videos.Image noise can have a significant impact on the image's quality and typically appears as scattered spots on a smooth surface.Random noise, banding noise, and noise with a fixed pattern are the most common types of image noise.Color shifts that are greater than the image's actual intensity are an indication of random noise.Long exposures and high temperatures result in the appearance of fixed pattern noise.When the camera reads data from the sensor, banding noise is created.This noise is directly related to factors in the camera's technology.Usually the noise above can be modeled as additive noise, multiplicative noise, Gaussian white noise, etc.
To reduce the effect of image noise, we need to adopt image and video filters on defected pictures.Most typical image filtering methods for instance median filter, mean filter, and gaussian filter are commonly used nowadays.Among the methods mentioned above, the gray value of each pixel is determined by the median of the gray values of all the pixels in a given neighborhood window of the point using the non-linear digital filtering technique known as the median filter.It is especially useful for reducing salt-and-pepper noise as well as speckle noise (impulsive noise).Unfortunately, its performance is not significantly superior than that of Gaussian blur when dealing with large amounts of noise [1].Mean filter and gaussian filter are linear smoothing filter which utilize average or weighted average value of a filtering window to replace the value of the center point of that window.They suffer from issues of poor contrast and fuzzy details due to the traditional nature of the image enhancement technologies they employ [2].Thus Tomasi and Manduchi proposed a bilateral filter which can preserve edges in some extent while denoising [3].Bilateral filter is widely used in multiple fields including denoising, tone management, video abstraction, texture editing and relighting etc.
However, classic bilateral filter also has its flaws.Users are required to choose the appropriate filtering parameters in traditional bilateral filtering algorithms including the filter window size and spatial/range filter strength σ_s/σ_r to get a better filtering result.Unreasonable parameters will lead to unsatisfactory filtering results.So relevant researchers have put forward many modified algorithms to improve its performance.Generally, these algorithms can be divided into two categories: adaptive parameters and adaptive window size.In reference [4] and [5], σ_s/σ_r are set to different values based on the values of neighborhood pixel.This method does not cause images to be blurry in light of improper parameters setting.An adaptive support window was produced as a result of reference [6].The size and shape of this window was determined by the local picture structure of the anchor pixel.
Although there are many algorithms to improve bilateral filtering, most of them are aimed at images, and the method of directly using images for bilateral filtering research on video has limited performance.This paper aims at studying representative optimization algorithms of bilateral filtering and innovatively raise the idea of both time and spatial domain based adaptive bilateral filtering algorithm so as to enhance the performance of video denoising.Experimental results show that the proposed algorithm can significantly improve the filtering effect.
The remainder of this essay is structured as follows.In Section II, related works are introduced.The proposed algorithm is presented in Section III.Its performance is given in Section IV and in Section V draws the conclusion.

Bilateral filter
Bilateral filter is a non-linear filter.It takes the intensity of each pixel and replaces it with a weighted average of the intensity values of neighboring pixels.On the basis of gaussian filter, bilateral filter introduces an another parameter to overcome the flaw of gaussian filter that it will blur the edge of image.The definition of bilateral filter is given as below, where  ′ () represents the intensity of pixel  after filtering.() represents the intensity of pixel  before filtering. represents the coordinate of current pixel to be filtered. represents the window centered in  .  () represents the range kernel for smoothing differences in intensities.  () represents the spatial (or domain) kernel for smoothing differences in coordinates   is smoothing parameter of spatial domain. is smoothing parameter of range domain.
The main advantages of the bilateral filter over other smoothing techniques, such as Gaussian blurs, is its ability to preserve sharp edges and details in an image while removing noise and low-frequency variations.This enables it well-suited for a variety of image processing tasks where edge preservation is important.
The block diagram of traditional bilateral filtering algorithm is depicted in figure 1.

Image Bilateral Filtering Algorithm
Reference [4] and [5] utilize different methods to achieve adaptive parameters   /  .In reference [4], when image gray scale is [0, 255], the researchers found that   /  fit the relationship as, where  is neighborhood size and its default value is 11.Δ is a value that reflects the fluctuations in nearby pixel values, it can be computed by, where  represents the amount of neighborhood pixel points, (  ) ̅̅̅̅̅̅ represents the average value of pixels in the neighbor area.By employed the relationship of equation ( 5), we can achieve better filtering result.The detailed process of this adaptive bilateral filtering algorithm is shown as below.
(3) Determine whether n is input or not.If not, n=11.
(4) Determine whether   is input or not.If not   = 3 • .
Reference [5] points out that the gradient's current value and the range parameter's value   have certain relationship.The researchers set as,   (, ) = (, ) * 1 (,) + (, ) (7) where (, ) represents the current pixel coordinate, (, ) and (, ) are the parameters which control the relationship of current   (, ) and current gradient (, ).In this processing, researchers set  = () and change  according to features of different images with certain tactic.The experiment result of reference [4] is shown as figure 2.
The entire procedure is described as, (1) We give the image a Gaussian gradient treatment in both the horizontal and the vertical directions separately.
(2) By solving the following equation, we are able to extract the gradient map of this image: (3) Get the   for each pixel based on the equation ( 7) and apply the bilateral filter adaptively.Experiment results show that both these two methods optimize picture quality as image are smoothed and edges are retained.In the edge or texture regions, they also significantly preserve more details than the conventional bilateral filter.

Video Bilateral Filtering Algorithm
Bilateral filter can also be applied on video denoising.One difference between videos and images is that videos contain time domain information, which means we can take advantage of the time feature of a video to improve the set parameters of filters.Reference [7] proposed an enhanced spatial-temporal combination-based video denoising algorithm.This algorithm firstly calculates Sum of Absolute Difference (SAD)， where(, ) is the displacement vector,   and  −1 signify accordingly both the current frame's and the reference frame's brightness values, and X×Y is a frame's size.If  > threshold, it applies temporal bilateral Kalman filtering.Others, it applies spatial bilateral ANL filtering.The block diagram of this video filtering algorithm is shown as figure 3.According to the findings of the experiments, the proposed algorithm is able to produce improved video denoising effects compared to some of the newest methods, particularly when dealing with additive white Gaussian noise.Additionally, the PSNR of video and visual quality may be enhanced by the proposed algorithm.

Rules of determination for parameters
To achieve adaptive the parameters for bilateral filter, we need to figure out how   and   work in filtering.First, these two values are the variance of kernel determined by equation (1).As the variance increases, the weight differs less.That means the corresponding factor in spatial domain(equation ( 3)) or range domain(equation ( 4)) influences less in filtering.If one of these two parameters approaches to zero, the filter will be invalid in smoothing.The larger   is, the more image gets blurred.The larger   is, the less edges will be retained.On the contrary, the smaller   is, the more filtering result is close to image before denoising.If   approaches to infinity, the whole process is equivalent to gaussian filter.Therefore, our model should follow the following principles.
(1) In area with strong noise, both   and   should be larger to make images smoother.
(2)   should be smaller in edge area and be larger in smooth area.
(3) In static area edges are distinct so   should be smaller while in dynamic area   should be larger for edges are indistinct because of motion blur.
To achieve the principles proposed above, it is necessary for us to conduct noise variance estimation for its noise level, edge detection for area texture complexity estimation and time domain motion estimation for motion magnitude detection.

Noise Variance estimation
The proposed algorithm uses Laplacian mask to estimate the noise variance.Because the image edge structure has a strong second-order difference property, the image is sensitive to the noise statistician of the Laplacian Mask.The algorithm performs the convolution operation through the kernel composed of two Laplacian operators, And the standard deviation of noise   is estimated by, where W and H represent the width and height of area to be processed.(, ) is the 3×3 window centered in (, ).

Edge Detection
Sobel operator is used in this algorithm to calculate edge intensity.The operator consists of two sets of 3×3 matrices, which are horizontal and vertical separately .By performing a plane convolution on the image, it is possible to determine the horizontal and vertical luminance difference approximations.If  represents the original image,   and   are the image detected by the horizontal and vertical edges, the following is the formula: Then the gradient (, ) can be obtained through equation (8).

Establishment of improved filtering model
Based on analysis and formulas mentioned above, we set the adaptive parameters as follows, where  is the estimated noise variance derived by equation (11) and equation (12). is the value derived by equation (9) to measure motion state. is the estimated value of gradient derived by equation (8) and equation (13).And , 1, 2 are coefficients to keep values of   /  with appropriate fitting accuracy.

Experiment result
In order to obtain suitable parameters for equation (14) and equation (15) above, we designed several experiments to determine specific values of , 1 and 2.A sequence named BQMall with size of 832x480 is employed here by 50 frames as shown in figure 4. The way to find optimal parameters is fixing one parameter and adding different levels of noise in each experiment.Then we change the coefficient of another parameter and study the relation between the variation of parameters and the performance (measure by PSNR improved after filtering).The experiment results are listed in Table 2 and Table 3 for different noise levels, respectively.
Table 2. Experiment to determine value of  with   = 20.In these two tables,  is the variance of original additive noise with PSNR0 as the anchor PSNR before filtering.  /  is fixed (equal to 2/20).We compute the PSNR improvements after filtering as the termΔPSNR in the tables to evaluate the effects of different parameters.It can be seen that, without loss of generality considering, we set the parameters  =1, 1 =2, 2 =2 here with acceptable performance gains.
To verify the final results and robustness of the proposed model, we set five noise levels and compare the proposed model with conventional fixed parameter bilateral filter (here we set   =2 and   =20).As we can find from the figure 5 above, when noise level is low, there is a very slight difference between the performance of these two methods.But as noise level increases, performance of fixed parameter bilateral filter declined rapidly while proposed method still maintains good performance with an improvement above 5dB.When noise level is extremely high, performance of proposed method descend quickly but still gains a better performance than conventional bilateral filter.Finally, the result images are shown as figure 6, figure 7 and figure 8.More specific details can be seen in figure 9.

Conclusion
This paper studied an adaptively optimal bilateral filter algorithm.We utilize noise variance estimation, edge detection and SAD to improve static or dynamic effect.Based on them, we can achieve adaptive bilateral filter strengths.As can be seen from the experiment results, the proposed technique works best in middle noise levels.In the future, we will investigate more efficient filtering methods in low or high noise levels.

Figure 1 .
Figure 1.Block diagram of bilateral filtering algorithm.

Figure 3 .
Figure 3. Algorithm process of one video filter.According to the findings of the experiments, the proposed algorithm is able to produce improved video denoising effects compared to some of the newest methods, particularly when dealing with additive white Gaussian noise.Additionally, the PSNR of video and visual quality may be enhanced by the proposed algorithm.

Figure 4 .
Figure 4. Original Sequence.The way to find optimal parameters is fixing one parameter and adding different levels of noise in each experiment.Then we change the coefficient of another parameter and study the relation between the variation of parameters and the performance (measure by PSNR improved after filtering).The experiment results are listed in Table2and Table3for different noise levels, respectively.Table2.Experiment to determine value of  with   = 20.

Figure 5 .
Figure 5. Performance in different noise levels.As we can find from the figure5above, when noise level is low, there is a very slight difference between the performance of these two methods.But as noise level increases, performance of fixed parameter bilateral filter declined rapidly while proposed method still maintains good performance with an improvement above 5dB.When noise level is extremely high, performance of proposed method descend quickly but still gains a better performance than conventional bilateral filter.Finally, the result images are shown as figure6, figure7and figure8.More specific details can be seen in figure9.

Figure 6 .
Figure 6.Sequence before filtering with noise level as var=0.008.

Figure 8 .
Figure 8. Sequence after the proposed adaptive filtering.

Figure 9 .
Figure 9. Segments contrast image of figure 7 (left) and figure 8 (right) in white pillar area.

Table 3 .
Experiment to determine value of 1 and 2 with   = 2.