Image detail extraction via dark region approximation

In this paper, we propose a simple but effective approximation—dark region approximation (DRA) to extract details from gray-scale images. The DRA is based on an assumption: there is little illumination in the dark regions of visible images. The Retinex model using the DRA is developed to improve the performance of preserving details from the dark regions in gray-scale images during reflectance estimation. Then, Gaussian field criterion is introduced to construct the objective function which could be solved by quasi-Newton method, in order to estimate the reflectance via the DRA-based Retinex model. The reflectance is considered as the final result of image detail extraction. Experiments on a variety of gray-scale images demonstrate the power of the DRA and the superiority of our method.


Introduction
Extracting details from gray-scale images (i.e. single-channel images) is a critical step for many applications, such as image enhancement [1], image fusion [2] and image compression [3], etc. The accuracy of detail extraction directly determines the performances of these applications. When the captured gray-scale images suffer from low illumination and low contrast, detail extraction will become more difficult. Moreover, compared to real color images (i.e. multi-channel images), grayscale images has poorer information, resulting in a tough challenge for detail extraction. Accordingly, we mainly focus on detail extraction from gray-scale images in this work.
In recent years, numerous methods of detail extraction have been developed owing to the fastrising demand. Multi-scale decomposition [4] based on various transform models performs well on feature representation and detail extraction. The transform models, such as wavelet [5], pyramid [6], curvelet [7], non-subsampled contourlet transform (NSCT) [8], and Gaussian filters [9], are often used to depict the spatial distribution of the variety of image intensity. Retinex decomposition [10], considering the captured image as the product of reflectance and illumination layers, is applicable to detail extraction, because the reflectance can be recovered from the captured image by various Retinex-based models. Typically, bilateral filtering (BF) [11] and Gaussian filtering (GF) [12] are employed to estimate the illumination for Retinex decomposition.
In this work, dark region approximation (DRA) is proposed for extracting details from gray-scale images. First, the DRA-based Retinex model is developed to improve the performance of preserving details from the dark regions in gray-scale images during reflectance estimation. Then, on the basis of the DRA-based Retinex model, reflectance estimation is formulated as an unconstrained optimization problem by Gaussian field criterion and solved by the quasi-Newton method, in order to achieve image detail extraction. Qualitative comparisons between our method and three other approaches are conducted on a publicly available dataset. Experiment results show that our method is superior on detail preservation, and thus it is able to improve the performance of low-light image enhancement, image fusion and image haze removal, etc.

Dark region approximation
The Retinex model considers a captured image as the multiplication of a reflectance layer and an illumination layer : where the operator ∘ means element-wise multiplication. In logarithmic space, the Retinex model could be rewritten as: where = log( ) , = log( ) and = log( ) . is considered as a map depicting the structural details of the captured objects. Thus, in the traditional Retinex-based methods of image enhancement, such as SSR and MSR, the reflectance is directly estimated by = − and is estimated from by Gaussian filtering. However, if the captured image is decomposed to bright and dark regions, we can find the problem of the Retinex model. The captured image consists of bright and dark regions so that = − becomes where , and are the dark regions of , and , respectively. , and are the bright regions of , and , respectively. Due to good illumination, the differences between and are salient. In other words, the details in bright regions are easily preserved in . But, because the intensity of is at low level, can only have low intensity. Therefore, compared to , the differences between and are relatively insignificant. This means that it is easier to lose the details of dark regions in . Figure 1 shows the illustration of reflectance estimation via the Retinex model. We can see that the details in the bright regions (the blue box) is more salient and distinguishable than the details in the dark regions (the red box). Accordingly, the directly estimated reflectance in the logarithmic domain typically loses the desired details. According to the observation on the dark regions of the captured images, we propose the DRA: it is assumed that there is little illumination in the dark regions of the captured images, i.e.
= . Thus, the model (3) becomes the DRA-based Retinex model as follow: On the basis of the DRA, the intensity distribution in the dark regions of the captured images can be considered as the reflectance in the dark regions. Compared to the model (3), the DRA-based Retinex model enhances the details of dark regions in since − . Hence, the trade-off between the details of bright and dark regions in is balanced by the DRA-based Retinex model. However, the model (4) cannot be directly used on reflectance recovery because it is difficult to distinguish from and . In the next Section, we will discuss how to apply the DRA-based Retinex model on reflectance estimation.

Objective function
As we all know, the reflectance is partly caused by the illumination on objects. Therefore, we consider the reflectance as a map function that depends on the illumination layer . Then, the model (2) becomes On the basis of the model (5), Gaussian fields criterion is used to construct an objective function, in order to solve the map function . The Gaussian-fields-based objective function is given by where and respectively denote the intensity values of the -th pixel in and , is a range parameter, is the total number of pixels in . The objective function enforces closeness between the map function and − . The Gaussian fields criterion is a good distance measure because it is continuously differentiable and has superiority on computational convenience. Furthermore, the Gaussian fields criterion in the objective function (6) is an important basis to distinguish from dark and bright regions in a single-channel image. This will be discussed later in this section.

Reflectance model
In this work, the reflectance is considered as a transformation result from the illumination. We assume that there is a regular pattern of the transformation from illumination to reflectance in the captured image. Thus, the map function (i.e. the reflectance model) can be formulated as n α nk x i k y i n−k + β n T i n (7) where [ T is the coordinate vector of the -th pixel in the captured image, and are the reflectance parameters, is the order of the reflectance model. The first term in the model (7) indicates the space distribution of the reflectance, and the second term describes the regular pattern between the reflectance and the illumination on objects.
Essentially, the proposed model is the mixture of the various polynomials that depend on coordinate vector and intensity value. Its high non-linearity is useful for the representation of the complex pattern in the reflectance.
The matrix form of the model (7) is given by 10 11 20 21 22 is the 1 × ( = ( +3) 2 ) dimensional vector containing all parameters of the first term in equation (7). = [ 1 2 is the 1 × dimensional vector containing all parameters of the second term in equation (7). Hence is the 1 × + dimensional reflectance parameter vector. is the 1 × dimensional vector containing all − and is the 1 × dimensional vector containing all . As a result, is the + × 1 dimensional polynomial vector of the -th pixel in . [ T denotes matrix transposition. Substituting equation (8) into equation (6), the optimization function becomes

Optimization
It is obvious that the optimization function (9) is always continuously differentiable with respect to the reflectance parameter . Thanks to the reflectance model which is defined in polynomial form, it is easy to write the corresponding derivative of equation (9) as follows: On the basis of the derivative (10), gradient-based numerical optimization approaches such as quasi-Newton method [13] can be employed to solve the optimal parameter . But before that, there is still a difficult problem in reflectance estimation: how to obtain the illumination layer . Estimating the illumination and reflectance layers from the captured image simultaneously is an ill-posed problem, which cannot be solved by using the objective function (9) and the derivative (10). Thus, the illumination layer must be determined before reflectance estimation.

Calculating the illumination layer
From the objective function (9) we can see that reflectance estimation is considered as a fitting problem in this work. The capture image and the illumination are the known data points. The reflectance model (7) is a fitting function and the objective function is a fitting criterion. Accordingly, the purpose of the objective function is to make the reflectance model approximate − . Since the reflectance model consists of polynomials, it can describe the regular pattern of the reflectance ( = − ). In the traditional Retinex-based methods such as the single-scale Retinex [14] and multi-scale Retinex [15], the illumination layer is typically obtained from the captured image by using Gaussian filter. If the Gaussian-based illumination layer obtained by G = log( G ) is used to the optimization of the objective function, the estimated reflectance may have the same disadvantage as the directly estimated reflectance (it has been discussed in Section 2.1).
The penalty curve of the Gaussian criterion (e.g. 1 − exp − 2 2 2 ) is shown in Figure 2. We can see that the Gaussian criterion has high tolerance for large . In other words, the Gaussian criterion has little response to the large value of . With this property, coarse blur is developed for obtaining the illumination, in order to reflectance estimation via the DRA-based Retinex model (4).

Figure 2. The penalty curve of the Gaussian criterion
Actually, the coarse blur is very simple: the Gaussian-based illumination layer G is further degraded by setting the pixel values whose and coordinates are both odd or even numbers to 0 (as shown in Figure 3). Here, C denotes the coarse-blur-based illumination layer and C denotes the pixel value in C . In the optimization of the objective function (9), by using the pixels with C 0 , the reflectance model is fitted to the differences between and G . According to the Retinex model, in the case of C 0 , the reflectance model prefers to represent the regular pattern of the reflectance in bright regions. Meanwhile, the term − C + C 2 of the pixels with C = 0 in the objective function becomes − C 2 . As can be seen in Figure 2, the objective function has little response to the large value of − C 2 . Therefore, by relaxing the range parameter to a proper value, we can make the reflectance fit to the low intensities of the capture image. In other words, in the case of C = 0, the reflectance model incline to depict the regular pattern of the intensity distribution of the dark regions.
With the objective function (9), we can firstly decompose a single-channel image into dark and bright regions by soft thresholding. Then, the reflectance layers of dark and bright regions could be modelled by = and = − respectively. Finally, the reflectance model is used to compromise the regular patterns of the reflectance in dark and bright regions. Thus, the optimal reflectance model, estimated by the objective function with the coarse-blur-based illumination layer, is able to describe the regular pattern of the reflectance via the DRA-based Retinex model = + − . When the optimal reflectance model is determined, a DRA-based reflectance layer C could be obtained by where C represents the optimal reflectancee model with order . Figure 3 shows an example of reflectance estimation based on DRA. Compared to the directly estimated reflectance in Figure 1, the DRA-based reflectance saliently enhances the details in the bright regions while illuminating the details in the dark regions. Our method is outlined in Algorithm 1.
IOP Publishing doi:10.1088/1742-6596/1707/1/012029 6 objective function (9). Parameter controls the nonlinearity of the reflectance model. In this work, the optimal setting is = 0ᙀ6 and = 4, which is determined thought multiple experiments. What's more, since we scale the gray range of the input image into [0 1 , the optimal value of parameter and in general will be similar on various samples. Figure 4 shows the qualitative comparisons of BF, GF, PCQI and our proposed method on four grayscale images. It is apparent that BF, GF and PCQI could to some extent extract the detail information from gray-scale images. By using BF, GF and PCQI, the textures in bright regions are easily preserved but with the loss of the details in dark regions. Compared to the competitors, the results of our method have more textural details and better contrast. Moreover, the contrast in the dark regions is increased significantly. The details with low visibility in the captured images are preserved and enhanced by out method, such as the windows in Figure 4(a) and (c), the shrubs and trees in Figure 4(b), the man in the shadow of Figure 4(d). These demonstrate that the DRA-based reflectance has superiority on detail preservation and the proposed DRA is helpful for extracting the detail information from the dark regions of images.