Study on reflection separation based on orthogonal polarization images

In this paper, a deep learning method based on polarization image is proposed. Firstly, the polarization characteristics of reflected light and transmitted light are analyzed by using the Fresnel’s law. Then, based on the polarization characteristics that the reflected light is polarized in the perpendicular direction and the transmitted light is polarized in the parallel direction, the polarization images obtained from the parallel direction and the perpendicular direction are input into the Encoder-Decoder net at the same time, and the net is trained by L1 loss. The experimental results show that by combining the polarization information and deep learning method, the reflected component and transmitted component can be effectively separated.


Introduction
There will be both reflection and transmission on the surface of transparent objects such as glass. When using a camera to image the scene behind the transparent object, the image obtained is a mixed image, which contains not only the transmitted light, but also the light reflected by the surface of the transparent object [1]. Because the reflection and the transmission exist at the same time, the reflection component and the transmission component will be superimposed and interact with each other [2]. Due to the fact that glass and other transparent objects are widely used, the problem of reflection removal of transparent objects has become a hot topic in the field of computer vision [3]. Since the known variable is less than the unknown variable, the reflection separation belongs to an ill-posed problem [4]. It is a very challenging task to separate reflected light and transmitted light from a single reflected light image. The traditional separation method is to use the prior information of reflected light to calculate the reflected components and transmitted component [5].
When the light is reflected or transmitted, the polarization state will be changed [6]. Therefore, the polarization characteristics can provide effective prior information for the reflection separation. The early reflection separation method is based on the principle that the perpendicular component of the reflected light is smaller than the parallel component [7]. By rotating the polarizer, the perpendicular component is filtered to achieve the suppression of reflected light and the extraction of transmitted light [1]. However, the perpendicular component of the reflected light is close to zero only in the Brewster angle, and when the viewing angle is far away from the Brewster angle, it is difficult to eliminate the reflected light completely only by rotating the polarizer. In order to eliminate the reflected light far away from Brewster angle, a separation method based on image correlation is proposed by Ohnishi N[8]. By using the principle of minimum correlation between reflection component and transmission component, the viewing angle corresponding to the minimum mutual information of reflection image and transmission image is obtained, and achieve the reflection separation. In order to  [9] explicitly used the polarization properties of light and presented a deep learning method.
Although there is great progress in the reflection separation, there are still the following problems. First, a common problem of many existing reflection separation methods is that strict assumptions are imposed on reflection. These methods can work well only in some specific conditions. When the experimental conditions and assumptions are inconsistent, the separation methods are often fail. Second, most existing reflection method based on deep learning only use the low-level information such as edge and texture, but do not make full use of the high-level physical information and the difference information between the reflection and transmission. So the reflection separation effect still has room for improvement. Aiming at the above problems in the reflection separation, this paper proposes a reflection separation method combining depth learning algorithm and polarization characteristics. Polarization information is the inherent physical property of reflected light. The separation effect between reflected component and transmitted component can be improved by using the physical constraints of polarization prior information and the deep convolutional network. By using the difference characteristics of reflected light in the perpendicular and parallel directions, a pair of polarization images in the parallel direction and the perpendicular direction are input into the depth network at the same time, so as to realize the effective reflection separation.

Polarization characteristics from the surface of an transparent object
The Fresnel's law shows that the polarization state of light will be changed when the reflection or transmission effect occurs on the surface of an object. There is an obvious difference between the polarization state caused by the reflection effect and transmission effect. The polarization characteristics can provide effective information to solve the problem of the separation of the reflected light and transmitted light from the surface of transparent objects such as glass.

Reflection polarization
Reflection can result in polarization. When the light reach the interface between two different medium, the polarization state of light will change. The polarization state of reflected light is related to the reflectivity in the parallel and perpendicular directions, which defined by Fresnel formula as following: The polarization degree of reflected light is calculated by:

Transmission polarization
According to energy conservation law and Kirchhoff's low, the sum of reflection rate R and emission rate  is equal to 1 [10].
The degree of polarization According to analysis above, the reflection polarization and emission polarization on the surface of glass can be simulated respectively as following.
(a) Reflection polarization (b) Transmission polarization Figure 1. The simulation curves of the degree of polarization generated by reflection and transmission of glass surface As shown in Figure 1, the simulations results of reflection and transmission polarization indicate that, for reflected light, duo to the fact that the perpendicular component is greater than the parallel component, the polarization direction of reflected light is perpendicular. With the increasing of viewing angle, the degree of polarization increases first, and in Brewster angle the degree of polarization reaching the maximum which is equal to 1, then the degree of polarization starts to decrease at large incident angle. In the case of transmission polarization, the perpendicular component of transmitted light is smaller than the parallel component, the polarization is polarized in the parallel direction. The degree of polarization generated by transmission increases with the increasing viewing angle.

Reflection separation based on deep learning
According to the polarization characteristics of light, the reflected light is mainly distribute in the parallel direction, and the transmitted light is mainly in the parallel direction. Therefore, the polarization images obtained in the parallel direction and the parallel direction contain different transmission and reflection information. By inputting the parallelly polarization image and the perpendicularly polarization image into the deep convolution net at the same time, the net can learn the physical information of the reflection and the transmission, so as to realize the effective separation of reflection and transmission.

Network Architecture
As shown in the figure above, the inputs of our network is a pair of polarization images obtained in the parallel and perpendicular directions, and the outputs is the transmitted light image and the reflected light image obtained after separation. Our network uses the U-net structure, which is composed of

Loss function
The loss L is equal to the sum of the losses in all scales: where n is the number of scales.

Qualitative evaluation
We use 5493 pairs of synthetic images to train our net,and use 85 pairs of images to test. These synthetic images is generated with the method in the [11]. As shown in Figure 3, the first row and second row are the ground truths of reflection images and transmission images respectively. The third row and fourth row are perpendicularly polarization images and parallelly polarization images. Our separation results are shown in Figure 4(a). It can be seen that our method based on polarization images can effectively separate the reflection component and the transmission component. In order to comprehensively evaluate the separation effect of the proposed method, we compare our method with the existing method. Figure 4(b) is the separation results with method in [1]. This method can only remove reflection whose viewing angle is close to Brewster angle. Figure 4(c) is the separation results with method in [8]. This method assumes that the polarization feature is spatial invariance, but in fact, the proportion of reflected light and transmitted light for different pixels in the image is different, and the polarization degree is variable, so the separation result of this method is limitation. The above separation results indicate that our method outperform these existing method in visual.   In order to compare the separation effect of different method quantitatively, we use the evaluation indexes of peak signal to noise ratio (PSNR), structural similarity (SSIM), image correlation (CORR) and mutual information (MUI) to evaluate the separation performance. As shown in Table 1. results show that these evaluation indexes of our method combining polarization features and deep learning are best,which verifies the effectiveness of the proposed method.

Conclusion
In this paper, a deep learning method is proposed to separate the reflected light and transmitted light. We analysis the polarization characteristics of reflected light and transmitted light. The Encoder-Decoder network is designed to separate the reflected light by learning the polarization characteristics from the polarized image in the perpendicular and parallel directions. The quantitative and qualitative experimental results show our method outperforms state-of-the-art methods on reflection separation.