Image perspective restoration considering multi-granularity distortion correction algorithm

Image restoration technology is a technology that uses the visible information on the current image or another image to fill in the occluded or damaged area on the image, so as to obtain a visually satisfactory effect. This paper proposes a multi-granular distortion correction algorithm, a The coarse-to-fine distortion correction algorithm performs perspective distortion correction on the image perspective and then is used to complement the damaged area on the target image. Finally, the Poisson imaged fusion algorithm is used to eliminate the ghost phenomenon between the repaired area and its surrounding pixels, To obtain a seamless repair effect. Experiments prove that the algorithm has obvious reliability.


Introduction
Image restoration technology is a technology that uses the visible information on the current image or another image to fill in the occluded or damaged area on the image, so as to obtain a visually satisfactory effect [1] . Because of the photo editing and rigging removal In addition to the wide application needs of special effects production and other aspects, image restoration technology has received a lot of attention in the past ten years, and many algorithms have emerged.
Traditional image restoration algorithms are mostly for single images, and are mainly divided into two methods based on partial differential equations and texture synthesis. The former defines known information pixels around the damaged area as boundary conditions, and image restoration is regarded as partial differential equations. Solve or variational problems [2][3] . Therefore, the process of image restoration is converted into a diffusion process of known pixel information to the cavity area. This type of method is generally only suitable for the restoration of small structural areas, but for highly textured areas. Sexual areas will be invalid. Regarding the known information area on the target image as a texture sample, the latter uses texture synthesis to generate a new image patch to fill in the information loss area. It is a very effective way to read the meter through the camera [4] . But for industrial and mining, petrochemical and electric power companies, the pointer meters are diverse and large in number, and it is difficult to equip each meter with a camera [5] . If the camera can rotate and scan to interpret various instruments in a 360° space, the computer vision-based instrument interpretation technology can be more practical and cost-saving. However, this method will cause each 2 dial to be tilted to different degrees in the camera, and the image will appear perspective distortion, causing serious interpretation errors [6] .
In response to the problems in traditional image restoration methods, there have been a small amount of work based on multiple images, especially large displacement viewpoint images, to perform image restoration. This paper proposes a multi-granular distortion correction algorithm. The image library-based method retrieves a huge image library Similar photos in the are used for the repair of holes, so the size and diversity of the image library become the key to determining the success of the repair.

Multi-granular distortion correction algorithm
Multi-granular distortion correction algorithm, that is, a new image restoration algorithm that uses a large displacement view to repair a large information loss area on the target image. The algorithm flow chart is shown in Figure 1.

Large displacement view deformation based on homography matrix
In order to speed up the convergence process of the algorithm, we first quickly obtain a globally optimal initial solution by assuming that the scenes in the two views are approximately on the same three-dimensional plane. The overlap between the two views obtained by the global transformation under the action of the homography matrix Regions are approximated as their common scene area.
The large displacement view deformation based on the homography matrix mainly includes the following three steps: (1) Feature detection and matching; (2) the solution of the homography matrix: due to the inevitable mismatch of the approximate nearest neighbor method, plus Due to the influence of image noise, there may be outliers in the feature matching points. We use the RANSAC algorithm to eliminate the excluded points, and the Lev2enberg2Marquardt (LM) algorithm robustly estimates the homography H that satisfies p=Hp', where p and p'are respectively The matching feature points on the target image and the large displacement view; (3) Large displacement view deformation: Transform the large displacement view S to the viewpoint of the target image T according to H.
The deformed large displacement view S'falls on the target image T to produce an overlapping area between the two. The part of S'that overlaps with the known area in the target image T establishes the initial pixel correspondence of the known common scene area Ω o . S'on The part that overlaps with the missing information area in the target image T provides an initial estimate of the void area Ω h .

Overlapping pixel correspondence based on energy optimization
When the flat scene is not satisfied, there will be a large number of mismatched pixels in the public scene area Ω o YΩ h , that is, residual distortion. Therefore, directly using the deformed large displacement view to fill the void will result in a very ugly effect. Here, it is necessary to change The known public scene area Ω o around the hole is used as the constraint and basis for correcting the initial 3 estimation of the missing information pixels, then the residual distortion in Ω o must be further corrected first. The problem of dense pixel correspondence between smaller views. Through the mismatch detection mechanism and dynamic weight parameters, we regard the distortion correction in Ω o as a pixel correspondence problem based on energy optimization, and propose a new optimization strategy for generating large Reliable pixel correspondence between displacement views. Let (p, p') represent the pixel matching point pair on the target image p∈T and the large displacement view p'∈S, Np represents the four-connected neighborhood of p in Ω o , and <p, q> is the neighborhood pixel pair q∈N p . Assuming that the entire image is a Markov Random Field (MRF), the attribute of a pixel can be uniquely determined by its neighboring pixels. Under the constraints of the color constancy of the corresponding pixel and the smoothness of the displacement field, We define the energy function as shown in equation (1): Taking into account the discontinuous motion boundary The strong suppression of, will result in a smooth transition field. The smooth term of the displacement field designed by us relaxes the penalty for large displacement changes, as shown in equations (2) and (3): 2 , The detailed optimization process corresponding to overlapping pixels is as follows: (1) Initialization. Use the inverse matrix H -1 of the homography matrix to initialize p', that is, p'~H -1 p, o p ∀ ∈ Ω ; the initial value of the dynamic weight λ is λ 0 .
(2) Energy function minimization. Use the conjugate gradient method to minimize the energy function to obtain p', o p ∀ ∈ Ω .

Lost pixel estimation based on energy optimization
Convert the hole pixel repair problem into the pixel corresponding optimization problem, that is, given the initial estimated value provided by the deformed large displacement view, the reliable pixel around the hole corresponding to M g is used to estimate the missing information pixel p∈Ω h on the large displacement view In order to obtain a satisfactory repair effect, three prior expectations must be met. h ∂Ω is set to represent the boundary of the hole, and is the pixel located in the known public scene area around the hole. NB p is a 3×3 image patch centered on pixel p. Considering the above a priori expectation, define the energy function of the restored pixel p As shown in formula (4)

Experimental results
In the character removal experiment in Figure 2, the image restoration algorithm based on texture synthesis repairs the target image. Figure 2(a) obtains Figure 2(b). It is obvious that the tree structure in the occluded area has not been reasonably restored. By introducing an image The large displacement view is shown in Figure 2(d), and we expect to get a satisfactory repair effect. The deformation result of the large displacement view based on the homography matrix is shown in Figure  2 Figure 2(g) ), it can be seen that the correction results in the large displacement view have been basically consistent with the corresponding parts on the target image. Finally, under the constraints of the reliable pixel correspondence in the known public scene area around, the initial estimation of the missing information area is corrected by energy optimization Figure 2(h) is the repair result of the method in this paper, which is significantly better than the effect in Figure 2(b) and Figure 2(f). The algorithm repaired about 9,000 on the target image of size in less than 1 minute. Occluded pixels.

Conclusions
This paper proposes a multi-granular distortion correction algorithm, which uses a coarse-to-fine perspective distortion correction algorithm to optimize and correct the large displacement view and then use it to restore the damaged pixels on the target image. Experimental results show that the proposed algorithm is better than traditional image restoration Algorithms can repair large damaged areas containing complex structural information.