Subpixel corner detection algorithm based on the plane chessboard

In this paper, the pixel-level corner position extracted by the Harris algorithm is used as the initial value of sub-pixel corner detection, and then it is used to detect the sub-pixel corner detection. The sub-pixel corner detection algorithm based on the inner product of the gray gradient has weak anti-noise and poor anti-distortion ability, and the sub-pixel corner detection algorithm based on Gauss quadric surface fitting has a high dependence on the initial value. In this case, a quadric fitting based on an iterative search thought algorithm of subpixel corner detection is proposed in this paper. The experiment shows that the improved algorithm has a stronger ability in anti-noise and anti-distortion and low dependence on the initial value. Using this algorithm to calibrate the camera, the re-projection error is smaller and the calibration accuracy is higher.


Introduction
Corner detection in images [1] is a fundamental topic within the realm of machine vision and computer science.In the domain of digital image processing, corner points [2] [3] have consistently served as a highly intuitive and crucial local feature.They are instrumental in capturing the contour characteristics of a subject, playing a decisive role [4] [5].Although there is no precise mathematical definition for corner points to date, they are generally considered to be locations with abrupt changes in brightness in a two-dimensional image or points exhibiting maximum curvature along the edge of an image curve.Corner points find extensive applications in fields like three-dimensional scene reconstruction [6], computer vision tasks such as motion estimation [7], target tracking, target recognition, image registration, and matching, where they assume a significantly important role.
In checkerboard calibration, pixel-level corner detection algorithms are primarily categorized into two distinct groups: those based on image grayscale and those based on image edges.The grayscalebased corner detection algorithm primarily focuses on examining grayscale variations within the pixel array, defining corners based on significant brightness contrasts relative to adjacent points.The existing relatively mature algorithms include the SUSAN algorithm, the Moravec algorithm, and the Harris algorithm.The accuracy of this algorithm depends on the parameter settings, and it is sensitive to image noise and distortion, so the robustness of the detection results is not high.The corner detection algorithm based on the edge of the image must first detect the edge of the image , then obtain the straight line through the Hough transform and other means, and then obtain the precise coordinates of the corner point through the intersection of the two straight lines.This method is simple and easy.The edge-based corner detection algorithm makes use of a large amount of edge information.Through multi-point averaging, it can effectively filter out noise point errors and improve the robustness of the results.The current edge detection algorithm adopts the entire pixel edge, and then calculates the intersection points of the line segments to obtain the checkerboard corner points.The rounding error caused by the whole pixel will be passed to the corner point calculation, reducing the detection accuracy.
The pixel-level corner detection algorithm has low accuracy and poor detection result robustness, and is not suitable for occasions with high accuracy requirements.Given this situation, in this paper, research is conducted on sub-pixel detection algorithms.First, the pixel-level corner position extracted by the Harris algorithm is used as the initial value for sub-pixel corner detection, and then sub-pixel corner detection is performed.First, the corner point response function is approximated to a quadratic surface, and the points within the local range of the corner point are used to form a system of overdetermined equations.The least squares method is used to solve the overdetermined equations to obtain the parameters of the quadratic function, and then the extreme point is found.The position of sub-pixel corner points reduces the effects of noise and distortion.Finally, the iterative search method is used to reduce the algorithm's dependence on the initial value.Experiments prove that the improved algorithm in this paper has strong anti-noise and anti-distortion capabilities and has low dependence on the initial value.

Method
Commonly used sub-pixel corner detection algorithms include the sub-pixel corner detection algorithm based on the gray gradient inner product and the sub-pixel corner detection algorithm based on Gaussian quadratic surface fitting.The sub-pixel corner detection algorithm based on the gray gradient inner product has weak anti-noise and poor anti-distortion capabilities, while the method based on Gaussian quadratic surface fitting has enhanced anti-noise ability, but is highly dependent on the initial value.Combining the shortcomings of the two algorithms, a sub-pixel corner detection algorithm based on quadratic surface fitting based on the iterative search idea is proposed in this paper.

The sub-pixel detection algorithm based on the gray gradient inner product
It is assumed that the initial position of the corner point is at a point, with the point as the center.A window is selected with a size of your own choice, each point in this window constitutes a point, and all vectors are detected.As shown in Figure 1(a), for a point on the edge, the vector is consistent with the edge direction, and the vector is perpendicular to the gradient direction at the point, so the dot product of the vector and the gradient at the point is 0. As shown in Figure 1(b), it is an area with uniform grayscale, the gradient at the point is 0, and the dot product of the vector and the gradient at the point is also 0. It is assumed that N combinations similar to vectors ⃗ can be searched around the point q, so that their dot product is 0: Due to the presence of noise, the dot product is not equal to zero, so Equation (1) can be transformed as follows: At this time, it can be transformed into a problem of finding the minimum value: The partial derivative of the Logarithm (3) should be determined so that it is 0: ⃗ = ( , ) is the position of the sub-pixel corner point sought, and if it is any position within the window range, the value can be derived.
The equation above is the principle of the sub-pixel corner detection algorithm based on the gray gradient inner product.It can be seen that the solution requires the gradient value and the coordinate value of the point.Because the gradient value needs to be calculated, the accuracy of the detection algorithm is greatly affected by noise, and its robustness to distortion is not strong.The selection of coordinate values is directly related to the detection window, so the selection of the detection window will also have a great impact on the accuracy of the detection algorithm.

The sub-pixel corner detection algorithm based on Gaussian surface fitting
The Harris corner detection algorithm extracts the position of the pixel-level corner point and requires the precise position of the sub-pixel corner point.It is assumed that the gradient of a pixel-level corner point and its neighborhood is approximated by a two-dimensional Gaussian surface, the formula can be expressed as: Since the Gaussian function is separable, a two-dimensional Gaussian function can be converted into two one-dimensional Gaussian functions, reducing the computational complexity.Equation ( 6) can be simplified as follows: By solving the position of the extreme point, the sub-pixel corner position can be obtained.The subpixel corner detection algorithm based on Gaussian surface fitting uses a quadratic Gaussian surface to fit the corner point and its surrounding gradient map, which can effectively eliminate the effects of noise and image distortion.However, it can be seen from Equation ( 7) that the detection algorithm is highly dependent on the initial value.

The sub-pixel corner detection algorithm based on quadratic surface fitting based on an iterative search idea
The sub-pixel corner detection algorithm based on the gray gradient inner product suppresses the influence of noise by increasing the range of the detection window, but the larger the window range is, the more complex the calculation will be.The sub-pixel corner detection algorithm based on Gaussian surface fitting can reduce the influence of noise, but it has a high position dependence on the initial value, so it was improved and is called quadratic curve fitting sub-pixel angle combined with iterative search.In the point detection algorithm, first, the corner response function is approximated to a quadratic surface, and the points within the local range of the corner point are used to form a system of overdetermined equations.Then the least squares method is used to solve the parameters of the quadratic function.Finally, the extreme point is found.The position of the corner point at the pixel level reduces the impact of noise and distortion on the accuracy of the detection algorithm.In order to further weaken the impact of the initial value on detection accuracy, an iterative algorithm can be used for optimization.Specific steps are as follows: 1. Quadratic surface fitting By approximating the local corner response function to a quadratic surface, Equation ( 6) can be written as follows: R(x,y) = Ax + By +Cxy+Dx+Ey+F (8 If there is a point, it should be written in the form of a matrix to express as: The least squares method is applied to solve the overdetermined equations and obtain the parameters of the quadratic polynomial.Then, by deriving Equation (8), the following can be obtained: In this way, the sub-pixel position of the corner point under this window is found.

Iterative search
Taking the sub-pixel corner detected for the first time as the center, the quadratic curve fitting method is used to further find a new point, that is, a point; then it will be used as the new area center, and the iterative solution will continue until the corner point is detected twice.If the position is less than or reaches the maximum number of iterations 10 times, the iteration is deemed to have ended.

Experimental comparative analysis
A 2*2 black and white checkerboard pattern is used in the experiment, with pixels of 200*200, and the ideal center corner position is (100, 100).The image is smoothed with a Gaussian filter to simulate an ideal picture without noise.The error criterion is the angle.The distance between the corner point position extracted by the point detection algorithm and the corner point position set by the simulation.For convenience of expression, the sub-pixel corner detection algorithm based on gray gradient inner product is referred to as the GI algorithm and the sub-pixel corner detection algorithm based on Gaussian quadratic surface fitting is referred to as the GCF algorithm, the sub-pixel corner detection algorithm of quadratic surface fitting based on the iterative search idea is called the CFI algorithm.
Experiment 1: Standard deviation Gaussian noise σ = 0.5~3 is added to an undistorted checkerboard pattern, and the GI algorithm is used to detect corner points under different detection window radii.It can be seen from Figure 2 and Table 1 that, for undistorted images with the same noise level, as the detection window increases, the average error of the GI algorithm becomes smaller, and the greater the noise is, the more obviously the detection window improves the accuracy.An increase in the detection window means more data to be solved and the algorithm is more stable, but the algorithm detection time will also be longer.Experiment 2: Standard deviation Gaussian noise σ = 0.5~3 is applied to the distorted and undistorted checkerboard images, and the GI algorithm is used to extract the sub-pixel positions of the corner points under (5×5) and (11×11) window radii respectively.As can be seen from Figures 3,4 and table 2,3 , under the same noise and detection window, the average error when the GI algorithm detects distorted images is greater than the average error when detecting undistorted images, which shows that distortion will affect the accuracy of the GI algorithm in extracting corner points.
Therefore, it can be concluded from Experiments 1 and 2 that the following factors mainly affect the accuracy of the GI algorithm: (1) The size of the detection window (2) Image distortion (3) Image noise Experiment 3: Standard deviation Gaussian noise σ = 0.5~3 is applied to an undistorted checkerboard image, and the GI algorithm, the GCF algorithm, and the CFI algorithm are used to extract the sub-pixel positions of the corner points.As can be seen from Figure 5, the three algorithms detect distortion-free images under the same detection window radius (5×5) and the same noise.The CFI algorithm has the smallest average error, followed by the GCF and the GI algorithms.Therefore, the conclusion can be drawn from Experiment 3: The order of the anti-noise capabilities of the three algorithms is CFI>GCF>GI.
Experiment 4: Standard deviation Gaussian noise σ = 0.5~3 is added to the distorted and undistorted checkerboard images, and the CFI algorithm (detection window radius 5) is used to extract the sub-pixel positions of the dots.As can be seen from Figure 6, when using the CFI algorithm to detect distorted images and undistorted images under different noises, the average corner point extraction error is almost unchanged.Therefore, the conclusion can be drawn from Experiment 4: The CFI algorithm is more robust to distortion.
Experiment 5: Standard deviation Gaussian noise σ = 0.5~3 is added to the distorted and undistorted checkerboard images, and the GCF algorithm (detection window radius 5) is used to extract the subpixel positions of the dots.As can be seen from Figure 7, when using the GCF algorithm to detect distorted images and undistorted images under different noises, the average corner point extraction error is almost unchanged.Therefore, it is concluded from Experiment 5 that the GFC algorithm has good robustness to distortion.
Experiment 6: When adding the standard deviation σ = 1, different initial values are set for detection, and the GCF algorithm is used to extract the sub-pixel positions of the dots.(The detection window radius is all set to 5).It can be seen from Figure 8 that the farther the selected initial value is from the actual value, the greater the average error value of GCF detection will be.Therefore, it is concluded from Experiment 6 that the detection accuracy of the GCF algorithm is greatly affected by the initial value.
Experiment 7: When adding the standard deviation σ = 1, different initial values are set for detection, and the CFI algorithm is used to extract the sub-pixel positions of the dots.(The detection window radius is all set to 5).It can be seen from Figure 9 that the average error value of the GCF detection is almost unchanged when the initial value is selected.Therefore, it can be concluded from Experiment 7 that the initial value has little impact on the CFI algorithm.It can be seen from Figure 10 that the farther the selected initial value is from the actual value, the greater the average error value of GI detection will be.Therefore, it is concluded from Experiment 8 that the detection accuracy of the GI algorithm is greatly affected by the initial value.

Camera calibration accuracy evaluation
Zhang Zhengyou proposed the "Zhang Zhengyou calibration method" in 1998, which is a camera calibration method based on a planar checkerboard .The key step of Zhang Zhengyou's calibration method is to detect the corner points of the checkerboard.The accuracy of corner point detection directly affects the accuracy of calibration.The accuracy of the calibration camera can be evaluated through the size of the reprojection error, thereby indirectly verifying the advantages and disadvantages of the improved algorithm proposed in this article with details as follows: Now it is assumed that the corresponding points in the two images satisfy: Among them, H is the homography matrix, and Xi' and Xi are the corresponding points of the left and right images, then the calibrated reprojection error can be written as ε=d X ,X +d (X ,X ) Among them, X =H X ,d is the image corresponding point error, is the estimated value of Xi, and is the estimated value of H. and are optimized to get the minimum heavy projection error.Based on the calibration results, the projection point of the three-dimensional space point on the image plane is calculated and compared with the coordinate point on the real image plane to solve the reprojection error, so the reprojection error can be used as an indicator to evaluate the calibration accuracy.
From the experiments above, the following conclusion can be drawn: with the same algorithm and the same standard deviation, as the detection window increases, the detection error will decrease.When the standard deviation of the detection window is the same, the error of the distorted image is greater than the error of the undistorted image.When the window size is the same and the detection algorithm is the same, the larger the standard deviation is, the larger the detection error will be.The three algorithms, when the standard deviation and detection window are the same, the errors from large to small are the GI algorithm, the GCF algorithm, and the CFI algorithm.The reprojection error after corner detection using the GI algorithm, the CFI algorithm, and the GCF algorithm on the same image is shown in the table 4 below.It is also consistent with our above experimental conclusion.

Conclusion
When there is distortion and noise in the image, the CFI algorithm has higher accuracy than the GCF and GI algorithms.Moreover, when the accuracy of pixel-level corner detection as the initial value is not high, the accuracy of the CFI algorithm is also very high.After detecting the corner points, the camera is calibrated according to Zhang Zhengyou's calibration method, and finally, the reprojection error is used to measure the accuracy of the calibration.It can be concluded that the improved algorithm proposed in this article has strong resistance to distortion and noise, and is less affected by the initial value.Using this algorithm for camera calibration, the reprojection error is small and the calibration accuracy is high.

Figure 1 .
Figure 1.Schematic diagram of algorithm based on the gray gradient inner product.

Figure 2 .
Figure 2. Error histogram of GI algorithm detection under different detection windows for undistorted images.

Figure 3 .
Figure 3.The average error of the GI algorithm under the (5×5) window with and without distortion.

Figure 4 .
Figure 4.The average error of the GI algorithm under the (11×11) window with and without distortion.

Figure 5 .
Figure 5. Average errors of the GI, the GCF and the CFI algorithms under distortion-free images.

Figure 6 .
Figure 6.The average error of the CFI algorithm under distorted and undistorted images.

Figure 7 .
Figure 7.The average error of the GCF algorithm under distorted and undistorted images.

Figure 8 .
Figure 8.The average error of GCF algorithm under different initial values.

Figure 9 .
Figure 9.The average error of CFI algorithm under different initial values.

Experiment 8 :
When adding the standard deviation σ = 1, different initial values are set for detection, and the GI algorithm is used to extract the sub-pixel positions of the dots.(The detection window radius is all set to 5).

Figure 10 .
Figure 10.The average error of GI algorithm under different initial values.

Table 1 .
Errors in GI algorithm detection under different detection windows for undistorted images.

Table 2 .
Errors of the three algorithms for undistorted images when the detection window radius is 5.

Table 3 .
Errors of the three algorithms for distorted images when the detection window radius is 5

Table 4 .
Calibrated reprojection errors using the GI algorithm, the CFI algorithm, and the GCG algorithm.