An affine transform center line extraction and discontinuity deviation correction method for non-orthogonal grid laser

The extraction of the laser center line is one of the key steps of laser scanning, which is related to the accuracy of the coordinate calculation. Non - orthogonal grid laser scanning has the advantages of a large measuring range and small translation error. For the extraction of the center line of the laser cross region of non-orthogonal grid, the discontinuity and deviation of the results obtained by the existing methods will cause the error of center line coordinate extraction. To solve these problems, according to the properties of image affine, an improved method of laser center extraction for non-orthogonal grid is proposed in this paper. Firstly, the image is divided into sub-images by affine transformation according to slopes, and the orthogonal grid laser image is obtained. Secondly, in order to solve the problem of centerline discontinuity, the orthogonal grid laser image was rotated 0° and 90°, and the Steger method was used to process images. Inverse affine transformation is performed on the results. Finally, the sub-images are clipped and splicing according to slopes, and the two rotation results are union to get the complete laser center line. By comparing various methods and 3D reconstruction experiments, the accuracy of the proposed method is verified in the center line extraction of grid laser cross-region.

In an on-line laser scanning system, the laser is projected from the laser onto the surface of an object to form a curved laser stripe.After capturing images with a camera, the center of the laser stripe is extracted to obtain the two-dimensional coordinates of the laser center line, which is converted into the three-dimensional point cloud under the world coordinate system.At present, most systems use single-line lasers, and the error in the translation system will be accumulated, which will reduce the measurement accuracy [11].In order to increase the measurement range and reduce the error caused by the outside world, many researchers now begin to use multi-line and grid-line laser systems [12,13].Multi-line and grid laser center line extraction becomes a key step.
There are many methods to extract fringe centers, such as the extremum method, threshold method, gray centroid method, etc.The gray barycenter method is to calculate the gray barycenter point along the axis of the laser fringe of the section of the center point position [14].The center line of the laser stripe can be obtained by representing the centroid point of each row or column.This method reduces the error caused by the uneven gray level of the strip, but it is easy to be affected by noise.When the image has bright spots, there will be a large error in the result.Moreover, when there is a cross part between two lasers, the two lasers will seriously interfere with each other during the process of extracting the center line, especially in the cross area.The direction template method is to construct four direction templates and uses four templates for image convolution operation, and the maximum response is the center point [15].However, the accuracy is often limited by the direction of the template, and the complex texture of the surface will make the fringe offset in more directions.The basic idea of the Steger algorithm is to obtain the normal direction of the optical strip through Hessian matrix, and then calculate the subpixel position of the center of the laser fringe by using Taylor polynomial expansion [16].This method has high accuracy and good robustness, but it has a huge amount of computation.Improper selection of convolution kernel size in Gaussian filtering will result in distortion of image information [17].When the center line of two non-vertical cross laser lines is obtained, there will be obvious dislocation deviation in the cross part.
According to the characteristics of laser fringe, He et al [18] proposed an algorithm to determine the direction of laser fringe by using the seed points of laser fringe.He et al [19] proposed an improved laser stripe extraction algorithm combining the center of mass method and Hessian matrix, which greatly reduced noise interference and improved efficiency.Li et al [20] proposed a center point and attitude detection method for cross-laser images based on Gaussian fitting and least square fitting, which has greatly improved the accuracy and processing speed.By improving the overall fitting accuracy of an optical plane, Li et al [21] made the measurement results of a multi-line laser scanning system more accurate.Yang et al [22] refined the laser stripe and calculated the cross center based on the characteristic that the sum of the pixels in the eight neighboring fields of the cross-line laser center is four, which improved the measurement accuracy.These methods can achieve specific multi-line or cross-laser fringe extraction to a certain extent, but the accuracy and continuity of the area around the intersection of the center line in the cross-region are not ideal, and the initial laser Angle will also bring a certain degree of influence on the extraction of the center line.
The methods of extracting the center line of laser fringe from a single-line, multi-line, and orthogonal grid have been mature.However, some methods, such as the traditional Steger method and skeleton extraction method, will have an obvious deviation in the cross region when the non-orthogonal laser lines exist, as shown in figure 1 below.The area where the two lasers cross can be viewed as a diamond shape.The offset caused by the traditional Steger method is because after Gaussian filtering, the gray value of the region near the acute corner of the diamond is higher than that near the obtuse corner.As a result, the extreme value of the gray value is mostly distributed near the long diagonal of the diamond, which is located in the red line of figure 1(d).The rectangular window of Gaussian filtering can exacerbate this phenomenon, as shown in figure 1(d).The range from blue to yellow in figure 1(e) represents an increase in gray values.The offset caused by the skeleton extraction method is because the corrosion speed in the short diagonal direction is faster than that in the long diagonal direction in the process of morphological corrosion.Therefore, when there is only one pixel in the short diagonal direction, the long diagonal direction has not been corroded to a single pixel, and it will continue to corrode at both ends, resulting in a shift in the long diagonal direction.So now we need a method that can handle non -orthogonal cross laser center line extraction.
In order to solve these problems, this paper proposes a cross-laser center line extraction algorithm based on image affine and the Steger method.The slope of the two laser lines is obtained by the Hofer transform of the image.The affine matrix of the image is obtained by the two slopes so that the two laser lines become vertical.The orthogonal laser image was rotated 90°, and the laser center line of the two images was extracted by the Steger method.The results of the Steger's method are transformed by an affine matrix, and finally the coordinates of the center line are obtained by union.
The main work of this paper is to improve the method of extracting the center line of the grid laser cross region.The slope of the laser edge line is used for the image affine.Then the center line is extracted by an orthogonal grid laser.Finally, the inverse transformation and the union are used to get the result.By comparing with the traditional method, it is proved that this method achieves good results.
The chapter arrangement of this paper is as follows: the second chapter is aimed at the problems and proposed methods.The third chapter introduces the improved algorithm.The fourth chapter is about experimental results and analysis.Chapter Five gives the conclusion.

Motivation and approach
When the existing methods were used to extract the center line of the cross laser, it was found that the Steger method almost did not have deviation for the extraction results of the center line of the orthogonal laser line placed vertically and horizontally, but there would be a discontinuity in the cross area, as shown in figure 1(c).The reason for the discontinuity is that the horizontal and vertical directions of the Gaussian convolution kernel cannot match the laser line widths in the horizontal and vertical directions.Therefore, if we can convert a nonorthogonal laser grid into an orthogonal laser grid, and the size of the Gaussian convolution kernel matches the laser line widths in both directions well.In this way, the laser center line can be obtained correctly.
Suppose there are two non-orthogonal intersecting lines a and b in the space, as shown in figure 2. The projection of the two lines on the plane P is a1 and b1 respectively, and when a1 and b1 are orthogonal, it can be shown that the two lines can change from non-orthogonal to orthogonal relationship by space transformation.
Based on this, the main ideas of this paper are: Firstly, the sub-images are sequentially intercepted on the original image, and the slopes of laser lines in two directions are obtained.Then, image affine was performed by slope conversion to obtain orthogonal laser lines, which were rotated 90°and extracted by the Steger method together with the original results.The image is transformed by inverse affine and clipped.Finally, the two results are combined to avoid the discontinuity caused by the Gaussian kernel mismatch, and the centerline extraction results of the original image are obtained.The method flow chart is shown in figure 3.

Centerline extraction of the grid laser stripe
3.1.The method for identifying the size of the sub-image When determining the size of the sub-image, the optimal state is that each sub-image contains laser cross regions, and the image should contain laser lines in both directions, as shown in the red box on the left of figure 4(a).In this way, when scanning the object, it will not be impossible to obtain the laser line slope because of the lack of laser lines in both directions in the sub-image.Since this area is difficult to identify in the image, it can be replaced by the black area wrapped by the laser line when calculating the size of the sub-image, as shown in the red box on the right of figure 4(a).
The black region can be identified by using inverse color transformation and connected domain detection.Calculate the average horizontal width and vertical height of the identified black area to obtain the size of the sub-image to be cut.The recognition results are partially shown in figure 4(b).The white lines in the figure are the laser edge lines, and the red box is the recognition results.

Image affine based on slope
The slope is obtained by fitting all the points in the image, which greatly increases the calculation amount of the algorithm.In this paper, the Hough transform is applied to edge points, and the slope is obtained in parameter space.In this section, we propose a method of obtaining an orthogonal grid laser based on the Hough transform and image affine.
Hough transform uses the duality between point and line to map the discrete pixels on the straight line in image space into curves in Hough space through parametric equations.The intersection points of several curves in the Hough space are mapped as parameters of the linear equation to straight lines in the image space.In the parameter space representation, the image coordinate space of the common-line points transform to the parameter space, in the parameter space intersects at the same point, and the obtained r and q are the parameters of the line, as shown in figure 5.In parameter space representation, conversion is carried out according to the following formula: In the formula, ( ) x y , i i and ( ) x y , j j are two points on the line, corresponding to the two curves r i and r j in the parameter space.The intersection of these two curves ( ) r q , 0 0 is the original line, q 0 is the slope of the line.The image after the Hough transformation is shown in figure 6.If the points in the original coordinate system are collinear or nearly collinear, most curves within the parameter space will pass through one point.In the parameter space, q of the point through which the curve passes the most times is the inclination angle of the first line.When the threshold is set according to the first inclination angle, q that is more different from the first point and passes through the curve more times is the inclination angle of the second line.The slope of the two lines can be obtained from the angle of inclination of the two lines.
Image affine transformation refers to the transformation of an image from an initial vector space to another vector space according to linear transformation and translation in geometric space.In general, rotation plus stretching is an affine transformation of the image.
As shown in figure 7, there are two lines in the original coordinate system with different directions, and they are translated so that they intersect at the origin.Now ( ) k 1, 1 and ( ) k 1, 2 are the direction vectors of the two lines.Using two direction vectors according to the following formula to carry out space affine transformation, make them orthogonal relationships, and get two orthogonal lines.The affine formula is as follows: k 1 and k 2 are the slopes of the two lines.The matrix is composed of m ij is an affine matrix.Equation (4) is the multiplication of the affine matrix and the direction vector of two lines in the coordinate system before  transformation, to obtain the orthogonal direction vector of the new coordinate system.Substituting equations ( 5) and (6) into equation (4) can keep the scale before and after transformation basically unchanged.The direction vector is determined by the Hough transform, and then the direction vector is substituted into the image affine to get nearly orthogonal cross laser lines.The specific steps are as follows: Step 1. Extract the edge of the image and save the coordinates of the edge points.
Step 2. Carry out the Hough transformation on the edge points, find out the point with the most times passed by the curve in the image under the transformed parametric coordinate system, and use the ordinate as the inclination angle of the first line.Continue to find the point where the number of passes of the curve is slightly lower than the previous point and the ordinate is quite different from the previous point.The ordinate is used as the inclination angle of the second line.
Step 3. Use the inclination angle to find the slope and get the direction vector of the line.An affine matrix is obtained by transforming the direction vector of a line into a vector along the coordinate axis.
Step 4. Image affine is performed on the original image according to the affine matrix to obtain the nearly orthogonal cross-laser line image.
After processing, the laser lines in the image are nearly orthogonal at this time, as shown in figure 8.

The completion method for discontinuities
Steger algorithm is based on Hessian matrix calculation, which can realize laser center sub-pixel precision positioning.The calculation steps are as follows: Step 1. Perform Gaussian filtering on the image.
Step 2. Calculate the Hessian matrix for each point with the following formula:

xx xy xy yy
Where r xx represents the second partial derivative of the image along the x direction, and other parameters are similar.
Step 3. Calculate the eigenvector corresponding to the eigenvalue of the Hessian matrix to obtain the normal direction of the laser line.Step 4. Perform Taylor expansion in the normal direction to obtain the subpixel position of the center point.In the process of image processing by the Steger method, firstly, it should be processed by Gaussian filtering, as shown in figure 9(a).The size of the Gaussian convolution kernel is usually divided by 3 by laser line width in the calculation of single-line and multi-line laser center lines.However, after the affine of the image, the laser line widths in the two directions are no longer equal.If the Steger method center extraction is carried out on the image at this time, the discontinuity mentioned above will appear.This is because in the process of Gaussian filtering, the size of the convolution kernel in both directions does not match the width of the laser line in both directions, resulting in one line covering the other line in the filtering result.With image pixel coordinates as the base and gray value as the Z-axis, the analysis from a three-dimensional perspective will be more vivid, as shown in figure 9(b).On the basis of this result, discontinuity appears in the process of calculating the maximum value of the gray value function in the cross region.
In order to avoid discontinuity, the size of the Gaussian convolution kernel in the horizontal and vertical directions should be consistent with the width of the corresponding laser line.However, the calculation and complexity will be greatly increased to obtain the corresponding laser line width.Therefore, the image is rotated 90°to perform the same filtering, so that the laser line that was previously covered in the bottom layer is covered over another line, as shown in figure 9(c).After processing by the Steger method, the two results can be combined to eliminate the discontinuity of the center line.

The correction method for offset
In the previous steps, in order to obtain the slope of the laser line, the original image was divided into several subimages for processing.During the stitching of sub-image processing results, the center line near the edge between sub-images will be offset, as shown in figure 10(a).This offset occurs at the edge of the sub-image, as shown in figure 10(b).
By simulating the intersection of laser line and image edge, four cases are summarized, as shown in figures 11(a)-(d).The red line in figure 11 is the offset laser center line, the blue line is the edge of the image, h is the laser line width, and L is the length to be cropped.These are the cases where the upper and lower edges intersect laser lines with slopes greater than zero and less than zero, respectively, and the left and right edges intersect laser lines with slopes greater than zero and less than zero, respectively.After geometric reasoning and summary, the clipping length can be obtained by the following formula: k is the slope of the laser line.The two cases of the upper and lower edges can be obtained by formula (8), and the two cases of the left and right edges can be obtained by formula (9).When calculating the cut length of the upper and lower edges, two slopes are substituted into equation (8) to get two cut lengths.In the case where the cross part of the laser line is at the edge, the two clipping lengths need to be compared with the vertical widths of the two laser lines, and the maximum value is the length of the upper and lower edges to be clipped.The two clipping lengths obtained by substituting the formula (9) are compared with the horizontal widths of the two laser lines, and the maximum value is the length of the left and right edges to be clipped.The above is the case where there is no laser crossing area at the edge of the image.When the laser cross area appears at the edge of the image, another offset is caused, as shown in figure 10(c).This is due to the incomplete cross area.
The partial absence of the cross area will cause the error of center line extraction, as shown in figure 10(c).Therefore, the laser cross area is approximately regarded as a diamond shape with equal sides, as shown in figure 12(a), and the length of the cut is the maximum projected length of the diagonal in the horizontal and     figure 12(c) represents two slopes one greater than zero and the other less than zero.Through the geometric relationship can be obtained: l is the length of the sides of an equilateral diamond.a and b are the inclination angles of the laser lines with slopes k 1 and k , 2 Lx and Ly are the maximum projected lengths of the diagonal lines in the horizontal and vertical directions, and h is the width of the laser lines.When the inclination Angle of the laser is obtained, the range should be 0°to 180°.Formula (11) ( 12) is the clipping length for the horizontal and vertical directions when the two slopes are of the same sign.Formula (11) ( 13) is the clipping length for the horizontal and vertical directions when the two slopes are different signs, where k 1 is the slope greater than 0.
Because the width of the laser cross area whether in the horizontal or vertical direction is always greater than the width of the laser line.Therefore, the width of the laser line in the horizontal and vertical directions is set as the threshold.The laser width at the edge is obtained by traversing the gray value of the pixels in the outermost circle of the image.If the width is larger than the threshold value, it is the laser cross area.The clipping length is calculated by the formula (10), ( 11), ( 12) and (13).If the width is less than the threshold value, formulas (8) and (9) are used for calculation.
In order to make the stitching result without missing parts, the overlap area of adjacent sub-images should always be larger than the crop area, so that there will be no missing parts when stitching.In this way, the complete non-orthogonal grid laser centerline extraction results can be obtained.

Experimental setting
The experimental system adopts a planar array camera of MER-503-36U3M of Daheng image.The resolution is 2448 × 2048 pixels.The matching lens is NH-3519-5M-C2/3X of Daheng image, and the laser is a two-channel laser of HOLON3D.Place the laser, camera, and shifting table in a non-collinear position.The camera lens is calibrated by the calibration plate, and the internal parameters, external parameters and transformation matrix are obtained.The laser projects the grid laser onto the surface of the measured object, forming curved laser streaks.The laser fringe image is taken by the camera and processed by the algorithm to get the center line.The experimental device is shown in figure 13.
Experimental results show that the proposed method is effective for extracting the center of non-orthogonal laser fringe and obtaining 3D point cloud.In the experiment, the proposed method is compared with the traditional gray centroid method and the Steger method, which proves the superiority of the proposed method in the center extraction of intersecting parts.

Experimental result
Figure 14 is the result of multiple methods of processing the same laser scanning local image.It can be seen from the results that in the region approximated as a parallelogram by the intersection of laser lines in two directions, the results obtained by the traditional Steger method, skeleton extraction method and gray centroid method all have obvious deviation, and the direction of deviation is along the long diagonal.
The result of the traditional Steger method is deviate to the long diagonal because the horizontal size of the Gaussian convolution kernel is too large and the size selection in the vertical direction is too small during the Gaussian filtering.The reason for the deviation of the result of the skeleton extraction algorithm is that in the process of morphological corrosion, the direction of the short diagonal is more easily corroded to a pixel than that of the long diagonal.So the image is identified as two parts, and the long diagonal is the line connecting the two parts.The reason for the deviation of the gray centroid extraction method is that the algorithm will identify the diagonal direction as the width of the laser during the row traversal or column traversal.The center line coordinates obtained will be offset to the area near the two diagonals according to row traversal or column traversal.
Figures 15(a)-(d) are the results of the original laser fringe image, skeleton extraction method, traditional Steger method, and the method presented in this paper.It can be seen that in the process of center line extraction, the first two methods can not effectively extract the center line of the cross area, and there are many wrong center points.In the cross region, the results of the skeleton extraction method and the traditional Steger method will be deviate in the diagonal direction.The figure shows that the results of the proposed method are better than those of the skeleton extraction method and the traditional Steger method for cross regions.
Figure 16 is the result of zooming in on the cross area.It can be seen that the results obtained by the method in this paper have almost no deviation for the edge of the laser line end region.
Figure 17 shows the error calculation results for a laser center line in the cross region.Figures 17(a) and (b) show the pixel errors of the skeleton extraction method and the traditional Steger method.It can be clearly seen from the image that the offset of the center line in the cross region is consistent with the offset distance of a similar sine function in figure 1.It can be seen table 1 that the proposed method is superior to the skeleton extraction method and the traditional Steger method in terms of maximum error and average error.By comparing the results, it is proved that the proposed method has high accuracy for the center line extraction in the cross region.

Acquisition of three-dimensional point cloud
Figure 18 shows the three-dimensional point cloud results of the center line extracted by various methods after scanning the adapter plate.Figure 19 shows the three-dimensional point cloud results of the center line extracted by various methods after scanning the upper part of the hexagon wrench.It can be seen from the two figures that the proposed method is superior to the skeleton extraction method and the traditional Steger method in point cloud density and abnormal burrs.Since the hex wrench is an irregular object, there is no precise specific size parameter, so only the point cloud results of the adapter plate are analyzed and evaluated.Table 2 shows the data related to the point cloud results of the adapter plate.Table 3 shows the point cloud related data of a single crossing area where the actual height of the adapter plate is 5.86mm.
It can be seen from the data in table 2 that under different relative error thresholds, the number of points obtained by the proposed method is more than that obtained by the skeleton extraction method and the traditional Steger method.From the point of view of the coefficient of variation, the stability of the results obtained by the proposed method is better than the other two methods.When the relative error threshold is enlarged, the more measurement data is retained, which conforms to the measurement data principle.From the perspective of average height, the results obtained by the method proposed in this paper are most close to the real value, and the stability is much higher than that of the skeleton extraction method and the traditional Steger method.

Conclusion
To achieve non-orthogonal grid laser centerline extraction, we put forward a comprehensive method based on Steger and other methods which are applied in many different fields.And did experiment verification to confirm     its accuracy and stability.The center line extraction, discontinuity phenomenon of non-orthogonal grid laser and edge deviation phenomenon of sub images could be realized by the method.The proposed method has the following advantages: (a) Based on Steger's method, this method uses an image affine to transform the image into an orthogonal grid laser fringe, which has a good effect on the extraction of the center line of the cross region.
(b) In view of the discontinuity of the center line in the cross region, the method uses the principle of Gaussian filtering to rotate the image 90°and process the original image together.The complete laser center line is obtained by the union of the obtained results.
(c) In order to solve the center line deviation caused by image segmentation, a method for calculating the length of edge clipping using the slope of laser line is proposed.The edges of sub-images are clipped to avoid the centerline deviation caused by image segmentation.
By combining existing methods with proper modification and optimization, our method implements unique functionality different than before.It's worth to study existing methods and improve them to find out more possibility of application in different situation.
In addition, the image affine based on the slope is used in this experiment, which may not be ideal for objects with large surface curvature or drastic changes.And in the process of processing, the algorithm needs to be optimized, and there is still room for improvement in the processing speed.Therefore, in future work, improvements can be made in slope acquisition, filtering window and 3D reconstruction accuracy to achieve better results.

Figure 1 .
Figure 1.The problem of center line extraction in cross regions.(a) Deviation of Steger method results.(b) Deviation of the results of the skeleton extraction method.(c) Centerline discontinuity.(d) Long diagonal position.(e) The result of Gaussian filtering.

Figure 2 .
Figure 2. Two lines change from non-orthogonal to orthogonal using projection.

Figure 3 .
Figure 3. Flow diagram of the proposed method.

Figure 4 .
Figure 4. Size recognition of sub-images.(a) Substitution of identification areas.(b) Results of identification.

Figure 5 .
Figure 5. Diagram of Hough transformation.(a) the original coordinate system.(b) parametric coordinate system.

Figure 6 .
Figure 6.Hough transform results in parameter space.

Figure 7 .
Figure 7. Image affine based on slope.(a) Two intersecting lines in the original coordinate system.(b) Direction vectors of the two lines.(c) A new coordinate system based on two directional vectors.

Figure 9 .
Figure 9.The results of filtering.(a) Laser cross part of the original image.(b) The result of Gaussian filtering in which the convolution kernel size is wider in the Y direction.(c) The result of Gaussian filtering by rotating the image 90°.
Figure 12(b)  shows both slopes greater or less than zero at the same time.Because the calculation process is the same, the picture only shows two slopes greater than zero at the same time.And

Figure 10 .
Figure 10.Center line deviation at the edge.(a) Centerline deviation when two sub-images are joined.(b) Center line deviation at the lower right edge of a sub-image.(c) Centerline deviation when the laser crossing area is at the edge of the image.

Figure 11 .
Figure 11.Center line edge deviation.(a) The deviation at the lower edge of the laser line whose slope is less than 0. (b) The deviation at the lower edge of the laser line whose slope is greater than 0. (c) The deviation at the left edge of the laser line whose slope is less than 0. (d) The deviation at the left edge of the laser line with a slope greater than 0.

Figure 12 .
Figure 12.Laser cross area.(a) Diamond laser cross area.(b) Laser crossing area when two slopes are the same.(c) Laser crossing area when two slopes are different.

Figure 13 .
Figure 13.Schematic diagram of experimental platform including laser, camera and translation platform.

Figure 14 .
Figure 14.(a) Part of the original image.(b) Results of traditional Steger method.(c) Results of skeleton extraction method.(d) Results of gray centroid method.(e) The method of this paper.

Figure 15 .
Figure 15.Laser centerline extraction results.(a) The original drawing.(b) Results of skeleton extraction method.(c) Results of traditional Steger method.(d) Results of this method.

Figure 16 .
Figure 16.A partial enlargement of figure 13.(a) The original drawing.(b) Results of skeleton extraction method.(c) Results of traditional Steger method.(d) Results of this method.

Figure 17 .
Figure 17.Pixel errors of each method in the cross region.(a) Pixel error of skeleton extraction method.(b) Pixel error of traditional Steger method.(c) Pixel error of the proposed method.

Figure 18 .
Figure 18.3D point cloud results.(a) Adapter plate (b) Results of skeleton extraction method.(c) Results of traditional Steger method.(d) Results of the method in this paper.

Figure 19 .
Figure 19.3D point cloud results.(a) Hex wrench.(b) Results of skeleton extraction method.(c) Results of traditional Steger method.(d) Results of the method in this paper.

Table 1 .
Maximum and average errors of each method.

Table 2 .
Related data about the point clouds of the adapter plate.

Table 3 .
Related data about a single cross region in the point cloud results.