Unmanned Sorting Site Combined with Path Planning and Barcode Identification

At present, the existing express sorting in the market has low accuracy and cannot be solved from manual handling. In addition, the management of express delivery sites is confusing, and the phenomenon of lost shipments is also endless. Regarding the issues above, we have proposed unmanned sorting site combining path planning and barcode identification. The site uses STM32 as the main control chip to realize barcode recognition and robotic arm control. Among them, barcode recognition uses calibration, image pre-processing and other related OpenMV technologies to realize fast identification, classification of express barcodes and obtain the location information of express shipments. The control of the robotic arm is achieved by using the D-H parameter method to locate each node of the robotic arm. And we are using introducing fuzzy adaptive PID algorithm to improve the control precision after path planning. After a large amount of experimental data analysis and comparison, the final conclusion is that the system can maintain high bar code recognition and fast picking accuracy under different conditions of different objects. And it can make up for many shortcomings of traditional express delivery sites and has certain market value.et value.


Introduction
With the rising of labor cost and the increasing of production demand, the mechanical arm is more and more involved in the industrial production process. At the same time, machine vision technology has been widely used in industrial production, which not only improves the production efficiency, but also realizes the production intellectualization, and makes a solid step to the age of the interconnection of everything.
Sorting operation is an indispensable stage in the classification of express delivery. However, the traditional sorting mode of stations is manual sorting. This kind of work has high repeatability and great labor intensity. After a long period of work, workers often make mistakes in sorting. Therefore, it inevitably brings some unnecessary economic losses.
In addition to the express delivery cabinet, the last step before delivery to the recipient still lack of intelligent sorting unmanned system [1]. The existing self-lifting cabinet not only needs manpower to put the express delivery into the express delivery cabinet [2], but also the occupied express delivery cabinet will slow down the subsequent express delivery into the cabinet when the express delivery can't take away in time.
In addition, through the research on the development of express industry at home and abroad, we find that there is less research on the use of new technology and new methods [3] for express terminal (express sorting station) at present , intelligent delivery sites are still rare.
In order to solve these problems, the application of the robotic arm with machine vision in express sorting system [4] can not only effectively reduce the economic loss caused by human operation, but also maximize the utilization ratio of the site space. Based on the above advantages and related theoretical technology [5], this paper designs a robotic arm unmanned express station based on machine vision, which uses STM32 as the main control chip At the same time, OpenMV module is used to quickly identify [6] and classify [7] the barcode of shipment, and the control of the robotic arm is optimized by combining the D-H parameter method and the fuzzy self-adaptive PID Algorithm. It is of great significance for the development of express industry to provide a set of feasible scheme for realizing the unmanned express station.

System architecture
The unmanned site [8] is composed of STM32F107VCT6 control board, robotic arm module, Machine Vision Module, Database, Short Message Module, Receiving Window and delivery mechanism. The system structure figure 1 is shown below. The system uses STM32F107 control board to analyze the information from each module and coordinate the communication between each module. The machine vision module identifies the barcode and obtains the corresponding information. The database is used to store the specific information of the express. The robotic arm is responsible for grabbing the express and sorting the specific position. In addition, the upper computer software on the Receiving Window is easy to operate for the person to input the pick-up information [9] to realize the interaction of the system.

Camera calibration
In order to accurately collect the specific location of the shipment in the warehouse to assist the robotic arm to make the correct sorting action, the machine vision module needs to be calibrated. This calibration process is actually the transformation process of the coordinate system.
The mathematical transformation formula of each coordinate is as follows: Among them, is the pixel coordinate system of the final recognition target, , , 0 , 0 are the internal parameters of the camera, R is the attitude of the camera relative to the world coordinate system, and T is the position of the camera relative to the world coordinate system, , , are the world coordinate system positions of the shipment. According to the placement of the camera, the calibration can be divided into Eye-in-Hand and Eyeto-Hand. Considering that the Eye-to-Hand solution is easy to be blocked by the robotic arm, we will adopt the former and fix the camera at the end of the robotic arm, so that both can remain relatively stationary, and the camera will move together with the end of the robotic arm to improve stability, get a wider range of sight and have better flexibility. In addition, we will place two cameras on the robotic arm for multi-angle calibration to improve the accuracy of the final coordinate system position.

Object recognition
In order to obtain the shape, size and physical characteristics of the shipment and enable the robotic arm to quickly recognize and complete the corresponding action, image processing in the later stage is particularly important. In addition, since the machine vision module will inevitably be affected by external interference during the shooting process and problems of system hardware will also affect the exposure time, and then the captured target picture will be mixed with a lot of noise, which will cause trouble for the later data matching, so it is necessary to preprocess the image in the early stage of object recognition.
(1) First, taking shipment one as an example, the maximum value graying method f(i, j) = max (R(i, j), (i, j), (i, j)) is used to weight the RGB of the image. The effect picture obtained not only obviously improves the contrast of the image, leaving more useful information, but also removing the unnecessary color information.
(2) Second, the Gaussian filtering method is used to remove the noise from the normal distribution in the image. The two-dimensional formula is as follows: This method can eliminate some isolated noise and make the outer contour of the image smoother.
(3) After the image grayscale and filtering, the express and background need to be separated to achieve more targeted identification. Therefore, the system will use the maximum inter-class variance method (image binarization) to do image segmentation to extract the region of interest. The idea is to select the appropriate segmentation threshold, and then divide the target's grayscale distribution map into two areas, and change the grayscale values to the maximum and minimum values, respectively. The image mapping relationship before and after segmentation is as follows: (4) Fourth, we need to use image edge detection. The binarized image contour can reflect the sharp change of gray value in the whole image, which is a combination of a set of critical points. Using this set to construct a differential operator can achieve edge detection. The extraction methods are mainly Roberts, Prewitt, Sobel and Canny algorithm. The Sobel algorithm is simpler and faster than other algorithms. It only needs to use two 3 × 3 convolution templates horizontally and vertically. This method is also often used in systems with low hardware requirements and high real-time requirements. Its formula is as follows: Θ=arc tan ( ) In view of the shortcomings of the traditional Sobel algorithm above, we propose an improved method, which is to add a template in the diagonal direction on the basis of the horizontal and vertical directions. Eventually form a convolution template in eight directions (0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°). The improved Sobel algorithm can realize large-area orbital convolution, which makes the algorithm more accurate for feature extraction in oblique directions.
(5) After the above operations, a target image separated from the background can be output. Because other target objects may be mixed around the target object, the center coordinates of the target object need to be determined. In addition, considering that the packages of express parcels are basically unique colors and the projection method is not computationally intensive, it can improve the real-time recognition of the system. Therefore, the system will use the projection method to determine the twodimensional coordinates of the target, and then use formula (1) to obtain Its real spatial coordinates.
(6) After the above operations, a set of space coordinates of the shipment can be obtained, but in order to improve the accuracy of the coordinates, the system will use two cameras to collect 10 consecutive coordinate values in parallel, a total of 20 sets of coordinate values, and weighted , And finally get the statistical coordinate value.

Bar code recognition
After the previous image processing, the size and position coordinates of the shipment can be determined. The next step is to identify the barcode on the shipment package. Although the barcode has a clear texture and its pattern characteristics are also different from other objects, our vision module does not take pictures face to face, the proportion of the barcode in its final imaging picture is very small, and the resolution may not be very High. Even sometimes, the wrinkle problem of express packages can affect the image quality of barcodes. Therefore, accurate positioning and error correction analysis of barcodes are required.
Step 1: After processing the input image, the system uses the ratio of the horizontal and vertical gray values of the barcode to determine its approximate position in the picture.
Step 2: After obtaining the approximate position of the barcode, it uses the MSER (Maximum Stable Value Area) algorithm to detect the angle of the barcode, and then uses reasonable filtering to enhance the effect and adjust the corresponding angle to obtain more complete barcode information.
Step 3: The next step is to extract the barcode features. Based on the previous two steps, the system will use the FRCNN object detection algorithm, a deep convolutional network, to train a model specifically for barcodes, and let it continuously keep self-learning (candidate box search) to accurately classify which part is Bar code[10] [10] .
Step 4: After the barcode position is determined, the barcode needs to be aligned. The system not only needs to calculate the midpoint position of the barcode area, but also needs to calculate the color variance of the rectangular border points, and then rotate the angle. In addition, in order to ensure a better smoothing effect in the final imaging, it will interpolate the gray levels of the four adjacent pixels.
Step 5: The system corrects the corrected barcode. According to the coding rules of the barcode, it will automatically allocate a few more adjacent pixels when identifying this determined vertical line, allowing the system to identify and obtain the information on the shipment.

Robotic Arm Path Planning
The robotic arm is a multi-degree-of-freedom mechanical structure that composed of a series of movable or rotatable joints and connecting rods, and the overall application function is determined by the end effector [11]. The system uses six steering gears as the rotating joints of the robotic arm, and uses a mechanical clip as the end effector to grip the express delivery. Before the operation of the robotic arm, the path planning algorithm of the robotic arm needs to be run in the STM32F107, which is to calibrate the camera to the spatial coordinates of the target shipment ( ) [12]. The inverse kinematics solution was used to obtain the deflection angles of the six servos [13], so as to control the robotic arm to grab the express [14]. First, a coordinate system is established for each joint node. The bottom of the arm is selected as coordinate system 0. From bottom to top, the sequence is 0, 1, 2, 3, 4, 5, 6. And the end is coordinate system 6, with a total of 7 coordinate systems.
After that, Let −1 be a transformation matrix from the coordinate system to the coordinate system. The joint transformation matrix is as follows: The D-H parameters of the robotic arm used in the system are shown in Table 1 Table 1 D-H parameters Knowing the space coordinates of the ends, we can calculate the rotation matrix formula T 6 0 . In addition (p x p y p z ) is the space coordinate of the shipment (X w Y w Z w ).According to the matrix algebra operation, we can still get 1 , 2 , 3 , 4 , 5 , 6 . The formula is as follows: Although the space coordinates at the end of the robotic arm are locked, the robotic arm can have different attitudes. The has multi-group solution, so it is necessary to get the only set of inverse solutions according to the actual situation and certain conditions. When the robotic arm of this system is running, it is necessary to ensure that all joints cannot exceed its maximum working range, and that each joint must have a minimum amount of movement during movement.

Robotic Arm Control
After the trajectory of the robotic arm (six degrees of freedom parameters ) has been obtained, the angular displacement of the six servos needs to be adjusted. This process is very important. If the accuracy cannot be controlled, the actual results after implementation will deviate greatly from the theoretical estimates. There are two main deviations in the movement process: a. Limited by the accuracy of the rotation angle of the steering gear, continuous accuracy errors may produce large erroneous results when the robotic arm is continuously executed.
b. The output of the control system can't converge quickly, and serious lag problems will occur. In response to the above problems, the control mechanism of the robotic arm uses STM32 as the control core of the robotic arm. And the fuzzy adaptive PID control algorithm [15] was introduced, which combined with the actuator's angle sensor to form a closed-loop control system.
The fuzzy adaptive PID algorithm is developed on the basis of the PID algorithm. It is a kind of automatic control decision with high precision and low complexity. It mainly consists of a PID controller and a fuzzy controller. It takes the error C as an input, and uses the fuzzy control rules of the fuzzy controller to adjust the parameters , , .The algorithm expression is as follows: 、 、 are the coefficient of proportion P, integral I, and derivative D. The error amount C(t) is the difference between the input amount i (t) and the feedback amount β i (t) of the sensor. Entering this closed-loop system, the input error amount is 0, the output amount remains stable, and the control time is reduced.

Experimental Results and analysis
In order to test the efficiency of the system, an unmanned express station model is built, as shown in figure 2. Figure 2 overall effect of the system The calibration experiment uses Intel quad-core processor, 8G memory computer and 640 * 480 Pixel camera as the hardware condition. And it uses MATLAB2018b for programming and makes timing after the second execution of the program.
In the process of calibration experiment, we use the camera to calibrate a total of 63 (30mm × 30mm) checkerboard in black and white between 9 cm long and 7 cm wide picture at different angles to obtain images. After that, we use CMERA Calibration Toolbox for Matlab to extract the corner information and calculate the camera's internal parameter Matrix A and distortion parameter KC. These two parameters as follows. With Formula (1) the corresponding, (X, Y, Z) can be obtained by the transformation among the world coordinate system, the camera coordinate system, the image physical coordinate system and the image Pixel coordinate system.

Bar code recognition
In the experiment of barcode recognition, we test the following objects in different positions many times, and get the corresponding recognition rate experiment results. This experiment mainly aims at the object's bar code recognition. The object Diagram and the final effect diagram are shown in the following figure. Algorithm, uploads the identified data sequence to the terminal database. According to the comparison, the sequence of numbers below the bar code on figure 3 is consistent with the sequence of numbers collected and uploaded to the terminal database by the machine vision module in figure 4.
In addition, in order to verify the accuracy of bar code extraction system, we use multi-group form to verify and comparatively analyze through the analysis of 10 to 60 courier identification rates. The data is as follows. As can be seen from table 2, the Identification rate of the machine vision module is relatively high, and can be maintained at about 90%, which provide a good foundation for the subsequent sorting operation of the robotic arm.

Grasping effect of mechanical arm
After image processing and trajectory of the robotic arm obtained by STM32F107, our system will carry out multi-group grab test for three sizes (small, medium and large). The success rate of three specifications is obtained after statistics. (Note: The dimensions of the shipment shown in table are in 3 ) Observing the data in Table 3, we can clearly see that the success rate of grabbing is higher when the number of experiments is less. When the number of grabs is 300, the success rate is significantly lower (tends to converge), but it can still be maintained at more than 70%. At the same time, it can be seen from the data that the system has the highest success rate in the medium size (40*40*40 3 )which converges to about 80% and the other two standard sizes only maintain about 75%. The comprehensive analysis shows that the configuration of all the system devices (software and hardware) is the best for the performance of medium-sized item, and the other size still need to be further optimized.
In order to further verify the success rate of the system, we analyze the different objects by using the data graph, as shown in the following figure. The data are from experiments in 5 to 30 clips, which are as follows.  Figure 5 we can conclude that the success rate of the system will change with the increase of the number of experiments. The success rate of the robotic arm in clamping objects of various sizes fluctuates in different degrees with the times of clamping objects. In general, the average success rate of clamping can be maintained at about 75%.

Conclusion
In this system, STM32 is used as the main control chip and FRCNN algorithm is used to train the barcode recognition and build a specific data model. The D-H parameter method of the robotic arm is used to locate the nodes of the robotic arm. The fuzzy self-adaptive PID algorithm is introduced to improve the control accuracy after the path planning and realize the adaptive path correction of the robotic arm.
The experimental results show that the data obtained is basically consistent with the expectation. And the system can keep the bar code recognition rate at about 90% and the success rate of grabbing about 75%. It has higher accuracy and practicability, but it needs to be improved in the whole time-consuming aspect, which will be the direction of our follow-up research.