Research and implementation of yield recognition of Citrus reticulata based on target detection

In order to effectively assess the yield of Citrus. This paper proposed a scheme to identify and calculate the size of the Citrus objects on the tree in order to achieve yield prediction which is based on the target detection technology and the principle of insinuation geometry. The results of the experiments show that, this scheme has good detection effect on the whole Citrustree or local multiple fruit on the tree in the field orchard whose correct rate of identification can reach 90% and recognition rate is as low as 1.006 frame per second. After manual measurement verification, the conFiguration achieved the accuracy of 80%. This paper is of great significance in the practicality of fast fruit counting, and the difficult problem of selecting Citrus, which consumes a supply of manpower, is solved preliminary.


Introduction
Forecasting the yield of Citrus is the condition of fine management of Citrus production. At the same time, forecasting the yield also plays an important role in the pre-sale transaction, because the fruit buyers can know how much Citrus the region can provide to them through the yield forecasting method, so as to make a more scientific and reasonable procurement plan. In the past, the yield of Citrus in a certain area was estimated by manual visual estimation or fruit picking statistics, but this method is timeconsuming and labor-consuming. In this paper, target detection technology based on deep learning and the principle of projective geometry are proposed. The scheme allows to automatically count the number of Citrus trees and calculate the single fruit size by uploading several photos of Citrus trees, and realize the yield estimation by combining the planting Mu and planting density.

Related research
According to the analysis of the influencing factors of yield prediction, the number of Mu and planting density are generally known and easy to know. The key of yield prediction is to accurately count the number of Citrus and the weight of single fruit. Therefore, the two major problems to be solved in this study are: (1) how to count the number of Citrus; (2) how to calculate the identified size of Citrus, so as to achieve the effect of yield prediction. Through the research, we found that there is no software solution based on deep learning target detection technology to identify and measure the size of Citrus, which can reduce the labor cost and promote the transformation of traditional agriculture to smart agriculture. Moreover, many current solutions do not consider the use of reference based method to detect the size of Citrus on the basis of target detection, but use a fixed ratio It has the advantages of large limitation, strong dependence and low practicability in hardware shooting equipment.

Our Scheme
Through the above research and analysis, this paper uses the target detection technology based on deep learning to recognize the target object of Citrus tree, and combined with the software method of recognition size, calculates the real size of the target object with a fruit on the tree as the reference. The advantage of this scheme is to consider the influence of the number and size of the target object on the yield factor without relying on the hardware equipment. At the same time, combined with the principle of projective geometry and the spatial attitude analysis of Citrus , the calculation formula of the short diameter pixel size of Citrus is deduced to realize the accurate calculation of Citrus size, improve the accuracy of yield prediction, and make it more convenient to use, which is in line with the requirements of practical application Demand. The design idea of this paper can be divided into two steps. The first step is to select an appropriate training model for Citrus object recognition, and then use the geometric conversion formula of virtual short diameter and reference object information to calculate the size of Citrus object, so as to realize the prediction of yield.

Data acquisition
Before the beginning of migration training, the production of data set is the most important, which directly affects the effect of migration training. In order to simulate the natural conditions, the data collected in this paper mainly take the whole tree as the object for shooting, including different light intensity, and fruit quality, such as large and small flower skin, large and small ulcer, sunlight, fruit cracking, abnormal fruit, a total of 1000 pieces. In addition, the collected data are rotated and clipped, which makes the data set expanded to 6 times of the original. Then the processed data set is unified in size. At the same time, lableimg marking tool is used to label the images, of which 1200 are used as test set.

Model selection
There are two kinds of target detection models based on deep learning, Faster R-CNN [9] and SSD [10], which are commonly used in multi-target small object detection. In order to choose a more suitable model for Citrus detection, this paper uses transfer learning method to combine Faster R-CNN with SSD Both R-CNN and SSD pre training models are migrated to Citrus field, including the whole Citrus tree, local multi fruit and local single fruit data set. Through verification, comparison and optimization, we get a more suitable model for Citrus practical application recognition, and count the number of target objects. After model recognition, each Citrus will be framed by a rectangle to predict its approximate position, And each Citrus will have a prediction score, because the model has only one recognition type: Citrus, so this is not the focus of this paper, and this paper agrees that when the confidence of Citrus exceeds 0.5, it is the target.

Algorithm for calculating short diameter of Citrus based on rectangular shortest path finding
The size of Citrus calculated in this paper refers to the short diameter of Citrus in practical application, not the area size, because in actual production, it is the fruit growers who measure Citrus one by one by using a hand-held grading fruit diameter measuring caliper. There are many dimensions (diameter, unit: mm) on the caliper. The standard is to put Citrus into the caliper from any angle and record the minimum size that can be passed. For the size of the fruit, the premise that the size of the fruit is approximately equal to the short diameter defined in the ellipse is that the projection of the ellipse in the caliper direction (i.e. the cross section) is regarded as a regular ellipse. After getting the approximate position of the rectangular frame of each target object through target detection, we can use the four corner coordinates of the rectangular frame to calculate the size of Citrus (short diameter) through geometric conversion. There are only two cases of rectangular box, namely rectangle and square. Before that, we need to understand the characteristics of an ellipse: "two parallel lines passing through the ellipse, and the line connecting the midpoint of the section inside the ellipse passes through the center of the ellipse". According to this characteristic, we can find out the center of the ellipse by using the parallel lines intersected by two pairs of extension lines, so we can use the two sides of the rectangle. The center of 3 the rectangle is the center of the ellipse. Because the recognized Citrus may not be completely tangent to the rectangle, the threshold value of this paper is 1.2. When the ratio of the long side to the upper short side is greater than 1.2, the rectangle is considered to be a rectangle; when the ratio of the long side to the upper short side is less than or equal to 1.2 When I was young, I thought the rectangle was a square. In Figures 1 to 5, the black box represents the detection box after target detection, and the orange ellipse represents the Citrus in the ideal state. When the rectangle is a rectangle, there are only two cases of Citrus's position in the rectangle. The first case is shown in Figure 1, where | |/| | 1.2|, with ℎ ℎ | |. In the second case, as shown in Figure 2 When the rectangle is a square, 1 max ℎ , ℎ ℎ min ℎ , ℎ ℎ 1.2, and the side length x is the average of width W and height H. there are only three cases of Citrus's position in the rectangle. The first case is shown in Figure 3. At this time, the ℎ ℎ ℎ ℎ width W /2 | |/| | /2 . How can we judge whether it is the current case? The current situation can not be determined by the coordinate information of the four points and the height h width W, that is, the side length information of the square can not determine the shape of the inscribed ellipse, which is inconsistent with the situation of the rectangle, and each rectangle can determine a unique ellipse.
In the second case, as shown in Figure 4, the short diameter is equal to twice the length of the blue line, and the calculation formula of the length is obtained according to the Pythagorean theorem, as shown in Formula 1, then the ℎ 2 √2 , how to judge whether it is this case at present? As in the first case, the current situation can not be determined by the coordinate information of the four points and the height h width W. In the third case, as shown in Figure 5, the short diameter is equal to twice the length of the blue line, and the calculation formula of the blue line is the same as that in the second case, so how to judge whether the ℎ 2 √2 , is this case at present? As in the first and second cases, the current situation can not be determined by the coordinate information of four points and the height, H and width W. therefore, in this paper, when the rectangular box is determined to be square, the short diameter is the average value of the short diameter in three cases, and the value of the short diameter is shown in formula 2.
Back to the target detection, with the support of the target detection platform of tensorflow [11] open source software library, the coordinate information of the identified targets can be easily obtained by obtaining the tensor of the detection frame. Its structure is a two-dimensional array. Each row of the MEMAT 2021 Journal of Physics: Conference Series 1820 (2021) 012128 IOP Publishing doi:10.1088/1742-6596/1820/1/012128 5 array represents ymin, xmin, ymax, xmax from left to right, that is, the coordinates of the upper left corner and the lower right corner of the detection frame. They are the pixels of the original image In order to get the pixel coordinates, we need to multiply the x-coordinate by the width W and the ycoordinate by the height h (in Python, the definition of the picture coordinate axis is that the upper left corner of the picture is the coordinate origin, the right horizontal direction is the positive direction of the x-axis, and the down vertical direction is the positive direction of the y-axis). The coordinates of points a and B are calculated by using the detection box tensor The coordinates of point a * , * , and the coordinates of point B are * , * . Finally, the virtual short diameter of each Citrus can be calculated by using the coordinates of points a and B, width W and height h according to the geometric conversion.

4.1Verification and comparison of target detection models
The most important measure of model recognition accuracy is mAP (mean average principle), which is the most suitable evaluation method to test the recognition effect and performance of target detection model. Target detection is different from ordinary classification. There may be many kinds of targets in each image, and there is a certain amount of data. Using mAP can not only obtain the recognition accuracy of each type in each image, but also obtain the overall average accuracy of the model.
As shown in Figure 6 and Figure 7. After a certain number of training steps, there is a curve tending to a fixed value, which is the mAP of the model. In the case of local multi fruit test, the mAP of Fast R-CNN model trained in this paper can reach 0.98, and that of SSD model can reach 0.88. In contrast, Fast R-CNN is more outstanding in the recognition accuracy of Citrus. Figure. 6 The mAP of Faster R-CNN model Figure. 7 The mAP of SSD model In the test set with the whole tree, partial multi fruit and single fruit, and the overall occurrence of green skin disease, the average time consuming of Fast R-CNN to identify a single image is 28.111 seconds, and the frame rate is 0.0036. The average time consuming of SSD model to identify a single image is 0.994 seconds, and the frame rate is 1.006 The r-cnn model is 28 times that of the SSD model, so this paper is based on the SSD as the pre training model, transfer learning to get the target detection model which can detect Citrus.

Size statistics validation
On the basis of target detection model recognition, we can know the coordinate information of each Citrus, that is, the coordinate value of each detection frame. On this basis, we can easily calculate the short diameter pixel size of each Citrus by using the calculation formula of the Citrus size derived from projectiology. Finally, we can calculate the real size and proportion of all Citrus by selecting and providing the real size of a reference fruit. Compared with the manual measurement method, as shown in Table 1, the total number of fruits identified in this paper is divided by the total number of artificial fruits, as shown in Table 1, which also verifies the accuracy of the model in identifying Citrus ; and the calculation method with high accuracy of size is as follows: firstly, the number of fruits identified in this scheme is divided by the number of artificial fruits Then, the sum of the correct rates of all the large and small fruits is taken as the average value to get the correct rate of wokan size in this scheme, that is, the sum of the correct rates of 40-45 fruits, 45-50 fruits and 85-90 fruits in the table is divided by 10 to get the correct rate of wokan size in this scheme, about 80%.

Conclusions
Based on the target detection, this paper deduces the calculation formula of short diameter pixel size of Citrus, calculates the number and size of identified fruits, makes the yield prediction more accurate, and does not rely on hardware equipment. The results show that this scheme can well solve the problem of consuming a lot of manpower to select and count Citrus, and can be applied to the whole Citrus tree or local multi fruit orchard production environment It is of great significance in the practicability of rapid fruit counting.