Identifying of unripe Ambon and Hijau banana fruits using computer vision and extreme learning machine classifier

The unripe Indonesian cultivar bananas of ambon kuning (Ambon) and ambon hijau (Hijau) after harvesting show a very close looking, green colour, similar size and shape, even Ambon one is costly than the Hijau. Hence in this study, identification was conducted using computer vision utilizing banana finger image taken with a mobile phone camera. The feature used as a differentiating feature is the shape feature and the skin texture feature of the fruit. The shape features were then extracted using morphological descriptor and convex hull, while the texture features were extracted using local binary pattern (LBP). The extreme learning machine (ELM) classifier was used to recognize both cultivars. A total of 76 banana finger imagery data were used in 3-fold testing. The test results showed that the combined use of shape and LBP features resulted in the highest accuracy, precision and recall values more than 93%. These results showed that the combination of the two features can effectively be used to distinguish the unripe Ambon and Hijau bananas.


Introduction
Banana cultivars are one of the most popular fruits that are widely cultivated and consumed in Indonesia [1]. Ambon banana is widely consumed as fresh ripe fruit because it provides a sweet taste and soft fruit meat [2]. Based on the data on the Ministry of Agriculture website [3], although the consumption of Ambon bananas decreased from the year of 2015 to 2020, but still reached the highest rank compared to other bananas.
There are two ambon banana cultivars that often found in the market namely ambon kuning (Ambon) and ambon hijau (Hijau). After harvesting, both has almost similar in size, shape and green colour, but they can be distinguished easily when it is ripe, where the skin of Ambon banana is yellow while the Hijau banana skin remains green. However, bananas are usually harvested still in raw to maximize the shelf life of the fruit, hence it is difficult to visually distinguish these banana cultivars. Although the shape is almost the same looking, the price of Ambon banana is higher than Hijau one. Reported from the website wartasolo.com [4], the latest price of August 2021 for Ambon banana is 40,000 IDR per tiers, while Hijau banana one is 15,000 IDR per tiers. Therefore, a quickly and accurately visual method to distinguish the unripe Ambon and Hijau banana is very important to be developed. The recent researches showed that computer vision technology with image data is quite effectively used in identifying fruit types with high accuracy [5][6][7][8][9]. Study by [10] classified banana cultivars of Cavendish, Lady Finger, and Pisang Awak using the shape feature and obtained the best accuracy using classifier Bayessian network. Study by [11] performed classification using the shape feature and support vector machine (SVM) to distinguish bananas from other fruits. While in the research conducted by [12] and [13] using banana tiers in the classification process. However, these two studies were not classified cultivars or banana species but determined whether the banana tier falls into the reject category or not. Study by [13] classified banana tiers in normal and reject categories using deep learning. While [12] classifies banana tiers into 4 classes namely extra class, class I, class II and reject class using RGB colour value feature and length size of the top middle finger of the banana tier. Classification of local banana cultivars such as ambon, kapok, raja, mas, susu, tanduk, awak was carried out by [14][15][16]. These studies used colour, shape, texture features and classifier of SVM and KNN [16], GLCM texture features and KNN classifiers [14], GLCM and HOG features and SVM classifiers [15]. These three studies used some cultivars that showed a visual different shape so that it was easier to be classified. However, the highest accuracy produced [14] was 80% and still needed to be improved considering each cultivar had a high different traits.
In contrast to previous studies, the purpose of this study was to identify unripe Ambon and Hijau bananas with a high similarity of green colour, size and shape. Although these two cultivars have almost the same looking shape, really there are differences in the texture of skin surface. Therefore, this study used shape features and texture features as a description and extreme learning machine (ELM) classifier. ELM is a classification method with the concept of feedforward learning that has a much faster training time than other neural network methods [17,18]. In addition, ELM also produces higher accuracy than SVM [19,20].

Banana image data
This study used 76 datasets of banana finger imagery consisting of 43 images of Ambon bananas and 33 images of Hijau bananas that were still unripe. A tier of each cultivar banana was purchased in the traditional markets. Banana fingers were then separated from the tier and the image taken from several sides. An example of a banana finger images of both cultivars is shown in Figure 1. The camera used for shooting was the front camera of Oppo F3 mobile phone with a resolution of 16 MP and a normal view angle of 78°. The imagery was taken in a room with sufficient sunlight (between 8 am and 3 pm). Bananas were placed on white paper and a camera distance of 20 cm above the fruit.

Method
The process of identifying banana cultivars using finger fruit imagery was done through several stages in computer vision and ELM classifier as shown in Figure 2. Input of the process was a fruit image with 3 a size of 4160x3120 pixels that is resized to a size of 20% of the original image to speed up computing without compromising accuracy. Here, the image size became 832x624 pixels.
The image that had been in the resized then processed further to get the descriptor of each class that was the shape and skin texture feature of the fruit. The shape feature was extracted using two methods namely morphological descriptor and convex hull, while texture feature was extracted using local binary pattern (LBP) method. These features were then used as input in the classification process using the ELM algorithm. In the classification process using ELM, learning and testing were required so that the dataset was divided into two, namely training data and test data [21]. ELM training process was conducted on the training data to obtain optimal learning weight by setting several parameters in ELM. The most optimal learning weights were then used as weights in the ELM testing process using test data. Performance of the classification process in identifying cultivars measured using accuracy, precision and recall calculated by confusion matrix [22]. 2.2.1. Shape features extraction. The shape features were extracted using morphological descriptors and convex hulls. Extracted morphological features included five basic features and six derivative features. Basic features include diameter, physiological length, physiological width, area and perimeter. While derivative features consist of form factor, aspect ratio, rectangularity, perimeter and diameter ratio, narrow factor, perimeter ratio with physiological length and physiological width [23]. Shape features extracted using convex hull were convexity and solidity [24]. The segmentation process with thresholding technique was done before the morphological extraction process to separate the fruit area with the background. This study used a combination of binary inverse and Otsu threshold. To provide optimal results in the thresholding process, the imagery was first filtered using Gaussian blur with a kernel size of 9x9. Furthermore, to perform the calculation of the convex hull feature was first done edge detection using Canny edge detection on the image that had been threshold. The process of extracting the shape feature was performed on the grey level image. Based on the test results it is known that the best channel used was the blue channel of RGB imagery, where thresholding can work optimally. An example of thresholding and edge detection imagery is shown in Figure 3.

Texture features extraction.
The texture feature was extracted using a local binary pattern (LBP) introduced by [25]. LBP operator depicts a two-dimensional surface texture using two measurements i.e., local spatial patterns and grayscale contrast. An illustration of the LBP operator calculation is shown in Figure 4. Labels on image pixels are formed by thresholding 3x3 neighbouring pixels against the middle pixel and the result is expressed in binary numbers. The texture descriptor is expressed in histograms of a number of 2 8 (256) different labels. The LBP histogram is calculated from the LBP image ( , )using formula 1. [26]

Extreme Learning Machine (ELM)
. ELM was used to classify banana cultivars using feature values that have been extracted from the imagery. The ELM architecture used in this study refers to [26], where the network has one hidden layer of feedforward with learning that works in a single epoch.

Results and discussion
Implementation and model simulation in this study using Python program by utilizing opencv-python library for pre-process and extraction of shape features, Python.scikit-image library for LBP feature extraction and Python Extreme Learning Machine library for ELM classification. There were three test scenarios: testing using the shape feature, testing using the LBP feature and testing using a combination of shape and LBP features. The entire test was conducted using k-fold cross validation with a value of k=3 and confusion matrix to obtain accuracy, precision and recall values. Each scenario was performed five times and then calculated the average accuracy, precision and recall values of the five tests to determine the performance of the three scenarios.
3.1. Classification testing using shape features ELM testing using the shape feature as input was done with the sigmoid activation function and the most optimal number of hidden neurons was 15. The result of classification using shape feature is shown at Classification test results with shape features showed a fairly good performance where accuracy and precision have almost the same value of 85%. Although recalls have a higher value, the difference was not more than 1% which means that the recognition was quite consistent. However, some data still cannot be recognized correctly. For example, confusion matrix of the fifth test can be seen in Table 2. In fold-2 and fol-3 it appears that almost all recognition error was in the class of Ambon banana (A) recognized Hijau banana (H). While in fold-1 the error occurs in both Ambon and Hijau bananas. However, from the three folds was known that many recognition errors occur in Ambon banana.

Classification testing using the LBP features
Testing with the LBP feature uses the sigmoid activation function, and the optimal number of hidden neurons was 20. The test results in Table 3 showed good level of recognition where the average accuracy, precision and recall obtained above 91%. Although the five tests have the vary of accuracy, precision and recall, but most was above 90% and, in each test, these three measurements have almost the same value which indicates that the resulting recognition can be said to be consistent. The error of recognition using the LBP feature is shown in the confusion matrix (Table 4). According to confusion matrix of the first test can be known that most of the recognition errors were occur on Ambon banana (fold-1 and fold-3). The recognition error in this feature was similar to that of the morphological feature usage.

Classification testing using a combination of shape and LBP features
Test with a combination of shape and LBP features also used the sigmoid activation function, while the optimal number of hidden neurons was 25. The test results in Table 5 showed excellent performance where the average accuracy, precision and recall were almost the same at 93.39%, 93.354% and 93.54% respectively. Of the five tests also showed fluctuating both on accuracy, precision and recall. But most of the results were above 94%. Some of the recognition errors seen in the confusion matrix of fourth test were also dominated by Ambon bananas (Table 6). Using combinations of features appears that most of the recognition errors found in Ambon bananas, this indicates that both the shape and texture of Ambon banana skin was close to Hijau banana.

Comparison of classification results
Classification using different features shows different results ( Figure 5). In the classification of Ambon and Hijau bananas, the use of shape features produces the lowest performance compared to others. This indicates that Ambon and Hijau bananas have a high similarity in shape [2]. The results of classification using shape features in this study proved to be better than [10] which used shape features extracted by scale invariant shape analysis. While in the study [11], although it produced higher accuracy, the classified banana and non-banana objects had significant differences in shape so they were easier to be identified.  Figure 5. Performance evaluation of classification.
Furthermore, the use of texture features extracted with LBP showed an increase in recognition performance. Visually, although the shape and size of these two cultivars were almost the same, but there were exist differences in skin texture. Ambon bananas usually have a smoother surface texture so that the hue of these two banana cultivars was more distinguishable than the shape [2]. An advantage used of texture feature in this study was found when compared to [14] and [15]. The use of LBP is proven to be able to differentiate Ambon and Hijau peel textures.
The combined use of shape and texture features provides the highest recognition performance and all measures of performance (accuracy, precision, and recall) showed almost the similar value. These results indicated that the combination of the two features was effective enough to be used in differentiating Ambon and Hijau cultivars. The result of this study also better than [16] which used a combination of colour, texture and shape features.

Conclusions
This study conducted identification to distinguish the cultivars of Ambon banana and Hijau bananas that were still unripe based on computer vision by using fruit imagery. ELM classification used a combination of shape and LBP features resulting in accuracy, precision and recall above 93% and was the highest result when compared to the use of shape features or LBP features only. The high accuracy, precision and recall values indicated that this technique was effective enough to be used as an alternative in the recognition of unripe Ambon and Hijau bananas. The combined use of the two features had also been shown the significantly improve in accuracy when compared to previous studies to identify local banana cultivars. However, this study only uses image datasets taken on one banana tier, therefore further testing needs to be done by adding banana datasets with more varied in sizes and shapes from different tiers.