Preliminary study on development of cocoa beans fermentation level measurement based on computer vision and artificial intelligence

The fermentation process is an important indicator of cocoa beans’ quality. The standard method used is the Magra test by splitting the cocoa beans and observing the color of the beans with the naked eye to estimate the degree of fermentation. Although, manual estimation systems require specific expertise, which can lead to inconsistency in predicting cocoa bean fermentation rate. This research aims to develop a classification model of two categories of cocoa, i.e., fermented and unfermented cocoa, using computer vision and a machine learning model. Image analysis has been carried out, and color features have been used to train and compare several classification models. After analyzing the data, it was found out that a model that can quantify the standard and accurate measurement of the degree of fermentation of cocoa beans using artificial neural network models so that it can segment, calculate, and grade classification by using color feature extraction, which is the average value of RGB and L*a*b. The Artificial Neural Network (ANN) Multilayer Perceptron (MLP) has been found to be superior compared to other models achieving training and validation accuracy of 94%.


Introduction
Cocoa is one of the plantation crops that is widely cultivated in tropical areas such as Indonesia. The high cocoa production capacity makes cocoa a national leading commodity that contributes the third largest foreign exchange contribution of 1.2 billion USD. The prospect of the world cocoa market is experiencing an increase in market demand throughout the world, especially in Asia and Africa, by 38% and 72%, respectively [1]. The quality of cocoa beans is highly dependent on several factors, especially on post-harvest processing. The post-harvest processing of cocoa includes several processes, especially the fermentation process, which greatly affects the quality of the final product, such as aroma precursors and the cocoa flavor [2].
It is unfortunate that the high production of cocoa beans from Indonesia is not balanced with the low quality of cocoa beans. This is due to the lack of knowledge of farmers in post-harvest processing, especially in the uncontrolled fermentation process. Generally, cocoa beans of good quality must be 2021 International Conference on Green Agro-industry and Bioeconomy IOP Conf. Series: Earth and Environmental Science 924 (2021) 012019 IOP Publishing doi: 10.1088/1755-1315/924/1/012019 2 perfectly fermented with a brownish-red color and have a moisture content of about 7.5% and have gone through a process of sorting the content of damaged beans. Fermented cocoa beans will experience a color change due to a decrease in the content of polyphenols (including anthocyanidins) which are oxidized and polymerized into insoluble high molecular weight compounds (tannins). In addition, fermented cocoa beans showed a decrease in pH and inactivation of enzymes that produced flavor precursors [3]. According to Davit M et al. [4], 85% of nationally produced cocoa beans are not fermented so that it has an impact on the selling price of cocoa beans in the international market. The application of the fermentation process is an important indicator to determine the quality and price of cocoa beans from Indonesia in order to compete in the world market [5]. Therefore, it is necessary to improve the process, both for the fermentation process and for measuring the quality of fermentation that is effective and efficient and needs to be widely available in Indonesia.
The application of manual and uncontrolled fermentation technology among farmers using wooden boxes poses a big challenge, especially in the process of measuring the degree of cocoa fermentation. This is because the environmental conditions of each wooden box are different, and the physical changes of each sample [6]. Fast and accurate measurement methods are needed in measuring the degree of fermentation. The standard method for evaluating the degree of fermentation of cocoa beans is to use a cutting test in the form of a bean cutting device called a Magra, and then an expert observer will assess the degree of fermentation of cocoa beans visually by observing the color of the flesh on beans that have been split open. This observation requires experts, and subjectivity can occur because the measurement of the level of fermentation is carried out qualitatively [7]. Along with technological developments, methods for measuring cocoa quality have emerged using spectrophotometry, chromatography, Fourier transforms infrared spectroscopy, and imaging systems. However, this method has several obstacles if applied on a large scale, such as it is time-consuming, requires a lot of human resources, expensive equipment prices, and is difficult to apply to practitioners, especially cocoa farmers [2].
Computer vision is a technology that is currently being developed as an effort to measure the quality of agricultural products, which has the advantage of being cheaper, faster, accurate, and non-destructive [2]. The use of the image analysis method with multivariate statistics has been successfully applied to agricultural products such as fruits and vegetables. In addition, the use of computer vision has a high accuracy rate of > 90% by using several methods such as Support Vector Machines (SVM), C4.5, AdaBoost (AB), k-Nearest Neighbors (KNN), Logistic Regression (LR), Stochastic Gradient Boosting Trees (GBDT), Extreme Learning Machines (ELM), Sparse Representation-based Classification (SRC), and Deep Learning (DL). The use of computer vision aims to produce a fast and accurate measurement process for classifying and predicting the quality of an agricultural product [3]. Based on this problem, this article aims to develop an image processing, feature extraction procedure, and a classification model of two classes of the cocoa bean, i.e., fermented and unfermented cocoa using computer vision and several machine learning models. This model is used as the first step in making a portable fermentation index measurement system so that it can be developed into a system that can be widely applied in the community.

Material and methods
The research was conducted from May to June 2021, using a mini studio located in the Faculty of Agricultural Technology, University of Brawijaya. The sample of cocoa beans (Theobroma cacao L. cv. Trinitario) was obtained from the Indonesian Coffee and Cocoa Research Institute, Jember, Indonesia. The samples of cocoa beans analyzed were 750 samples of the fermented and unfermented beans category ( Figure 1). The fermented samples have been fermented for 4-5 days at 55 ºC and dried for 1-3 days to obtain a moisture content of 7.5%. Note that the surface appearance could not be used to determine the fermentation level. Thus, the samples were cut, and the color of the internal beans was observed.

Image acquisition
The image acquisition process is carried out by taking image data using standard lighting (mini studio) and a color chart as a reference. The type of camera used includes a DSLR camera using ISO 100, an aperture value of F5.6, and a manual exposure value of 1/13. Primary data used in the form of image data of cocoa beans from two categories, namely fermented and unfermented, as many as 140 images, and each image has 5 pieces of split cocoa beans. The image used is an image file with a format Joint Photographic Group (.JPG) measuring 3456 x 5184 pixels with a depth of 72 dpi, as shown in Figure 2. The mini-studio and the result of image acquisition used to look like the picture below.

Image processing and color feature extraction
Pre-processing data aims to change the image size by 30% from 5184 x 3456 pixels to 1555, 1036. In addition, the pre-processing of this data includes the process of converting BGR (Blue, Green, Red) colors to RGB (Red, Green, Blue). Color parameters used in feature extraction include RGB and L*a*b colors. Pre-processing can be carried out using standardized binarization in the separation of different cocoa bean backgrounds [8]. The pre-processing stage is carried out by changing the size of the RGB image by 40% to facilitate the process of converting RGB color to HSV [9]. This stage aims to convert the RGB image to HSV (Hue, Saturation, Value). Hue images show colors such as red, blue or yellow, while saturation is used to determine how pure the color of the image is, and the value shows a measure of how much brightness/light comes from color. The image results after converting to black and white into one threshold value will cause missing information. The Otsu method is useful for automatically dividing the histogram image into two different areas. Using a discriminant analysis approach by determining a variable that can distinguish between two or more groups that appear. The formulation used is as follows: for example, the threshold value to be searched is with the letter k. The value of k ranges from 1 to 255 [9]. The Otsu method is a segmentation method that uses a histogram to group the pixels in the x-and y-axis images. The x-axis in the Otsu method is a different level of intensity, while the y-axis represents the number of pixels that have that intensity value [8].
Feature extraction is the process of taking the color you want to analyze, including RGB and Lab colors. The value of each sample of cocoa beans Characteristics from the images was carried out because it was estimated that they would have different values for each class of cocoa beans fermentation level. Taking these values is carried out on all image pixels that are included in the segmentation so that only the value of the fruit image is obtained without the image background [10]. CFs include color mean value and excess RGB index. Color mean value can be described as follows [11]: where: color value can be defined as the range of each color space in the pixel such as red, green, blue, grey, L*, a*, b*. M is the total number of pixels in the masked region of the image. The complete image analysis step has been shown in Figure 3. The activation function of ReLU has been used for the hidden layer, and Sigmoid has been used for the output layer. The weight vector value initialization method uses random values and Adam optimizer has been used to optimize the model [12]. Binary cross-entropy and binary accuracy have been monitored through the training and validation process.

Image pre-processing and features extraction
Based on the results of the analysis of 1443 images of cocoa beans from both of category, it shows that there are differences in the value of each color feature. After preprocessing feature extraction, the color feature value of each sample has been obtained as shown in Table 1. The threshold analysis will provide a threshold value for each image between the foreground and the background. The threshold value uses half the value of the global threshold automatically to reduce noise from each image category. In addition, in doing image segmentation, it is known that several different results are known when analyzing images directly from RGB colors and when doing the grayscale conversion process from each category. From Otsu's method, this determines the threshold value based on stats image information of cocoa beans from each category by minimizing the number of weighted group variances, where the weight is the probability of each group [13]. According to Bhahri and Rachmat [14] in the segmentation process, the first step is to divide the image into several parts to determine the boundaries and continue with the stage of giving an index to the color of each pixel that shows the part in a segmentation. Otsu's method works by separating objects with a discriminant analysis background that will maximize the variables visualized in a histogram. The histogram is used to determine the number of pixels and the threshold for each gray level.

Machine learning models
The average value of each color channel of the masked area of the images has been labeled as R_ave; B_ave; G_ave, L_ave, a_ave, and b_ave. Based on research conducted by Oliveira, et al., [3] explained that the application of RGB imagery in classifying the fermentation rate of cocoa beans has several advantages such economical and can thus be applied by farmers and small processors as a low-cost alternative to the traditional cut-off test with accurate prediction results on the fermentation grade of cocoa beans. These features then were used as an input of a machine learning model. For the machine learning model other than ANN, we tried to compare the result of the model with data scaling (standard scaler) and with no scaling. Table 2 shows the training and validation accuracy results of several machine learning models besides ANN. Among those models, none could achieve more than 90% accuracy.

Prediction using an artificial neural network model
ANN-MLP model with different architecture has been trained and validated. From these data the value in making the classification and modeling of an artificial neural network is influenced by several factors such as the selection of the number of neurons in the hidden layer and the appropriate learning rate value of each data classification model. The addition of the number of neurons in the hidden layer, the longer the running time of data processing, but will increase the accuracy value obtained. Addition and subtraction of a matrix can be obtained using the confusion matrix which will affect the accuracy and  [15]. The method which is simple but with high accuracy will be preferable to avoid model complexity curse. The simple model also mean that it could be executed faster with lower memory and storage required. From the results, it has been found that the first model which is using 2 hidden layers (16-16) provide the best results ( Figure 4). Interestingly ANN-MLP model which used more than two layers, i.e., three and four could not achieve accuracy more than the two hidden layers models. From the results of the preparation of the model using artificial neural network, there are several important points. Factors to choose the type of activation function, the number of iterations and the number of hidden layers of each model need to be considered. In addition, the number of hidden layers must also be adjusted to the learning rate of a model. ANN modeling using these 3 models is very capable of being developed in a system to classify the level of cocoa bean fermentation with an accuracy value of >90%. Table 3 shows the results of the accuracy and validation of the best model for each ANN MLP architecture.

Conclusions
From 1143 cocoa beans image data from two categories of fermented and unfermented cocoa beans, it is obtained if there is a color relationship, especially RGB in determining the correlation value between parameters. The use of Otsu's method will simplify the image segmentation process for each category due to the loss of noise in the form of light disturbances and unwanted image objects to be detected. Image segmentation is the first step in developing a detection system for the degree of fermentation of cocoa beans through image images by dividing an image into several regions, which in an area have similar attributes. In making modeling using ANN, it is necessary to pay attention to several factors such as the number of iterations, the value of learning rate and the number of neurons in a hidden layer. From the model that has been made to classify the level of fermentation of cocoa beans using an artificial neural network, it has an accuracy value of >90% from 2 categories of fermentation levels. Especially in the 1st model with 2 hidden layers at node values (16,16) and 0.002 learning rate value, through the output activation function in the form of Sigmoid, has the highest validation accuracy value of 94%. These results show that it is relatively easy to classify the fermented and unfermented cacao beans with any machine learning model using a few color features. Further challenges are to predict the fermentation levels in more detail in higher resolution using classification with more classes or a regression model.