Image segmentation methods for quick characterization of ore chip using RGB images

The ability to forecast geometallurgical properties during resource extraction is essential to optimize the mine to mill process. Models for mine planning thus often incorporate attributes related to processability. The analysis of these attributes in a laboratory can be time- and cost intensive. Only a limited number of data may be available. During production, grade control drilling may provide access to many more samples. Conducting laboratory analysis to each of these samples would be not realistic. If there was an opportunity to quickly obtain related proxy data, as physical characteristics that can stand in for direct measurements, then these indices could be estimated, certainly less precise but with a significantly increased spatial density. A moderately simple approach to acquire data from grade control drilling is to take digital Red, Green and Blue spectral bands images (RGB images) in from core trays. Although these capture only three spectral band regions, images can contain valuable texture and colour related information. A first necessary step is to automatically extract from an image and analyse objects, that represent ore particles or mineral content. This study aims to investigate the performance of different available segmentation methods under field conditions. First an overview of methods for image segmentation as a basis to create objects is presented. Objects can be related to single grains and minerals within the grains. The aim is to provide a basis for texture feature extraction related to granular rock, such as found in chip trains. Modern image analysis provides a large number of methods for segmentation and classification of objects. This work focuses on evaluating performance on images of 3 levels of complexity of pixel- based segmentation for complex or less noisy images and object-based segmentation (Watershed, Simple Linear Iterative Clustering and Quickshift) as a more advanced and universal method.


Introduction
Improving the geometallurgical behaviour of material streams from mine to mill or processing plant can create substantial economic and efficiency benefits. Image processing is one of the techniques that can contribute to achieving such an improved understanding. It allows the characterization of textures and mineral related properties of ore in material streams. Compared to other off-line techniques such as laboratory analysis, image processing can improve our knowledge without time-consuming investigation. Studies of using image sensors during the material transportation process from mine to mill show perspectives in fast capturing rock mass characteristics. That allows to improve the process efficiency by adjusting downstream process parameters [1][2][3][4].
The goal of an image-analysis-based approach is to quickly characterize samples along the material stream. This contribution focuses on RGB-images taken in a more or less static environment such as taken for the drill chip in tray samples or at stockpiles. Without loss of generality, methods discussed here can also be applied to RGB images taken in a more dynamic environment, such as above conveyor belts. Typically, colour and texture-based indices are determined for samples shown on the images. Texture features indices rely on the frequency of the pixel or edge indicators repetitiveness. Depending on the segmentation goal, the choice of the texture or size features extraction method will differ. Here, two general approaches are distinguished, pixel-and object-based methods. The differences between two and their main aspects become clear in the process of collecting the information about previous studies and methods execution.
Object based methods first define objects, such as single rock chips or grains and then derive texture or colour-based indices for each object. Difficulties in forming objects result from reflection, lighting conditions and shadows. While forming the objects such as rock chips or sand particles, texture features will be concentrating on the repetitive properties of the edge (methods based on multifractal theory [5] that makes it easy to detect. In certain cases, the object-based segmentation will most likely give false results, by picking up noise as a separate object or will not detect specific formations as separate at all (e.g. quartz veins, Greisen-type mineralization in geology or bone structure deformations in medicine). In those cases, pixel-based features (contrast, homogeneity, angular second moment etc.) provide more reliable information as they are based on the repetitiveness of the grey level pixel values on the chosen distance (that gives opportunity for detailed look at the ore characteristics) and under certain angles. When the target components (mineral composition or ore particles needed to be detected) are too complicated for correct edge detection, pixel-based segmentation is more suitable since it determines indices based on pixel statistics involving its surroundings.
The aim of the paper is to assess granular rock such as sands, broken ore or drill chips. In this case object-based methods (in particular Watershed, Simple Linear Iterative Clustering (SLIC) and Quickshift techniques as more advanced) give not only texture information, but also geometrical characteristics e.g. particle area, perimeter, major and minor axis.

A brief review of image processing applications
The potential of the image processing use, specifically segmentation, was documented and studied in many fields including mining, medicine and other fields.
In a considerably big amount of documented cases (e.g. Maitre 2019) concerning image analysis application for ore characterization, the imaging setup was laboratory like, which simplified the analysis by a defined background, sufficient lighting and shadow control etc. Unfortunately, it is not the case for the usual, field-made images of the ore. The heterogeneous operational environment requires algorithms that can deal well with the considerable amount of noise and set low requirements on the image taking process.
Maitre [6] proposes an original computational approach to automate mineral grain recognition from high-resolution RGB images obtained with an optical microscope. SLIC superpixel-segmentation was applied to divide each mineral grain composing the sand and provide a more sufficient result compared to convolutional neural networks (CNNs) and classical segmentation methods.
Yi and Zhang [7] presented a Multi-Features Fusion for SLIC segmentation algorithm of the rock surface of the basalt data, created from the Terrestrial Laser Scanning point cloud and digital images. This technique defines a regional dissimilarity through neighbouring superpixels clustering indices and is capable of getting rock surface features with high efficiency.
Previous work had been limited to the laboratory conditions and high-resolution close-up images.
Oppositely Thurley [8] presented measurement and image analysis of 3D surface profile data of blasted rock piles in an open-pit mine for determining the size distribution of the visible rocks on the pile by watershed segmentation and morphological operators.
Raju [9] presented a satellite image segmentation system by K-mean clustering and Watershed segmentation. Research demonstrated the k-means algorithm's downside e.g. depends on the correct assumption of the initial parameters, and the watershed algorithm's advantage in creating image segmentation.
Besides application of image analysis in rock and land characterization, large advances have been made in the field of medical sciences. Although it is a different application domain, developed methods are of high relevance for the topic of this contribution. A Thresholding estimation algorithm with Watershed transforming Sobel filter for detection of different cells in microscopic images of blood has been proposed by Biswas and Ghoshal [10]. In this study, the proposed algorithm has used 30 numbers of blood microscopic images as test images and obtained around 93% accuracy results to identify and count the different blood cells.
Nassir Salman [11] proposed a method of improving the Watershed algorithm by initially creating K-mean clustering segmentation and edge strength technique on the images of brain MRI-scans. Despite the accurately located boundaries, the technique is highly dependent on the correct performance of the K-mean clustering algorithm.
By Ibrahim and Mohammed [12] a Quickshift segmentation method applied on breast cancer thermal images to improve the process of extracting regions of interest (breast area) by eliminating the unnecessary region, such as parts of underarms and neck. The results show that the proposed algorithm significantly enhances the segmentation results of healthy and unhealthy cases images using the dataset from Mastology Research with Infrared Image.
Since the importance of the principal features, such as particle size, homogeneity and texture, is similar in characterizing biomaterial and ore material, the investigated method would fit both in the medical and in the mining field.
In summary to everything stated above, five pixel-and object-based segmentation algorithm performance on the images of 3 levels of complexity (industrial sand, crushed ore and drill chips) are chosen to be reviewed, compared and evaluated. Pixel-based segmentation is presented by Threshold and K-mean clustering, while object-based segmentation is presented by Watershed, SLIC and Quickshift segmentation. Results evaluated by hard and soft criteria described in part 3.2.

Methods and data
The research aims to evaluate the potential of using suitable segmentation methods to characterize broken ore or sand. To accommodate for varying and heterogeneous operational environments while image taking, the algorithms will be tested on three different images that represent different conditions, which will be described in more detail in the section test data set. The images are generally presented as a matrix of N pixels, whereas each pixel has an intensity value for the red, green and blue channels (bands).
The simpler of the chosen methods concentrate on the pure colour-based segmentation for each pixel individually (Threshold, K-mean), others more sophisticated methods use gradient filters for the edge detecting (Watershed, SLIC, Quickshift). The most commonly used methods, Watershed and Kmean, will be compared to the less used superpixel ones (SLIC, Quickshift). While the Watershed approach is designed to work with overlapping objects, superpixel segmentation provides a convenient and compact information about the content of the ore, texture and grain size distribution.
Next, the compared methods will be described.

Threshold segmentation
Commonly Thresholding is used in object tracking when the targeted pixel value is known in advance or easy to define, i.e. images with defined background, light conditions and visually defined objects.
There two main types of Threshold segmentation: • Global Thresholding, with one set threshold (Formula 1(a)) • Multiple Thresholding, with multiple set thresholds (Formula 1(b)) In general image Thresholding set pixels into categories: • Those to which some property measured from the image falls below a threshold T.
• Those at which the property equals or exceeds a threshold T. On this basis, the output image x' can be obtained from the original x by applying the threshold T to each pixel.
a) Thresholds can be applied to all three RGB channels. The method is moderately simple to implement. A challenge is the correct determination of the thresholds. An incorrect choice of these values results in an incorrect image segmentation. Especially under varying conditions this is difficult and would require future algorithms to automatically determine the threshold values [11] .

Clustering segmentation
The clustering-based techniques are colour-index based methods, which segment the image into clusters having pixels with similar characteristics.
Hard Clustering is a technique that divides the image into a set of clusters i.e. one pixel can only belong to only one cluster. [13,14].
Fuzzy Clustering allows one pixel to belong to more than one cluster, and this degree of belonging is described by membership values [15][16][17][18].
For this case study fuzzy clustering is used as a more flexible approach that is capable of producing close to truth results and omits the noise.
Fuzzy K-mean Clustering focuses on minimizing the objective function of J by calculating degree of membership and computing clusters centres: (2) Where: x -whole dataset (set of pixels); n -number of data items (pixels); c -Number of clusters; v jk -degree of membership; q -Weighting exponent, q∈[1;∞] ; vi -prototype of centre cluster; d 2 (xk,vi) -distance between xk and cluster vi.
The k-means algorithm initially has to define the number of clusters c and randomly choose cluster centres (v,i). Then the distance between each pixel to each cluster centre is calculated (d 2 (x_k,v_i)). Single pixels are compared to all cluster centres and assigned to the particular cluster. Then the centres are re-estimated. Again, each pixel is compared to all cluster centres. This calculation continues until the centre converges [19].

Object-based segmentation
This method group is based not only on colour-indices, but also on the gradient of the intensities within the picture. That makes these techniques highly useful in creating and detecting separate objects in medical (brain tumours, blood and body tissues) and in geological fields (ore particles) [10][11][12].
Edge detection techniques locate the edges, where either the first derivative of intensity is greater than a particular threshold or the second derivative has zero crossings. For this pre-process in edgebased methods, gradient filters are helpful.
The Sobel filter calculates the gradient of image intensity at each pixel within the image, by using a kernel matrix 3x3 for two directions (x and y). It finds the direction of the largest increase from light to dark and the rate of change in that direction [1,20].
The Laplacian filter measures the rate at which the first derivatives change. This operator shows how much the value of the measured area differs from its average value taken over the surrounding points [1,20].
Principles of gradient filters performance is shown on Figure 1. The Watershed segmentation algorithm views a grayscale image as a topographic surface where high intensity denotes peaks and hills, while low intensity denotes basin. Algorithms use the label matrix that contain integers matching to the location of each basin, where the zero-valued pixels are positioned along the watershed, and the distance from every pixel to the nearest non-zero is measured.
This algorithm is great for defining the edges of overlapping objects by using the concept of previously described gradient filters (Sobel or Laplacian).
Simple Linear Iterative Clustering (SLIC) segmentation generates superpixels by k-means clustering pixels based on their colour similarity and proximity in the image plane. Superpixels, as a group of pixels that share common characteristics, are used to replace the pixel-grid in order to speed up the algorithm [3].
The algorithm uses the approximate number of labels in the segmented output image, compactness that balances colour-space proximity and pre-processing with width of Gaussian smoothing kernel for each dimension of the image.
Starting from sampling a regularly spaced cluster centre and associating each pixel group with the nearest cluster centre whose search area overlaps this group. Afterwards a new centre is computed as the average colour (x,y,z) vector of all the pixels that belong to the cluster. These calculations are repeated until convergence [3].
The Quickshift Segmentation segments an image by identifying clusters of pixels using the Quickshift mode-seeking algorithm in the colour(x,y)-spatial space [4,22].
Influential on the algorithm parameters are: • width of Gaussian smoothing kernel for smoothing the sample density; • cut-off point for data distances; • ratio that balances colour-space and image-space proximities. Object-based segmentation has several advantages e.g. simplicity, speed and generality. These are modern and advanced methods to form homogeneous pixel clusters and therefore detect their edges and extract features.

Test data set
To investigate the performance of the presented methods under the field conditions, digital RGB images of three distinct samples are used (Figure 2). Each of the images differs in rock type and origin and present a different amount of noise in the picture, due to image clearance, patches of reflected light, shadows etc. Following a description of the different cases is provided.
Case 1: The image presents ore chips in a tray from a gold mine originating from a grade control program. The chips vary in size (form small grains to the samples triple in size), and consist of two main minerals. Some surfaces are wet and some are dry, causing a different degree in reflection. In addition, there are areas of shadow resulting in different intensities of reflected light. These circumstances represent a typical noise field environment without much effort in preparing the sample for image acquisition. Prior to the analysis, these conditions may be classified as difficult.  Case 3: image 3 is taken from a mineral sand sample of a small but homogeneous particle size and colour. The image is taken under laboratory conditions resulting in good light conditions and minimization of shadows due to particle placement.

Methodology of result evaluation
Criteria to evaluate the segmentation performance of the different methods can be divided into the categories "soft" and "hard".
Soft criteria include: • Visual accuracy • Processing time • Versatility The visual accuracy criteria imply the correct finding of edges and identifying the objects that can be seen with "naked eye". Although subjective, eventually the algorithms try to mimic the expert in evaluating the sample. In this way, a simple visual inspection is quite a complex and strict criterion.
Processing time criteria refers to the computational complexity of the method. The benchmark will be a comparison of the run time between different methods on a personal computer used. Since the images represent a typical size of a digital image, scaling is not considered here. In this case, an asymptotic runtime analysis of the algorithms should be performed.
Versatility implies the suitability for different application cases that vary in size, colour, complicity etc. In addition, this criterion captures the ability to improve results by combining an approach with other ones.
The hard criteria are represented by quantitative statistical indices calculated from the results of the segmentation applications. The idea behind these indices is to check for the homogeneity of created objects. It is assumed that objects exhibit inherently homogeneous colour properties. In an ideal segmentation the variance of the gray-scale values within an image are split in objects, which do show none to little variability. In other words, the efficiency of the image segmentation is judged on the proportion of variability of RGB values within the image that can be explained by the differences between the objects. The residual variability is the variability within the object. This is similar to the volume-variance-relationship used in geostatistics.
Following indicators are used: The Variance is a measure of the average spread of RGB intensities within one object; The Kurtosis will explain if this spread is due to single isolated outlier pixel values or if the spread is homogeneous over the complete object.

Threshold segmentation
Multiple minerals within the analysed ore requires the use of multiple thresholds of the RGB intensities of an image pixel. Since the intention of the core tray was to discriminate between two minerals contained within the grains, two masks are created. Pixels not classified with one of these masks are classified as background, which contains noise and unidentified pixel values. Using a set of training pixels within known minerals, RGB histograms are calculated for each mineral. Thresholds are histogram values ranges in Red, Green and Blue bands respectively and by this values masks are created. (Fig 3). Also results can be biased towards bigger and brighter objects. That makes it often ignore less illuminated objects, so segmentation ends up missing out a lot of elements.
Although results appear somewhat accurate, this approach is insufficient and very restricted due to the histograms, since it is hard to predict the same RGB values for all of the training images. The method is not universal and needs time-consuming adjustments for different study cases.
Considering the approach complicity and a small amount of the indicators, such as discrete particle size, that can be extracted out of the Thresholding, this method is not compared and evaluated with other, more advanced approaches.

Clustering segmentation
The performance of K-mean clustering depends on the amount of the objects (clusters) needed for certain cases.  For the drill chip it is obvious that up to 5 clusters are needed for the Mineral 1, Mineral 2, light/shadow parts and background ( Figure 5), for the crushed stone case selecting the cluster numbers will be more dependent on the goal and specific objects that wanted to be achieved.
As for the sand, with small size grains, it is better to use this method for finding groups of the same mineral grains, rather than separate grain analysis ( Figure 6). K-mean clustering segmentation is biased to the user assumption of the number of cluster groups that is disadvantageous compared to the object-based segmentation.

Object-based segmentation
Object-based (edge-based) segmentation methods focus on finding the discontinuities in the grayscale between dark and light pixels in a neighbourhood. For finding and outlining these discontinuities as an edge, those methods need additional filters to evaluate the grey-scale gradient.

Figure 7. Watershed Edge Detection
Watershed Edge Detection: In this study as the distance gradient was used the Sobel filter. Even due to the test results Laplancian filter reduce the amount of markers (array assigning the basins values to the Label matrix) and give smooth polygons to the objects, it created very detailed over-segmentation (a division into too many segments, as when attempting to recognize parts of an image), that create additional noise (Figure 7).

Figure 8. SLIC Edge Detection
As seen from the Tabl.2 Watershed segmentation created a lot of objects presented with 1 pixel, that was created by picking up noise and the centre of basins. As it is evident from combination of all criteria and low variance value, Watershed algorithm worked best for the industrial sand sample. SLIC Edge Detection: SLIC method strongly depends on the contrast of the input image and object's shape. Smaller values create bigger segments and less precise or correct edges.
The benefit of this method is versatility. Depending on the aim of the segmentation, the same values of all three components are suitable for all 3 different cases with different object size and picture quality (Figure 8).  Figure 9. Quickshift Edge Detection Table 3 shows evaluation results for the SLIC segmentation, by which drill chips samples are segmented the most accurately in terms of created objects and particle shape and low variance. In the case with industrial sand segmentation has the least amount of noise in soft criteria but creates objects that overlap several sand particles. Quickshift Edge Detection: Despite the visually correct segmentation creation, Quickshift segmentation is one of the most difficult and demanding approaches among all tested through this research (Figure 9).
In order to avoid oversegmentation, cut-off points for data distances should have higher values, increasing the processing time.
For example, if for the segmentation of the ore chips in a tray maximum distances should have been at least 200 pixels, for the mineral sand it was possible to take an amount of 30 with the same colour-space and image space ratio and kernel size.
For the ore chip it took much longer processing time than for the industrial sand, which has smaller simple shaped particles and because of this requires creation of smaller segments ( Table 4). Amount of the noise also requires application of the additional modification such as blurring.  For the drill chips Watershed creates a lot of objects with one data point, mainly in the Red band. That could be caused by the amount of noise (as shadow and wet/dry particles) in the image. SLIC segmentation of the samples is quite uniform in Variance values. But creates quite few data extremes in kurtosis value by creating larger objects with a few distinct fractions. Quickshift segmentation scatter shows the clusters of the outliners for the drill chip, as well as crushed ore, that correspond to the objects that was incorrectly formed and consist from the intersections between: • A very dark spots and the part of the very light in colour stone that located underneath the others • Different in contrast (light/dark) minerals fragments within one stone(object). Considering all of the above the SLIC segmentation handles the amount of noise in the sample image the most effective.  Figure 11. Crushed ore segmentation result evaluation For the crushed stone that presents similar particle size but in more controllable noise condition and colour consistency, the graph is more uniform. The outliers present objects with a fraction of other minerals. Tests and comparison of the most used segmentation methods show their performance is different by the level of complicity images. From the results is noticeable that even if the easy methods, i.e. Threshold and K-mean clustering are fast and easy in processing, the amount of the needed modifications makes these approaches complicated to use with complex images containing a big amount of data (a lot of noise, mineral count, different picture quality etc.) As for the edge detection methods, since they work on the principles of the k-mean clustering, they have advantage in form of precision and possibility of the combination with other methods and improvement in order to get more accuracy.
Watershed segmentation is useful in cases with little noises and where the overlap of the object is present. But the amount of the adjustments for different images makes it less correct for some of the complicated images. The different story is for the superpixel segmentation, which shows itself as a more universal method with a processing time as disadvantage.
The result of this work is important for correct cluster analysis and the creation of the classifier for further image analysis and mineral content detection.
The statistic obtained from the segmented objects, by the Superpixel segmentation, makes it possible to choose any of the classification methods that fulfil the purpose of the ore analysing.