Algorithm for automatic detection and measurement of Vickers indentation hardness using image processing

In manufacturing processes, the hardness test is used to measure the basic mechanical properties of materials, including metals, ceramics and polymers. For instance, the Vickers indentation hardness scale is widely used to predict the hardness of both novel and conventional materials [1–5]. For this purpose, durometers are hardness testing machines that use pyramid shaped diamonds to apply a predetermined force onto a material's surface. The diagonal lengths of the rhombus-shaped footprints, known as indentations, are manually measured by a human operator, using a camera connected to the durometer. Based on the average of the two diagonal lengths for each indentation, the Vickers hardness (HV) of the specimens is then calculated, using the equation reported in the literature [6–8]. To ensure the high quality of the microscopic image, the specimens must be cleaned with various kinds of chemical substances prior to Vickers hardness testing. This is due to the likelihood of contaminants on the surface of the specimens. Other difficulties in hardness testing are caused by the mechanical treatment of the specimen, which can generate indentation deformations. In addition, variations in the indentation shape can be produced by different indenter strengths, indentation time lapse, and disturbances in the illumination conditions. Moreover, indentation images can have low contrast, which complicates the measurement of the Vickers hardness number of the specimens. Operator inexperience and working conditions can also adversely affect the results of indentation testing. To overcome this problem, several researchers [9–16] have developed algorithms to estimate the Vickers hardness number, using digital image processing. However, the implementation of image processing algorithms for indentation measurement requires a system to acquire and store the microscopic indentation images. This is due to the fact that many conventional durometers do not have image acquisition systems or the accessories to adapt to commercial frame grabbers.

The computational algorithms for Vickers hardness testing are used to detect the indentation and evaluate its diagonal lengths using image processing, and considering footprints with a square or rhombus shape. However, the position, dimension, and exact orientation of the indentation in the image are not known. Some algorithms for Vickers indentation testing have used segmentation-like binarization via thresholding to detect and locate the indentation in the image [9, 12]. In addition, morphological filters have been employed to eliminate speckles in the segmentation [8, 12]. Several methods locate the indentation vertices and estimate the Vickers hardness number based on the average diagonal length of the indentation. These methods have considered the axes projection [8, 9], edge point detection, template edge matching [10, 13, 17] and dual resolution active contours segmentation [18]. In addition, some researchers applied the Hough transform and least squares approximation of straight lines to the edge points of indentation images in order to obtain data on the indentation edges and vertices [8, 10]. These methods are suitable for indentation images with high contrast and straight edges [13, 14]. However, other methods are required for indentation images with low contrast and more irregular edges. Moreover, algorithms based on edge and line oriented contour detection [19] have challenges such as high computational complexity, multiple input parameters specified by the user, and contour detection that may collapse in images with low contrast.

Methods based on template matching [14, 16] are able to detect the indentation even in the presence of noise in the indentation image, under conditions of low contrast variable brightness. These methods demonstrate stability even when the material surface is affected by grooves or speckles. For instance, Maier and Uhl [15] reported a method of detecting small indentations segmenting the image by means of operators known as the decrementing area map and the incrementing area map. This method has limitations in terms of indentation images with very low contrast in relation to the adjacent image area. In addition, these operators must be evaluated to differentiate many indentation images groups, as reported in [15]. In industrial environments, modern durometers [20] can control the indentation orientation, but for conventional durometers, the camera and microscope positions are fixed. Some researchers [8–12] have developed algorithms for indentation measurement without considering orientation. Nevertheless, the light emitted by the durometer microscope causes illumination variations during image acquisition, which affect the performance of these algorithms. Recently, Domínguez-Nicolás et al[ 21] reported an algorithm for the analysis of Vickers hardness testing images, in which the binarization applied to the indentation image is based on maximum and average gray values. In addition, they employed an algorithm to detect corners, rather than edges, so as to locate the indentation vertices. However, their image processing is semi-automatic, and the algorithm has limitations for those indentation images with shading effects generated by the microscope-integrated light source. This occurs when the specimen has a specular-polished surface. Furthermore, this algorithm is not suitable for indentation images with very low contrast in relation to the adjacent image. To overcome these problems without the intervention of a human operator, we propose an algorithm with automatic image processing for Vickers indentation hardness, as shown in figure 1. Our algorithm uses the criterion of binarization from each input image with the mean gray values and extreme gray values, exchanging the mean gray values relating to automatic analysis for a standard histogram equalization, in order to determine the optimal binarization threshold.

Zoom In Zoom Out Reset image size

Our algorithm can evaluate a wide range of indentation features, where the edges of the indentation are not exactly straight lines, and where the gray values in both the indentation and background of the image are not uniform. This algorithm can measure the Vickers indentation hardness of specimens with indentation images with very low contrast in relation to the adjacent image area, in comparison with the limited capabilities of other algorithms [13–17, 21] to detect indentation vertices. We used a Mitutoyo model HM-125 microdurometer with a diamond tip [22] to generate the indentation images that were subsequently tested using the proposed algorithm. A system based on a field-programmable gate array (FPGA) and a graphic interface [23] is used to convert the analog video signal originated by the microdurometer to a digital video signal in real time, storing any frame-like image in BMP format. We compare the results of our algorithm (e.g. vertex coordinates, diagonal lengths and Vickers hardness for each input image) with those manually obtained by specialists in Vickers hardness testing. These results are estimated based on 230 indentation images, taken from specialist data available online at www.uv.mx/personal/saudominguez/208-2/, known as the UV-Vickers Indentation Images Data Bank [24].

This paper is organized as follows: section 2 examines the limitations of the algorithms for indentation hardness testing. Next, section 3 describes the proposed algorithm for Vickers indentation hardness testing. Section 4 includes the experimental results for our algorithm. Finally, our conclusion and ideas for future research are reported in section 4.

Hardness testing systems determine the average diagonal length of indentation images in order to estimate the Vickers hardness number of material samples. For instance, the active contours algorithm [19] can predict the Vickers hardness number of indentation images. However, if the starting condition is not carefully selected, the active contours can estimate a local minimum instead of the best global minimum. Lima-Moreira et al[ 25] developed a new method of measuring the Vickers hardness number, based on the active contour, which demonstrated advantages over segmentation methods such as region growing and watershed. This method was applied to a small number of indentation images , without considering stains, scratches and texture porosities. Other methods [14, 17, 19, 20] have also been used to determine the Vickers hardness number of indentation images. However, large gradient values near indentation borders cannot be confirmed, because the edges of the indentation footprints are not straight lines. Thus, these methods are not suitable for indentation images with low contrast and edges not formed by straight lines. In comparison to modern methods based on active contours, our algorithm can estimate the Vickers hardness number of indentation images even in cases of low contrast and edges without straight lines. Other algorithms use a binarization criterion based on extreme gray. Our algorithm uses different binarization techniques with respect to those reported by other researchers [9, 12, 17, 20], in that they do not use extreme and average gray values as binarization criteria. In addition, other algorithms [21] have employed similar binarization criterion to that employed in our algorithm, but without automatically determining the optimal binarization threshold from each input indentation image. Thus, the algorithm reported in [21] can semi-automatically determine the binarization criterion, but presents many outlier images in the vertices indentation detection. Figure 11 shows a comparison between indentation vertices detected using our algorithm, and the semi-automatic algorithm reported in [21]. Furthermore, table 2 and figure 11 show that our algorithm achieves better results in comparison with other algorithms proposed in the literature [8, 21].

Our algorithm processes each input indentation image as a 2D monochromatic digital image, where the gray values are between 0 (black) and $Max$(white). Indentations of near-rhombic shape are formed in each image, where the position, size, and exact orientation of the indentations are unknown.

In indentation images, the indentation is commonly identified as a dark area on a brighter background. In many cases, the indentation edges are virtually straight lines. However, our algorithm does not make this assumption. Figure 2 depicts two indentation images taken from the UV-Vickers Indentation Images Data Bank [24], in which the vertices and diagonals of the indentation are marked in white. The up vertex is defined as $U = ({U_x},{U_y})$, the down vertex as $D = ({D_x},{D_y})$, the left as $L = ({L_x},{L_y})$, and the right as $R = ({R_x},{R_y})$. Thus, the diagonals of the indentations are given as $\mathop {UD}\limits^{\_\_\_\_}$ and $\mathop {LR}\limits^{\_\_\_\_}$, and the average diagonal length $l$ to calculate the Vickers hardness number $HV$, based on the equation reported in [6–8], is determined as:

$$\begin{equation}l = ({l_1} + {l_2})/2\end{equation} \tag{ 1 }$$

Zoom In Zoom Out Reset image size

with

$$\begin{equation}{l_1} = \sqrt {{{({U_x} - {D_x})}^2} + {{({U_y} - {D_y})}^2}} ,{l_2} = \sqrt {{{({L_x} - {R_x})}^2} + {{({L_y} - {R_y})}^2}} \end{equation} \tag{ 2 }$$

3.1. Image segmentation

The first step in our algorithm is segmentation, which starts with the binarization of the input image (see figure 2). Using the average gray value, a standard histogram applied to the input image, and the difference to the maximum gray values, we determine the threshold values ($\tau$) for binarization. These values represent the global characteristics for each input image. The highest frequency of occurrence determined by the histogram, average, and maximum gray values are global characteristics determined by the input image $G$, which is defined as $G = \left( {g\left( {x,y} \right)} \right),{\text{ }}{x_1}, \cdots ,{x_{max}},{y_1}, \cdots ,{y_{max}}$ (see figure 3). The average gray value of $G$ is given by ${G_{mean}} = (1/{x_{max}}{y_{max}})\mathop \sum {\left\{ {g\left( {x,y} \right):x = {x_1}, \cdots ,{x_{max}};y = {y_1}, \cdots ,{y_{max}}} \right\}}$, the maximum gray value is given by ${G_{max}} = max\left\{ {g\left( {x,y} \right):x = {x_1}, \cdots ,{x_{max}};x = {y_1}, \cdots ,{y_{max}}} \right\}$ and the highest frequency of occurrence of the gray value is defined as ${G_h} = max{\text{ }}h\left( i \right)$, where $h\left( i \right)$ is the histogram of the image with a number $i$ of gray values.

Zoom In Zoom Out Reset image size

For every input pixel $p = \left( {x,y} \right)$, our algorithm uses the following binarization criterion, where $p$ is considered a pixel of interest if:

$$\begin{equation}\left| {G(x,y) - {G_{\max }}} \right| > \tau \end{equation} \tag{ 3 }$$

Binarization can be resumed at the following stages:

• 1)  
  Apply histogram to input image to obtain ${G_h}$.
• 2)  
  Obtain ${G_{max}}{\text{ }}and{\text{ }}{G_{mean}}$ from input image.
• 3)  
  Initially $\tau$ acquires the ${G_{mean}}$ value, such that $\tau = {\tau _0} = {G_{mean}}$.
• 4)  
  Apply binarization criterion (3).
• 5)  
  The result is a binary image ${G_b} = \left( {g{\prime}\left( {x,y} \right)} \right)$ with $x = {x_1}, \cdots ,{x_{max}}$, $y\, = \,{y_1}, \cdots ,{y_{max}}$, where each pixel of interest is represented as black, and other pixels are white. Thus, a subset of the black pixels will represent the indentation region, which in the original image is coincident with the fact that the indentation region is dark.
• 6)  
  Our algorithm applies morphological filtering to ${G_b}$, as reported in [21].
• 7)  
  Finally, the image segmentation ${G_s}$ is obtained via region growing, applying the 8-connectivity reported in [16].
• 8)  
  Are indentation vertices detected?Yes: end of algorithmNo:If ${G_{mean{\text{ }}}} < {G_{max}}/2$, then $\tau = {\tau _0} + \left| {\left( {{G_h} - {G_{mean}}} \right)} \right|$, elseIf ${G_{mean{\text{ }}}} > {G_{max}}/2$, then $\tau = {\tau _0} - \left| {\left( {{G_h} - {G_{mean}}} \right)} \right|$
• 9)  
  ${\tau _0} = \tau$
• 10)  
  Return to point 4.

For indentation images with gray levels distributed along the dynamic range of $h\left( i \right)$ (see figure 4), the indentation vertices are detected using (3) with $\tau = {\tau _0} = {G_{mean}}$. Furthermore, in these images, ${G_{mean}} \cong {G_h}$ and ${G_{mean}}{\text{ }}$ are located to approximately half of the dynamic range $h\left( i \right)$.

Zoom In Zoom Out Reset image size

In dark-field images of indentation, the histogram presents a landslide of their gray levels, with the highest frequency of occurrence on the left of the dynamic range $h\left( i \right)$, as shown in figure 5. Thus, the binarization in $\tau = {\tau _0} = {G_{mean}}$ prevents our algorithm from detecting the indentation vertices. Thus, equation (3) is evaluated for $\tau = {\tau _0} + \left| {\left( {{G_h} - {G_{mean}}} \right)} \right|$ until good image segmentation can be obtained, and the indentation vertices can be determined.

Zoom In Zoom Out Reset image size

Histograms with a landslide of gray values, where the highest frequency of occurrence falls to the right of the dynamic range $h\left( i \right)$, are indentation images with light gray levels (see figure 6). In these images, the indentation vertices are not found, due to failings in the binarization in $\tau = {\tau _0} = {G_{mean}}$. Therefore, equation (3) is evaluated for $\tau =$ ${\tau _0} - \left| {\left( {{G_h} - {G_{mean}}} \right)} \right|$ until better image segmentation is achieved. Our algorithm is then able to determine the indentation vertices.

Zoom In Zoom Out Reset image size

3.2. Location of the indentation

Effective binarization can obtain a maximum area black 8-component of the image, binarized and morphologically filtered by our algorithm. To complete the indentation detection, the gravitational center of the indentation region $M$ is calculated, and defined as the center point $q = \left( {{q_x},{q_y}} \right)$ of $M$: if ${p_i} = \left( {p_x^i,p_y^i} \right),{\text{ }}i = 1, \ldots ,m$ denotes all pixels of ${\text{ }}M$, then

$$\begin{equation}{q_x} = \frac{1}{m}\sum\limits_{i = 1}^m {p_x^i},\ {q_y} = \frac{1}{m}\sum\limits_{i = 1}^m {p_y^i} \end{equation} \tag{ 4 }$$

Figure 7 shows examples of indentation centers, which are marked by our algorithm with a green dot.

Zoom In Zoom Out Reset image size

3.3. Determination of the indentation vertices

The indentation center coordinates are used as auxiliary data to accurately determine the indentation vertices. Our algorithm uses the bounding box (circumscribed rectangle) in the indentation region, as reported in [21], where the algorithm employs Harris-Stephen corner detection to obtain the indentation vertices. However, our algorithm uses the Laplacian of Gaussian filter (LoG) to average the two digital gradients comprising the Harris-Stephen matrix defined in [21], where the average scale changes in the image to make it independent of the image resolution. This is due to the circular symmetry of the LoG filter, which in our algorithm is adapted in the Harris-Stephen matrix to detect corners in the indentation image.

The Gaussian function is therefore given by:

$$\begin{equation}{h_g}(x,y) = {e^{\frac{{ - ({x^2} + {y^2})}}{{2{\sigma ^2}}}}}\end{equation} \tag{ 5 }$$

where $\sigma$ represents the standard derivation.

The Gaussian scale-space of the image is represented as:

$$\begin{equation}S(x,y;\sigma ) = {G_s}(x,y) * {h_g}(x,y;\sigma )\end{equation} \tag{ 6 }$$

where, $*$ is the convolution operator.

The Laplacian operator is given by:

$$\begin{equation}{\nabla ^2} = \frac{{{\partial ^2}}}{{\partial {x^2}}} + \frac{{{\partial ^2}}}{{\partial {y^2}}}\end{equation} \tag{ 7 }$$

This Laplacian operator is then applied to $S\left( {x,y;\sigma } \right)$ in order to generate the Laplacian of Gaussian filter, which is first computed and then convolved with the image segmentation. Thus, its scale-space representation is calculated by

$$\begin{equation}{\nabla ^2}{h_g}(x,y) = \frac{{{x^2} + {y^2} - 2{\sigma ^2}}}{{2\pi {\sigma ^6}}}{e^{\frac{{ - ({x^2} + {y^2})}}{{2{\sigma ^2}}}}}\end{equation} \tag{ 8 }$$

Finally, our algorithm applies an evaluation (as reported in [16]) of the corners, as determined by Harris-Stephen, to obtain the indentation vertices $U,D,{\text{ }}L,{\text{ }}R$ (see figure 1).

We generated an image bank of Vickers indentations using a microdurometer (Mitutoyo model HM-125^®) and a microscope with up to 100-fold magnification, as shown in figure 8. An FPGA is used to convert the analog video signal supplied by the charge-coupled device to a digital video signal, linked to a video graphics array monitor. Using this graphic interface, the frames of the digital video signal can then be stored as 2D indentation images in gray-scale BMP format. Both the video signal converter based on FPGA, and the graphic interface have previously been reported in [23]. Our algorithm is implemented in MATLAB software, where the indentation images were handled as gray values corresponding to a range between 0 and 1.

Zoom In Zoom Out Reset image size

The indentation images reproduced by the microdurometer (Mitutoyo model HM-125^®) contain only one Vickers indentation, without rotation. The indentation images were obtained at a resolution of 640 × 480 pixels, and at 50-fold magnification. Our algorithm then converted the initial size of the indentation images from pixels to micrometers( see table 1). Finally, the algorithm determined the indentation image vertices and the Vickers hardness numbers.

The following image groups (see Appendix) were tested by our algorithm:

Group 1: specular-polished surface, comprising 68 samples of steel-316 and 2 of hafnium nitride (HfN) (see figures 9 or 10).

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Group 2: roughly polished surface, comprising 49 samples of steel-316 and 21 samples of HfN, see image of third row and column in figure 11.

Group 3: deformed indentations, comprising 50 indentation images, including 24 samples of HfN and 26 of steel-316 (see image in fifth row and third column in figure 11).

Group 4: 40 indentation images with strong imperfections (scratches, grease stains, and texture porosities), comprising 32 samples of steel-316 and 8 of HfN (see image in seventh row and third column in figure 11).

For the purposes of this experiment, a load of 500 gf was applied by the indenter to the steel samples, and a load of 300 gf was applied to the HfN specimens.

For all images, the diagonal lengths and Vickers hardness numbers were calculated by obtaining the ground true vertex coordinates via a conventional manual process implemented by a specialist in Vickers hardness testing. The relative errors in terms of the average diagonal length for each group of indentation images were compared with the corresponding results determined by our fully automatic algorithm. Table 2 shows that neither group has images where the resulting relative error was greater than 3%. Thus, there are no outlier images in the groups, unlike those reported in [21]. Table 2, comprising samples from Group 1, Group 2, Group 3, and Group 4 from the UV-Vickers Indentation Image Data Bank [24] shows the results for our algorithm, compared with those manually obtained by a specialist in Vickers hardness testing.

Many images from group 1 present a light spot in the indentation center, generated by the microscope's integrated light source. This problem is caused by the highly light-reflective characteristics of the specular-polished material surface. The segmentation of these images, as presented in [21], had a maximum area 8-component which did not coincide with the indentation. They registered a large number of corner points as potential indentation vertices, taking up a great deal of data processing time. In addition, the relative error increases in the detection of indentation vertices, as shown in figure 9. However, as a result of the image segmentation automatically obtained by our algorithm, the maximum area 8-component coincides with the indentation footprint. Thus, our algorithm determines fewer corner points as prospective indentation vertices, reducing the data processing time. Moreover, the relative error significantly decreases in the detection of indentation vertices (see figure 10). Group 1, Group 2, Group 3 and Group 4 from the UV-Vickers Indentation Images Data Bank [24] depict indentation vertices with a relative error in terms of average diagonal length of less than 1%. Figure 11 shows images from Groups 2, 3, and 4, whose segmentation, based on the results in [21], generated a high number of diagonal length errors. This is mainly due to low-contrast images and incorrect detection of their vertices. Using the same images, our algorithm detects the indentation vertices with relative errors ranging from 1% to 3%, as shown in figure 11. Results for groups 1-4 from the UV-Vickers Indentation Images Data Bank [24] show indentation vertices with a relative error, in terms of average diagonal length, of between 1% and 3%.

Figure 12 depicts an indentation image from [21], in which the morphological filter is insufficient to eliminate pixels detected as false positives for the indentation region. This is caused by inadequate processing of an input indentation image with low contrast. For this image type, our algorithm is robust and fully automatic in that it can determine the optimum value of $\tau$ in each image segmentation, and thereby detect the indentation vertices. Figure 13 shows the performance of our algorithm applied to the indentation image shown in figure 12, in which the four indentation vertices are detected.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Table 3 shows the relative error in the measurement of the average diagonal length of indentation vertices determined by different algorithms, and compared with those manually measured by a specialist in Vickers hardness testing. For all indentation image groups, our algorithm registered the smallest relative error as compared to the other algorithms. Moreover, the proposed algorithm was applied to indentation images with low contrast due to the high degree of light reflectivity on specular-polished material surfaces (see figure 14). Our algorithm has a relative error level below 3%. On the other hand, the algorithms reported in [8, and 21] create outlier images in their detection of indentation vertices. Figure 15 shows the indentation vertices determined by our algorithm for images where neither the indentation region nor the background can be categorised as having uniform gray values. This decreases the level of relative error in the detection of indentation vertices in comparison with results of other algorithms [8, 13–17, 21]. Thus, our algorithm is more efficient in detecting vertices in diverse types of Vickers indentations. Figure 16 shows outlier indentations obtained by our algorithm in relation to indentation imageswhere segmentation difficulties arise due to fractures in the specimen.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

In this paper, we have presented an algorithm for the automatic detection and measurement of Vickers indentation hardness, using image processing. This algorithm employed binarization criterion for each input image, together with mean gray values and extreme gray values, altering the mean gray values via automatic analysis with standard histogram equalization to determine the optimal binarization threshold. Morphological filtering was applied to the binarized image, followed by a segmentation on the growing region. The result obtained is a maximum area black 8-component of the image segmented by our algorithm. Thus, the indentation detection is based on the gravitational center of the indentation image. The proposed algorithm considered the Laplacian of Gaussian filter to average the two digital gradients contained in the Harris-Stephen matrix, where the average scale changes in the image, making it independent of the image resolution. In addition, the algorithm generates a smooth image segmentation, removing much of the noise in the indentation image.

Our algorithm evaluated a wide range of indentation images, in which the indentation edges are not exactly straight lines, and the gray values in both the indentation and the background of the image are not uniform. In addition, our algorithm was applied to indentation images with very low contrast in relation to the adjacent image area, and where the indentation image presented some shading, thereby resolving illumination problems in the case of specular-polished material surfaces. Thus, our algorithm has demonstrated its effectiveness and accuracy in determining indentation vertices in a variety of Vickers indentation images. Future research in this area will involve measuring the average roughness of the samples in at least two directions for each group of samples. Furthermore, our algorithm will be applied to Vickers indentation images reproduced with loads of less than 30 gf. In addition, our algorithm will be embedded using a FPGA platform to obtain both the video signal converter and algorithm in real time.

Indentation images of Group 1, Group 2, Group 3, Group 4, Group 5, results estimated by our algorithm and specialist annotations may be accessed at www.uv.mx/saudominguez/208-2.