Fault Diagnosis of Wind Turbine Blades Based on Image Fusion and ResNet

In the diagnosis of wind turbine blade faults, the information provided by a single sensor is limited. To address this issue and take advantage of complementary features among multiple fault information sources, while enhancing fault diagnosis accuracy, a method for diagnosing wind turbine blade faults is proposed. This method combines Image Fusion Convolutional Neural Network (IFCNN) with the ResNet network. Firstly, the time-frequency representation of vibration data is obtained using wavelet transform. The time-frequency representation and blade fault images are input into the IFCNN to obtain fused images containing two categories of fault features. Next, the ResNet convolutional neural network is employed to automatically extract non-linear features from the fused images, establishing a classification model for blade fault images. Experimental results demonstrate that, with limited training data, the classification accuracy of this method can reach 86.7%, outperforming fault diagnosis models trained with single fault information. This approach offers a new perspective and method for the fusion of multiple fault information in the field of wind turbine blade fault diagnosis


Introduction
With the aggravation of climate change and energy crises, wind power generation has gained widespread application and development as a clean and renewable energy source [1].By 2026, the installed capacity of wind power is projected to reach 557GW.However, as the installed capacity of wind power grows rapidly, the issue of faults in wind turbine (WT) has also become one of the urgent problems in this field [2], [3], [4].
Currently, the main focus of WT component failures is on the gearbox [5], generator [6], bearing [7], and blade [8], as their damages directly affect the power generation capacity of the WT [9].With the continuous increase in WT power output, blades have progressively become larger to generate more electricity, leading to a yearly increase in the proportion of blade failures [10].Therefore, conducting research on WT blade fault diagnosis is of significant importance in reducing the proportion of blade failures and extending their service life [11].
The fault diagnosis of wind turbine (WT) blades mainly includes contact and non-contact methods.Joshuva, A [12] et al. used piezoelectric accelerometer to obtain vibration signals of WT blades and employed an inertial classifier combined with histogram features for the multi-fault classification of WT blades, aiming to improve the accuracy and efficiency of fault diagnosis.Wang [13] et al. utilized a multi-channel convolutional neural network (MCNN) for detecting faults in WT blades.By automatically and effectively capturing fault features from raw vibration signals and identifying their states in multiple convolutional neural network (CNN) models, the experimental results showed that this method achieved an accuracy rate of 87.8% for the classification of four fault types in large-scale wind turbine blades.Kushwah [14] et al. collected vibration signals under different blade conditions through experiments and applied SVM, KNN, and decision tree classifiers for fault classification.The results showed that the SVM algorithm achieved the highest accuracy rate of 87% in fault classification, indicating that machine learning techniques can effectively be used to monitor the health condition of wind turbine blades.These contact-based diagnostic methods involve installing various sensors on the wind turbine to obtain fault data.However, during the actual operation of the wind turbine, the signals from different sensor components can be coupled, and there may be strong background noise, which limits the accuracy of data collected by sensors, thereby affecting subsequent diagnostic algorithms.
Fremmelev [15] et al.studied the development of defects in a 52m blade after various artificial damages using acoustic emission signals.By comparing the acoustic emission sensor with other sensors, they found that the acoustic emission sensor can detect the process of fatigue damage growth caused by blade damage sources.Ashim Khadka [16] et al. used a digital image correlation (DIC) system installed on a drone as a sensing technology.Through a multi-angle stitching method, they successfully obtained the dynamic characteristics of the entire length of the blade, realizing non-contact monitoring of offshore wind turbine blades.Guo [17] et al. combined Haar-AdaBoost and CNN to process the WT blade captured by UAVs and other devices, achieving fast and efficient identification of WT blade damage.These methods utilize non-contact devices to acquire image data, making data collection more convenient.However, they rely on subsequent processing algorithms, requiring a high level of algorithm accuracy.
In response to the above problems, this paper proposes a leaf fault diagnosis method based on image fusion and ResNet.First, we use wavelet transform (CWT) to convert the time-domain signal into a wavelet time-frequency image.At the same time, we use a camera to capture image data of the leaves.Then, we use the image fusion neural network (IFCNN) to merge the two types of image data into one category, and construct a dataset based on this.Finally, ResNet is used for fault classification and recognition.

Continuous Wavelet Transform
Continuous wavelet transform (CWT) performs inner product computations between wavelet functions and signals, obtaining wavelet coefficients at different scales and frequencies [18].These coefficients reflect the local features of the signal at different scales.The following calculation is performed to achieve this： In this equation, x(t) represents the input signal, a is the scale parameter, b is the translation parameter, and ψ * is the complex conjugate of the wavelet function.CWT can analyze the signal in both time and scale domains, providing an intuitive representation of the relationship between the frequency components of the signal and time.

Image Fusion Convolutional Neural Network
Image fusion can integrate prominent features from multiple input images into a comprehensive image.However, current image fusion models lack generalization ability and can only perform well on specific types of images.On the other hand, Image Fusion Convolutional Neural Network (IFCNN) is capable of fusing various types and quantities of input images [19].IFCNN adopts an element-wise fusion method, which does not involve any parameters and can fuse any number of input images.The fusion rule is chosen based on the characteristics of the image dataset [20] (element-wise maximum, elementwise summation, or element-wise average).The fusion formula is as follows: ( , ) ( ( , )),1 In the formula, fuse represents the fusion rule, j i f represents the i feature map extracted from the j input image, and j f represents the j feature map of the fused feature maps.In addition to the fusion part, IFCNN also includes the input image feature extraction and fused image reconstruction parts.The specific structure is as follows:

Deep residual network
In deep neural network, increasing the number of network layers can lead to the problems of gradient vanishing and gradient explosion.ResNet introduces residual learning to address this issue [21].It adds residual blocks between the input and output, where the residual blocks achieve the residual purpose by summing the input and output.The structure of the residual block is shown in Figure 2:

IFCNN-ResNet framework for fault diagnosis
The process of fault diagnosis for wind turbine blade using the model is shown in Figure 4. Firstly, vibration data collected by the vibration sensor is gathered.The collected vibration data is segmented and enhanced using the sliding window sampling method.WT blade data is captured using a 5-million pixel camera.Then, the segmented vibration data is transformed into a 2D wavelet time-frequency map of size 256×256×3 using CWT (cmor3-3 wavelet as the mother wavelet).At the same time, the 2D wavelet time-frequency map of the same fault type is combined with the fault blade image.The IFCNN is used to merge the two types of images to obtain a fusion image that contains the characteristics of both image types.The final step is to create a dataset of the fusion images according to the fault types.

Data acquisition and processing
To verify the effectiveness of the proposed fault diagnosis model for WT blades, experimental validation was conducted in the laboratory using a small wind turbine generator with a power of 100W.Actual vibration data and blade image data of the WT were collected for the experimental verification.
During the data acquisition process, to verify the effectiveness of the fusion features of IFCNN, vibration data was collected using a low sampling rate vibration sensor with a sampling frequency of 1Hz.However, due to the rotational characteristics of the blades during WT operation, direct monitoring of blade vibration was not possible.Therefore, the sensor was placed at the top of the WT main shaft, which is connected to the root of the blades, to collect radial vibration data.The image data of the blades was obtained through a camera.The data acquisition process is shown in Figure 5.

Results and discussion
After we get the data set, we use the ResNet34 network for fault diagnosis.The hyperparameters are as follows : BatchSize = 8, epoch = 15, learning rate is 0.0001.The confusion matrix of the test set after training is shown in Figure 8 :  From Figure 10, it can be observed that using the IFCNN to fuse fault data and then applying the ResNet for classification produces significantly better results compared to the other two diagnostic methods.In the CWT-AlexNet method, the three types of data are mixed together, making it difficult to accurately differentiate between fault types;Although CWT-ResNet shows some improvements over CWT-AlexNet, there are still overlaps among the different fault classes; The boundary between broken state and normal state in IFCNN-ResNet is ambiguous.This phenomenon occurs due to the limited number of training samples, but it achieves relatively good classification results.In conclusion, the performance of IFCNN-ResNet is superior to the other two methods.

Conclusion
In order to address the limited information provided by using a single sensor to diagnose faults in wind turbine (WT) blades, this paper proposes a method that uses IFCNN to fuse vibration data spectrograms with blade fault images, and then uses the ResNet network for fault classification.This method achieves feature complementarity between different types of fault data, thereby improving the accuracy of fault diagnosis.The effectiveness of the method is validated by collecting vibration data and image data of blade faults from a small-scale wind turbine with a power of 100W.Experimental results are compared with training AlexNet and ResNet networks only using vibration data spectrograms, demonstrating the superiority of combining IFCNN and ResNet.The training set of the proposed method is only compared with half of the training set of the comparative method, which indicates that using the IFCNN preprocessing method can enable the fault diagnosis model to achieve better performance.In addition, we will attempt to incorporate more diverse types of fault data for training the neural network in the future.Furthermore, more experiments are needed to validate the proposed method, especially in the actual scenario of fault diagnosis for wind turbine blades.

Figure 1 .
Figure 1.IFCNN structure diagram.In Figure 1, CONV1 and CONV2 are used to extract the features contained in the input images individually.After feature fusion, CONV3 and CONV4 are used to reconstruct the fused image from the fusion of convolutional features.IFCNN possesses strong generalization ability and can achieve fusion of various image features, making it applicable in the field of fault diagnosis.

Figure 2 .
Figure 2. Residual block structure.ResNet consists of several residual blocks, which makes the network very deep and can extract more image features, thereby increasing the accuracy of image classification.In this paper, ResNet34 is used for WT blade fault diagnosis.The structure is shown in Figure 3 :
The dataset is proportionally divided into training and testing sets.The ResNet34 model is constructed and trained using the training set to obtain the WT blade fault diagnosis model.The model is then used to diagnose the faults in the testing set, and the classification results of the testing set blade fault diagnosis are outputted.

Figure 4 .
Figure 4. Architecture of the proposed method..

Figure 5 .
Figure 5. Data collection process.According to the existing fault types of WT blades, we set up two types of faults that are most likely to occur in WT blades, and obtain three types of WT blade states 1) normal state ;2) broken state ; 3) cracking.The vibration signals of WT blades in different states are shown in Figure 6 :

Figure 6 .
Figure 6.Time domain vibration signal of WT blade.After obtaining the vibration data, 256 points are selected as data samples.The one-dimensional vibration data is transformed into a two-dimensional wavelet time-frequency map using CWT.At the same time, the IFCNN is used to fuse the obtained time-frequency map with the blade image data.This fusion generates an image that contains both the time-frequency characteristics of blade vibration and the visible fault features, thereby constructing a new fault diagnosis dataset.Forty sets are chosen as the

Figure 7 .
Figure 7. Time-frequency diagram, visible light diagram and fusion image of different fault types.

Figure 8 . 9 .
Figure 8. Fault classification confusion matrix.The horizontal and vertical coordinates of the confusion matrix represent the real label and the predicted label, respectively.It can be seen that the classification accuracy of the ResNet34 network for the three types of states reached 80 %, 100 %, and 80 %, respectively, and the overall accuracy was 86.7 %.In addition, when training the network, the ratio of training set to test set is 40 : 20, which also shows that the data set constructed by using IFCNN to fuse vibration time-frequency map and blade image data can achieve feature complementarity, so that the fault diagnosis network can obtain more fault features under the condition of few training samples, so as to obtain high classification accuracy.In order to verify the superiority of IFCNN fusion image in WT blade fault diagnosis, this method is compared with the fault diagnosis model trained only by time-frequency diagram of vibration data.The time-frequency diagram is sent to AlexNet and ResNet34 network training to obtain the diagnosis model.