Effects of Attack Interference on Radio Signal Classification

Signal recognition and classification are urgent issues in the field of radio monitoring. Although there have been many academic publications on the automatic modulation classification of radio signals, these studies have focused on how to use the new algorithms to increase signal recognition accuracy, and there is a lack of consideration of the effect of interference on classification accuracy. Based on publicly available datasets, RF datasets for Machine Learning, this paper uses the deep learning algorithm to generate an attack interference dataset and studies the impact of attack interference and noise on the classification accuracy of radio signals. The experimental results are as follows. Firstly, the presence of attack interference generally reduces the accuracy of radio signal classification. Secondly, the power of attack interference has a direct relationship with classification accuracy. Thirdly, under the condition of attack interference, the classification performance trends of the three open-source public datasets are consistent. This further illustrates how optimizing the modulation classification performance in the presence of interference is the key to applying the algorithm to practice.


INTRODUCTION
The goal of radio monitoring is to solve problems such as "what type of radio wave, where it comes from, and whether it is compliant with the spectrum allocation rules".Concerning their intangibility, invisibility, and free-space propagation, radio waves are extremely sensitive to interference and noise.To accurately identify the modulation type of radio signals, numerous problems must be resolved in the space electromagnetic environment.Previously, the type of modulation had been identified by statistical techniques [1], which had few practical values due to low recognition accuracy or lack of generalization.Recently, much attention has been drawn to automatic modulation classification by using deep learning models, especially the release of the open-source GNU radio ML dataset [2].In terms of the dataset, a lot of research has been conducted to improve the accuracy of modulation classification.For example, to discriminate between various modulated signals, convolutional neural networks (CNNS) were proposed [3].The conclusion indicts that the CNN model can achieve an accuracy of 87.4%.Convolutional Long Short Term Deep Neural Networks (CLDNN) with deep structure [4] achieve 88.5% classification accuracy.Rajendran et al. [5] proposed a simple LSTM with inputs for amplitude and phase, which improved the signal recognition rate to 90%.Meanwhile, by using the RML 2016.10adataset, our laboratory achieved an accuracy of 93.7% by pre-processing the data of the DPM-SCNN algorithm [6].
However, these studies focus on how to improve the accuracy of signal modulation type identification, and there is a lack of reports on the impact of interference on classification accuracy.Fully weighing the classification accuracy and computational complexity, and combining various forms of feature extraction methods, the proposed Gated Graph Convolutional Neural Network (GGCNN) [7] can achieve an accuracy of nearly 100%, and it is difficult to improve the classification performance.Although the use of deep learning algorithms greatly improves the probability of correct classification of modulated signals, it is in an ideal situation.When wireless signals are transmitted in electromagnetic space, there are various challenges, such as spectrum utilization, signal fading, the signal being disturbed by noise and interference, etc.However, the public RadioML datasets now available only take into account the noise and ignore the presence of interference, which is detrimental to communication security in the actual modulation recognition application.Therefore, considering the existence of interference and noise in the received signal will be more in line with the actual transmission of the signal.

SIGNAL MODEL
The attack interference presented in this paper is achieved by finely perturbing the features of the input benign sample to fool classifiers.Figure 1 illustrates how the modulated signal from the transmitter is transmitted to the receiver.If the system just takes noise and interference into account, the received signal can be modeled by using the following equations: where   denotes the transmitted signal, ℎ  denotes the time-varying pulse of the wireless channel, * denotes the convolution method,   denotes the noise, which is usually known as Gaussian white noise,   denotes the interference and is assumed to remain constant during the signal transmission, and   denotes the received signal.The intelligent receiver faces a challenge in figuring out how to identify and determine which modulation types that a wireless signal belongs to.
The two directions of in-phase I and orthogonal Q for each sample of a modulated signal denoted by  / can be expressed as follows: Figure 1.Interference model

INTERFERENCE DATASETS
In a wireless system, the modulated signal received from one (or more) authorized receivers is referred as the original signal x when attack interference is absent.The received signal is defined as x x  when the attack interference  appears.To study the impact of interference on classification accuracy, how to simulate interference is the key to this study.This can be described as [8]: where  is attack interference, ‖•‖ represents the  norm which is considered to be a measurement of the difference between the original signal and interfered signal,  ,  denotes the output of f (,;θ) for kth class.The detailed method for producing attack interference has been given in Algorithm 1.The core of the algorithm is principal component analysis (PCA) [8], which can find the best interference direction, supposing we have a random subset of input  ,  , ⋯ ,  , associated with the true label The RadioML datasets including RML2016.10a_dict,RML2016.10b, and 2016.04C.multisnr are commonly utilized datasets for modulation classification that are available at https://www.deepsig.ai/datasets/,but the impact of interference is not considered.According to the above method, the attack interference superimposes the samples in the datasets to create three RML2023 datasets: RML2023A_int, RML2023B_int, and RML2023C_int, which are publicly available at https://github.com//RMLI_DATASET/.As shown in Table 1, the RML2023 datasets increased in size by 11-fold as a result of the addition of INR.

CLASSIFICATION MODEL
To understand the effects of attack interference on radio signal classification, a CNN2-style convolutional neural network architecture [3] named ConvNet is used in this study.The structure of ConvNet is illustrated in Figure 2, denoted as ConvNet Model.It is a 7-layer network with three convolutional layers and four dense fully connected layers.Except for the Sigmoid activation function of the output layer, the activation function of the other layers is Rectified linear unit (ReLU).
The ConvNet is firstly trained on the original signal dataset without interference.Adam is utilized as the optimizer and the cross-entropy is chosen as the loss function.Then the dataset of the signal with attack interference is fed to the trained model for modulation type classification.This is aim to simulate the phenomenon that the system is disturbed by attack interference, which may reduce the classification accuracy.

RESULTS AND DISCUSSION
Since the existing RadioML datasets take into account the presence of Gaussian noise, both noise and interference affect the classification accuracy in our RadioML2023 datasets.
To evaluate the effect of noise and attack interference on the modulation recognition rate, we used the signal-to-noise ratio (SNR) and interference-to-noise ratio (INR) to quantify the metric.SNR measures the strength of desired signal relative to background noise.INR is the ratio of interference intensity to noise intensity and is used to quantitatively describe the magnitude of the interference.Experiments were carried out with three generated datasets with attack inference.The classification accuracies of the modulated signal under the condition of noise and attack interference are shown in Figure 3, Figure 4, and Figure 5.
In Figure 3 (a), it can be concluded that the higher the SNR is, the better the classification accuracy is.Moreover, when the INR is large, the classification accuracy increases slowly.When INR ≤ 0 dB, the classification accuracies are nearly coincident at the SNR of 18 dB, indicating that small attack interference does not work at high SNR.In Figure 3 (b), the classification accuracy decreases with increasing INR.When the SNR is large enough, the classification accuracy decreases slightly at higher INR.The attack interference is considered imperceptible when INR ≤ 0 dB, meaning that the power of the noise is higher than the power of the interference.Similar conclusions could be drawn from Figure 4 and Figure 5.The confusion matrices offer crucial information about primary modulation types that lead to severe classification errors.When INR = -2 dB, a representative confusion matrix is selected in Figure 6, and the matrix at SNR = 8 dB is selected in Figure 7.The two figures are respectively composed of 4 confusion matrices.Each matrix has a vertical axis that represents the true types of signals and a horizontal axis that represents the signal type predicted by the classifier.The darker the color of the squares is in each matrix, the greater the number of samples that the classifier judges the corresponding signal type on the left as the signal type at the bottom.We summarize the observations below:  In Figure 6, with the increase of SNR, the number of correctly recognizing modulation types increases except for 16QAM and 64QAM, which have overlapping constellation points [9].
 In Figure 6, with the increase of SNR, it is possible to classify a sizable portion of WBFM modulation data as AM-DSB.Both WBFM and AM-DSB's data come from speech signals with off-time and occasional interludes [3], causing classification errors.
 Figure 7 corresponds to the inverse process of Figure 6, illustrating that attack interference, like noise, reduces the number of modulation types correctly identified.
The experiment takes the size of the input data N = 500, and the comparison of the classification accuracy with and without attack interference is summarized in Table 2, where RML2016.10a_dict and RML2023A_int denote datasets of type A, where RML2016.10b and RML2023B_int denote datasets of type B, and where 2016.04C.multisnr and RML2023C_int denote datasets of type C. It reveals that when attack interference was added to the three datasets, their maximum, minimum, and average recognition accuracies changed in the same trend.To demonstrate the generalizability of our RML2023 datasets with attack interference, we experimented on the convolutional neural network (ConvNet) [3], the residual neural network (ResNet) [10], and long short term memory (LSTM) [5] with RML2023A_int, which gets attack interference by using ConvNet model.As shown in Figure 8, the solid line denotes no attack interference, and the dotted line denotes the attack interference of intensity of 0dB.It is clear that the attack interference performs well on both LSTM and ResNet models even though it is developed by using ConvNet.Overall, the attack interference makes the classification accuracy of all models drop sharply, especially at  0, where the accuracy of modulation classification is maintained at about 10%.

Figure 2 .
Figure 2. Interference principle and classifier model

Figure 8 .
Figure 8.Comparison of classification accuracy of representative models with and without attack interference on RML2023A_int 6. CONCLUSION Firstly, the generation method of attack interference for the radio signal is studied.Then the attack interference is added to the RadioML data set to build a new data set, and on this basis, ConvNet is used to conduct modulation classification, and the impact of interference is analyzed.The results show that the attack interference will greatly reduce the classification accuracy of the radio signal, especially when the INR is greater than 0 dB, the classification accuracy of 84% can only be achieved at an SNR higher than 18 dB.Radio signals are transmitted in open space, the signals may be disturbed by various external factors during transmission, including attack interference.Therefore, to apply the method of modulation classification in practical radio monitoring systems, improving the classification accuracy and the robustness of the deep learning model in the presence of attack interference and reducing or even eliminating the impact of attack interference are the focus of our future research.

Table 1 .
,  , ⋯ , and interference directions  ,  , … ,  , where the normalized gradient   ,   ,  ∑  log  .Noting that  ∈  ,  ∈ 1,  , so  ,  ,  , … ,  is a matrix with dimension   .By performing a singular value decomposition on the matrix, the first principal component can be found.Since the first principal component with the largest variance will account for as much variability in  ,  ,  , ⋯  as possible, we use its direction as the direction of attack interference.Comparison of old RML2016 and new RML2023.

Table 2 .
Comparison of the modulation classification accuracy