Compact convolutional neural networks for classification of asynchronous steady-state visual evoked potentials

2.1.1. EEGNet: compact convolutional neural network (Compact-CNN).

To assess the utility of deep learning approaches for SSVEP BCI, our analysis used our Compact-CNN (the EEGNet architecture) that was designed for classifying raw EEG when only limited amounts of data are available, such as in SSVEP BCI experiments [35]. Our Compact-CNN approach efficiently represents EEG signals in a compact manner by first performing temporal convolutions, with the convolutional kernel weights being identified from the data. The first layer of the network then performs a temporal convolution to mimic a bandpass frequency filter, a result supported by the convolution theorem [42]. Our approach then uses depthwise spatial convolutions that act as spatial filters to reduce the dimensionality of the data. The main benefit of depthwise convolutions is reducing the number of trainable parameters to fit, as these convolutions are not fully-connected to all previous outputs. When used in EEG-specific applications, this operation provides a direct way to learn spatial filters for each temporal filter, thus enabling the efficient extraction of frequency-specific spatial filters. The Compact-CNN also uses separable convolutions to more efficiently combine information across filters [43]. The main benefits of separable convolutions are (1) reducing the number of parameters to fit and (2) explicitly decoupling the relationship within and across outputs by first learning a kernel summarizing each output individually, then optimally merging the outputs afterwards. As described in table 1, each convolution layer is followed by batch normalization, 2D average pooling, and dropout layers. In the first two layers, the exponential linear unit (ELU) non-linearity (as opposed to other non-linear activation functions such as sigmoids or rectified linear units) was employed since it resulted in superior performance for EEG classification [35]. The fourth and final layer is connected to a dense layer with a softmax activation function for classification. Our Compact-CNN is trained using a categorical cross-entropy loss function (shown in equation (1)):

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle L_i = -\sum_{j} t_{i,j} log(\,p_{i,j})\label{equ1} \nonumber \end{align} \tag{ 1 }$$

where p are the model predictions, t are the true labels, i denotes the sample number, and j denotes the class. The model was implemented in Tensorflow [44], using the Keras API [45]. The model was trained for 500 iterations using the Adam [46] optimizer, with a minibatch size of 64 trials. The dropout probability was set to 0.5 for all layers. For this application, the model learned 96 temporal-spatial filter pairs (F₁  =  96, F₂  =  96 and D  =  1). The complete model architecture of our Compact-CNN is shown in table 1; our implementation can be found at https://github.com/vlawhern/arl-eegmodels.

Table 1. Compact-CNN architecture, where C  =  number of channels, T  =  number of time points, F₁  =  number of temporal filters, F₂  =  number of separable filters (here, $F_1 = F_2$), D  =  number of spatial filters to learn per temporal filter and N  =  number of classes, respectively.

Layer	Layer type	# filters	Size	# params	Output	Activation	Options
1	Input				(C, T)
	Reshape				(1, C, T)
	Conv2D	F1	(1, 256)	256 * F1	(F1, C, T)	Linear	Mode  =  same
	BatchNorm			2 * F1	(F1, C, T)
	DepthwiseConv2D	D * F1	(C, 1)	C * D * F1	(D * F1, 1, T)	Linear	Mode  =  valid, depth  =  D, max norm  =  1
	BatchNorm			2 * D * F1	(D * F1, 1, T)
	Activation				(D * F1, 1, T)	ELU
	AveragePool2D		(1, 4)		(D * F1, 1, T // 4)
	Dropout				(D * F1, 1, T // 4)		Rate  =  0.5
2	SeparableConv2D	F2	(1, 16)	$16 * D * F_1 + F_2 * (D * F_1)$	(F2, 1, T // 4)	Linear	Mode  =  same
	BatchNorm			2 * F2	(F2, 1, T // 4)
	Activation				(F2, 1, T // 4)	ELU
	AveragePool2D		(1, 8)		(F2, 1, T // 32)
	Dropout				(F2, 1, T // 32)		Rate  =  0.5
3	Flatten				(F2 * (T // 32))
Classifier	Dense	N * (F2 * T // 32)			N	Softmax

2.1.2. Standard CCA.

Canonical correlation analysis (CCA) is a statistical analysis technique that finds underlying correlations between two multidimensional datasets. Given two multidimensional variables X and Y, CCA seeks to find weight vectors W_x and W_y such that their corresponding linear projections $x = X^TW_x$ and $y = Y^TW_y$ have maximal mutual correlation. These projections are found by solving the following objective function:

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle \max_{W_s,W_x} \rho (x,y) = \max_{W_s,W_x} \frac{E[W_x^TXY^TW_y]}{\sqrt{E[W_x^TXX^TW_x]E[W_y^TYY^TW_y]}}. \nonumber \end{align} \tag{ 2 }$$

For SSVEP detection, CCA computes the canonical correlation between multichannel EEG data and a set of reference signals Y_k composed of sines and cosines matching the fundamental and harmonic frequencies of the target stimulus. Reference signals are made for each of the K frequency stimuli where each set contains N_h harmonics of the fundamental frequency f_k.

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle Y_k = \left(\begin{array}{@{}c@{}} \sin(2\pi f_kt) \nonumber \\ \cos(2\pi f_kt) \nonumber \\ \vdots \nonumber \\ \sin(2\pi N_h\, f_kt) \nonumber \\ \cos(2\pi N_h\, f_kt) \end{array}\right).\label{equ3} \nonumber \end{align} \tag{ 3 }$$

EEG data is canonically correlated with each set of reference signals, and SSVEP detection is made on the basis of selecting the frequency set which provides the maximum canonical coefficient, $f_k = \max_k \rho(x, f_k)$, where $f_k=f_1, f_2, ... f_K$. In this analysis, K  =  12 since data was used from a 12-class SSVEP BCI [39] with numbers on the virtual keypad flashing at frequencies ranging from 9.25 Hz to 14.75 Hz in steps of 0.5 Hz.

2.1.3. State-of-the-art Combined-CCA.

Combined-CCA is a recent extension to the CCA method that combines the reference signals from traditional CCA with prototype responses derived from participant EEG data to better account for the imperfect match between actual brain EEG data and the approximate sine and cosine signals [18, 19]. In the Combined-CCA method, participants complete a calibration session before using the SSVEP BCI to collect training data for each stimulus frequency, and EEG data for each frequency is averaged across the training trials. In our recent extension of Combined-CCA [20], we introduced a pooled transfer approach that averages SSVEP trial data from other subjects to eliminate a calibration session for the current user.

To create the prototype responses, $\bar{X_k}$, data were taken from 9 of the ten subjects and averaged across trials and subjects for each stimulus frequency k. Using these prototype responses $\bar{X_k}$ as well as the sine and cosine reference signals Y_k (equation (3)), the Combined-CCA classifies the data from the test subject, X, by fusing together all pairs of canonical correlations between $\bar{X_k}$, Y_k and X using a weighted average resulting in a combined correlation coefficient for each class, p_k. The frequency that maximizes this weighted correlation value is selected as the SSVEP target:

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle f_k = \max_k \rho(k), \quad k = 1, 2, \dots K. \nonumber \end{align} \tag{ 4 }$$

To assess whether deep learning approaches improve classification of SSVEP signals, performance of the Compact-CNN was compared to the multivariate statistical analysis technique, CCA, and its current state-of-the-art variant, Combined-CCA. The classification accuracy for each method is shown for each of the ten participants in figure 2, and results demonstrate the robust performance improvement for Compact-CNN relative to the two comparison methods. Compact-CNN proved to be particularly beneficial for subjects whose performance was notably poor when CCA based methods were used (i.e. subjects 1, 3, 9 and 10).

Zoom In Zoom Out Reset image size

When evaluating the overall mean performance (figure 2, right), results demonstrated that Compact-CNN significantly outperformed CCA and Combined-CCA methods as indicated by paired t-tests (Compact-CNN versus Combined-CCA: $t(9) = -10.5, p<0.0001$; Compact-CNN versus CCA: $t(9) = -8.7, p<0.0001$). Surprisingly, the state-of-the-art Combined-CCA method performed significantly worse than CCA, and as subsequent analyses will show, a likely explanation is that the method is not prepared to work with asynchronous data that is not phase locked.

Our analysis next examined the underlying learned representation of the Compact-CNN, investigating the diagnostic features of the data that may account for the superior performance. Our first analysis along this effort focused on interpreting the learned temporal kernel weights in layer 1. Figure 3 visualizes a subset of the temporal kernels learned by the Compact-CNN model for one randomly-selected fold. Here we see that the Compact-CNN identifies narrow-band frequency activity along a spectrum of frequencies, both slow-wave (Kernel 63, at approx. 9 Hz) and fast-wave (Kernel 95, approx. 14 Hz). These frequencies closely align with that of the SSVEP experimental stimulus frequencies, suggesting that the model is capturing task-relevant oscillatory activity.

Zoom In Zoom Out Reset image size

Our next analysis uses a data reduction and visualization technique (t-SNE) to investigate the hidden unit activation structure across all layers of the Compact-CNN. The activation in each layer was similar, so only the t-SNE projections of the activation in layer 3 of the Compact-CNN were plotted in figure 4 for all training observations (across the nine training subjects) in fold 1.

Zoom In Zoom Out Reset image size

This visualization is a projection of the data down to two dimensions, allowing us to estimate discriminability among training trials and to infer the properties of the learned representation that account for classification performance. Each element of the t-SNE plot represents a single trial with a color that indicates different known features. In figure 4, trials were labeled based on their class (one of 12 numbers on the virtual keypad), the participant ID (one of 10 unique individuals), and the trial order (one of 15 trial bins ordered from first to last). Only the class plot revealed coherent clusters of colored elements, indicating that the learner identified the 12 unique classes of the numeric keypad. The lack of clusters in the subject plot confirmed that the variability between individual participant brain signals does not account for the feature representation, and lack of clusters in the trial plot demonstrated that there was not a time-on-task effect (e.g. fatigue, drowsiness, inattention) that differentiated the trials across the duration of the experimental session.

Next, our analysis examined the composition of the clusters themselves. Interestingly, the class plot revealed separable clusters that were colored with the same class label. In figure 4(A), elements colored yellow belong to the trials where the participant fixated on the number 8 that flickered at 12.25 Hz. This reveals that the learner differentiated different trials from the same class. This unexpected separation is not predicted from the derivation of the model, as the objective function in equation (1) only seeks to separate the 12 classes. Consequently, our analysis investigated whether these within-class clusters may be related to another known feature of the dataset, namely the 1 s segments of the original 4 s trial epochs.

In figure 5(A) (right), the elements in the 12.25 Hz class (yellow trial elements) were recolored according to their segment of the original 4 s trial, where 0–1 s is blue, 1–2 s is red, 2–3 s is green, and 3–4 s is orange. The segments account for the separable clusters within-class. This indicates that the deep learner is influenced (and learns) EEG features other than the trained class-level differences.

Zoom In Zoom Out Reset image size

Next, our analysis characterized the signal properties of raw SSVEP responses for each trial with a phase and amplitude analysis shown in figures 5(B) and (C), respectively. The radial phase plot in figure 5(B) shows that the EEG trial segments have separable phases with the 3rd and 4th segments clustering at opposite phases to the 1st and 2nd segments of the trial. In contrast, figure 5(C) shows similar amplitudes across the four trial segments and only Dim2 values of figure 5(A) seem to denote trial segment separability.

While figure 5 illustrates phase and amplitude features for the 12.25 Hz class from channel Oz, the subplots of figure 6 confirm that these patterns are common across other stimulus classes and channels. The average estimated phase (left column) and amplitude (right column) are depicted for the 9.25 Hz class (top row), the 12.25 Hz class (middle row), and the 14.75 Hz class (bottom row). The bottom diagram in figure 6 shows the corresponding channel layout and coloring. Phase and amplitude estimations were averaged over each trial and subject for the corresponding stimulus frequency and are shown for each electrode as colored line plots. Segment-wise differences are strongest for phase, but they can also be seen for amplitude features across all EEG channels. Here, only a few classes were illustrated for brevity, but this pattern is consistent across classes and scalp locations.

Zoom In Zoom Out Reset image size

Deep learning approaches allow computational models to learn representations of data with multiple layers of abstraction. This in turn can allow for the discovery of rich features or structure within large datasets. However, since these layers of features are learned instead of designed by human engineers, interpreting the meaning of these features remains a critical challenge in the field of deep learning [47, 48]. In this paper, our analysis utilized the representational learning capabilities of multilayer CNNs and show that the model learned interpretable features from EEG data that are consistent with the SSVEP literature. The Compact-CNN identifies phase features that are invariant to both subject and trial (time-on-task) differences.

Our results extend previous studies that have applied deep learning to SSVEP BCI paradigms. These prior approaches leveraged domain specific knowledge when constructing their feature extraction layers [36–38]. The first approach applied a CNN for the classification of SSVEP signals by using the fast-fourier transform (FFT) to convert time-domain representations into frequency-domain representations [36]. The second, more recent approach transformed SSVEP data into the frequency domain as a pre-processing step before classification with a CNN [37]. A similar approach was taken in [38], where the authors used an FFT-based hidden layer for classifying SSVEP signals. These approaches incorporate domain-specific knowledge of the stimulus frequencies by utilizing the FFT to transform raw time-domain EEG signals to the frequency domain since spectrally focal power differences are known to be discriminative features in the context of SSVEP-based decoding. All these approaches incorporate frequency specific knowledge, but no phase information. In contrast, our Compact-CNN operates on broadly-filtered EEG signals, and the relevant features (frequency and phase) are learned and extracted directly from these input signals. This end-to-end approach might be particularly advantageous for use-cases where domain-specific knowledge is unavailable.

Additionally, our results demonstrated that the Compact-CNN was able to discover deeper structure within the SSVEP dataset in the form of phase information that reflects features of the 1 s segments of the original 4 s SSVEP trial epochs. This learned structure is present in the t-SNE projections of the model activations in the hidden layers. Thus, the Compact-CNN provides a powerful tool for interpreting intrinsic structures within EEG datasets, and critically, this type of structure discovery would not be possible using standard linear methods, such as CCA or Combined-CCA.

SSVEP signals can be generated and classified using either synchronous or asynchronous temporal coding paradigms [49]. In synchronous paradigms, the onset and offset of the user gaze towards the stimulus are known: the user is cued to look at a flashing stimulus in a specified time-window so that the user's gaze is synchronized with the onset of stimulus flashing. In contrast, asynchronous SSVEP paradigms allow users to freely gaze/attend to any stimulus at any time. Asynchronous paradigms are generally more flexible and are more amenable to practical BCI. For some applications, asynchronous temporal coding paradigms can make BCI interaction faster, more intuitive, and/or ergonomical as the user can directly communicate a specific command of choice instead of waiting for the respective system cues in synchronized approaches. EEG-based decoding however, is typically more difficult in asynchronous SSVEP paradigms, particularly if phase information of the stimulus is not known.

Here, our division of the 4 s SSVEP trial epoch into 1 s segments simulates an asynchronous SSVEP BCI from a discretized stimulation paradigm. These segments approximate a less-controlled paradigm wherein a participant may randomly attend to a stimulus frequency. A segment length of 1 s represents a fast-paced BCI design where classifications can be made every second. On this data set, the Compact-CNN method outperformed state-of-the-art approaches in simulated asynchronous operation. The Compact-CNN learner extracted relevant frequency and phase features directly from the broadly-filtered EEG signals even when specific phase information of the SSVEP trials was unknown. This improvement could potentially benefit a variety of asynchronous SSVEP BCI applications including control of wheelchairs [17, 50], neuroprosthetics [51], exoskeletons [52], or interaction with virtual [53, 54] or augmented reality [55]. The observed performance improvement may also make this asynchronous approach attractive for applications like spellers [19], where synchronous paradigms have been traditionally preferred for their higher information bandwidth at the cost of lower flexibility.

Additionally, achieving calibrationless BCI classification is paramount for developing more practical BCI systems as it eliminates the need for time-consuming training sessions for the user [56]. Using a leave-one-subject out transfer protocol, our Compact-CNN approach outperformed the baseline CCA approaches without any user-specific calibration. This is further reflected in t-SNE projections which indicate that our deep learning model is able to learn subject invariant features, which is critical for transfer across subjects. Based on its promise for asynchronous BCI paradigms and the fact that user-specific calibration is not required, this Compact-CNN approach gives BCI designers additional options to tailor SSVEP BCI systems to the needs of specific user-groups or other requirements.

The Convolution Theorem states that convolutions of signals in the time-domain relate to multiplication in the frequency-domain. Since the first layer of our Compact-CNN is a temporal convolution (whose weights need to be learned from the data), our network has the capability to learn frequency-specific temporal filters, including EEG features as shown in our previous work [35]. We showed in figure 3 that the Compact-CNN model is capable of extracting narrow-band task-specific slow-wave and fast-wave frequency activity. We believe that our network is also capturing information correlated to frequency through the use of the average-pooling layer in layer 1 of our model. The sequence of operations in layer 1 (temporal convolution, ELU non-linearity then average-pooling) is similar to the methodology of [57] for calculating event-related synchronization and desynchronization features. In their work, they narrow-band filter, then square, then average over a moving window the signal to obtain an estimate of frequency power. The similarity of this approach to the operations in layer 1 of the Compact-CNN suggests that our model is calculating features at least correlated to that of frequency power.

In addition, convolutional neural networks have the property of equivariance to local translation [58–60]. In particular, when the data is a time-series signal, local translations correspond to phase offsets in the signal. This means that the output of the convolution operation (a dot product between the convolutional kernel and the signal) will preserve the phase information in the signal relative to the phase of the convolutional kernel (i.e. when the convolutional kernel is in phase or in anti-phase with the data). The ability of convolutional neural networks to capture translation-invariant features (in the case of a time-series signal, phase-invariant) was proven in [59]; it was also shown that pooling layers and model depth were critical to achieve this (additional theoretical treatment of convolutional neural networks and their ability to extract translation-invariant features can be found in [60]). It is through this property that our Compact-CCN approach was able to extract phase information from the signals of interest. Figure 5, which shows a t-SNE projection of the hidden unit activations in layer 3, provides additional evidence that our network can capture this phenomena.

While this paper largely focuses on the relevance for BCI applications, the Compact-CNN's ability to detect phase holds incredible potential to augment our understanding of visual processing in common SSVEP experimental paradigms. For example, SSVEP experimentation has revealed the temporal dynamics and spatial constraints of spatial attention when divided across locations, refining theories of how we attend to specific locations in our environment [61]. Likewise, phase information about oscillations of alpha activity within coordinated populations of neurons in the visual cortex has been shown to predict whether stimuli will be detected in the environment, suggesting that phase information drives coordinated excitation that facilitates perception or synchronized inhibition that prevents stimulus detection [62–64].

Here, the Compact-CNN identified phase information in the SSVEP response that was irrelevant for the 12-class classification but robust in the neural response, both across stimulus frequencies as well as channels. The ability to detect nuanced phase variability in EEG signals holds incredible potential to augment our understanding of the role of phase information in neural processing more generally. Variability in phase measured with EEG has been shown to modulate the neural response to a stimulus [65–68], influence reaction times [69, 70], and determine whether multisensory input is bound as a coherent percept [71]. Phase information within the neural response has been suggested to carry precise informational content [72, 73] and may even be a component of the neural code [74]. Our results demonstrate that a deep learner may provide an innovative way to reveal robust phase variability between stimuli that underlie the intricate coordination of neural activity that gives rise to cognition.

Within this dataset, the impaired performance of Combined-CCA likely arose from insufficient representation of the precise phase information across the 1s segments. Future research could investigate whether Combined-CCA could be adapted or used differently to better handle asynchronous operation. For example, provided that the BCI has knowledge of the phase of the stimulus, the template could be shifted so that its phase matches the phase of the stimulus. This may improve performance of Combined-CCA in an asynchronous setup. It is noteworthy though, that phase information of the stimulus is not available in all applications. Our Compact-CNN does not require phase information, suggesting its versatility for both experimental use and novel BCI applications.

Finally, our experimental setup did not include a non-control state, which is typically utilized in self-paced BCI systems. Future work could investigate the efficacy of our method in the presence of a non-control state in a closed-loop setting, which could lead to even more flexible and intuitive SSVEP-based human machine interaction.