Improving EEG-based decoding of the locus of auditory attention through domain adaptation^*

This paper is structured as follows: section 2 explains Riemannian geometry, and section 3 describes LoA classification with EEG. DA and PT are explained in section 4. The experimental setup, preprocessing of the EEG data, model evaluation method and statistical analysis are discussed in section 5. Results and discussions are presented in section 6. Lastly, section 7 sums up the paper with conclusions. A pipeline from data to classification accuracy is provided in appendix A and a more in depth explanation of PT is presented in appendix B.

The covariance matrix $\mathbf{P}_{s,i}\in \mathbb{R}^{d \times d}$ is defined as:

$$\begin{equation} \mathbf{P}_{s,i} = \mathbb{E} \left[ \left(\boldsymbol{x}_{s,i} \left(t\right) - \mathbb{E}\left[\boldsymbol{x}_{s,i}\left(t\right) \right] \right) \left(\boldsymbol{x}_{s,i} \left(t\right) - \mathbb{E}\left[\boldsymbol{x}_{s,i}\left(t\right) \right] \right)^\mathrm{T} \right] \end{equation} \tag{ 1 }$$

where $\boldsymbol{x}_{s,i}(t)$ is the recorded EEG time series for subject s and trial i. Each trial in the EEG dataset used in this study is structured in a t × d matrix, where t is the number of samples and d is the number of EEG channels. Each element in the d × d covariance matrix $\mathbf{P}_{s,i}$ describes the covariance between the corresponding channels. Preprocessing of the EEG data is further explained in section 5.2.

One way to describe the curvature of the Riemannian manifold $\mathcal{M}$ is through a so-called sectional curvature. The manifold has a tangent space $\mathcal{T}_\mathbf{P}\mathcal{M}$ at the point $\mathbf{P} \in \mathcal{M}$, and the sectional curvature is defined by the point P and a two-dimensional subspace of the tangent space. Hence, the sectional curvature depends on two linear independent tangent vectors, and it is therefore possible to view the (symmetric) covariance matrices on the Riemannian manifold as vectors in the Euclidean tangent space. These vectors are used as features in the classification, further explained in section 3. In this paper, the point P is a covariance matrix and S is the vector in Euclidean space. The shortest path between two covariance matrices $\mathbf{P}_1, \mathbf{P}_2 \in \mathcal{M}$ on a Riemannian manifold is called a geodesic curve, and it is given by;

$$\begin{equation} \varphi\left(t\right) = \mathbf{P}_1^{\frac{1}{2}} \left(\mathbf{P}_1^{-\frac{1}{2}} \mathbf{P}_2 \mathbf{P}_1^{-\frac{1}{2}} \right)^t \mathbf{P}_1^{\frac{1}{2}}, \quad 0 \unicode{x2A7D} t \unicode{x2A7D} 1. \end{equation} \tag{ 2 }$$

The length of the curve above, referred to as the Riemannian distance, is unique and is given by (Yair et al 2019);

$$\begin{align} d^2_\mathrm{R}\left(\mathbf{P}_1,\mathbf{P}_2\right)& = \left\Vert{\log\left(\mathbf{P}_1^{-\frac{1}{2}} \mathbf{P}_2 \mathbf{P}_1^{-\frac{1}{2}} \right)}\right\Vert^2_\mathrm{F}\nonumber\\ & = \sum_{i = 1}^n \log^2 \left( \lambda_i \left(\mathbf{P}_1^{-\frac{1}{2}} \mathbf{P}_2 \mathbf{P}_1^{-\frac{1}{2}} \right)\right) \end{align} \tag{ 3 }$$

where $\Vert{\cdot}\Vert_\mathrm{F}$ is the Frobenius norm, $\log(\mathbf{P})$ is the matrix logarithm and $\lambda_i(\mathbf{P})$ is the ith eigenvalue of P.

The Riemannian mean M_s for subject s, also referred to as the Fréchet mean, is the sum of distances to the point on the manifold that is closest to all the subject points (Yger et al 2017, Yair et al 2019):

$$\begin{equation} \mathbf{M}_s \triangleq \mbox{arg} \min_{\mathbf{P}_s \in \mathcal{M}} \sum_i d^2_\mathrm{R}\left(\mathbf{P}_s,\mathbf{P}_{s,i}\right) \end{equation} \tag{ 4 }$$

where $d^2_\mathrm{R}(\mathbf{P}_s,\mathbf{P}_{s,i})$ is the Riemannian distance defined above. In this paper, each subject point is a covariance matrix and the Riemannian mean is therefore a symmetric matrix, which can be interpreted as finding the center of mass in a high-dimensional Riemannian geometric figure. The Riemannian mean of two covariance matrices $\mathbf{P}_1,\mathbf{P}_2 \in \mathcal{M}$ is the midpoint $\varphi (1/2)$ in equation (2). The Riemannian mean for more than two covariance matrices can be computed by an iterative algorithm 1 developed by Barachant et al, where $\mbox{Log}_{\mathbf{M}}\left( \mathbf{P}_i \right)$ and $\mbox{Exp}_{\mathbf{M}}\left( \mathbf{S} \right)$ are defined in (B.1) and (B.2).

Algorithm 1. The Riemannian mean iterative algorithm for more than two SPD matrices (Barachant et al 2013).
Input: a set of SPD matrices $\left\{\mathbf{P}_i \in \mathcal{M} \right\} ^n_{i = 1}$ where n is the number of trials
Output: the Riemannian mean matrix M
1. Compute the initial term $\mathbf{M} = \frac{1}{n} \sum^n_{i = 1} \mathbf{P}_i$
2. while $\Vert{\mathbf{S}\Vert}_\mathrm{F} \gt \epsilon$
2.1. Compute the Euclidean mean in the tangent space $\mathbf{S} = \frac{1}{n} \sum_{i = 1}^n \mbox{Log}_{\mathbf{M}}\left( \mathbf{P}_i \right)$
2.2. Update $\mathbf{M} = \mbox{Exp}_{\mathbf{M}}\left( \mathbf{S} \right)$
end while

This paper focuses on EEG-based locus of auditory attention classification (left vs. right). The classification method involves four steps: (1) computing the covariance matrix, (2) projecting it onto the tangent plane of the Riemannian manifold, (3) vectorizing the covariance matrix to create a feature vector, and (4) employing a linear support vector machine (SVM) for classification. Although we also explored four alternative classification methods—k-nearest neighbor, regression tree, decision tree, and neural network with various configurations—the regression and decision trees yielded unsatisfactory results, and the neural network exhibited a similar accuracy to SVM but with considerably longer training time.

While conventional methods like temporal response functions (TRFs), canonical correlation analysis (CCA) or match-mismatch are commonly used for classification, our paper explores an alternative approach using vectorized covariance matrices as the data features in the SVM classifier. This choice offers several benefits. First, our approach is audio-free, addressing challenges when audio data is not available. Second, computing covariance matrices is relatively straightforward as they are based on and capture linear relations. Third, past research (Yair et al 2019, 2020) demonstrates the effectiveness of SPD matrices (such as covariance matrices), particularly with simpler classifiers like SVMs, which are less complex and computationally more efficient compared to deep neural networks. Covariance matrices have been successfully used as features in diverse fields such as medical imaging, ML and computer vision (Tuzel et al 2008, Sra and Hosseini 2013, Freifeld et al 2014, Bergmann et al 2018). In the context of physiological signal analysis and medical imaging, Riemannian geometry of covariance matrices has been utilized (Pennec et al 2004, Barachant et al 2013).

The training data $D_\mathrm{S}$ comes from a source domain $\mathcal{D}_\mathrm{S}$ and its predictive learning task is denoted $\mathcal{T}_\mathrm{S}$. The testing data $D_\mathrm{T}$, as well as all other data used after the training, comes from a target domain $\mathcal{D}_\mathrm{T}$ and its predictive learning task is denoted $\mathcal{T}_\mathrm{T}$. All notations are presented in table 1 (Weiss et al 2016).

Table 1. Notations used in this section (Weiss et al 2016).

Notation	Description	Notation	Description
$\mathcal{X}$	Input feature space	$D_\mathrm{S}$	Source domain data
$\mathcal{Y}$	Label space	$D_\mathrm{T}$	Target domain data
$\mathcal{T}$	Predictive learning task	P(X)	Marginal distribution
Subscript S	Denotes source	P(Y$|$X)	Conditional distribution
Subscript T	Denotes target	X	Particular learning sample
$\mathcal{D}_\mathrm{S}$	Source domain	xi	Feature vector i
$\mathcal{D}_\mathrm{T}$	Target domain	yi	Class label i

In classical ML, the domains and the tasks of the source and target are the same, hence $\mathcal{D}_\mathrm{S} = \mathcal{D}_\mathrm{T}$ and $\mathcal{T}_\mathrm{S} = \mathcal{T}_\mathrm{T}$. This is the same as fulfilling the conditions:

$$\begin{equation*} \begin{aligned}\mathcal{X}_\mathrm{S} = \mathcal{X}_\mathrm{T} \,\,\,\,\, &\mbox{same feature space} \\ \mathcal{Y}_\mathrm{S} = \mathcal{Y}_\mathrm{T} \,\,\,\,\, &\mbox{same label space} \\ \mbox{P}\left(\mbox{X}_\mathrm{S}\right) = \mbox{P}\left(\mbox{X}_\mathrm{T}\right) \,\,\,\,\, &\mbox{same marginal distribution} \\ \mbox{P}\left(\mbox{Y}_\mathrm{S}|\mbox{X}_\mathrm{S}\right) = \mbox{P}\left(\mbox{Y}_\mathrm{T}|\mbox{X}_\mathrm{T}\right) \,\,\,\,\, &\mbox{same conditional distribution}. \\ \end{aligned} \end{equation*}$$

When one or more of these conditions are not satisfied, generalization methods built on a learning-to-learn principle need to be used. This is most commonly referred to as transfer learning. The specific case when $\mathcal{X}_\mathrm{S} = \mathcal{X}_\mathrm{T}$, $\mathcal{Y}_\mathrm{S} = \mathcal{Y}_\mathrm{T}$ and, the mismatch between the source and target solely comes from the probability distributions is called DA. This is the situation studied in this paper (Kouw and Loog 2018). The definition of DA presented by (Weiss et al 2016) is:

Definition 4.1 (DA). 'Given a source feature space $\mathcal{X}_\mathrm{S}$ with corresponding source label space $\mathcal{Y}_\mathrm{S}$ and a target feature space $\mathcal{X}_\mathrm{T}$ with corresponding target label space $\mathcal{Y}_\mathrm{T}$, DA is the specific case of transfer learning when $\mathcal{X}_\mathrm{S}$ = $\mathcal{X}_\mathrm{T}$, $\mathcal{Y}_\mathrm{S}$ = $\mathcal{Y}_\mathrm{T}$ and the mismatch between source and target comes from P(X$_\mathrm{S}$) $\ne$ P(X$_\mathrm{T}$) and/or P(Y$_\mathrm{S}|$X$_\mathrm{S}$) $\ne$ P(Y$_\mathrm{T}|$X$_\mathrm{T}$).'

After the transportation, the marginal and conditional distributions are merged into the joint distribution P(X,Y) = P(Y$|$X)P(X). In DA, this joint distribution can be broken down into two different cases (Kouw and Loog 2018):

$$\begin{align}& \mbox{P}\left(\mbox{X}_\mathrm{S}\right) = \mbox{P}\left(\mbox{X}_\mathrm{T}\right) \quad \text{and}\nonumber\\ & \mbox{P}\left(\mbox{Y}_\mathrm{S}|\mbox{X}_\mathrm{S}\right) \ne \mbox{P}\left(\mbox{Y}_\mathrm{T}|\mbox{X}_\mathrm{T}\right) \quad \mbox{concept shift}\nonumber\\ & \mbox{P}\left(\mbox{X}_\mathrm{S}\right) \ne \mbox{P}\left(\mbox{X}_\mathrm{T}\right) \quad \text{and} \nonumber\\ & \mbox{P}\left(\mbox{Y}_\mathrm{S}|\mbox{X}_\mathrm{S}\right) = \mbox{P}\left(\mbox{Y}_\mathrm{T}|\mbox{X}_\mathrm{T}\right) \quad \mbox{covariate shift}.\end{align}$$

The brain's electrical pulses, which can be detected by EEG electrodes, are captured as time series data. These time series are typically non-stationary due to factors such as electrode placements, fluctuations in attention levels, environmental influences, eye and motor movements. As a consequence, comparing EEG signals across different trials or sessions often reveals covariate shifts (Razaa et al 2019). In the context of comparing subjects (as in this study), anatomical variations among individuals may introduce differences in the conditional distribution, leading to concept shifts (Albuquerque et al 2019).

The DA method proposed in this study is grounded in the concept of PT (Yair et al 2019). It specifically addresses the challenge of covariance matrices residing in different regions of the manifold, which commonly arises when data is gathered from multiple subjects and/or sessions. Previous applications of this method in the field of EEG include emotion recognition (Wang et al 2021) and seizure detection and prediction (Peng et al 2022).

PT uses Riemanninan geometry to:

• (i)  
  Compute the Riemannian mean M_s of all covariance matrices for each subject s.
• (ii)  
  Compute the Riemannian mean D of all M_s from step (i).
• (iii)  
  Project all the covariance matrices from the Riemannian manifold onto a Riemannian tangent plane at M_s for each subject s. This is done with a logarithm map which is illustrated in figure 2(a) and further explained in appendix B.
• (iv)  
  Move all the data to D using PT, which is illustrated in figure 2(b) and further explained in appendix B.
• (v)  
  Project the covariance matrices from the Riemannian tangent plane back to the Riemannian manifold. This is done with an exponential map, which is illustrated in figure 2(a) and further explained in appendix B.
• (vi)  
  Project the covariance matrices to the Euclidean tangent space for classification and plotting purposes. This step is also computed for the baseline algorithms (explained in sections 5.3 and 5.4) which do not use DA.

Visualization plays a significant role in comprehending high-dimensional data, yet condensing such data into two or three dimensions for visualization purposes may result in the loss of significant information. A dimension reduction technique called t-distributed stochastic neighbor embedding (t-SNE) (van der Maaten and Hinton 2008) is used in this paper. The accompanying figure 1 provides an illustration of PT. It should be noted that the Riemannian mean D is represented as a $64 \times 64$ matrix, but due to the dimensional reduction using t-SNE to two dimensions, it may not visually appear as the mean in the figure.

Zoom In Zoom Out Reset image size

The exponential and logarithm maps are illustrated in figure 2(a). The gray/black line between the two points x₀ and x on the Riemannian manifold $\mathcal{M}$ is the minimum length curve. u is a vector on the tangent plane of x₀. The exponential map projects u to a point $x \in \mathcal{M}$ in the direction of u. The logarithmic map is the inverse, where the point $x \in \mathcal{M}$ is projected to the tangent plane $u \in \mathcal{T}_{x_0}\mathcal{M}$ (Calinon 2020).

Figure 2(b) shows an illustration of the PT method of the vector $u \in \mathcal{T}_g\mathcal{M}$. The goal is to transport u, along infinitesimally close tangent spaces, to the tangent space $\mathcal{T}_h\mathcal{M}$ of the point h on the manifold $\mathcal{M}$. The black vectors show the direction of the transportation in each tangent space. Using infinitesimally close tangent spaces gives a smooth transportation with preserved features of the vector u (Calinon 2020).

Zoom In Zoom Out Reset image size

EEG data were collected from a group of 19 participants who were native Danish speakers with normal hearing, aged between 19 and 30 years, and had no reported neurological disorders. The data were recorded using d = 64 scalp electrodes during 60 trials, with a sampling rate of 512 Hz. In each trial i, participants listened to a pair of competing speech stimuli (one male, one female) presented at 65 dB. The speech streams were narrated by storytellers and recorded at 44.1 kHz in an anechoic chamber. To reflect real life situations, some of these recordings were simulated in a mildly reverberant room and some in a highly reverberant room using the Odeon room acoustic modeling software. The three environment scenarios included anechoic, mildly reverberant and highly reverberant. Following each trial, participants answered multiple-choice questions related to the content of the attended speech.

The EEG data was preprocessed through the FieldTrip (Oostenveld et al 2011) and COCOHA (Wong et al 2018) toolboxes in MATLAB. The procedure included removal of artifacts (eye blinks, muscle movements, heart beats etc), filter out 50 Hz line noise and downsampling to 64 Hz. The script preproc_data.m which was used for the preprocessing can be downloaded from zenodo.org (Wong et al 2018, (Fuglsang et al 2017).

In scenarios with several talkers, an increase in oscillatory alpha (frequency band 7–14 Hz) power due to the brain activity of ignoring the unattended speakers (Paul et al 2020) has been reported. This alpha activity is an epiphenomenon of spatial attention, and an additional Butterworth bandpass filter of order 6 with frequency band [1–30] Hz was therefore applied to the EEG data.

Three different classification tasks are performed; baseline classification, before transportation (BT) and after PT. The last two tasks are illustrated in figure 3. Each datapoint in the figure represents the covariance matrix for a specific trial where attention was either to the left side or to the right side of the subject. All three classification tasks are made on the tangent plane of the Riemannian manifold, hence step 6 in the PT-pipeline (section 4.3) and table B2 is also computed for baseline and BT classification.

Zoom In Zoom Out Reset image size

The baseline LoA classification accuracy is computed for each subject using leave-one-out cross-validation (LOO CV) with an SVM classifier. This first classification task gives a LOO CV accuracy for each subject and helps identify candidate subjects who exhibit low classification accuracy and may benefit from DA. Reference subjects, on the other hand, are identified as the subjects with the highest classification accuracy. Detailed information on the selection of candidate and reference subjects are provided in section 6.1.

Candidate subject data is augmented with data from the reference subjects. Two different approaches are used to evaluate whether data augmentation improves the classification performance. The first approach, referred to as BT, involves temporal concatenation of the reference and candidate subject datasets. This is visualized in figure 3(a), where the candidate subject #2 data is augmented with the data from reference subjects #9 and #15. The second approach utilizes DA via PT. This is used to evaluate the benefits of transfer learning on classification accuracy of the candidate subjects, and is illustrated in figure 3(b). As in the baseline condition, LOO CV with SVM is used for evaluating both BT and PT. Further details on these approaches are provided in section 6.2.

The experiment involved 60 trials with two competing talkers (one male and one female) positioned at spatially-separated angles (${\pm}60^\circ$). Each trial lasted for 50 s and correct classification rate was computed using LOO CV.

5.4.1. Baseline classification

5.4.2. BT classification

5.4.3. PT classification

To derive the statistically significant threshold (i.e. empirical chance level), the binomial inverse cumulative distribution function (Combrissona and Jerbi 2015) was computed by using the MATLAB function $x = \mathrm{binoinv}(y,n,p)$, where y = 0.95 is the significance level, n is total number of trials and p = 0.50 is the theoretical chance level for binary classification problems. Hence, the result x is the smallest integer such that the binomial cumulative distribution function (computed at x) is equal or greater than y. Dividing $p_\mathrm{c} = x/n$ gives the level of chance for binary classification problems with n trials at the $95\%$ confidence interval. The last step is to assign the variable f to 0 (statistically significant) or to 1 (not statistically significant) by comparing the computed classification accuracy Acc with $p_\mathrm{c}$.

The main results in this section show that PT increases the classification accuracy compared to both the baseline and BT. It indicates that only adding more subjects (adding more data) is not enough but a DA method, such as PT, is needed to reach statistically significant results.

Figure 4 shows the LoA classification results for each subject. The level of chance is 60% for a two-classes classification problem with n = 60 trials and a significance level of p = 0.05. This is computed with the binomial cumulative distribution function (Combrissona and Jerbi 2015). As seen, there is a large variability between subjects and only six of them have a classification accuracy above the significance level, while the highest value reached 80%. The dashed lines indicate the $mean \pm standard$ deviation $(std) = 56.11 \pm 10.92$ of the classification accuracy across all subjects.

Zoom In Zoom Out Reset image size

The candidate subjects $Sub_\mathrm{cand}$ = (${\color{red}{1}},{\color{red}{2}},{\color{red}{5}},\color{red}{16}$) are selected as subjects with a classification accuracy below or equal to round (mean − std). The three subjects with highest classification accuracy are chosen as reference subjects $Sub_\mathrm{ref} = ({{\color{Green}{8}}},{{\color{Green}{9}}},{{\color{Green}{15}}})$. The first goal is to increase the classification accuracy of the candidate subjects with data from different combinations of reference subjects, investigated in section 6.2.

Thereafter, we conducted an additional analysis to explore the practical task of classifying LoA in subjects whose EEG data the model had not seen. To tackle this, a two-step analysis was made. First, the best combination of reference subjects was selected. Then, BT and PT classification accuracy was performed for each subject in the dataset and the mean was computed. The second goal of this paper is therefore to investigate if augmentation with the best combination of reference subjects and PT increases the mean classification accuracy over all other subjects. This is investigated and discussed in section 6.3.

The classification accuracy rates for the candidate subjects $Sub_\mathrm{cand}$ = (${\color{red}{1}} ,{\color{red}{2}} ,{\color{red}{5}} ,\color{red}{16}$) with different combinations of reference subjects $Sub_\mathrm{ref} = ({{\color{Green}{8}}},{{\color{Green}{9}}},{{\color{Green}{15}}})$ are shown in figure 5. The solid black line shows the candidate subject baseline.

Zoom In Zoom Out Reset image size

The blue solid line shows the BT classification accuracy when data from all subjects except the candidate is used as reference. Hence, this line shows the result with data from 17 reference subjects and one candidate subject. Candidate ($\color{red}{16}$) achieved the same BT classification accuracy as the subject-specific baseline classification, resulting in one black line representing both accuracies in the subfigure. The classification accuracy around 50% indicates that only adding data from more subjects does not give any statistically significant results. All four subfigures show the benefit of PT from fewer reference subjects compared to the blue solid line.

In general, BT marginally increased the classification accuracy compared to baseline, indicating that augmenting the data this way has a small impact on the accuracy. This is further outlined with the mean over the candidates shown in table 2, where $mean_\mathrm{BT}$ slightly increased the accuracy in all cases except two when compared to $mean_\mathrm{baseline} = 45.84\%$. However, no reference combinations with $mean_\mathrm{BT}$ reached above the level of chance. Both figure 5 and table 2 show the benefit of DA. All combinations increased the classification accuracy rates compared to both baseline and BT, although the optimal set of reference subjects highly varies across candidate subjects. In particular, references (${{\color{Green}{15}}}$) and (${{\color{Green}{9}}},{{\color{Green}{15}}}$) significantly benefited from PT, since their mean accuracy increased from baseline of $45.84\%$ to $66.25\%$ and $67.92\%$ respectively with PT. It is important to acknowledge that the obtained accuracy of around 70% for AAD with DA using 50 s trials might seem less favorable when compared to other methodologies such as SR results. While we have omitted the inclusion of other methodologies in this paper, our focus remains on demonstrating how EEG data from subjects with lower classification accuracy can be enhanced to yield improved results, thereby enhancing the overall performance of AAD.

Table 2. The mean of the classification accuracy for the candidates shown in figure 5. The baseline mean for the candidates is 45.84%.

Reference subjects	$\color{Green}{{8}}$	$\color{Green}{{9}}$	$\color{Green}{{15}}$	$\color{Green}{{8}},\color{Green}{{9}}$	$\color{Green}{{8}},\color{Green}{{15}}$	$\color{Green}{{9}},\color{Green}{{15}}$	$\color{Green}{{8}},\color{Green}{{9}},\color{Green}{{15}}$
Mean BT	45.83	50.00	42.08	51.25	44.58	46.67	46.67
Mean PT	59.17	61.25	66.25	60.42	65.00	67.92	63.33

Two metrics were studied to evaluate which combination of references induced the best improvement of the candidates:

• (i)  
  $mean(Acc_\mathrm{PT} - Acc_\mathrm{BT})$: the mean over all candidates of the accuracy difference between PT and BT. The desired outcome would be a large positive value, indicating a large benefit with PT.
• (ii)  
  $mean(Acc_\mathrm{PT})$: the mean over all candidates of the PT accuracy, also presented in table 2. This is used to assess the significance of the results.

These metrics resulted in (${{\color{Green}{15}}}$) and (${{\color{Green}{9}}},{{\color{Green}{15}}}$) as the two best reference combinations. The classification accuracy for these two combinations together with subjects #$(3, 4, 6, 7, 8, 10, 11, 12, 13, 14, 17, 18)$ are shown in figure 6. Hence, all subjects in the dataset except the reference subjects and the candidate subjects in figure 5 are presented. Note that also subject $\#8$, which was a reference subject in figure 5, is included. For the sake of comparison, the significance level at 60% and $mean \pm std$ of the classification accuracy over all subjects (same as in figure 4) are also presented. Subjects $\#3$ and $\#11$ stand out, as the classification accuracy decreases significantly when adding PT. Furthermore, subjects $\#12$ and $\#17$ slightly decrease the classification accuracy compared to baseline. Interestingly, subjects #$(3, 12, 17)$ all achieved a baseline accuracy above the significance level and performed worse when adding PT, indicating that higher performing subjects might not benefit as much as low performing subjects when applying DA. Nevertheless, the high performing subject $\#8$ performed equally well with PT as their baseline classification.

Zoom In Zoom Out Reset image size

However, eight out of 12 subjects reached an accuracy above or equal to the significance level after PT. In particular, subjects #6, #14 and #18 could greatly benefit from PT as the classification accuracy for these subjects increased from $48.33\%_\mathrm{baseline} \rightarrow 75.00\%_\mathrm{PT,({\color{Green}{15}})}$, $51.67\%_\mathrm{baseline} \rightarrow 83.33\%_\mathrm{PT,({\color{Green}{15}})}$ and $51.67\%_\mathrm{baseline} \rightarrow 80.00\%_\mathrm{PT,({\color{Green}{15}})}$ respectively. Again addressing the issue of new label-free datasets, the most interesting metric is the mean over these twelve subjects, which is shown in the last column in each subfigure. The two reference combinations with PT resulted in mean classification accuracy rates of $64.44\%$ and, $63.33\%$, respectively, both above the significance level.

EEG holds promising potential as a mean to objectively assess hearing abilities and evaluate the effectiveness of hearing devices. Dealing with subjects who exhibit low classification rates and performance poses significant challenges. However, enhancing EEG data by augmenting suboptimal recordings from one test participant with high-quality EEG data from a different test participant, thereby refining the signal quality. This process may enable the use of EEG as a reliable tool for objective assessment, overcoming the challenges encountered in such cases.

In order to integrate EEG into hearing devices, there are several challenges that need to be addressed. Among these challenges are the reduction of the number of electrodes and specific investigation of the electrodes close to the ear. If similar results can be obtained with fewer electrodes near the ear, a practical approach could involve training the model using data from well-performing reference subjects. Subsequently, the trained model could be utilized to augment data from candidate subjects with low classification performance. This could then serve as feedback to the hearing aid, enabling real-time adjustment of noise reduction algorithms implemented in hearing devices.

In summary, the practical implication of our results lies in the potential of DA to bring us closer to using EEG in neuro-steered hearing devices. It is crucial to emphasize that significant efforts are required to actualize the development of brain-controlled hearing aids, but the benefits make it an attractive pursuit.

This section gives some more in depth explanation of parallel transport, and more information of the method can be found in the article by Yair et al (2019).

The logarithm map $\mathbf{S}_{s,i}$ and exponential map $\mathbf{P}_{s,i}$ for subject s and trial i is defined as:

$$\begin{align} \mathbf{S}_{s,i} = \mbox{Log}_\mathbf{P} \left(\mathbf{P}_{s,i}\right) = \mathbf{P} ^ {\frac{1}{2}} \mbox{log} \left(\mathbf{P}^ {-\frac{1}{2}}\mathbf{P}_{s,i}\mathbf{P} ^ {-\frac{1}{2}} \right)\mathbf{P} ^ {\frac{1}{2}} \,\,\,\, \in \mathcal{T}_\mathbf{P}\mathcal{M} \end{align} \tag{ B.1 }$$

$$\begin{align} \mathbf{P}_{s,i} = \mbox{Exp}_\mathbf{P} \left(\mathbf{S}_{s,i}\right) = \mathbf{P} ^ {\frac{1}{2}} \mbox{exp} \left(\mathbf{P}^ {-\frac{1}{2}}\mathbf{S}_{s,i}\mathbf{P} ^ {-\frac{1}{2}} \right)\mathbf{P} ^ {\frac{1}{2}} \quad \in \mathcal{M} \end{align} \tag{ B.2 }$$

where $\mathbf{S}_{s,i}$ is a symmetric matrix. Table B2 and algorithm 2 shows the MATLAB pseudocode for the six steps of the parallel transport method outlined in section 4.3.

Algorithm 2. Domain adaptation using parallel transport for SPD matrices (Yair et al 2019).
Input: $\left \{\mathbf{P}_{1,i} \right \}_{i = 1}^n, \dots, \left \{\mathbf{P}_{s,i} \right \}_{i = 1}^n, \dots, \left \{\mathbf{P}_{N,i} \right \}_{i = 1}^n$ where $\mathbf{P}_{s,i}$ is the covariance matrix for subject s and trial i.
Output: $\left \{\tilde{\mathbf{S}}_{s,i} \right \}_{i = 1}^n, \dots, \left \{\tilde{\mathbf{S}}_{s,i} \right \}_{i = 1}^n, \dots, \left \{\tilde{\mathbf{S}}_{N,i} \right \}_{i = 1}^n$ where $\tilde{\mathbf{S}}_{s,i}$ is the new representation of $\mathbf{P}_{s,i}$ in a Euclidean space.
1. For each $i \in \left\{1,2,\dots,n \right \}$, compute the Riemannian mean Ms of the subset $\left \{\mathbf{P}_{s,i} \right \}$
2. Compute D, the Riemannian mean of $\left \{\mathbf{M}_s \right \}_{s = 1}^N$
3–5. For all s and i, apply projection and parallel transport using equation (B.3): $$\begin{equation*} \mathbf{P}_{s,i}^{\mathbf{D}} = \mathbf{E} \mathbf{P}_{s,i}^{\mathbf{M}_s} \mathbf{E}^\mathrm{T}, \quad \mathbf{E} = \left(\mathbf{D} \mathbf{M}_s^{-1}\right)^{\frac{1}{2}} \end{equation*}$$
6. For all i, project the transported matrix to the tangent space via: $$\begin{equation*} \tilde{\mathbf{S}}_{s,i}^{\mathbf{D}} = \mbox{log} \left(\mathbf{D}^{-\frac{1}{2}} \mathbf{P}_{s,i}^{\mathbf{D}} \mathbf{D}^{-\frac{1}{2}} \right) \end{equation*}$$

Steps 3–5 can be combined, which gives the projection to the tangent plane, transportation along the tangent planes and projection back to the manifold in one equation:

$$\begin{align} \mathbf{P}_{s,i}^{\mathbf{D}} = \mbox{Exp}_{\mathbf{D}} \left(\Gamma_{\mathbf{M}_s \rightarrow \mathbf{D}} \left(\mbox{Log}_{\mathbf{D}} \left(\mathbf{P}_{s,i}^{\mathbf{M}_s} \right) \right) \right) = \mathbf{E} \mathbf{P}_{s,i}^{\mathbf{M}_s} \mathbf{E}^\mathrm{T}, \end{align} \tag{ B.3 }$$

where $\mathbf{E} = (\mathbf{D} \mathbf{M}_s^{-1})^{\frac{1}{2}}$ (Yair et al 2019). Step 6 is also computed for the baseline algorithms (explained in sections 5.3 and 5.4) which do not use domain adaptation.