Does bimodal stimulus presentation increase ERP components usable in BCIs?

Visual stimuli were presented on a Samsung 19'' TFT monitor (1280 × 1024, 60 Hz) and consisted of yellow–green arrows starting from the blue disc in the current hexagon and pointing to a neighbouring hexagon (see figure 1(a)). These stimuli were sequentially presented for 200 ms (on time) and not visible in the following 120 ms (off time). Participants were allowed to gaze directly at the visual stimuli. The visual display was positioned horizontally (flat on a table) such that the visual and tactile stimuli were congruent in direction and corresponded with natural 2D navigation directions (see Thurlings et al 2012 for the importance of directional congruence in ERP-BCIs). The viewing distance was approximately 50 cm from the eyes to the centre of the visual display.

Participants wore a vest (which could be adjusted to body dimensions) with integrated tactors. The tactors were custom built and consisted of a plastic case with a contact area of 1 × 2 cm containing a 160 Hz electromotor (TNO, the Netherlands, model JHJ-3, Van Erp et al 2007). To prevent the participants from perceiving auditory information from the tactors, we let them listen to pink noise via in-ear headphones during the experiment.

For this study, the tactors were arranged in groups of two at six locations in a circular layout on a horizontal plane (around the torso at approximately navel height). The locations of the tactors corresponded to the navigation directions of the task: left, right and two directions in between along both the frontal side and backside (see figures 1(a) and (b)). The on and off times of the tactors were the same as those for the visual stimuli (200 ms on and 120 ms off).

Bimodal visual–tactile stimuli were created by triggering corresponding visual and tactile stimuli simultaneously. Because the visual and tactile stimuli have varying technical properties, the corresponding intensity–time functions are different. This maximum intensity of the tactile stimulus is reached within 50 ms after the visual stimulus has reached its maximum intensity.

EEG was recorded from 60 nose-referenced scalp electrodes that used a common forehead ground (Brain Products GmbH, Germany). The impedance of each electrode was below 20 kΩ, as was confirmed prior to and during the measurements. EEG data were recorded with a hardware filter (0.05–200 Hz) and sampled at a frequency of 1000 Hz.

To prepare the recorded EEG for further processing, the data were once again bandpass filtered (0.5–30 Hz) and sampled with a sampling frequency of 100 Hz. Such a bandpass filter reduces influences from artefacts such as eye blinks (e.g., Guger et al 2009, Tereshchenko et al 2009). We are interested in the clean effects of endogenous attention on the ERP and attempted to isolate such effects in the ERP. To this end, we used two strict rejection criteria, which were only applied to select data for the ERP analysis and not for the classification analysis. Finally, a difference measure was computed to isolate endogenous effects. The rejection criteria will be described next, followed by the difference measure.

The first rejection criterion was that EEG was not used for the ERP analysis if it was expected to be contaminated by previous or following target responses relative to the stimulus response of current interest using an approach similar to that of Treder and Blankertz (2010). To this end, we selected only EEG data associated with (non)targets when the three preceding and the three following presented stimuli were nontargets, i.e. when there were no (other) target responses between –960 and 1280 ms relative to (non)target onset. For the selected (non)targets, epochs from all electrodes were extracted from –320 to 750 ms relative to stimulus onset. This resulted in an average number of 246 target epochs (range 230–262) and 189 nontarget epochs (range 167–214) per participant. Similar numbers of epochs were selected per condition. The epochs were baseline corrected relative to the average voltage during the 320 ms preceding the stimulus onset.

As a second rejection criterion, we discarded all epochs within one repetition if epochs recorded at both frontal electrodes (Fp₁ and Fp₂) contained amplitude differences exceeding 100 µV, indicating eye movements. This was the case for 5.6% of the measurements. For all conditions, this left us with averages of 227 target epochs (range 107–262) and 183 nontarget epochs (range 130–209) per participant. Subsequently, the selected target and nontarget epochs were averaged per participant, per condition and per electrode.

Finally, we subtracted the averaged clean nontarget epochs from the averaged clean target epochs for each participant, each condition and each electrode. In this step, we removed exogenous (involuntary or automatic) attention effects. Further analyses were performed regarding this difference ERP (or endogenous ERP).

We aimed at identifying and quantifying all ERP components triggered by endogenously attended stimuli occurring during the interval until 750 ms after stimulus onset. First, we identified significant endogenous effects by performing a sample-by-sample t-test on the difference ERP for each electrode and condition. To correct for multiple testing, the method of Guthrie and Buchwald (1991) was applied (Molholm et al 2002, Rugg et al 1995, Vidal et al 2008). In our case, this implied that (at least) four consecutive samples (equivalent to 40 ms) had to be significantly different from zero to consider the corresponding samples as a stable segment. Second, we clustered these segments over electrodes (hierarchical clustering of pairwise objects) to label the elicited endogenous ERP components using the cluster's time interval, topography and polarity. The clustering of segments was based on the beginning and end of their time periods and their averaged amplitudes (using the following standard parameters: Euclidean distance in feature space, single linkage and maximum 15 clusters; Webb 2002). We also applied the method of Guthrie and Buchwald (1991) to correct for multiple testing with respect to the electrode dimension. Clusters were considered robust if they contained segments of at least four electrodes. Robust clusters that had overlapping intervals and comparable averaged amplitudes were assumed to be subcomponents of the same ERP component and were combined. These combined robust clusters defined the topographic distribution and the interval of the endogenous ERP components, taking the beginning of the earliest segment and the end of the latest segment in the clusters as ERP component intervals.

After identifying the elicited endogenous ERP components, we quantified these to compare them between conditions. To capture the strength of a local (at a certain electrode site) ERP component, regardless of its shape, we used the area-under-the-curve values (AUC; Allison et al 1999, Luck 2005, Puce et al 2007). To also allow for topographic distributions of an ERP component as a measure of the component's magnitude, we determined the sum of AUCs from the electrodes included in a defined ERP component to quantify endogenous ERP components in each condition and for each participant, i.e. AUCs from stable segments included in an ERP component were calculated and summed. We refer to the sum of topographic distributed AUCs from an ERP component as the tAUC. With the tAUC, we can describe the magnitude of an ERP component not only by taking the averaged amplitude and duration of the component into account but also by considering the topographic distribution. Endogenous ERP components with overlapping intervals and equal polarities between conditions were considered to reflect the same ERP component. Associated tAUCs were statistically compared. Note that this measure corresponds to perceptual and cognitive processes but not necessarily to discrimination, as in BCI classification, because the information of neighbouring electrodes in broadly distributed components can be redundant.

Additionally, we compared ERP components over conditions using the more traditional peak-picking (Luck 2005) to validate the tAUC value and to allow for a comparison of ERP components to be made between conditions when these were absent in one or more conditions. We looked for peaks in condition-specific intervals as defined in the previous step. If an ERP component was only detected in one or two but not in all three conditions, the (overlapping) interval of the detected ERP component was used to determine peaks in all conditions. Six electrodes were chosen for the determination of peak amplitudes for each ERP component based on the topographic distribution of that component.

Classification accuracies were analysed offline by linear discriminate analysis (LDA) because previous BCI research showed good results using this relatively simple method (Blankertz et al 2011, Krusienski et al 2008, Zander et al 2011). We applied a stepwise LDA (SWLDA) with similar parameter settings as in Krusienski et al (2008): a maximum of 60 features employed in the model, a p-value of < 0.1 for features included in the model initially and a p-value of > 0.15 for features removed from the model backwards. Features were extracted from each electrode, and voltages were averaged within a subwindow of a specific ERP interval, which was determined according to the method described in section 2.5.2. For each condition, we divided each specific ERP interval into four subsequent subwindows of similar lengths. In total, the number of features prior to selection corresponded to the number of subwindows (4) * the number of ERP components involved (1, 2 or 3) * the number of electrodes (60). The training set was based on the first half and the test set on the second half of the recorded data from each participant and each condition and followed preprocessing steps similar to those described for the ERP analysis; however, no data were rejected to realistically assess potential online classification performance. To investigate how conditions interacted with the amount of data necessary to improve the signal-to-noise ratio, we investigated the effect of the number of repetitions for averaging over test repetitions (1–10).

A goal of this study was to determine whether increased bimodal (compared to unimodal) ERP components are usable in BCI, which we have approached both theoretically and practically. For the theoretical approach, we focused on the effect of bimodally increased ERP components on BCI performance only. In contrast, for the practical approach we included all ERP components that could boost BCI performance, to evaluate the effect of stimulus modality. The theoretical approach may produce relevant information for further development of bimodal BCIs, whereas the practical approach directly evaluates the visual–tactile BCI in a practical sense.

For the theoretical approach, we compared classification accuracies from all conditions based on the same ERP components, which were enhanced for bimodal, and based on the same number of repetitions. The number of repetitions included in this analysis was established according to a criterion by which successful BCI control could be expected. Successful BCI control was considered when a threshold for classification accuracies of 70% or higher was obtained (e.g., Schreuder et al 2010) by at least 80% of the participants.

For the practical approach, for each condition separately, we determined the most optimal classifier choices, i.e. on which ERP components (separately or combined) the classifier should be based and the number of repetitions that are required to result in the highest bitrates, while successful BCI control could be achieved. We calculated bitrates (Serby et al 2005), based on each participant's classification probability for each repetition. Using the most appropriate classifier and number of repetitions for each condition, bitrates were calculated and statistically compared.

As a measure of task performance, we determined the counting accuracies as the percentage of steps in which the number of targets was counted correctly for each participant and each condition.

ERP components' tAUC values and peak amplitudes, classification and counting accuracies were statistically analysed using Statistica 8.0 (StatSoft, Tulsa, OK, USA). We used one-way repeated-measures ANOVA with modality (three levels) as the independent variable to test for effects of stimulus modality on tAUCs, on classification accuracies and on counting accuracies (or paired t-tests if suitable). Two-way repeated-measures ANOVAs were applied to analyse peak amplitude (with the electrode as the second independent variable). We report the main effects of modality and interaction effects if these revealed local effects of modality. Tukey post-hoc tests were applied when appropriate.

The N1 was only identified for the visual–tactile condition, with its tAUC values (figure 4(f)) differing significantly from zero (t₍₉₎ = 5.18; p < 0.001). Additionally, a comparison of peak amplitudes (figure 4(a); determined at electrodes: CP_z, P_z, PO_z, O_z, FT₈, Fp₈) between all conditions, using the interval values of the visual–tactile condition, reveals an effect of modality (F_(2,18) = 4.77; p < 0.05) caused by higher peak amplitudes for visual–tactile than for both unimodal conditions (both p < 0.05).

The shape, latency and distribution of the N2 were remarkably similar for the visual and visual–tactile conditions. Indeed, statistical analyses of tAUC values (figure 4(g)) and peak amplitudes (figure 4(b); determined at electrodes: T₇, TP₇, CP₅, T₈, TP₈, CP₆) revealed an effect of modality on the N2 (F_(2,18) = 24.86; p < 0.001 and F_(2,18) = 5.39; p < 0.05, respectively), which was caused by a stronger N2 for visual and visual–tactile compared to tactile conditions (for respective analyses, both p < 0.001 and both p < 0.05). The significant interaction effect of modality × electrode (F_(10,90) = 2.6; p < 0.01) on the N2 peak amplitude showed that the main effect is caused by higher amplitudes at electrodes TP_7/8 and CP_5/6 (all p < 0.05) and not T_7/8.

A main effect of modality was found on the tAUCs (figure 4(h)) of the P300 (F_(2,18) = 16.35; p < 0.001). The post-hoc analysis showed that the P300 was stronger in the visual compared to visual–tactile and tactile conditions (both p < 0.01). The main effect of modality on the P300's tAUC was confirmed by an effect of modality on the P300 peak amplitudes (figure 4(c); determined at electrodes C_z, CP_z, P_z, PO_z, CP₁, CP₂; F_(2,18) = 15.15; p < 0.001). Similar to the tAUC value, the post-hoc analysis showed a stronger P300 for the visual compared to the tactile condition (p < 0.01), but also revealed higher P300 peak amplitudes for the visual–tactile compared to the tactile condition (p < 0.01). This highlights the different characteristics of the P300 with respect to modalities; the tactile P300 has lower amplitudes but lasts longer compared to the visual and visual–tactile P300. Such information is important for feature selection in the classification analysis. Additionally, an interaction effect of modality × electrode on the P300 peak amplitudes was found (F_(10,90) = 3.49; p < 0.001). The post-hoc analysis revealed the same main effects of modality, indicating that the visual and visual–tactile P300 amplitudes were higher than the tactile amplitudes (reproducing the main effect of modality presented above) and that the visual P300 amplitudes were higher than visual–tactile amplitudes (p < 0.05 for all). The latter effect (indicating that adding a tactile stimulus to the visual stimulus did not enhance but reduced the P300) confirmed the results of the former analysis.

The N400 was only detected to be robust in the visual condition, with a tAUC value (figure 4(i)) differing significantly from zero (t₍₉₎= 3.71;p < 0.01). Additionally, we compared the N400 peak amplitudes (figure 4(d); determined at electrodes: F_z, F₃, F₄, FC_z, FC₃, FC₄) with respect to the conditions using the interval detected for the visual N400 and found an effect of modality (F_(2,18) = 13.63; p < 0.001). However, the post-hoc analysis indicated enhanced visual and visual–tactile amplitudes relative to the tactile amplitudes (both p < 0.001).

Offline classification analyses (see section 2.5.4) were based on the N1, and on a combination of the N1 and N2. These ERP components were selected, because the visual–tactile N1 was increased with respect to the unimodal conditions, and although the visual–tactile and the visual N2 did not significantly differ, the N2 should provide a neutral basis to increase classification accuracies. Figures 5(a) and (b) show the classification results for N1-based and N1&N2-based, respectively. All participants scored 100% classification accuracy in the visual–tactile condition when data from sufficient repetitions were used (six or more) using the N1&N2-based classifier. Such homogeneous top performance was not reached in either of the unimodal conditions. To investigate the effect of stimulus modality, classification accuracies for the N1-based classifier were compared when successful BCI control is expected (80% of the participants achieved accuracies of 70% or higher). This criterion was met at the eighth repetition for the visual–tactile condition, and not met at all for the visual or tactile condition (figure 5(a)). A comparison of N1-based classification accuracies at the eighth repetition reveals a significant effect of modality (F_{(2, 18)} = 32.73, p < 0.001), and the post-hoc analysis showed that tactile classification accuracies were lower than for visual and tactile (both p < 0.001). For the N1&N2-based classifier, the criterion was already met at the first repetition for both visual–tactile and visual (figure 5(b)). The statistical analysis showed a significant effect of modality (F_{(2, 18)} = 63,76 p < 0.001), and the post-hoc analysis also indicated that tactile classification accuracies were lower than for visual and tactile (both p < 0.001). However, the enhanced trend in visual–tactile compared to visual classification accuracies was not significant.

Zoom In Zoom Out Reset image size

In addition to the N1-based and N1&N2-based classification, classification accuracies were also calculated with a P300-based (figure 5(c)) and N1&N2&P300-based classifier (figure 5(d)). Based on all classification results, bitrates were calculated as a function of the number of repetitions. In general, for all three conditions, averaged bitrates (see figure 6) were highest at the first repetitions and decreased when multiple repetitions were used. However, BCI operation at the first trial was not considered successful in all cases, because the corresponding criterion was not met (see section 2.5.4.) We determined the highest bitrates achieved for each condition when at least 80% of the participants had classification accuracies of 70% or higher. For visual 90% and for visual–tactile 100% of the participants exceeded the threshold using the N1&N2&P300-based classifier at the first repetition resulting in bitrates of 62.45 (SD 20.05) and 64.93 (SD 18.18) bits min^–1, respectively. The best performance and successful BCI control may be expected for tactile at the seventh repetition: 6.52 (SD 3.18) bits min^–1, since then 80% of the participants exceeded the threshold when the P300-based classifier was used. To statistically analyse the effect of modality on BCI performance for practical use, these condition-specific classifiers were used and the corresponding bitrates were compared. The effect of modality on bitrates was significant (F_{(2, 18)} = 91.37, p < 0.001), and post-hoc analyses confirmed that bitrates for visual and visual–tactile were higher than for tactile (both p < 0.001), but did not differ from each other. However, similar to the theoretical approach using the N1&N2-based classifier, also with the practical approach using the N1&N2&P300-based classifier, only for the visual–tactile condition was a 100% classification accuracy achieved by all participants (for repetitions four and higher).

Zoom In Zoom Out Reset image size

Early processing of attended visual–tactile stimuli resulted in the elicitation of the N1. This bimodal N1 cannot be explained by either of the unimodal stimuli alone because the activity in the corresponding interval is significantly higher for attending to visual–tactile stimuli compared to either of the unimodal stimuli. The input from both sensory modalities contributed to the bimodal N1 through the (sum of the) two separate processes and possibly by additional integration activity due to crossmodal links in sensory processing at that early stage (Foxe et al 2000). The latency of our bimodal N1 is well in line with previously reported multisensory integration effects modulated by spatial (selective) attention: Talsma and Woldorff (2005) reported the first effect to peak at approximately 100 ms, and ours was present between 70 and 130 ms after stimulus onset. Furthermore, Talsma et al (2007) showed that it is required that both modalities of bimodal stimuli are attended for early ERP multisensory integration effects to occur and are not present if one modality is explicitly ignored. This suggests that the participants in our study correctly followed the instructions to attend to both modalities and did not ignore one modality in the bimodal condition.

The effect of endogenous attention on the visual–tactile N1 in this study was already clearly present 70 ms after stimulus onset, which is earlier than what was observed for the unimodal stimuli, for which the earliest attentional effects were only locally present at 110 ms for visual stimuli and at 210 ms for tactile stimuli. This finding is in line with that of Forster et al (2009), who reported that attended unimodal stimuli are processed faster when the input of more than one modality is processed and crossmodal links are involved.

To investigate the theoretical advantage of visual–tactile stimulus presentation on classification accuracies, we analysed offline classification accuracies that were only based on the features related to early ERP components. Our results showed a nonsignificant positive trend for bimodal compared to unimodal stimuli. If ERP components are enhanced, we also expect higher classification accuracies, when the classifier is sensitive enough and the data have not reached a ceiling yet. Therefore, the absence of a significant effect of visual–tactile compared to visual classification accuracies may indicate that the classifier was not sensitive enough to fully employ the visual–tactile ERP component.

If we would hypothesize that in the future technology and knowledge will develop such that with a combination of appropriate electrodes and classifiers the difference between targets and nontargets could be detected almost immediately after the brain response, then the bottleneck for a fast response would be biologically determined. The small latency of the visual–tactile N1 is in that case especially interesting for applications for which (near) real-time interaction is desired, such as in gaming or steering and control tasks that are hampered by delays in the control loop. Using the N1-based classifier, only the visual–tactile BCI could be considered to be successfully operated, although bitrates were considerably higher when the classifier was also based on other ERP components.

Recent studies using a visual ERP-BCI showed that when participants gazed at a central point of fixation compared with gazing at targets directly, BCI performance dropped tremendously (Brunner et al 2010, Treder and Blankertz 2010). This result suggests that gaze-dependent BCIs are not (solely) based on the P300 and make use of low-level visually evoked potentials that are elicited by foveating targets, which is confirmed in the present study. Moreover, it appeared from the investigated relation between time and classification error by Treder and Blankertz (2010) that for gaze-dependent BCIs, the classification error was minimal at approximately 200–250 ms, an interval during which both early and late ERP components were present. A comparison of classification accuracies based on the early (including an early negative ERP component comparable to our N2) versus the late (including the P300) part of the ERP was made for a non-representative participant with a strong early negative ERP component, for whom the early part of the ERP appeared to be more influential (Bianchi et al 2010). We observed a similar dominance of the early ERP components for visual and visual–tactile ERP-BCI performance (compare visual and visual–tactile traces in figure 5(d) to those in 5(b) and 5(c)).

Besides gaze-dependent BCIs, also gaze-independent BCIs may benefit from the early ERP components. In our study, the visual ERP-BCI was gaze-dependent, and only elicited an N2, while the visual–tactile ERP-BCI elicited an N1 and an N2. Thus, by adding a tactile, gaze-independent stimulus, more early ERP activity was enhanced. Whether this N1 depends on gaze or not is not clear and we are currently investigating this. The tactile N2, however, is by definition gaze-independent. Interestingly, we detected this tactile N2 in our previous studies only when vision was involved for determining the target direction (Thurlings et al 2012), but not when the target direction was presented in the tactile modality (Brouwer and Van Erp 2010). Therefore, it seems plausible that the tactile N2 in this study also has some type of multisensory nature. In line with our observations are the results of Forster et al (2009), who found earlier effects of spatial attention on the tactile ERP when vision was involved, compared to when it was not. Our tactile N2 may contribute to the effectiveness of a tactile BCI, if the classifier is only based on the N2 and the P300, and cannot make use of earlier information. Thus, we hypothesize that the application of a tactile BCI when using a visual task, such as navigation, could result in enhanced BCI performance compared to if the visual modality is not involved at all.

In our study, the bimodal stimuli were semantically congruent because the indicated directions of the visual and tactile stimuli were the same. Temporally, the visual and tactile stimuli within a bimodal stimulus were triggered congruently and may be considered temporally congruent. However, the course of stimulus intensity slightly differed between visual and tactile stimuli because of technical properties with a difference of less than 50 ms between the latencies of maximum stimulus intensity. Because previous research showed that multimodal signals are perceived as emanating from the same source if presented within a delay of approximately 100 ms (Vroomen et al 2004), we believe that this temporal difference is unlikely to be the cause of the degraded late multisensory effects.

The semantic and temporal relations within the bimodal stimuli are unlikely to explain the absence of an advantage of bimodal stimuli on task performance, but the spatial relation may do so. The locations of the stimuli indicated directions, which referred to the participants' bodily anteroposterior axes for the tactile stimuli and to the participants' displayed current position on the route for the visual stimuli. Therefore, the indicated directions of the stimuli were congruent, but the stimuli had different locations and thus were location incongruent (for the effect of directional congruency in ERP-BCIs, see Thurlings et al at press).

Previous research has reported contradicting results regarding the necessity of spatial congruency for multisensory integration. While some researchers have reported enhanced task performance for spatially incongruent bimodal stimuli compared to unimodal stimuli (Gondan et al 2005b, Philippi et al 2008, Teder-Salejarvi et al 2005), others have reported the opposite (Ho et al 2009). We believe that the spatial incongruence of bimodal stimuli only hampers multisensory integration if the task involves spatial selective attention, which is the case in both our study and that of Ho et al (2009).

The location incongruence within the visual–tactile stimuli may have degraded task performance because participants had to divide their attention between two locations (and modalities). The bimodal P300 in our study does not reflect a tactile P300 (which occurred relatively late and lasted long), suggesting that tactile targets were not specifically endogenously selected and attended in the visual–tactile condition. Although the bimodal N1 indicated that both modalities were attended in the early stage, endogenous selection occurs only after the first multisensory processing effects that occur within the (prelocated) focus of attention. Thus, spatial attention could have shifted between those two processes around the intermediate stage. One reason for such a shift may be that the tactile stimuli were distracting rather than cooperative and attracted attention away from the visual stimuli so that attention became more dispersed instead of focused on the target location(s). The effects of spatial congruency under various circumstances were previously related to higher perceptual-cognitive processes well after our N1 but around the latency of our N2 in the intermediate stage (Gondan et al 2005b, Teder-Salejarvi et al 2005, Zimmer et al 2010).

In section 2.5.2, we discussed the potential-added value of the bimodal N1 to decrease the response times in future BCIs. In this study, we could not show significant improvement of performance for the visual–tactile compared to the visual condition, neither from a theoretical approach (on classification accuracies when only early ERP components were used, which were increased for bimodal compared to unimodal), nor from a practical approach (on bitrates when using the most appropriate ERP-based classifier and number of repetitions for each modality separately). Nevertheless, in both cases we did find some indications that bimodal BCIs could help the effectiveness of a BCI, as for example only for bimodal was 100% top performance reached for all participants. Future research should point out whether significant improvement can be reached when other classification techniques are used, when bimodal and visual BCIs are not gaze-dependent (and overall performance will be lower) and when bimodal stimulus presentation is offered location congruently. We are currently investigating both latter factors.

A visual–tactile ERP-BCI as outlined in this study makes use of additional ERP components that are elicited or enhanced due to multisensory integration. In addition to the potential-added value of bimodal stimulus presentation on the effectiveness of a BCI, bimodal ERP-BCIs possess other potentially interesting properties. As discussed previously, the gaze-dependence is a problem for users with impaired vision or eye muscles and in situations in which gaze is needed elsewhere (e.g., while driving or gaming). However, the advantage of gaze-dependent visual ERP-BCIs is that these are associated with high BCI performance while performance of gaze-independent ERP-BCIs (visual, tactile or auditory) is relatively low. In a bimodal ERP-BCI, participants do not necessarily have to attend to both stimulus modalities, but they may decide to attend only one modality, thus alternating between the two ERP-BCIs. In that way, the advantages of both modalities may be exploited alternately, as appropriate for a specific situation. This flexibility during usage could be of interest to both able-bodied and disabled users and fits with the growing interest in flexibility by combining different BCIs, also referred to as hybrid-BCIs (e.g., Allison et al 2010). To realize such an alternating bimodal BCI, an additional classifier may be needed to detect the currently attended modality if the ERPs would differ when one or the other modality is attended, so that the appropriate classifier for the currently attended modality can be applied to distinguish targets from nontargets. Furthermore, for less able-bodied users with, for instance, a neuromuscular disorder, it might be helpful to use a bimodal ERP-BCI during the period when their ability to use their muscles is degrading so that a smooth transition from a visual gaze-dependent to a tactile BCI can be realized (Klobassa et al 2009).

In this study, we investigated one aspect which is of importance to the evaluation of any system: effectiveness of the BCI. However, other aspects such as usability are relevant as well. Usability may be influenced by the limitations induced by the hardware required. For the use of the tactile or bimodal BCI, tactors have to be attached to the body. In this study, we used a vest in which tactors are integrated, but are connected with cables and other equipment. The limitations of this hardware seem less restricted though than the hardware necessary to record EEG. Also, recently a tactile wireless belt was developed (Eaglescience⁴ and TNO), which is as easy to wear as a regular belt.