Generalizable cursor click decoding using grasp-related neural transients

Two participants provided informed consent prior to performing any study-related procedures. This study was conducted under an Investigational Device Exemption from the Food and Drug Administration and approved by Institutional Review Board at the University of Pittsburgh (Pittsburgh, PA), registered at ClinicalTrials.gov (NCT01894802). The first participant (P2) is a man with tetraplegia caused by C5 motor/C6 sensory ASIA B spinal cord injury. The participant has some residual upper arm and wrist movement, but no hand function. Approximately five years prior to data collection for the current study, two 88-channel microelectrode arrays (Blackrock Microsystems, Salt Lake City, UT) were implanted in the hand and arm areas of motor cortex.

The second participant (P3) has a C6 ASIA B spinal cord injury. He retains some residual arm and wrist movement, but has no hand function. Approximately four months prior to data collection, two 96-channel microelectrode arrays (Blackrock Microsystems, Salt Lake City, UT) were implanted in the hand area of motor cortex. Both participants also had two 64-channel arrays implanted in somatosensory cortex (Flesher et al 2016), which were not used for this study.

Due to contact restrictions caused by the COVID-19 pandemic, most sessions by P2 were performed using an at-home portable iBCI system (Blackrock Microsystems, Weiss et al 2019). The portable system uses digital Cereplex-E headstages and a portable NeuroPort Signal Processor (Blackrock Microsystems), connected to a medical-grade tablet. As reported previously, this system achieves comparable performance to typical in-lab systems. Briefly, neural signals collected by the portable system were filtered using a 4th order 250 Hz high-pass filter, logged as threshold crossings (−4.5 RMS) and binned at 50 Hz. The binned counts were then convolved online with a 440 ms decaying exponential filter to provide a smoothed estimate of firing rate. An in-house Matlab-based program (Mathworks Inc.) was used to automate the session, progressing through decoder calibration and evaluation tasks. Caregivers were trained to connect the digital headstages and perform battery changes for the tablet when necessary. However, the participant performed all other elements of system operation using the tablet touchscreen, which mostly entailed running the program, selecting the task from a dropdown menu, and pressing on-screen buttons to progress through each phase of the task. Once the participant was comfortable using the system, he could perform sessions with no assistance from researchers. All at-home data for P2 (total of 18 sessions) was collected between 1795 and 1909 days post-implant. Two additional laboratory sessions were used to achieve optimized decoder control and compare performance with the current standard approach (see section 3.5). In the laboratory, data were recorded with the Cereplex-E headstages and standard NeuroPort Neural Signal Processors using the same parameters as the home sessions, except that the initial neural data filtering was done with a 1st order 750 Hz high-pass filter.

All data collected from participant P3 (total of eight sessions) were collected in the laboratory, between 115 and 192 days post-implant. As for P2, two of these sessions were used for control optimization and comparison with the current standard click decoding method.

Recent work in nonhuman primate neurophysiology indicates that neural population activity in motor cortex during upper limb movement can be captured by a relatively small number of correlation patterns (Churchland et al 2012, Sadtler et al 2014, Gallego et al 2017, Degenhart et al 2020). To mitigate concerns of overfitting during decoder training and take advantage of the apparent stability of low-dimensional components (Gallego et al 2018, 2020), we reduced the neural activity to a 20-dimensional (20D) state space using factor analysis. On each session we calculated factor weights from data collected during each initial decoder calibration routine (observation), and then used those weights to reduce all data to a 20D state space. The activity within this reduced state space was used to train the click decoders. We tested different dimensionalities (e.g. 10 and 15) during offline analysis, but found no significant differences in cross-validated performance.

For both participants, we tested two types of calibration routines for translation and click: (a) discrete click center-out, and (b) sustained click center-out. For each session, we selected one calibration type. Decoder calibration occurred at the beginning of the session and consisted of two components: observation followed by partially assisted brain control (see section 2.9). During the observation period, the participants observed the cursor as it moved under computer control between targets on the screen and transitioned between click states. The participants were asked to perform covert (i.e. imagined) movements with their right arm corresponding to the cursor translation (e.g. move arm to the left when the cursor moves to the left), and attempted grasp corresponding to cursor click (i.e. grasp for click, release for unclick). The participants' hands are naturally in a clenched, palmar grasp posture due to hypertonia. Both reported that their motor imagery for 'grasp' and 'release' corresponded to isometric force production ('squeeze' and 'relax') rather than finger flexion and extension.

2.4.1. Discrete click calibration task

A schematic of the task is shown in figure 1(a). Each trial (40 total) began with the cursor in the central target. One of eight outer targets then appeared, and after a short delay (0.5 s) the cursor moved with a bell-shaped velocity profile to the outer target. A voice then cued the participant to 'click', followed approximately 3 s later by 'release'. During the clicked period, the cursor changed from an open circle to a filled circle. After release, the cursor returned to the center target to begin a new trial. Due to click and release occurring consecutively at the outer target, this calibration task only included cursor translation while the cursor was in the unclicked state (figure 1(a), bottom). This calibration design closely mirrors ones used previously for demonstrating point-and-click control (Kim et al 2011).

Zoom In Zoom Out Reset image size

2.4.2. Sustained click calibration task

We aimed to test the ability of the click decoders to generalize across the two main functions necessary for full click function: point-and-click (discrete click), and click-and-drag. To improve user engagement, the tasks were stylized as helicopter-based arcade games. Schematics of the two tasks are shown in figures 1(c) and (d). During each session, the participants first completed 16 trials of the click-and-drag task with each decoder, followed by 16 trials of the point-and-click task with each decoder. This sequence then repeated, resulting in 32 trials for each decoder/task combination.

2.5.1. Point-and-click

2.5.2. Click-and-drag

To inform the development of our novel click decoder, we first aimed to identify prominent neural activity patterns related to click (attempted/covert grasp) observed during the discrete click calibration task. To do this, we performed an extensive grid search for unique activity patterns across all potentially relevant temporal windows encompassing grasp and release (figure 2). For each step of the search we selected a window start and window end relative to each grasp event (100 ms increments) and assigned class labels to the neural factors (class A if within window, class B otherwise). We then fit a linear discriminant analysis (LDA) classifier on the resulting dataset, thus attempting to isolate the activity observed within the selected window. Using ten-fold cross-validation, we obtained the resulting performance of the classifier, measured using Matthews correlation coefficient (MCC):

$$\begin{align*}{\text{MCC}} \!=\! \frac{{{\text{TP}} \cdot {\text{TN}} - {\text{ FP}} \cdot {\text{FN}}}}{{\sqrt {\left( {{\text{TP}} \!+\! {\text{FP}}} \right)\left( {{\text{TP}} \!+\! {\text{FN}}} \right)\left( {{\text{TN}} \!+\! {\text{FP}}} \right)\left( {{\text{TN}} \!+\! {\text{FN}}} \right)} }},\end{align*}$$

Zoom In Zoom Out Reset image size

where ${\text{TP}},{\text{ TN}},{\text{ FP}}$ and ${\text{FN}}$ represent the number of true positives, true negatives, false positives, and false negatives, respectively. Due to its symmetry, the MCC metric provides a good indicator of classification performance even for highly unbalanced datasets (as is the case here for very small temporal windows). However, MCC can still display biases due to class imbalances (Zhu 2020). To address this limitation and improve comparisons of classification performance across window sizes with different inherent class imbalances, we introduce a slightly adjusted version of MCC:

$$\begin{align*}{\text{MC}}{{\text{C}}_{{\text{adj}}}} = { }\frac{{{\text{MCC}} - K}}{{1 - K}},\end{align*}$$

where $K$ is the minimum MCC achieved across all classifications with the same class imbalance (i.e. window size).

After sweeping across all windows from 1 s pre-click to ∼1.5 s post-release, we observed three local maxima—indicating three separate neural responses related to click/grasp—which are highlighted in figure 2. The first was a window starting around the time of click onset and ending around the time of release (figure 2, purple), indicating a neural component related to sustained grasp. The second was a window starting just before release and ending about half a second after release (figure 2, orange), and the third was a window starting just before click and ending about half a second after click (figure 2, blue). These two components represent transient responses at the offset and onset of grasp, respectively. Note that absolute MCC_adj score does not necessarily indicate the magnitude or salience of the associated neural response. Small trial-by-trial variation in the timing of an attempted action (and the corresponding neural response) will have a significantly greater impact on classification performance for short time windows (e.g. figure 2 orange or blue) than for long windows (figure 2, purple). However, this sensitivity to temporal variability is only present during offline classifier training, which assumes fixed relationships between the cues and the neural responses, and does not predict the performance during asynchronous online control.

The sustained neural response that occurs during grasp (identified in figure 2) has been used in previous approaches to click decoding, in which a classifier is used to directly output the click state (Kim et al 2011, Simeral et al 2011, Jarosiewicz et al 2015, Pandarinath et al 2017). We replicated this approach to provide a baseline comparison of existing click decoder functionality. To calibrate the decoder, we used the click state (clicked/unclicked) at each point in the calibration task to assign class labels and then trained a simple LDA classifier on the 20D neural state (factors). During online control, we applied the classifier at each time point (20 ms) and directly mapped the output to the click state. This simple linear implementation does not reflect the most state-of-the-art approach to click decoding (e.g. Pandarinath et al 2017). However, we chose to focus on simple LDA for both the transient and sustained decoders to create the most direct link between neural activity and click behavior. In a following section (section 3.5) we introduce a more advanced implementation that uses a hidden Markov model (HMM) framework for both decoder types to gauge the upper limit of control provided by each decoder type.

While previous approaches to click decoding utilized sustained neural responses during grasp, we aimed to instead use the transient responses observed at the onset and offset of grasp (figure 2). To do this, we trained two independent classifiers—one for decoding grasp onset and one for decoding grasp offset—which then ran concurrently during online control to update changes in click state. To train each transient classifier, we first identified the optimal time window for a given session. From the results in figure 2, we found that the transient responses (observed in the smoothed estimate of neural firing rates, see section 2.2) were only about half a second long. These short time windows meant that idiosyncrasies in the user's approach during calibration—which might vary across days or across subjects—could significantly impact the classification. For example, a user might attempt to grasp immediately upon hearing the audio cue to 'click', or might wait until visual feedback of the cursor changing from the unclicked to clicked state. For this reason, on each session we trained the transient classifiers using a limited grid-search approach similar to the one outlined in the section 2.6. However, rather than search the entire parameter space, we restricted the search to smaller time periods. For both grasp and release, we swept through a range of window centers (−1.0 s to +1.0 s relative to onset/offset, 0.1 s increment) and window widths (0.2–2 s, 0.1 s increments). For each window, we computed the output probabilities from the resulting LDA classifier (ten-fold cross-validation) and calculated the MCC for probability thresholds between 0.1 and 0.9 (increments of 0.1). We then selected the time window with the greatest cumulative MCC (summing across all thresholds).

During online control, we applied a simple heuristic to convert the transient classifier outputs to click function. If the cursor was unclicked and the grasp onset transient probability exceeded both 0.2 and the release transient probability, the cursor entered the clicked state. If the cursor was in the clicked state and the release transient probability exceeded both 0.2 and the grasp onset transient probability, the cursor entered the unclicked state. We chose a threshold value of 0.2 since it was the approximate threshold at which we observed maximum MCC during calibration. Rather than set optimal thresholds for each session, we instead settled on a single threshold value in order to remove it as a possible source of session-by-session performance variability.

The focus of this study was decoding click, rather than cursor translation. To maintain consistent translation performance in combination with both tested click decoders, we used an optimal linear estimator (OLE) approach to decode cursor velocity. We have previously used this kinematic decoding approach to demonstrate high-quality control for two-dimensional cursor movement (Weiss et al 2019) and up to ten-dimensional arm/hand movement (Collinger et al 2013, Wodlinger et al 2014).

Following our previous approaches for kinematic decoding, we followed up each of the calibration routines (discrete click or sustained click) with a set of partially assisted online brain control trials. These trials (40 total) followed the same center-out format, but with movement velocity controlled by the OLE translation decoder (restricted to the target axis). The data collected during this assisted set was used to fit a new OLE translation decoder, but was not used for any aspect of click decoder calibration.

During the point-and-click task, the sustained click decoder was only effective when trained on the discrete click calibration task (figure 4(a), left). This condition is equivalent to previous demonstrations of point-and-click control, including the same training paradigm, decoding approach, and evaluation task (Kim et al 2011). Almost all trials of this type fell into two categories—success and early click—with roughly equal probability. The relatively low success rate in comparison to previous studies is likely because we only examined the first click on each trial when determining success or failure rather than allowing multiple attempts following an errant first click. When trained on the sustained click calibration task, the sustained decoder failed to provide adequate control, with a high occurrence of early clicks. This indicates that sustained grasp-related neural activity is not easily isolated from translation-related signals during simultaneous control, which matches results from a previous study from our group (Downey et al 2018).

Zoom In Zoom Out Reset image size

Unlike the sustained click decoder, the transient-based click decoder was successful at providing point-and-click functionality regardless of calibration routine (figures 4(c) and (d)). It did display a higher incidence of 'drag out' failures (i.e. failing to unclick before leaving the target). However, this can likely be attributed more to limitations in translation control than to a failure of click control. From the click locations shown in figure 4(c), the 'drag out' failures almost exclusively occurred on trials where the participant clicked on the outer edge of the target. Thus, those trials appear to reflect a failure in stabilizing the cursor, and not necessarily a failure in release control.

The sustained click decoder was unable to provide any meaningful drag functionality, regardless of calibration routine (figures 5(a) and (b)). However, the types of failures depended on the calibration routine. When trained on discrete click calibration, the participant was able to successfully reach the outer target on a significant number of trials. However, he was unable to maintain click during translation back to the center target, and almost every trial ended with an early drop. This failure to maintain the click is a result of the limitations caused by the calibration routine. The discrete click calibration routine only included interleaved translation and click. Thus, the resulting classifier was not able to generalize to the condition in which translation coincided with maintained click. When trained on the sustained click calibration routine, the sustained click decoder behaved equally poorly, but the failures almost entirely resulted from early click—this failure type corresponds to false positives during grasp classification.

Zoom In Zoom Out Reset image size

The transient-based decoder provided high click-and-drag functionality regardless of calibration routine (figures 5(c) and (d)). As during the point-and-click task, the most common failure was an early click before reaching the target. However, especially for the decoder trained following the sustained click calibration routine (figure 5(c), right), the majority of early clicks occurred just outside of the outer target, and thus appear to reflect inadequacies in translation control rather than click control. The decoder trained from sustained click calibration also had a lower incidence of drops (8% vs 17%, figure 5(d)), suggesting slightly better release control. Together, these results indicate that while the performance of a transient-based decoding approach is largely invariant to the calibration task, a calibration routine involving sustained click (i.e. translation in both clicked and unclicked states) may lower the incidence of both unintentional clicks (figures 4(d) and 5(d)) and unintentional releases (figure 5(d)).

The results in figures 4 and 5 reflect the strictest possible success criteria. As described in section 2.5, early click errors during task performance did not actually trigger trial failure. To evaluate performance in a more forgiving framework, we recalculated the overall performance metrics after allowing for multiple clicks (figure S1 (available online at stacks.iop.org/JNE/18/0460e9/mmedia)), which is equivalent to the participant's online experience while performing the tasks. For the transient decoder, this resulted in an increase in point-and-click success rates to 72% (discrete click calibration) and 57% (sustained click calibration) and click-and-drag success rates to 72% and 82%. For the sustained decoder, point-and-click success rates increased to 91% and 90%, but click-and-drag rates increased only to 5% and 16%. The most striking change in performance was the improvement in point-and-click performance by the sustained decoder (from 10% to 90%). However, the participant performed many unnecessary clicks in this condition, with 46% of successful trials containing at least five clicks (figure S2). Trials with the transient decoder (trained using either calibration routine) contained only one or two clicks on >95% of trials, indicating that even though the success rate was lower than the sustained decoder, it provided more reliable and consistent control.

In addition to total successes, we also investigated the temporal aspect of control achieved by each decoder. To summarize general performance speed, we calculated the target acquisition rate (number of successful trials divided by the total task time, excluding intertrial periods) for each 16-trial block of the point-and-click and click-and-drag tasks (figure 6(a)). Acquisition rates varied considerably even within condition, which reflects cross-session variability in both click control and translation control. The sustained click decoder only achieved consistent, meaningful control on the point-and-click task, and only when trained using discrete click calibration (median rate of 3.6 successes/minute). This rate was not significantly different from the rate achieved by the transient decoder trained using discrete click (p = 0.63, Mann–Whitney U-test) or sustained click (p = 0.66, Mann–Whitney U-test) calibration. The variance in performance was also not significantly different (discrete click: p = 0.29, sustained click: p = 0.08, F-test).

Zoom In Zoom Out Reset image size

For the point-and-click task, the transient click decoder achieved a median acquisition rate of 3.1 successes/minute when trained using discrete click calibration, and 3.0 successes/minute when trained using sustained click calibration. Thus, the calibration paradigm had no effect on median acquisition rate (p = 0.76, Mann–Whitney U-test). However, performance on this task was more consistent across sessions when trained using discrete click calibration (σ = 1.2 compared to σ = 2.7, p = 0.006, F-test; open vs closed circles in figure 6(a)). For the click-and-drag task, the transient click decoder achieved a median acquisition rate of 3.7 successes/minute when trained using discrete click calibration and 4.4 successes/minute when trained using sustained click calibration. These rates were not significantly different from each other in median (p = 0.21, Mann–Whitney U-test) or cross-session variance (p = 0.87, F-test). These results indicate that the transient decoder, unlike the sustained decoder, provided generalizable control, and allowed the participant to achieve consistent performance across tasks.

To better analyze the behavior of each click decoder during control, we calculated the latency of click and release responses across all successful trials on both tasks (figure 6(b)). On each successful trial we identified the time lag between when click or release was possible, and when it actually occurred. For both tasks, click latency thus represents the time between when the cursor entered the outer target and when click occurred. Release latency during the point-and-click task is simply the time between click and release, and for the click-and-drag task it is the time between completion of the drag (outer target coinciding with center target) and the release. The average click latencies were not different between the two decoders (p = 0.88 Mann–Whitney U-test), with a median click latency of 0.58 s for the sustained decoder and 0.56 s for the transient decoder. However, the release latencies differed significantly (p < 10⁻⁴⁰, Mann–Whitney U-test), with a median release latency of 0.12 s for the sustained decoder and 0.56 s for the transient decoder. Note that for the transient decoder, the release latencies mirrored the click latencies (p = 0.61, Mann–Whitney U-test), indicating that both likely resulted from similar volitional control. Sustained decoder release latencies were skewed significantly lower than the click latencies (p < 10⁻²⁸, Mann–Whitney U-test). The distribution of release latencies comprised almost entirely point-and-click trials (see breakdown of successful trials, figures 4(a) and (b)), and indicates that for point-and-click function, a sustained click decoding approach can provide shorter click durations and improve the speed of button clicking. However, this advantage comes at the cost of generalizability, as the click cannot be maintained during cursor translation.

The behavioral results from the point-and-click and click-and-drag tasks indicate that a transient-based approach to click decoding can provide more generalizable control of cursor click. Here we examined the neural response features used by each decoding approach to better understand the cortical control of grasp and its application to decoding.

On each session, we aligned neural responses (during calibration) to click events and projected the neural activity onto the three (normalized) LDA axes (see figure 2) used by the decoders (figures 7(a) and (b)). The resulting traces thus correspond to the main cortical responses observed during attempted grasp (see section 2). Across all sessions and calibration routines (discrete click: figure 7(a), sustained click: figure 7(b)), we found consistent and reliable responses related to all three grasp-related responses: onset (blue), offset (orange) and sustained (purple) grasp. However, the salience (magnitude) of the responses was not equal, with the sustained component consistently weaker than either transient response.

Zoom In Zoom Out Reset image size

To compare response magnitudes across the three identified neural axes, we calculated on each trial the maximum deviation along each axis. Specifically, we found the range (difference between the maximum and minimum) observed on each trial after projecting along the onset, offset, and sustained axes (figures 7(c) and (d)). We found that both transient responses were consistently stronger than the sustained response. The maximum deviations along the grasp onset axis were greater than the maximum deviations along the sustained axis during discrete click calibration (median onset = 3.7 a.u. median sustained = 3.4; p < 10⁻²⁰, paired t-test) and sustained click calibration (median onset = 4.1 a.u. median sustained = 2.9; p < 10⁻⁵⁵, paired t-test). The maximum deviations along the grasp offset axis were also greater than the maximum deviations along the sustained axis for discrete click (median offset = 4.3 a.u. median sustained = 3.4; p < 10⁻⁵⁷, paired t-test) and sustained click (median offset = 4.4 a.u. median sustained = 2.9; p < 10⁻⁶², paired t-test) calibration. Results for participant P3 were qualitatively similar (figure S3), however the onset and offset responses were not significantly larger in magnitude than the sustained response during discrete click calibration (p > 0.5, paired t-test). During sustained click calibration, the transient responses were both significantly larger than the sustained response (p < 10⁻¹⁰, paired t-test). These results highlight the significance of transient cortical responses during grasp control, responses that were previously excluded from grasp-driven approaches to click decoding.

For the majority of our data collection, we chose to use a basic linear classification method (LDA) so that the performance of both decoders would be simply and directly linked to the salience of the underlying neural activity. However, within either decoding framework (i.e. decoding transient onset/offset vs decoding sustained grasp), more complex methods can be applied to improve control. A recent demonstration of high-performance point-and-click decoding used a HMM framework, which proved to significantly reduce the incidence of inadvertent clicks compared to a simple linear approach (Pandarinath et al 2017). Yet it is unclear whether the same implementation of this state-of-the-art approach (which was based on a sustained grasp response) could also provide useable click-and-drag function.

For both the sustained and transient-based approaches, we adjusted the decoders to incorporate an HMM framework as described by Pandarinath et al. Briefly, an HMM improves classification by incorporating the probability of each transition between classification states. This adds 'inertia' to the current state, and helps prevent misclassifications due to brief fluctuations (e.g. due to noise) in the neural activity. Each participant completed two sessions of the click-and-drag task (sustained click calibration) using the HMM versions of the sustained click (equivalent to Pandarinath et al 2017) and transient decoders.

We found that even with the HMM implementation of the sustained decoder, the participants were still unable to reliably perform the click-and-drag task (figures 8(a)–(d), left). However, they were both able to use the transient HMM decoder to complete the task (figures 8(a)–(d), right). While P3 only ever used the HMM decoders, P2 used both the linear and HMM decoder implementations, and achieved higher success rates using the transient HMM decoder (approximately six successes per minute, figure 8(b)) than the linear transient decoder (approximately 4.5 successes per minute, figure 6(b), right). Even after hyperparameter optimization, the sustained HMM decoder was unable to match the click-and-drag performance of the transient HMM decoder (figures S4 and S5).

Zoom In Zoom Out Reset image size

Participant P2 (who performed most experiments remotely) was encouraged throughout the experimental period to use the system for applications outside of the experimental tasks outlined in this study. After performing a calibration routine (as outlined in the section 2), he was free to select one of the decoders (named 'HybridRIOLEdGraspState_v2' and 'HybridRIOLedOnOff_v2' for transient-based and sustained-based decoders, respectively) and use it as a mouse emulator for any application of his choosing. Over the course of the data collection period, he used the system for this purpose 15 times. On every occasion he chose to use the transient-based click decoder over the sustained click decoder. He was not told the specifics of how each decoder worked, but when asked to describe the performance of each, he stated, 'One of them works, and the other doesn't'. One common application he performed with the decoder was digital painting, which requires the ability to perform discrete clicks (for selecting brushes, colors, etc) and also the ability to click-and-drag (for drawing lines, etc). An example painting showcasing this control is shown in figure 9. In addition to painting, he also used the BCI system for playing a card-based computer game that also requires the ability to click-and-drag.

Zoom In Zoom Out Reset image size