Cursor control by Kalman filter with a non-invasive body–machine interface

Subjects were presented with a cursor that moved on the monitor and were asked to move their shoulders with the cursor, as if they were controlling it. They were instructed to 'follow' up and down cursor movements by moving their right shoulder up and down (elevation and depression respectively), and to follow right and left cursor movements by moving their left shoulder up and down respectively.

The 1 cm diameter cursor moved through a predetermined centre-out path with a cosine velocity profile so that the cursor's position history while moving right /left followed the function:

$$\begin{eqnarray}&&{{x}_{t}}=\;5{\rm sin} \left( \frac{\pi }{4}t \right),\end{eqnarray} \tag{ 1 }$$

$$\begin{eqnarray}&&{{y}_{t}}=\;0.\end{eqnarray} \tag{ 2 }$$

Here $t\;=\;k{\rm d}$t with k = 1,...,200 and dt = 0.02, so that the total duration of each cursor movement from the centre to the right direction and back lasted for a total of 4 s. A 6 × 6 cm box enclosed the cursor's movement range so that subjects knew when the cursor was going to reach the edge and come back to the centre, and they could plan to move their shoulders accordingly. Each of the four directions was reached six times for a total calibration time of 96 s.

Cursor and body motion data were logged during the calibration phase. The position, velocity, and acceleration of the cursor were recorded at each time step k (every 20 ms) as the cursor's state, i.e. ${{{\boldsymbol{s}} }_{{\boldsymbol{k}} }}=[x,\;y,\;{{v}_{x}},{{v}_{y}},{{a}_{x}},{{a}_{y}}]_{k}^{T}$ where s_k $\in$ R6 × 1. The IMUs Euler angles, angular velocities, and linear accelerations were recorded as the body observation in a 24-dimensional vector z_k = {[2-Euler angles (roll, pitch) + 2-gyroscope + 2-accelerations]*4 sensors} at each time step k as the subjects followed the cursor with their body. Both data were fed into a Kalman filter to learn the mapping between body motions and cursor kinematics.

The main goal of the Kalman filter is to make an estimation of the cursor's state, at every instant in time. The Kalman model assumes the cursor's state at time k to be linearly related to the future state at time k + 1 via the stochastic linear function

$$\begin{eqnarray}&&{{{\boldsymbol{s}} }_{{\boldsymbol{k}} +{\bf 1}}}={{A}_{k}}{{{\boldsymbol{s}} }_{{\boldsymbol{k}} }}+{{w}_{k}},\end{eqnarray} \tag{ 3 }$$

where k = 1,2,...,M, A_k $\in$ R^6 × 6 is the matrix that linearly relates the cursor's kinematics between successive time steps, w_k represents the process noise term, which we assumed to have zero mean and to be normally distributed, i.e. w_k ∼ N(0,W_k), W_k $\in$ R^6 × 6, and M is the total number of time steps.

Due to subjects following the cursor with their body as if they were controlling it with their shoulders during the calibration phase, we assumed the body motion observation to be linearly related to the state at each point in time via the stochastic linear function

$$\begin{eqnarray}&&{{{\boldsymbol{z}} }_{{\boldsymbol{k}} }}={{H}_{k}}{{{\boldsymbol{s}} }_{{\boldsymbol{k}} }}+{{q}_{k}},\end{eqnarray} \tag{ 4 }$$

where z_k $\in$ R^C × 1 is the vector containing the IMUs' observation at each time step k. C is the dimension of the observation vector (24 in this case, but will change for other groups as explained in 3.2.3. Familiarization). H_k $\in$ R^C × 6 is the coefficient matrix that linearly relates the cursor's state to the body motion, and q_k is the measurement noise term, i.e. q_k ∼ N(0,Q_k), Q_k $\in$ R^C × C.

In principle, A_k, H_k, W_k, and Q_k might change at each time step k. However, we made the common simplifying assumptions that they remain constant. Therefore, we can estimate each matrix from calibration data using least squares (for details, see (Wu et al 2002)). After the model's parameters were estimated, the cursor's kinematics and body motion were now encoded by equations (3) and (4) respectively, and subjects could now control a cursor by moving their shoulders.

After the calibration phase, subjects were randomly assigned to one of three groups to perform the rest of the experiment, so that each group had eight subjects. Each group had a different amount of information in their observation vector used to control the cursor. Subjects in the first group had only Euler angles in the observation vector (group E), subjects in the second group had Euler angles, and angular velocities (group EV), and subjects in the third group used Euler angles, angular velocities, and linear accelerations to control the cursor (group EVA). Subjects were then allowed to try the mapping in a familiarization phase for 1 min.

They were asked to move through their entire range of motion during the calibration phase. However, performing these types of movements for the whole duration of the experiment would cause exhaustion. Therefore, we reduced the effective range of motion by amplifying the measured motion signals by 300% for the rest of the experiment. This meant that subjects would have to elevate their right shoulder to 33% of their range in order to reach a target located 5 cm above the origin.

Typical joysticks have a mass spring damper system that allows them to come back to the resting position when no force is applied, so we implemented a filter to obtain a similar behaviour. The cursor's position, or the x and y in the state, were modelled as forces acting orthogonally on a mass spring damper system described by the equation of motion

$$\begin{eqnarray}&&{\boldsymbol{u}} _{k}^{^{\prime\prime} }+\frac{b}{m}{\boldsymbol{u}} _{k}^{^{\prime} }+\frac{k}{m}{{{\boldsymbol{u}} }_{k}}=0,\end{eqnarray} \tag{ 5 }$$

where ${{{\boldsymbol{u}} }_{{\boldsymbol{k}} }}\;=\;[x^{\prime} ,\;y^{\prime} ]_{k}^{T}$ represents the cursor's new, filtered position coordinates. Values for the mass, spring, and damper coefficients were tuned so that the system had a resulting damping ratio of

$$\begin{eqnarray}&&\zeta =\;\frac{b}{2\sqrt{mk}}=0.1,\end{eqnarray} \tag{ 6 }$$

There was no specific goal for the familiarization phase, but the subjects were told to try to move the cursor up, down, left, and right several times, to check that they had control of its movements and check that they could bring the cursor back to the origin, and finally, to make sure that they could reach the four corners of the screen. The calibration procedure was repeated if a subject was not comfortable with the map.

Twenty-four subjects performed the first reaching task. Once subjects familiarized themselves with the map, they performed five blocks of a centre-out reaching task. Subjects controlled a blue, 1 cm diameter cursor to reach four targets 4 cm in diameter appearing in random order on the screen 5 cm below, above, to the right, or to the left of the origin. While this study was limited for practical purposes to a few reaching targets, the interface, after the calibration phase, allowed them to move in all directions and to reach all points of the computer display. Subjects were allowed a total of 6 s to complete the task before the target disappeared. Subjects had to remain inside the 4 cm diameter origin target for 200 ms for a new yellow target to appear. The subjects were instructed to reach the targets as quickly and accurately as possible and remain inside them for 1 s. The targets turned green while the cursor was inside them and turned red after the 6 s 'deadline', where the trial was logged as a failed attempt and the target returned to the origin.

Subjects performed 24 trials per block, with random target order comprised of exactly six trials in each direction. The experiment consisted of five blocks and there was a 1 min resting period between blocks. This protocol allowed us to chart an explicit learning curve for different performance measures for each of the 24 subjects.

A group of four subjects performed a second reaching task using the map that resulted on the best performance for the Four-Target Reaching Task. After following the same Calibration and Familiarization procedures, subjects performed a set of five training blocks of centre-out reaching, with generalization blocks before and after training. The training and generalization task schedule is shown in table 1.

Training Trials consisted on subjects controlling a mouse pointer to reach five targets 2.22 cm in diameter appearing in random order on the screen. Unlike the first reaching task where subjects had to move only one shoulder at a time to reach a target, the five target locations for this task required subjects to combine and coordinate shoulder motions in order to reach them. Subjects were allowed a total of 1 s to reach the target before it changed colour. However, contrary to the first reaching task, subjects had unlimited time after the target changed colour to make corrections and complete the trial by remaining inside the target for 1 s. Subjects performed 20 training trials per block, with random target order comprised of exactly five trials in each direction.

Blind Trials occurred in random order within the same block as the Training Trials in order to test if subjects formed an inverse model of the shoulder-to-cursor map, or if they were relying purely on visual feedback to control the cursor. The blind trials were to the same locations as the training trials, except that cursor feedback was removed for the first second of the trial. Subjects performed five blind trials per block, with random target order comprised of exactly one trial in each direction.

Generalization Trials occurred before and after the five blocks of training. These trials consisted on five additional targets that were not seen during training. The targets were a rotated and scaled version of the training targets (figure 7, bottom row). Target distances were scaled down by 75% in order to ensure that subjects would be able to reach them. The different target locations required subjects to make different combinations of shoulder motions than the ones used during Calibration and the ones used during training. In order to prevent a training effect, subjects performed ten generalization trials per block, with random target order comprised of only two trials in each direction.

After completing all blocks the Five-Target Reaching Task, subjects were placed in a virtual environment developed by our laboratory using a commercial- grade, 3D gaming engine (Unreal Development Kit, Epic Games, USA). Controlling a virtual wheelchair in the simulated environment provided a safe environment where participants could learn and practice simulated driving tasks without the risks of collisions or serious accidents. The simulator was adapted to use the 2D output of the body–machine interface as the virtual wheelchair's joystick input. The custom environment reproduced a series of task features that mirrored those that the participant would need to perform in a real wheelchair. These tasks were a modified version of the Wheelchair Skills Test (version 4.1, http://www.wheelchairskillsprogram.ca).

As participants drove the virtual wheelchair along the environment, instructions appeared on the screen telling them where to go and what to do. A research assistant also provided feedback and guided the participant through the tasks. All subjects completed seven tasks one time. The tasks were done in the following order for all participants:

• (1)  
  Driving forward in a straight line, turning 90° counter-clockwise, driving in a straight line, turning 90° clockwise, driving forward in a straight line, opening a door by pressing a proximity switch, and entering the doorway before the door closed after 10 s.
• (2)  
  Parallel parking between two wheelchairs and driving through an open doorway.
• (3)  
  Driving forward in a straight line, turning 90° clockwise, driving in a straight line, turning 90° counter-clockwise, driving forward in a straight line, opening a door by pressing a proximity switch, and entering the doorway before the door closed.
• (4)  
  Driving in slalom form between a set of three barrels and driving through an open doorway.
• (5)  
  Driving forward in a straight line, turning 90° clockwise, driving in a straight line, turning 90° counter-clockwise, driving forward in a straight line, opening a door by pressing a proximity switch, and entering the doorway before the door closed.
• (6)  
  Driving in a circle around a barrel with seven outside barrels as a barrier and driving through an open doorway.
• (7)  
  Driving forward in a straight line, turning 90° counter-clockwise, driving in a straight line, turning 90° clockwise and driving forward in a straight line.

Error Frequency was defined, for each block, as the ratio of failed attempts to total allowed attempts (24 per block) i.e. a ratio of 0.8 would indicate that 80% of the trials in one block were not successfully completed. This measure indicated overall performance.

Movement Time was computed as the time between a target appearing on the screen and the target disappearing after 1 s of the subject being inside it (successfully completing the task). This measure indicates the speed with which a subject was able to complete the task.

Movement Variability was computed as the standard deviation along the axis orthogonal to the direction of the target. This was a measure of the extent to which the sample points lay in a straight line along that axis.

Path Length Ratio was defined as the sum of the Euclidian distance between time consecutive cursor points along the path of one trial, divided over the ideal distance for that trial. This measure indicated the 'straightness' or 'effectiveness' of the movement. A path length ratio equal to one would indicate that the subject moved ideally from the origin to the target.

All performance measures were averaged over all trials to obtain four values per block (one for each reaching direction) for each subject. This resulted in a total of twenty values per subject for the whole experiment. Together, these performance measures allowed us to elicit differences in the cursor's path control within each subject, within a group, and between the three different groups. Other performance measures (average distance to target, average movement perpendicular error, maximum perpendicular error, dimensionless jerk) were also computed, but they were highly correlated to these four, so these four were enough to characterize movement and performance.

A two-way mixed model analysis of variance (ANOVA) was performed on each performance measure with BLOCK (1–5) as the within participant factor and GROUP (E, EV, EVA) as the between-participant factor. Violations of sphericity were corrected by the Greenhouse–Geisser method. A post-hoc comparison using a Tukey correction was performed to test the null-hypothesis that the mean between groups at each block was the same. In order to determine if subjects from one group were better than subjects from another group after the five blocks, a post-hoc pairwise comparison using Bonferroni correction was performed to test the null-hypothesis that the mean between two groups at the fifth block was the same. These tests were repeated for each performance measure and allowed us to reject the null hypothesis at each block at p < 0.05.

A paired t-test was used to analyse a group's overall improvement in performance. There was an average performance for each subject on the first and on the last blocks. We tested the null hypothesis that the mean difference between paired observations of the two blocks was zero. We repeated this test for each performance measure and it allowed us to reject the null hypothesis at p < 0.05.

We asked the question of how much calibration time is necessary for our subjects to perform well enough. Ideally, we could calibrate the Kalman filter by having the subject move only once in each direction (or 16 s). In reality, subjects might need to move more than once in each direction in order to perform as well as if they moved the six times to each direction (or 96 s) as in the Calibration phase. We repeated the calibration of the filter (learning the mapping matrices) by using six different calibration-phase durations: 16, 32, 48, 64, 80, or 96 s (note that it takes 16 s for the subject to move once to each of the four directions).

We tested each of the six maps on a testing set that consisted of the body-movement observations for the last 16 s of calibration data. We 'fed' those observations into the maps in order to make a prediction of the state, and we called this the reconstructed state. We then computed the correlation between the reconstructed state and the actual state. There was one correlation coefficient for each of the dimensions of the state ($x,\;y,\;{{v}_{x}},{{v}_{y}},{{a}_{x}},{{a}_{y}}$). This allowed us to test the performance of the calibration for each of the six calibration times.

All subjects were able to complete the five blocks of the reaching task. Differences in performance between blocks for the three different groups are shown in figure 3. There was an overall improvement in performance after practice for subjects in the three groups. However, the level of improvement was not consistent across the groups.

Zoom In Zoom Out Reset image size

A reduction in error frequency across the five blocks was apparent for most subjects in all three groups (figure 3, row 1). This indicated that, with practice, subjects became more accurate and were able to control the target well enough to perform the task in the allotted time. There was a block effect on error frequency (p < 0.01). However, group differences did not depend on block (ANOVA interaction effect block*group p = 0.158). The between-groups difference in error frequency was only statistically significant between groups E and EVA, as demonstrated by the results of the Tukey multiple comparison (p = 0.016). This value is shown on top of the line connecting both groups' error frequency panels. Differences in error frequency between groups E and EV, and differences between groups EV and EVA were not significant (p = 0.063 and p = 0.785 respectively). It is important to notice that most subjects on group EVA had an initial error frequency of less than 0.1, so even though they reduced their error frequency to zero, there was not much change in their performance.

Subjects were allowed a total of 6 s to complete the task before the target disappeared, however subjects in all three groups learned to reach the target in much less time. With practice, the general trend of all three groups seemed to be to reduce the movement time (figure 3, row 2). By the fifth block subjects in all three groups had a shorter movement time than their first block. This indicated an improvement in the control and familiarization with the 2D cursor using their shoulders. There was a block effect on movement time (p < 0.01). However, group differences did not depend on block (ANOVA interaction effect block*group p = 0.398). The between-groups difference in movement time was statistically significant between groups E and EVA (p = 0.011). Differences in movement time between groups E and EV, and differences between groups EV and EVA were not significant (p = 0.083 and p = 0.583 respectively).

A reduction in movement variability across the five blocks was also apparent for most subjects in all three groups (figure 3, row III). Subjects on group E had higher movement variability than subjects on groups EV and EVA for the first block. This result suggested that subjects learned to decrease the variability of their movements and converged to one movement strategy. Accordingly, they possibly learned to reduce movements that were unnecessary or caused the cursor to move in an undesired direction. There was a block effect on movement variability (p < 0.01). However, group differences did not depend on block (ANOVA interaction effect block*group p = 0.474). The between-groups difference in movement variability was only statistically significant between groups E and EVA (p = 0.031). Differences in movement variability between groups E and EV, and differences between groups EV and EVA were not significantly different (p = 0.191 and p = 0.603 respectively).

As mentioned in the Reaching Test section, subjects moved towards a straighter trajectory of the controlled cursor. This was quantified by an overall reduction of path length across the five blocks of the experiment. There was a block effect on path length (p < 0.01). However, group differences did not depend on block (ANOVA interaction effect block*group p = 0.220). Interestingly, the between-groups difference in path length was not statistically significant between any two groups. The difference between groups E and EVA was not significant (p = 0.118). Differences in path length between groups E and EV, and differences between groups EV and EVA were also not significant (p = 0.447 and p = 0.671 respectively).

The results for the post-hoc pairwise comparison at block five can be seen in table 2. Significant differences in performance on the fifth block were observed mostly between groups E and EV and between groups E and EVA. However, there was not significant difference in performance at the fifth trial between groups EV and EVA.

Most subjects were able to learn and improve their performance as evaluated by all four measures. Learning averages for each group are shown in figure 4. Each group's bar represents the mean of the differences between the first and the last block for all subjects in that group. A positive value indicates an improvement in performance. The results of the paired t-test are shown by the p-value on the top of each plot. An asterisk and bold number indicate a significant learning for that group. The learning effect was not consistent between groups. Groups E and EV seem to show a stronger learning effect than group EVA. This was mostly due to a ceiling effect, because subjects on group EVA might have been at a ceiling of performance since the very first block.

Zoom In Zoom Out Reset image size