Neural activity shaping utilizing a partitioned target pattern

A LNL model is used for phenomenological prediction of neural activation by electrical stimulation [11–13] (figure 1(a)). The model uses scaled units, where (S) represents spikes, (T)—time, (I)—current, and (L)—length. The linear part of the model used a dimensionless transfer matrix, ${\mathbf{W}}$, to calculate a generator signal, $\vec g$ (I), at each pixel in the retina that results from a set of electrode currents, $\vec s$ (I),

$$\begin{equation}\vec g = {\mathbf{W}}\vec s.\end{equation} \tag{ 1 }$$

The vector, $\vec s$, has the same number of elements as there are electrodes in the implanted electrode array, ${N_{\text{e}}}$, and the vector, $\vec g,$ has the same number of pixels that we wish to model in the retina, ${N_{\text{r}}}$. (As an aside, if the same electrodes were used to record neural response to electrical stimulation so as to estimate the model [13] then ${N_{\text{r}}} = \,{N_{\text{e}}}$, but we make no such assumption here.) The columns of the matrix, ${\mathbf{W}}$, which is of size ${N_{\text{r}}} \times {N_{\text{e}}}$, quantify the weighting of the contribution of each electrode to the generator signal associated with the neurons in each pixel in $\vec g$. Each electrode contributes a spread of neural activity that results in the perception of phosphenes. For simplicity, this spread is modelled as Gaussian with uniform standard deviation, $\sigma$ (L). Other choices are possible, including estimating ${\mathbf{W}}$ by recording responses, this is an area for future work.

The spike rate at each pixel in the retina, $\vec r$ (ST⁻¹), is assumed to be the magnitude of the generator signal at that pixel (figure 1(a)); this is the nonlinear part of the model,

$$\begin{equation}\vec r = \left| {\vec g} \right|.\end{equation} \tag{ 2 }$$

This proportionality assumption simplifies the following derivation while retaining the important qualitative properties of the experimentally measured two-sided nonlinearities. Specifically, these nonlinearities are observed to be symmetrical, with a minimum of neural activity at a generator signal value of zero [13].

The basic function of a retinal implant is to activate electrodes to induce perception of a given image. The conventional approach to selecting electrode stimulation strengths, given a particular target neural activity pattern, ${\vec r^*}$, is to first downsample the target pattern. Each electrode is then stimulated in proportion to the local intensity of the downsampled target neural activity. This approach is independent of the shape of the individual phosphenes. In this study, an enhanced version of this conventional approach is implemented by making the selection of each electrode's current amplitude dependent upon the shape of its associated neural activity profile. In this enhanced conventional approach, each electrode current is proportional to the projection of the target pattern of neural activity onto phosphene/ERF associated with that electrode,

$$\begin{equation}\vec s = k.{\vec r^{*^{\text{T}}}}{\mathbf{W}},\end{equation} \tag{ 3 }$$

where ${\vec r^*}$ is the given target pattern of neural activation and $k$ is a normalization constant. This process mimics the conventional process but improves upon it by using information about the neural activation pattern induced by each electrode. This acts as a conservative benchmark against which we can compare the proposed methods.

An improved approach is to use the inverse of the above LNL model in order to calculate the electrode activation pattern, $\vec s$, required to induce a given target pattern of neural activation, ${\vec r^*}$. The above LNL model (equations (1) and (2)) can be combined to give

$$\begin{equation}{\vec r^*} = \left| {{\mathbf{W}}\vec s} \right|,\end{equation} \tag{ 4 }$$

which is equivalent to

$$\begin{equation}{{\mathbf{R}}^*}\vec \varphi = {\mathbf{W}}\vec s,\end{equation} \tag{ 5 }$$

where

$$\begin{equation}\vec \varphi = {\text{sgn}}\left( {{\mathbf{W}}\vec s} \right),\end{equation} \tag{ 6 }$$

and ${{\textbf{R}}^{\boldsymbol{*}}}$ is a diagonal matrix whose diagonal elements are the vector, ${\vec r^*}$, and $\vec \varphi$ is here referred to as the partitioning vector, whose entries are ±1. We then multiply both sides of equation (5) by the pseudoinverse of ${\mathbf{W}}$ to obtain the desired set of electrode currents,

$$\begin{equation}\vec s = {{\mathbf{W}}^ + }{{\mathbf{R}}^*}\vec \varphi ,\end{equation} \tag{ 7 }$$

where the '+' superscript symbol represents the pseudoinverse of the transfer matrix, ${\mathbf{W}}$, obtained via singular value decomposition (SVD) implemented by function 'svd' of MATLAB (version 2019a, MathWorks, MA, USA).

The sign of the generator signal for each pixel, given by the contents of the partition vector, $\vec \varphi \,$, are free to be chosen. The unpartitioned neural activity strategy simply sets every element of the partition vector to +1 (figure 1(c)). The problem investigated in this paper is to find the optimum values for each of the elements of $\vec \varphi$ in equation (7) (figure 1(d)). There are ${2^{{N_{\text{r}}} - 1}}$ possible (non-degenerate) combinations of elements of $\vec \varphi$. The optimum combinations are those that give the most visually useful representation of the target neural activation, ${\vec r^*}$. We generally assume that this is achieved by minimizing the mean squared error $\varepsilon \,$ (S² T⁻²) between the model predicted pattern and the actual target pattern of retinal activity,

$$\begin{equation}\varepsilon = {\left\| {{{\vec r}^*} - \vec r} \right\|^2}.\end{equation} \tag{ 8 }$$

As has been shown previously [11], this leads to the objective function,

$$\begin{equation}{{\Theta }} = { }\mathop {\min }\limits_{{{\vec \varphi }_i}\in \left\{ { - 1,1} \right\}} { }{\vec \varphi ^{\text{T}}}\left[ {{{\mathbf{R}}^*}\left( {{\mathbf{I}} - {\mathbf{P}}{{\mathbf{P}}^{\text{T}}}} \right){{\mathbf{R}}^*}} \right]{ }\vec \varphi ,\end{equation} \tag{ 9 }$$

where ${\mathbf{I}}$ is the identity matrix and ${\mathbf{P}}$ are the left eigenvectors, from the SVD of ${\mathbf{W}}$. This fits the form of a binary optimization problem, $\mathop {{\text{min}}}\limits_{{{\vec v}_i}\in \left\{ { - 1,1} \right\}} {\vec v^{\text{T}}}{\mathbf{M}}\vec v$, which is NP-hard for a general matrix, ${\mathbf{M}}$, which is of size ${N_{\text{r}}} \times {N_{\text{r}}}$.

A potential (i.e. locally optimal) solution to the binary optimization problem in equation (9) is provided by a Hopfield network of ${N_{\text{r}}}$ nodes using the elements of the symmetric matrix ${\mathbf{M}} = {{\mathbf{R}}^*}\left( {{\mathbf{I}} - {\mathbf{P}}{{\mathbf{P}}^{\text{T}}}} \right){{\mathbf{R}}^*}$ as the strengths of connections between the nodes and $\vec \varphi$ as the states of the nodes [25]. Since the diagonal elements of the matrix, ${\mathbf{M}}$, are the coefficients to the terms, $\varphi _i^2 = 1,{ }$these do not contribute to the minimization of $\varepsilon$ and can be set to 0. This avoids self-connections in the Hopfield network, which does not function to minimize the objective function with self-connected nodes.

A random set of initial values for $\vec \varphi$ are used and then updated according to the iterated Hopfield network rule, ${\vec \varphi _{i + 1}}\mathop \leftarrow \limits^{} {\text{sgn}}\left( {{\mathbf{M}}*{{\vec \varphi }_i}} \right)$, until the vector, ${\vec \varphi _i}$, is static and new iterations do not lead to any change. As is required for Hopfield optimization, the multiplication, ${\mathbf{M}}*{\vec \varphi _i}$, is not performed in the normal, parallel way, but instead sequentially, in which each row of ${\mathbf{M}}$ is multiplied by the vector, ${\vec \varphi _k}{ }$, and the appropriate value is then updated within the vector, ${\vec \varphi _i}$, before the next row is multiplied. This process is referred to as an 'iteration' of the Hopfield network. Each iteration of this rule leads to a reduction in the error, $\varepsilon$, and is repeated until it reaches a (local) minimum. In the present study, the entire process of repeated iterations of the Hopfield network is referred to as a 'cycle' of the Hopfield network. The Hopfield network underlies the three approaches used in this paper. However, in practice, this single-cycle Hopfield network typically results in a detrimentally high error, $\varepsilon ,$ for the partitioning (see Results and figure 2(c)). This is due to being caught in local minima; consequently, it is not used on its own.

A single cycle of the Hopfield network frequently leads to local minima in the minimization. These local minima are often associated with distracting artefacts in the visual field (e.g. see figure 2(c)). One approximate solution to this is to simply perform single-cycles multiple times with new random conditions, with each new cycle beginning with different random initial conditions. The process completes when a chosen computational limit, ${C_{{\text{lim}}}}$, has been reached and the Hopfield cycle with the lowest $\varepsilon$ is chosen as the best result. This approach is referred to as the repeated Hopfield network strategy and has the practical advantage of being implementable in parallel because each cycle of the Hopfield network is not dependent on other cycles. In principle, parallelization provides a dramatic increase in speed when implemented using an appropriate architecture (e.g. a field programmable gate array, FPGA), though this is not done here.

The repeated Hopfield network approach with random initial conditions does not build on the results of previous cycles. Simulated annealing uses the outcome of previous cycles as a starting point for new cycles of Hopfield network iterations. Specifically, each of the cycles commences with an initial guess, $\vec \varphi$, that is associated with the lowest error solution of all the previous Hopfield cycles and combined with some randomization of the values of $\vec \varphi$. This randomization is implemented by reversing the sign of a fraction of the elements of the partition vector, $\vec \varphi$, with probability, ${F_{{\text{rev}}}}$. This process begins with maximum levels of randomization and then reduces to minimum levels of randomization, allowing the solution to 'relax' to its lowest level in the standard annealing fashion.

The probability, ${F_{{\text{rev}}}}$, is updated throughout the optimization according to the following annealing trajectory that interpolates between the maximum and minimum possible values of ${F_{{\text{rev}}}}$,

$$\begin{equation}{F_{{\text{rev}}}}\left( n \right) = 0.5 - \left( {0.5 - \frac{1}{{{N_{\text{r}}}}}} \right){(C/{C_{{\text{lim}}}})^x}.\end{equation} \tag{ 10 }$$

The maximum randomization is associated with a sign flip probability, ${F_{{\text{rev}}}} = 0.5$, because the error, $\varepsilon$, is the same with either $\vec \varphi$ or $- \vec \varphi$, and the minimum amount of randomization is a single element reversal, i.e. ${F_{{\text{rev}}}} = 1/{N_r}$. In equation (10), $C$ is the number of computations completed and $x$ determines the departure from linearity of the annealing trajectory. If $x = 1$, the trajectory is linear. If $x < 1$, the value of ${F_{{\text{rev}}}}$ initially decreases more quickly than a linear trajectory, leading to more cycles with less randomization. If $x > 1$, the value of ${F_{{\text{rev}}}}$ initially decreases more slowly, leading to more cycles with higher randomization. The optimum value of the parameter, $x$, was found to be approximately 2 and this is the value used in the investigation. The annealing cycles continue until the computation limit, ${C_{{\text{lim}}}}$, has been reached, and the cycle with the lowest $\,\varepsilon$ is chosen as the solution.

Instead of using a random initial condition for each Hopfield network cycle, we introduce an alternative approach that uses non-random initial conditions. We base the choice for the initial condition on the observation that, in practice, the results of using approaches 1 and 2 tend to partition the target activity into contiguous regions of high neural activity that are achieved with the same generator signal sign. This qualitative empirical observation can be described as two principles:

• Principle 1: two adjacent retinal locations of high neural activation that are not separated by lines of near-zero activation should be stimulated with generator signals of the same sign. In this way, the generator signal in the retina on the path between those two locations will not pass through zero; passing through zero would introduce undesired artifactual lines of near-zero neural activation (figure 2(c)).
• Principle 2: two adjacent retinal locations of neural activation separated by lines of low neural activation should be achieved with generator signals of opposite sign. This forces the generator signal on the path between these two locations to pass through zero, thus creating the desired region of low neural activation (figure 2(b)).

It is apparent that these two principles describe conventional image thresholding followed by image segmentation. These are implemented for the target pattern (figure 3(a)) by initially binarizing it into high and low neural activity regions by thresholding at 33% of the maximum brightness level in that target pattern (figure 3(b)). Note that this threshold parameter is later explored for its effect on the results. The high neural activity pixels are grouped according to their direct adjacency to other high neural activity pixels using the function 'bwconncomp' in MATLAB (figure 3(c)). Regions with a low number of pixels, fewer than either 16 pixels or 0.5% of the total number of pixels, are removed (figure 3(d)). Low neural activity pixels are initially set to be +1 in the partition vector and high neural activity pixels are initially set either +1 or −1 using an exhaustive search procedure described below. (Note an entirely equivalent approach would be to reverse the sign so that low neural activity pixels are set to be −1 and high neural activity pixels are initially set either −1 or +1).

Zoom In Zoom Out Reset image size

The advantage of the segmentation approach is that it is not necessary to search every possible combination of pixel sign, but instead test each possible combination of region sign. This reduces the exhaustive search space from ${2^{{N_{\text{r}}} - 1}}$ to ${2^{{N_{{\text{reg}}}} - 1}}$, where ${N_{{\text{reg}}}}$ is the number of regions in the target retinal pattern, ${\vec r^*}$, separated by connected lines of low neural activity. In general, this represents a dramatic reduction in the search space because there are typically fewer regions in the target activity pattern than elements in the target activity pattern vector (i.e. ${N_{{\text{reg}}}} \ll {N_{\text{r}}}$). In the example illustrated in figure 3, the search is reduced by a factor of ${\sim}10^{10}$. Each combination is used as the initial condition of a single-cycle Hopfield network.

The principle of the segmentation search is to decide which high-activity regions should be achieved with opposite partition polarities. The combinations of regions are tested in a particular order (figure 3(e)) to ensure that the earliest combinations tested are those most likely to have the largest influence on $\varepsilon$ and thus ensure these combinations are tested within the imposed computation limit. First, all high neural activity regions are set to partition vector values of −1 and the value of $\varepsilon$ calculated, then the largest region is flipped to be +1 and the value of $\varepsilon$ is recalculated. Then, the largest region is returned to −1 and the second largest segment is set to +1. This process continues until the second smallest region has been tested. There is no need to vary the signs in the partition vector associated with the smallest region as these degenerate combinations are redundant. Table 1 illustrates this process for a three-region pattern. If there are many combinations, it is possible that not all possibilities will be tested because, when the computation limit, ${C_{{\text{lim}}}}$, is reached, the process terminates. After termination, the partition vector associated with the lowest $\varepsilon$ is chosen as the solution.

Unless otherwise stated, the following model considers a 7 × 7 electrode array with a generator signal spread with a standard deviation of 1.5 times the distance between electrodes ($\sigma = 0.21$). This was value was chosen to ensure that the generator signals from each electrode overlapped. Lower and higher generator signal spreads of 1 and two times the distance between the electrodes ($\sigma = 0.14$, and $\sigma = 0.29$) were also tested.

Single neural activity pattern analysis is completed with a 41 × 41-pixel target neural activity pattern. Results across a set of images are completed with 1000 images from the CIFAR 100 database [26]. This database is used because it is low resolution, allowing for fast computation speed, and allows comparison of the partitioning algorithms across many images of different objects and environments.

Each partitioning algorithm is run for a set amount of central processing unit (CPU) time as calculated by the function 'cputime' of MATLAB. This is equivalent to 1.04 × 10¹⁰ CPU clock cycles. The maximum CPU time, ${C_{{\text{lim}}}},$ is set to 1.56 × 10¹¹ clock cycles, because this is found to deliver sufficient number of Hopfield network cycles to see the impact of the search process, while also being low enough to allow for testing of 1000 neural activity patterns in a feasible timeframe. This computational limit is implemented as a soft boundary; the current Hopfield cycle is allowed to continue until it reaches its own termination condition.

Given the dimensions of the matrix ${\mathbf{M}}$ and vector ${\vec \varphi _i}$ each iteration of the Hopfield network process requires $N_{\text{r}}^2$ multiplication operations. This makes the problem of computing each iteration $O\left( {{n^2}} \right)$. This applies to all three methods, the repeated Hopfield network, Hopfield network with simulated annealing, and Hopfield network with image segmentation. However, given that we impose a computational limit on the algorithm termination, increasing the number of pixels reduces the number of iterations possible within the time available for each algorithm. In general, this is expected to make segmentation more beneficial, given that it prioritizes salient solutions.

Three standard image pre-processing approaches were applied to the 1000 image CIFAR database to test their suitability when combined with image partitioning.

• (a)  
  Each pixel below 33% of the maximum pixel intensity was set to 0 intensity. This emphasizes dark regions of the image increasing perceptual prominence.
• (b)  
  Sobel edge detection was applied to the image using MATLAB's 'edge' function. This edge detected version was then superimposed on the original image. This increases the perceptual prominence of borders in the image.
• (c)  
  All pixels below 33% intensity of the maximum intensity were set to 0 intensity and all pixels above this were set to maximum intensity. After edge detection was applied, this technique leads to the appearance of connected lines of the kind that the partitioning approach can create in the neural activation pattern. These lines were superimposed on the original image. This increased the perceptual prominence of borders in the image in a way that is particularly suited to the partitioning approach to stimulation.

In approaches B and C, the edges were emphasized as regions of low intensity so as to be suited to the lines of low-intensity created by the partitioning approach.

In practice, there are many possible initial partitions for a given neural activity pattern, specifically ${2^{{N_{\text{r}}}}}$. In order to visualize the results of a Hopfield network search process, a 'toy' scenario with 3 × 3 target pixels and 2 × 2 electrodes is considered for illustrative purposes. The electrode array is modelled to have current spread wider than the electrode separation. A random target vector, ${\vec r^*}$, of length, ${N_{\text{r}}} = 9$, is used, giving a partition vector, $\vec \varphi$, of length 9. This random target image can be best achieved using a particular partition of the generator signal pattern into negative and positive regions. Given that ${\vec \varphi _i}\in \left\{ { - 1,1} \right\}$, there are ${2^9} = 512$ combinations of values of the vector. Each of these is labelled with a unique number from 1 to 512. A single attempt to find this optimum pattern using the Hopfield network is completed and each step in this search illustrated (figure 4(a)). The Hopfield network begins with a particular guess (labelled node 181), then proceeds iteratively to make small changes to its initial guess each of which reduce the error associated with the solution. The search proceeds until it cannot find any more improvements (node 224). This solution may or may not be the optimum solution. The Hopfield network based on equation (9) is then run for every vector combination as initial condition and the trajectory of each run are mapped (figures 4(b) and (c)).

Zoom In Zoom Out Reset image size

It can be seen that there are six groups, each made up of vertices that flow from nodes of high $\varepsilon$ to nodes of lower $\varepsilon$, finishing at a local minimum. Thus, the groups can be considered to be basins of attraction, each associated with one of six local minima, of which two are global minima. The six groups are made up of three sets of mirrored pairs. This mirroring effect arises due to $\vec \varphi$ and $- \vec \varphi$ representing identical degenerate solutions to the inversion problem with inverted partition values.

The plots in figure 4(b) show the Hopfield network results numerically, with the trajectories of $\varepsilon$ through each node shown as a function of the number of iterations of the Hopfield network for different initial conditions. The different plots in figure 4(b) correspond to the three distinct basins of attraction. Note that the first basin of attraction has the lowest $\varepsilon$ of the three but can also take the longest to converge depending on the initial condition.

Figure 4(c) shows detailed results associated with the set of iterations associated with an example single cycle of the Hopfield network. Although a single cycle of the Hopfield network does discover nodes of lower error compared to the initial guess (or sometimes equal error), it does not reliably discover the best solution from a random initial condition but can easily become trapped in a local minimum. In cases where an exhaustive search is not possible and the Hopfield network is applied just once, this will often lead to poor outcomes. For this reason, optimization with only a single cycle of a Hopfield network was rejected.

It is possible to perform a repeated but non-exhaustive Hopfield network search (section 2.6). The Hopfield network minimization is repeated multiple times, with each referred to as a cycle. This is tested with a target neural activity pattern of 41 × 41 pixels and a 7 × 7 electrode array. Each begins with a random initial condition. Cycles are undertaken until the CPU clock cycles limit, ${C_{{\text{lim}}}}$ = 1.56 × 10¹¹, is reached (section 2.9) and then the current Hopfield cycle is allowed to continue to its own termination, and then the lowest $\varepsilon$ outcome is picked as the solution.

This approach is applied using a neural activity target with three obvious regions (figure 5(a), top insert labelled 'Target'). The result of applying an unpartitioned LNL inverse model (figure 1(c)) can be seen to appear as a blurred version of the target (figure 5(a), bottom insert labelled 'unpartitioned'). In the time available, three cycles of the Hopfield network are completed. The results of each iteration during the three cycles are shown in the main plot, while their final neural activity patterns are shown in the inset at right of figure 5(a). The time gaps between the starts of the run and the starts of the first Hopfield network cycles represent the set-up times of the search process. The result with the lowest $\varepsilon$ is labelled '3'. The unusual regions in the resulting pattern are highly artifactual, which is a typical sign of a Hopfield network being trapped in detrimental local minima. These artefacts are similar to the Ising model phase transitions between magnetic or crystal domains as explained in the introduction and illustrated in figure 2(c). Avoiding these outcomes efficiently is one of the aims of this publication.

Zoom In Zoom Out Reset image size

The repeated Hopfield network (figure 5(a)) approach effectively throws away information from the results of previous cycles. An alternative approach, described in section 2.7, is to use the lowest $\varepsilon$ solution of previous cycles as the basis of a new initial guess for a new cycle of the Hopfield network (figure 5(b)). The initial guess is additionally perturbed by randomly flipping the sign of some values in the partition vector with the contribution of the randomization reducing during later cycles (equation (10)). This approach of reducing randomization under repeated trials is a form of simulated annealing. The same maximum number of CPU clock cycles, ${C_{{\text{lim}}}} = \,$ 1.56 × 10¹¹, was used. In this particular implementation with this target neural activity pattern, the simulated annealing approach gave $\varepsilon =$ 0.065, an improved result over the repeated Hopfield network approach ($\varepsilon =$ 0.068) within the time allowed (figure 5(b)). This can be confirmed by comparing the qualitative neural activity patterns resulting from the repeated Hopfield network (figure 5(a), '3') with the Hopfield network with simulated annealing (figure 5(b), '4'). Figure 5(b) also illustrates that the amount of improvement over successive annealing runs reduces markedly for later runs.

The Hopfield network with image segmentation approach is applied to the same three-region target pattern, as described in section 2.8. It can be seen in figure 5(c) that this results in a greatly reduced number of computational operations and, in this particular case, lower $\varepsilon = 0.026$. The solution, shown in the inset (figure 5(c), '4'), is also the obvious optimal outcome for this particular target.

It can be seen by examining the delays before the first iteration of the first cycle of the Hopfield network (figures 5(a)–(c)) that the additional image segmentation step did not significantly impact upon the results of the image segmentation approach. It can also be seen, by comparing the number of computations within each cycle of the Hopfield network, that each cycle of the image segmentation approach (figure 5(c)) is much faster than those associated with the repeated Hopfield network (figure 5(a)) and the Hopfield networks with simulated annealing (figure 5(b)). This is because these networks are provided with an initial guess driven by image segmentation that is already close to a local minimum of the Hopfield network.

The results shown in figure 5 deal with only a single simple target image. The same process is completed for 1000 images taken from the test set of the CIFAR database [26]. A 7 × 7 electrode array and target neural activity pattern of 32 × 32 pixels are chosen for investigation. The CPU clock cycles limit was kept at the same value, ${C_{{\text{lim}}}} =$ 1.56 × 10¹¹. The minimal $\varepsilon$ for each of the 1000 images is compared for the repeated Hopfield network and the Hopfield network with image segmentation (figure 6). The results are normalized to the $\varepsilon$ associated with the unpartitioned approach (figures 6(a) and (b), black line, largely obscured in figure 6(a)).

Zoom In Zoom Out Reset image size

The results across the image database are shown in figure 6. Figure 6(a) shows results for each algorithm ordered by $\varepsilon$ of the image segmentation approach. Since algorithm choice for retinal implants is not performed on an image-by-image basis, but rather across a representative sample of salient images, the results are also shown in figure 6(b) ordered by $\varepsilon \,$for each algorithm independently. This allows for an algorithm-by-algorithm comparison. When the results are shown in these ways, it becomes obvious that image segmentation provided the best overall results but was not the best in all cases.

Of the 1000 neural activity patterns, all partitioning approaches gave the same unpartitioned result in 682 cases. Of the 318 neural activity patterns that were improved through partitioning, 169 were most improved using the image segmentation approach, 86 using the Hopfield network with simulated annealing approach, and 63 using the repeated Hopfield network approach.

The comparison of image segmentation with simulated annealing, the next best solution, is shown by calculating the ratio between $\varepsilon$ for each target neural activity pattern resulting from simulated annealing vs. image segmentation. In figure 6(c), the images are ordered by the magnitude of the resulting ratio. A ratio greater than 1 indicates that, for a particular image, image segmentation provided a better outcome than the repeated Hopfield network. It can be seen that for those images that have lower $\varepsilon$ for image segmentation, the benefit is often substantial with one image with a ratio greater than 2, while for the smaller number of images that had a lower $\varepsilon$ for simulated annealing, the benefit was more limited with $\varepsilon$ ratio up to 1.36.

The data presented in figure 6 provide quantitative evidence that, based on mean squared error, $\varepsilon$, image segmentation is a superior choice of algorithm. However, given that ultimately it is image perception that is important for retinal implant users rather than a numerical measure of pixel-by-pixel spike rate differences, it is relevant to look at qualitative results. Figure 7(a) shows a selection of three favourable examples of image segmentation with low values of $\varepsilon$ relative to the unpartitioned approach. It can be seen that the particular target neural activity patterns for which image segmentation is particularly relevant are those target images with narrow connected lines of low intensity. Median outcomes and poor outcomes for the image segmentation approach are also shown (figures 7(b) and (c)).

Zoom In Zoom Out Reset image size

The results above relating to the image segmentation approach (figures 5–7) are for a particular set of parameters, namely: a threshold in image segmentation of 33%, electrode number of 49, and generator signal spread of $\sigma = 0.21$ (where the length of the electrode array is set to be 1). The value of these parameters may conceivably influence the results, and so the effect of varying them is examined by comparing the improvements in outcomes in the relative $\varepsilon$ across the 1000 images. Figure 8(a) shows that there is an optimal range for the choice of target image threshold level between 25% and 40% of the maximum brightness level in the image. These selections produced the greatest number of images that were improved by the partitioned approach, while the choice of 10% or 50% reduced the numbers of improved images. Increasing the number of electrodes improves the outcomes for both the partitioned and unpartitioned strategies but reduces the benefit of partitioning relative to the unpartitioned strategy (figure 8(b)). This is because the low-activity lines induced by partitioning are more influential in improving low-resolution images. The extent of phosphene overlap, as defined by the spread of generator signal, does not influence $\varepsilon$ relative to the unpartitioned solution under the same conditions (figure 8(c)).

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Emphasizing features in images provides more opportunity for implant users to benefit from the low information provided by visual prostheses. Pre-processing is a standard approach in visual prostheses. By applying a set of standard image pre-processing strategies, the partitioning approach was tested for its suitability (figure 9).

In the context of pre-processed images, the value of the partitioning approach to stimulation is dramatically increased (figure 9(d)). This outcome appears to be due to the fundamental property of the partitioning approach in representing isolated low-intensity regions of a target image (figures 9(a)–(c)).

Using pre-processing approach C, different computational limits were explored to discover the influence on the selection of the partitioning algorithm. It was found that image segmentation was the most effective strategy choice with lower computation limits (figure 10).

Zoom In Zoom Out Reset image size