Performance of neural networks for localizing moving objects with an artificial lateral line

The data sets used in this research resemble the information perceived by the artificial lateral line organ and consist of simulated hydrodynamic data. For the construction of the data sets the fluid velocities caused by a source object (sphere) moving in a 2D plane in a 3D volume through water are calculated. Figure 1 shows this scheme.

Zoom In Zoom Out Reset image size

The sensor array is located along the trajectory running from coordinate ($-D$, 0) to (D, 0). The sensors are equally spaced along this line. The first sensor is located at coordinate ($-D$, 0) and the last sensor at (D, 0).

An experimental justification for using potential flow in a similar set up as assumed in the present study has been obtained in studies on fish. Both the shape of the wavelets and specifically the distance coding in its spatial characteristics predicted by potential flow have been observed in the biological lateral line responses of fish (Ćurčić-Blake and van Netten 2006). It was shown that the boundary layer of the stationary fish was not affecting the flow field to a high extent, which can also be expected using artificial lateral line sensors. This observation is supported by a theoretical study by Goulet et al (2008) and by a computational fluid dynamics study (Rapo et al 2009).

In Franosch et al (2005) a similar method to find the fluid velocity distribution for a moving source object (sphere) with a constant speed is considered. This method too was used for the case of inviscous potential flow. The velocity field distribution described in that work can be shown to be equal with the distribution in equation (2.1). This implies that the present work can equally well be applied to a vibrating source at different locations as well as moving sources.

The fluid velocity component parallel to the array at position s on the sensor array, $v(s)$, is given by

$$\begin{eqnarray}&&v(s)=C\left({{\psi}_{o}}\sin \phi -{{\psi}_{e}}\cos \phi \right),\end{eqnarray} \tag{ 2.1 }$$

with

$$\begin{eqnarray}&&C=\frac{W{{a}^{3}}}{2{{y}^{3}}},\end{eqnarray} \tag{ 2.2 }$$

where W is the velocity of the source object and a is the radius of the sphere. The angle of the source with respect to the sensor array in radians (see figure 1) is ϕ, and the even wavelet ${{\psi}_{e}}$ and the odd wavelet ${{\psi}_{o}}$ are described respectively by

$$\begin{eqnarray}&&{{\psi}_{e}}\left(s,x,y\right)=\frac{1-2{{\left(\frac{s-x}{y}\right)}^{2}}}{{{\left[1+{{\left(\frac{s-x}{y}\right)}^{2}}\right]}^{\frac{5}{2}}}},\end{eqnarray} \tag{ 2.3 }$$

$$\begin{eqnarray}&&{{\psi}_{o}}\left(s,x,y\right)=\frac{-3\left(\frac{s-x}{y}\right)}{{{\left[1+{{\left(\frac{s-x}{y}\right)}^{2}}\right]}^{\frac{5}{2}}}}.\end{eqnarray} \tag{ 2.4 }$$

Here (x,y) denotes the instantaneous coordinate of the moving object. The equations (2.3) and (2.4) show that the shape of the even and odd wavelets solely depends on the location of the source object with respect to the sensor location. The shape of these wavelets is shown in figure 2.

Zoom In Zoom Out Reset image size

In Ćurčić-Blake and van Netten (2006, p 1551) it was noted that the spatial variations along the x direction, as described by the even and odd wavelet functions, scale linearly with the distance of the source y. It was also shown that the maximum amplitude of the even wavelet is reached at the point of the lateral line that is closest to the source and that the odd wavelet is zero at this position. This is the location on the lateral line that is equal to the x coordinate of the source object.

Given the implicit memory present in the ESN architecture, the hydrodynamic data for all architectures is presented in trajectories rather than discrete locations so that for predicting the current location, the internal representation of past detections can be used.

The source object starts at time $t=0$ at a random location (x₀,y₀) in the Cartesian system where x₀ is taken from a uniform distribution with range $\left[-D,D\right]$ and y₀ is taken from the uniform distribution with range $\left[0,D\right]$. The object remains located in a $2D\times D$ area to one side of the lateral line (see figure 1).

For the data sets it is assumed that the source object moves with a constant velocity of 0.1D per time step. The direction ${{\phi}_{t}}$ in radians in which the object moves at time step t is

$$\begin{eqnarray}&&{{\phi}_{t+1}}={{\phi}_{t}}+A.\end{eqnarray} \tag{ 2.5 }$$

Here, A, in radians, is taken from the uniform distribution with range $\left[-1,1\right]$. The change in angle per time step is therefore limited to about 60 degrees.

The next location at time $t+1$ is selected by moving the sphere in the direction denoted by angle ${{\phi}_{t}}$ over a distance 0.1D.

When the source object would move outside of the area boundaries, the next movement direction is altered as shown in figure 3. If the object would still cross a boundary when it is moved in the resulting direction, the movement direction is similarly changed again.

Zoom In Zoom Out Reset image size

Each data point in the data sets represents the fluid velocities detected by sensors that are equally spaced along a one-dimensional sensor array. This concatenation of fluid velocities is used as input for the neural networks.

In fish neuromasts, two types of velocity detecting hair cells are present. The first type only detects water flow in one direction, while the second type only detects flow in the opposite direction (Flock and Wersäll 1962). The two sorts of information perceived by the two types might be transmitted separately to the central nervous system (Münz 1989, p 290). If this is so, the fish is able to discern between positive and negative fluid velocities.

As a preprocessing step, this biological parallellarization of information is imitated; each sensor reading is represented using two values. The first value only represents positive velocities. It is zero for negative velocities. For the second value, the same applies, but vice versa. Since lateral lines with 16 and 32 neuromasts are simulated, this results in using 32 and 64 inputs. We used this input doubling because of the observed enhanced performance in source localization.

To test the noise robustness of the networks, different levels of noise are added to the fluid velocities computed for test sets.

The noise is taken from a normal distribution with a mean of zero. Different noise levels are applied via variation of the standard deviation of the noise. The noise level is defined in this paper in terms of the signal noise ratio (SNR), which is given as:

$$\begin{eqnarray}&&\text{SNR}=10{{\log}_{10}}\frac{{{A}^{2}}}{{{\sigma}^{2}}}~\text{dB}\end{eqnarray} \tag{ 2.6 }$$

In this equation, A is the maximal magnitude of deviation from zero of an input excitation pattern after scaling (section 2.1.5) has been applied. ${{\sigma}^{2}}$ is the variance of the normal distribution out of which the noise values are taken.

The distance from the source to the artificial lateral line y affects the amplitude of the signal in the excitation pattern, see equation (2.2). To investigate the effects of noise independent of y, noise is added to all sensor signal inputs to obtain the required signal to noise ratio and then scaled, according to equation (2.7).

Adding noise affects the range of the values in an input excitation pattern. In real-life applications in which noise quantities are unknown, all excitation patterns would be scaled to the same value ranges, regardless of the quantity of noise that is present. As a consequence, the contribution and level of the original signal to the normalized excitation pattern differs for each noise level.

In order to make the localization process robust to amplitude variance and to let it focus on the spatial characteristics of the excitation pattern, it is first scaled before the input is presented so that the largest magnitude in the excitation pattern becomes 1. For this, all excitation pattern values are changed according to equation (2.7).

$$\begin{eqnarray}&&\text{new}\;q[n]=\frac{q[n]}{\max \mid q\mid}\end{eqnarray} \tag{ 2.7 }$$

The discrete signal q is the excitation pattern at each individual sensor calculated with equation (2.1) and index n is the sensor number. This scaling causes the values of q to always be within the range [$-1$,1]. This causes information about the y coordinate to be present only in the spatial scaling of the normalized excitation pattern, while information about the x coordinate is present in the location of the pattern along the sensor array.

The MLP used in this paper is a fully connected network (see figure 4). The input layer has a variable size (32 or 64 nodes) and the output layer has two nodes to represent the location x, y of the source. Equation (2.8) shows the way in which the node activation values are computed.

$$\begin{eqnarray}&&x_{i}^{m+1}=f\left(\underset{j=1}{\overset{{{N}^{m}}}{\sum}}\,\left(w_{ij}^{m}x_{j}^{m}\right)+b_{i}^{m+1}\right)\end{eqnarray} \tag{ 2.8 }$$

Zoom In Zoom Out Reset image size

Here, $x_{i}^{m}$ denotes the activation of the ith node of the mth layer, where counting starts at the input layer. $w_{ij}^{m}$ denotes the weight connecting the jth node in the mth layer to the ith node in layer $m+1$. $b_{i}^{m}$ denotes the bias of the ith node in the mth layer. N^m is the number of nodes in the mth layer. The activation function $f(x)=tanh(x)$ is used to calculate the activation of the hidden nodes. For the output nodes, the identity function is used as activation function (see figure 4). During training, when computing the activation value of a node in the hidden layer, some noise sampled from a uniform distrubution with range $\left[-{{10}^{-3}},{{10}^{-3}}\right]$ is added, which amounts to 46 dB.

The network was trained with the incremental learning version of the backpropagation algorithm (Rumelhart et al 1985). The weights are initialized according to the normalized initialization procedure described in Goulet and Bengio (2010). The bias weights are initialized to zero.

When overfitting occurs, a network is trained to specifics of a training set instead of general features. This causes the network to perform better on the specific training set, but worse in general cases.

To determine whether overfitting occurs, during training, the Mean Squared Error (MSE) on the training and a validation set was monitored. This is the usual error that is minimized during training neural networks. The MSE for a network on a specific data set is

$$\begin{eqnarray}&&\text{MSE}=\frac{1}{{{D}^{2}}M}\underset{n=1}{\overset{M}{\sum}}\,\frac{1}{N}\underset{i=1}{\overset{N}{\sum}}\,{{\left({{t}_{i}}(n)-{{o}_{i}}(n)\right)}^{2}}.\end{eqnarray} \tag{ 2.9 }$$

In this equation D is half the length of the array, M is the number of samples in the data set, $N=2$ is the dimensionality of an output sample, t is the target output and o is the network output.

An additional measure for reporting is the Mean Euclidian Distance (MED):

$$\begin{eqnarray}&&\text{MED}=\frac{1}{DM}\underset{n=1}{\overset{M}{\sum}}\,\sqrt{\underset{i=1}{\overset{N}{\sum}}\,{{\left({{t}_{i}}(n)-{{o}_{i}}(n)\right)}^{2}}}.\end{eqnarray} \tag{ 2.10 }$$

The MED provides a more intuitive relative distance measure of error as it is defined as a fraction of the depth of field D.

Like an MLP, an ELM is a feedforward neural network (see figure 4). It differs from the MLP in that only the weights from the hidden to output layer are trained. Weights from the input to the hidden layer are initialized randomly and not altered during training (Huang et al 2006).

An ELM with a perfect performance on the training set would have a weight matrix W, that describes the weights from the hidden layer towards the output layer, that satisfies:

$$\begin{eqnarray}&&WH=T.\end{eqnarray} \tag{ 2.11 }$$

Here, the teacher output T is the matrix that is built up from a consecutive series of columns, in which each column consists of the correct output for all output nodes to the training pattern. Hidden layer output matrix H is built up from the activation values of the units in the hidden layer at all the time instances during the training that are the result of presenting the training data to the ELM.

Training of the ELM consists of finding a least squares solution for W from this linear system, given H and T. This is computed using:

$$\begin{eqnarray}&&W=T{{H}^{\dagger}},\end{eqnarray} \tag{ 2.12 }$$

where ${{H}^{\dagger}}$ is the MoorePenrose generalized inverse of matrix H.

Because determining the matrix W is very fast, ELMs can be efficiently trained in little time compared to MLPs (Huang et al 2006).

While experimentally determining parameter settings, ELMs with internal recurrent connections were also tested. The resulting ESN (Jaeger 2002) architecture effectively introduces memory into the neural network, which could help in predicting the current position of a source based on past detections.

Figure 5 shows the schematic representation of an ESN that consists of an input layer, a dynamic resevoir (DR) and an output layer. Each input node connects to each node in the DR and each node in the DR connects to the output layer through weights. The hidden nodes in the DR are sparsely connected with each other and themselves. This causes the DR to contain a high dimensional representation of the input.

Zoom In Zoom Out Reset image size

The weight matrix representing the internal weights of the DR must be constructed so that the spectral radius, which is the largest absolute eigenvalue of the weight matrix, is smaller than 1. As a result input is echoed and dies out over time, which is called the Echo State Property. This gives the ESN short term memory.

Due to the short term memory, the output of the first few steps is often inaccurate. To wash out the effects of the initial network state, the network output of the first 50 samples is discarded during training and testing.

For optimizing the MLPs, the varied parameters were the learning rate, number of hidden layers and layer sizes. All MLPs in the initial parameter study were trained for 300 epochs. The best performing networks for both 16 and 32 sensors were then trained with a learning rate of 0.01. They had two hidden layers of 80 and 40 nodes respectively. For these optimal parameter settings, new networks were trained for 3000 epochs.

The best performing and thus chosen hidden layer sizes were 5000 and 7000 nodes for ELMs with 16 and 32 sensors respectively. The performance of the ESN architecture increased when the recurrent connection weights were chosen smaller. Since the performance was best in a neural network architectures without recurrent connections, the best performing ESNs are effectively ELMs, and only ELMs are discussed for high signal to noise input.

In order to ascertain and compare the noise robustness of the ESN architecture, ESNs with DR sizes of 5000 for 16 inputs and 7000 nodes for 32 inputs were also tested with a spectral radius of $0.1$.