Flying over uneven moving terrain based on optic-flow cues without any need for reference frames or accelerometers

2.1.1. Airframe

2.1.2. Electronics

2.1.3. Test-rig

Let us take a local motion sensor facing an object located at distance D(Φ) in the direction d(Φ) characterized by the elevation angle Φ (Φ is the angle between the direction of the local motion sensor d(Φ) and the perpendicular with the eye's equator (see figure 6). The movements of the local motion sensor can always be decomposed into a translation vector $\vec{T}$ and a rotation vector $\vec{R}$. The motion field or OF in that direction $\overrightarrow{\omega (\Phi )}$ was defined by Koenderink and Doorn (1987) as follows:

$$\begin{eqnarray}\begin{array}{rcl} \overrightarrow{\omega (\Phi )} & = & -\frac{(\vec{T}-(\vec{T}.d(\Phi ))d(\Phi ))}{D(\Phi )} \\ {} & {} & -\vec{R}\times d(\Phi ). \\ \end{array}\end{eqnarray} \tag{ 1 }$$

If we assume that the BeeRotor robot is moving only in translation $(\vec{R}=0)$, we can obtain from (1) :

$$\begin{eqnarray}&&\overrightarrow{\omega (\Phi )}=-\frac{(\vec{T}-(\vec{T}.d(\Phi ))d(\Phi ))}{D(\Phi )}.\end{eqnarray} \tag{ 2 }$$

By expressing the translation vector $\vec{T}$ in the orthogonal basis formed by d(Φ) (the viewing direction) and $\frac{\overrightarrow{\omega (\Phi )}}{\parallel \overrightarrow{\omega (\Phi )}\parallel }$ (the normalized OF vector), Whiteside and Samuel (1970) established that equation (2) can be expressed in the form:

$$\begin{eqnarray}\begin{array}{rcl} \omega (\Phi ) & = & \frac{\parallel \vec{V}\parallel }{D(\Phi )}{\rm sin} (90{}^\circ +\Psi -\Phi ) \\ {} & = & \frac{\parallel \vec{V}\parallel }{D(\Phi )}{\rm cos} (\Phi -\Psi ), \\ \end{array}\end{eqnarray} \tag{ 3 }$$

where Ψ is the angle between the eye's equator and the direction of the speed vector $\vec{V}$ (see figure 2(c)).

Zoom In Zoom Out Reset image size

The common feedback control loops in BeeRotor I and II can be described as follows:

First OF feedback: the feedback control loop that modifies the altitude was inspired by ethological findings obtained on mosquitoes, desert locusts, moths and honeybees, which showed that flying insects use OF cues (Kennedy 1951, Kuenen and Baker 1982, Baird et al 2006, Portelli et al 2010a) to adjust their height by maintaining a preferred retinal velocity.

As we can see from equation (3), the OF measured by a local motion sensor pointing towards the ground depends on the distance between the aerial robot and the ground and on the robot's ground speed. In particular, by fusing the outputs from the two ventral OF sensors, we can deduce from equation (3) that:

$$\begin{eqnarray}\begin{array}{rcl} {{\omega }_{{\rm MaxOF}}} & = & \omega (\Phi )+\omega (-\Phi ) \\ {} & = & \frac{2\parallel \vec{V}\parallel .{\rm cos} (\Phi ).{\rm cos} (\Psi )}{D(\Phi )}, \\ \end{array}\end{eqnarray} \tag{ 4 }$$

where ω(Φ) is the OF in the forward FOV and ω(−Φ) is the OF in the backward FOV.

Geometrically, it can be proven that the distance D(Φ) depends on the slope of the surface α, the distance between the robot and the ground in the direction normal to the slope of the ground D_Down/slope, the angle θ_Eye/slope between the eye's equator and the slope of the nearest surface and the orientation of the local motion sensor Φ

$$\begin{eqnarray}&&D(\Phi )=\frac{{{D}_{{\rm Down}/{\rm slope}}}}{{\rm cos} (\Phi +{{\theta }_{{\rm Eye}/{\rm slope}}})}.\end{eqnarray} \tag{ 5 }$$

In this case, equation (4) can be written:

$$\begin{eqnarray}\begin{array}{rcl} {{\omega }_{{\rm MaxOF}}} & = & 2.{\rm cos} (\Phi ).{\rm cos} (\Phi +{{\theta }_{{\rm Eye}/{\rm slope}}}) \\ {} & {} & \times \frac{\parallel \vec{V}\parallel .{\rm cos} (\Psi )}{{{D}_{{\rm Down}/{\rm slope}}}}. \\ \end{array}\end{eqnarray} \tag{ 6 }$$

We can then see that the sum of the OFs can be used to adjust the ratio between the speed V and the distance between the robot and the ground in the direction normal to the slope of the ground D_Down/slope. We obtain exactly the same result if the robot is following the ceiling, and we then compute the sum of the two dorsal OFs.

To adjust the robot's altitude, it is therefore possible to regulate the maximum OF generated by the nearest surface and thus maintain a safe distance from this surface. During our experiments, we actually used the information from the forward OF sensors to act on the altitude of the rotorcraft as it has proven to improve the ability of the autopilot to avoid forthcoming obstacles.

Second OF feedback: the feedback loop controlling the speed was inspired by experimental findings on honeybees travelling along a tapered corridor, where they were found to adjust their forward speed in order to regulate 'the sum of the two opposite OFs in the horizontal or vertical planes' (Srinivasan et al 1996, Portelli et al 2011).

In the present case, we can write:

$$\begin{eqnarray}\begin{array}{rcl} {{\omega }_{{\rm SumOF}}} & = & \omega (\Phi )+\omega (-\Phi )+\omega (180{}^\circ +\Phi ) \\ {} & {} & +\omega (180{}^\circ -\Phi ) \\ {} & = & 2.{\rm cos} (\Phi ).{\rm cos} (\Psi ).\parallel \vec{V}\parallel \\ {} & {} & \times \left[ \frac{{\rm cos} (\Phi +{{\theta }_{{\rm Eye}/{\rm slopeDown}}})}{{{D}_{{\rm Down}/{\rm slope}}}} \right. \\ {} & {} & +\left. \frac{{\rm cos} (\Phi +{{\theta }_{{\rm Eye}/{\rm slopeUp}}})}{{{D}_{Up/slope}}}) \right], \\ \end{array}\end{eqnarray} \tag{ 7 }$$

where D_Up/slope is the distance between the robot and the ceiling in the direction normal to the slope of the ceiling, and D_Down/slope is the distance between the robot and the ground in the direction normal to the slope of the ground.

The BeeRotor I robot (see figure 2(a)) was equipped with the full cylindrical CurvACE visual sensor presented in figure 2(b), which is composed of two CurvACEs mounted back to back, each of which has a FOV of 180° × 60° and is composed of three planar layers: a microlens array, a neuromorphic photodetector array, and a flexible printed circuit board that are stacked, cut, and curved to produce a mechanically flexible imager (Floreano et al 2013). With its 360°FOV, the rotorcraft can assess the angular speed in several RoIs with great sampling accuracy and cope with any increase in the OF due to the presence of obstacles on its path. It can therefore autonomously adjust its altitude and forward speed so as to achieve a collision-free trajectory, based solely on the measurements performed in the frame of reference associated with the robot's body $\left( B,{{x}_{b}},{{y}_{b}},{{z}_{b}} \right)$.

Figure 2(c) shows the BeeRotor I robot flying in the environment described in section 2.1.3. Due to the robot's motion, the contrasted ceiling and floor generate an OF that is measured by the CurvACE sensor. Among all the pixels in the full cylindrical CurvACE, the four RoIs shown in blue in figure 2(c) composed of 6 × 3 pixels and covering a FOV of 24° × 12° were selected in order to compute the angular speed between adjacent pixels, using 15 two-pixel local motion sensors (LMSs). In each RoI, we then determined the median value of the OF, which was denoted ω_median^DF, ω_median^DB, ω_median^UF and ω_median^UB. As we have shown in a previous study (Roubieu et al 2012a), by taking the median value of the output of several LMSs, we stronlgy increase the accuracy of the OF measurements and therefore of the visuomotor control loops as it strongly decreases the noise and the unreliable measurements which are unavoidable in natural environments. We have chosen to take four RoIs as it appears that it was the minimal number in order to implement the control loops presented in section 3.2.

As we saw in the section 2.3, the OF measurements delivered by the full cylindrical CurvACE sensor make it possible to:

• adjust the aerial robot's altitude based on the OF generated by the nearest horizontal surface, and
• adjust its forward speed based on the sum of the ventral and dorsal OFs.

The BeeRotor I's autopilot (see figure 3) is therefore composed of two intertwined OF feedback loops that modify: (i) the mean speed of the propellers Ω_Rotors (common-mode) and (ii) the differential thrust of the propellers ΔΩ_Rotors (differential-mode).

Zoom In Zoom Out Reset image size

In order to robustly avoid any forthcoming ground obstacles, the feedback control loop shown in green (see figure 3) adjusts the mean rotor speed on the basis of the maximum forward OF generated by the ground and the ceiling:

$$\begin{eqnarray}&&\omega _{{\rm MaxOF}}^{{\rm meas}}=\frac{1}{{{{\rm cos} }^{2}}(\Phi )}.{\rm max} \left( \omega _{{\rm median}}^{{\rm DF}},\;\omega _{{\rm median}}^{{\rm UF}} \right)\end{eqnarray} \tag{ 8 }$$

and this value is compared with the maximum OF setpoint ω_setMaxOF in order to set the rotors' mean speed Ω_Rotors, which will eventually determine $\frac{{{D}_{{\rm Down}/{\rm slope}}},{{D}_{{\rm up}/{\rm slope}}}}{{\rm cos} (\Phi +{{\theta }_{{\rm Eye}/{\rm slope}}})}$ (see equation (6)), where D_Down/slope is the distance between the aerial robot and the ground in the direction normal to the slope of the ground and D_Up/slope is the distance between the aerial robot and the ceiling in the direction normal to the slope of the ceiling. When the OF generated by the nearest horizontal surface is greater than the maximum OF setpoint ω_setMaxOF, depending on which surface is detected, the altitude controller will either increase or decrease the rotorʼs speed command in order to drive the robot away from this surface.

The median value measured by each VMS was used to compute the ventral and dorsal OFs, which are denoted ω_Vtrl and ω_Drsl:

$$\begin{eqnarray}&&\omega _{{\rm Vtrl}}^{{\rm meas}}=\frac{1}{2.{{{\rm cos} }^{2}}(\Phi )}\cdot (\omega _{{\rm median}}^{\Phi }+\omega _{{\rm median}}^{-\Phi }),\end{eqnarray} \tag{ 9 }$$

$$\begin{eqnarray}&&\omega _{{\rm Drsl}}^{{\rm meas}}=\frac{1}{2.{{{\rm cos} }^{2}}(\Phi )}\cdot (\omega _{{\rm median}}^{{{180}^{{\rm o}}}+\Phi }+\omega _{{\rm median}}^{{{180}^{{\rm o}}}-\Phi }),\end{eqnarray} \tag{ 10 }$$

where Φ = 23° is the angle corresponding to the median direction of each RoI.

The second OF feedback control loop determining the pitch and eventually the forward speed (shown in blue) regulates the following OF sum:

$$\begin{eqnarray}&&\omega _{{\rm SumOF}}^{{\rm meas}}=\omega _{{\rm Vtrl}}^{{\rm meas}}+\omega _{{\rm Drsl}}^{{\rm meas}}.\end{eqnarray} \tag{ 11 }$$

The ω_SumOF^meas is compared with the sum OF setpoint ω_setSumOF in order to adjust the robotʼs airspeed.

Two nested feedback loops regulate the pitch rate of the aerial robot $\dot{\theta }$ via a proportional integral derivative (PID) controller based on the rate gyro measurements and the airspeed via a PD controller based on the measurements delivered by the airspeed sensor. The second OF feedback control loop then modifies the differential thrust of the propellers of the aerial robot ΔΩ_Rotors, which eventually determines the speed $\parallel \vec{V}\parallel .{\rm cos} (\Psi )$ (see equation (6)) via the forward dynamics. When the sum ω_Vtrl^meas + ω_Drsl^meas is greater than the forward speed OF setpoint ω_setSumOF, the forward controller will decrease the airspeed setpoint, which is then compared with the current airspeed measurement. This results in a decrease in the setpoint of the pitch rate feedback loop and the differential thrust of the propellers ΔΩ_Rotors. In our experiments, in the absence of wind, the measurements of the airspeed sensor is equal to the groundspeed and provides therefore the robot with metric information in the inertial frame of reference. However, we can predict that, in the case of tailwind, the measured airspeed would be greater than the groundspeed which would result in a transient slowing down of the robot that would be quickly compensated by the 2nd OF feedback loop adjusting the forward speed of the rotorcraft. In the case of headwind, the robot would first transiently accelerates (inner loop) before reaching back the speed required on steady state thanks to the same 2nd OF feedback loop. The autopilot therefore does not require any metric information about its speed, altitude or attitude in the inertial frame of reference.

Thanks to the coupling of these feedback loops using respectively the maximum OF generated by the closest surface and the sum of the ventral and dorsal OFs, the forward speed of the robot will be automatically proportionnal to the size of the tunnel (see equation (15) in Serres et al 2008).

It is worth noting that this autopilot, which works mainly on the basis of OF measurements, enables an aerial robot to fly autonomously without having to estimate its attitude (yaw, pitch and roll), altitude or groundspeed.

The controllers implemented onboard BeeRotor I are described in detail in appendix D.

As described in section 2.1.3, the floor of the experimental tunnel could be either rotated or moved up and down by means of two actuators in order to test the BeeRotor robot's performances in a highly variable environment. Figures 4(a)–(d) show the BeeRotor I's altitude when flying over a stationary floor, a floor moving at 30 cm s⁻¹ in the same direction as the robot, a floor moving at 30 cm s⁻¹ in the opposite direction and a floor oscillating up and down (see extension 1). Each curve corresponds to the altitude of the aerial robot during a 40 m long trajectory with ω_setMaxOF = 200° s⁻¹ and ω_setSumOF = 250° s⁻¹ except when the aerial robot was rotating in the same direction as the aerial robot (figure 4(b)) where ω_setMaxOF = 180° s⁻¹ and ω_setSumOF = 250° s⁻¹. In the latter case, the perceived OF decreased due to the motion of the ground and the altitude feedback loop decreased the mean thrust of the propellers accordingly, and we had to decrease the maximum OF setpoint value to prevent the robot from crashing onto the ground. On the other hand, when the floor was rotating in the opposite direction (figure 4(c)), the perceived OF increased and the aerial robot's altitude increased accordingly so that the OF was kept at the setpoint value similarly to what have been observed on honeybees with lateral moving walls Srinivasan et al (1991). Due to the changes in the perceived OF generated by the movement of the ground, the sum of the ventral and dorsal OFs also varied. This resulted in an increase in the robot's forward speed when the ground was moving in the same direction because the sum of the OFs decreased, and vice versa.

Zoom In Zoom Out Reset image size

Figure 4(d) gives the altitude of the aerial robot while it was flying over a floor oscillating up and down, reaching heights ranging between 0 and 64 cm with a frequency of 0.05 Hz. Despite this severe disturbance, the aerial robot consistently followed the ground and avoided colliding with the sloping ground. In addition, the robot's forward speed varied constantly due to the changes in the height of the tunnel. In particular, it can be seen that the robot flew at a higher speed when the altitude of the ground was minimal, which explains why the rotorcraft's altitude did not decrease as much as the ground's altitude because the altitude feedback loop adjusted the propellers' speed in order to maintain the ventral OF equal to the maximum OF setpoint value.

Thanks to its auto-adaptive pixels automatically adapting to background light level, the CurvACE sensor proved to be able to robustly compute the optic flow under several decades of illuminance Floreano et al (2013). Figure 5 shows the trajectories of the BeeRotor Iʼs autopilot while flying autonomously over more than two decades of background light level (photo-current of the photodiode ranging from 1.24 × 10⁻⁵ A to 3.61 × 10⁻³ A). To measure the effective illuminance of the scene scanned by the CurvACE sensors, a custom-made illuminance sensor based on an OSRAM BPX65 photodiode facing downward was connected to an analog amplifier circuit operating in the photovoltaic mode. The instantaneous photocurrent I_ph of this illuminance sensor was determined as follows: ${{I}_{{\rm ph}}}=({{{\rm e}}^{{\rm Vout}/0.125}}-1)\cdot {{I}_{{\rm dark}}}$, where the dark current I_dark is equal to 0.1 nA and V_out is the amplifierʼs output voltage.

Zoom In Zoom Out Reset image size

Despite the considerable differences existing from one experiment to another in terms of the background light level, as can be seen from the photographs on the right of the figure 5 and the last experiment in extension 1, the aerial robot was consistently able to automatically adjust its altitude and its forward speed based on the OF and to robustly avoid the obstacle of the ground.

The BeeRotor I aerial robot equipped with a full cylindrical CurvACE proved to be able to automatically adjust its altitude and its forward speed based on OF measurements. The performances of the robot were assessed under several decades of background light level in an environment oscillating up and down or moving along the direction of the robot flight or its opposite. The BeeRotor I's autopilot, which uses only measurements performed on the robot's own body frame of reference, manages to navigate robustly in a rugged environment, avoiding all the obstacles encountered.

However, as we can see from equations (6) and (7) and figure 3, this autopilot will adjust not only the speed $V$ and the distance between the aerial robot and the nearest surface ${{D}_{{\rm Down},{\rm Up}/{\rm slope}}}$, but also other parameters depending on the direction of the speed vector $\Psi$ and the angle between the eye and the slope of the nearest surface ${{\theta }_{{\rm Eye}/{\rm slope}}}$. This obviously affects the performances of BeeRotor Iʼs autopilot, which was designed to automatically reach a 'safe height and safe forward speed' based on OF measurements. These quantities that depend on the structure of the unknown environment can be regarded as additional nonlinear disturbances affecting the performances of the BeeRotor Iʼs autopilot. Indeed, when flying over very irregular terrain, the flying robot proved to be unable to avoid very steep obstacles. To improve the proficiency of the aerial robot without adding any sensors giving measurements defined in the inertial frame of reference, we decided to use the OF measurements to visually realign the eye so as to keep it parallel to the slope of the surface followed and thus obtain a more straightforward OF defined in a new frame of reference, i.e. the local slope of the surface followed. One would say that thanks to the panoramic FOV of the full cylindrical CurvACE sensor in the vertical plane, it would have been possible to virtually realign the eye by changing the chosen RoIs. But, due to an interommatidial angle greater than 4° (Floreano et al 2013), the artificial compound eye (CURVACE) spatial sampling was not high enough to process the OF in the relevant RoIs. This is the reason why a second quasi-panoramic eye composed of four VMSs oriented precisely thanks to a stepper motor towards specific RoIs was used. By coupling the stepper motor with a 1/120 reductor, we were able to adjust the orientation of the eye with a $0.02{}^\circ /{\rm steps}$ resolution.

4.1.1. Theoretical advantages of reorienting the eye

BeeRotor is assumed here to be flying at a constant horizontal speed V_x above a tilted terrain (with angle α) including a discontinuity (see figures 7(a) and (c)). This aerial robot is equipped with a mechanically decoupled eye composed of OF sensors looking down forward and down backward (each of them is oriented at an angle Φ relative to the normal of the plane defined by the eye's equator). Depending on the angle of the eye's equator which is zero in the case of a non reoriented (NR) eye and α in the case of a decoupled eye which is perfectly reoriented (R) in parallel with the surface, the ground discontinuity (which occurs at the position where the ground becomes flat) reflected in the OF measurements will be detected respectively at the positions:

$$\begin{eqnarray}\begin{array}{rcl} {{x}_{{\rm NR}}} & = & \Delta h\times {\rm tan} (\Phi )\ \ {\rm and} \\ {{x}_{{\rm R}}} & = & \Delta h\times {\rm tan} (\alpha +\Phi ), \\ \end{array}\end{eqnarray} \tag{ 12 }$$

where Δh is the difference between the altitude of the robot and the altitude of the ground after the obstacle.

Zoom In Zoom Out Reset image size

As the BeeRotor is flying at a constant speed V_x, this distance differential will be converted into a prediction time horizon:

$$\begin{eqnarray}\begin{array}{rcl} \vartriangle t & = & \frac{{{x}_{{\rm R}}}-{{x}_{{\rm NR}}}}{{{V}_{x}}}=\frac{\Delta h}{{{V}_{x}}} \\ {} & {} & \times ({\rm tan} (\alpha +\Phi )-{\rm tan} (\Phi )). \\ \end{array}\end{eqnarray} \tag{ 13 }$$

In a realistic case where the aerial robot is flying over a tilted terrain (α = 15°) at the horizontal speed V_x = 1 m s⁻¹, and the altitude Δh = 1 m over the ground discontinuity with a local motion sensor looking in the direction Φ = 23°, the reorientation of the eye will enable the aerial robot to detect the discontinuity 0.35 s earlier in comparison with a fixed eye. In particular, it is worth noting that the reorientation of the eye increases the prediction time horizon of the system by the ratio:

$$\begin{eqnarray}&&\frac{{{\tau }_{{\rm R}}}}{{{\tau }_{{\rm NR}}}}=\frac{\Delta h}{{{V}_{x}}}\times \frac{{\rm tan} (\alpha +\Phi )}{{\rm tan} (\Phi )}.\end{eqnarray} \tag{ 14 }$$

In the case of the above example, this ratio is equal to 1.84, which means that the discontinuity will be detected almost twice as fast when the eye is decoupled and oriented parallel to the ground. This example clearly shows that the eye-reorientation system helps to detect obstacles earlier than previously. When following an obstacle with a negative slope, the eye will of course be oriented backwards, thus delaying the detection of any discontinuities. However, we have observed experimentally that although the detection of obstacles is delayed when the aerial robot is following a descending ramp, the reorientation of the eye in parallel with the surface improves the flight performances because the OF will be underestimated in this case with a fixed eye, resulting in a sharp rebound or even a crash at the end of the descending ramp. Thanks to this reorientation, the measured OF does not depend on the slope angle of the closest surface (see equations (29) and (30)) which has proven, in any case (positive or negative slopes), to improve the performances of the autopilot to avoid obstacles.

4.1.2. OF-based method of estimation of the eye's reorientation angle

As we wanted to use mainly OF on our robot, we tested how its eye could be oriented relative to the surface using only a few LMSs. One simple solution which came to mind was to determine the direction in which the OF was maximal because the distance D would be minimal. In a computer simulations performed, the robot was found to be able to reorient its eye automatically, but it was over-sensitive to OF measurement errors. In particular, as our OF sensor showed a loss of accuracy at high speeds, the reorientation of the eye became noisier when the OF increased.

A particularly robust regression method involving a set of data containing outliers, the 'RANdom SAmple Consensus' algorithm (Fischler and Bolles 1981), might have been suitable here, but it was too costly in terms of computational ressources for the BeeRotor's microcontroller because of the large number of iterations necessary to obtain a good fit with the data.

Since the least squares regression method fits the microcontroller's computational abilities, it was implemented here onboard BeeRotor to determine the angle between the slope of the surface and the eye. The OF patterns measured by the ventral OF sensors when there is an angle α between the surface and the eye's orientation and when the eye's equator and the nearest surface are parallel are shown in figures 7(b) and (d), respectively.

In equation (3), it was established that the OF varies with the orientation of each local motion sensor Φ relative to the eye's frame of reference as follows:

$$\begin{eqnarray}&&\omega (\Phi )=\frac{\parallel \vec{V}\parallel }{D(\Phi )}.{\rm sin} (90-\Phi -\Psi ),\end{eqnarray} \tag{ 15 }$$

where Ψ is the angle between the eye's equator and the direction of the speed vector (see figure 6(b)). If we assume that the local motion sensor is facing downward, we can determine geometrically the distance D(Φ). Indeed, D(Φ) depends on the slope of the surface α, the distance between the aerial robot and the ground in the direction normal to the slope of the ground D_Down/slope, the angle θ_Eye/slope between the eye's equator and the slope of the ground and the orientation of the local motion sensor Φ, as follows:

$$\begin{eqnarray}&&D(\Phi )={{D}_{{\rm Down}/{\rm slope}}}\frac{{\rm cos} (\alpha )}{{\rm cos} (\Phi +{{\theta }_{{\rm Eye}/{\rm slope}}})}.\end{eqnarray} \tag{ 16 }$$

From equations (15) and (16), we can deduce that:

$$\begin{eqnarray}\begin{array}{rcl} \omega (\Phi ) & = & \frac{\parallel \vec{V}\parallel }{{{D}_{{\rm Down}/{\rm slope}}}.{\rm cos} (\alpha )}.{\rm cos} (\Phi +\Psi ). \\ {} & {} & {\rm cos} (\Phi +{{\theta }_{{\rm Eye}/{\rm slope}}}). \\ \end{array}\end{eqnarray} \tag{ 17 }$$

As we were looking for the direction of the maximum OF in order to determine the angle between the eye's equator and the slope of the nearest surface θ_Eye/slope, we differentiated the equation (17) with respect to Φ:

$$\begin{eqnarray}\begin{array}{rcl} \frac{{\rm d}w(\Phi )}{{\rm d}\Phi } & = & \frac{-\parallel \vec{V}\parallel }{{{D}_{{\rm Down}/{\rm slope}}}.{\rm cos} (\alpha )} \\ {} & {} & .\;\left[ {\rm sin} (\Phi +\Psi ).{\rm cos} (\Phi +{{\theta }_{{\rm Eye}/{\rm slope}}}) \right. \\ {} & {} & +{\rm cos} (\Phi +\Psi ). \\ {} & {} & {\rm sin} \left. (\Phi +{{\theta }_{{\rm Eye}/{\rm slope}}}) \right]. \\ \end{array}\end{eqnarray} \tag{ 18 }$$

The maximum value of the cosine function is then obtained with:

$$\begin{eqnarray}&&\frac{{\rm d}w(\Phi )}{{\rm d}\Phi }=0\;\Longrightarrow \;{{\hat{\theta }}_{{\rm Eye}/{\rm slope}}}=-\frac{\Psi +{{\theta }_{{\rm Eye}/{\rm slope}}}}{2}.\end{eqnarray} \tag{ 19 }$$

As we can see from equation (19), when the aerial robot is moving, the maximum value of the OF will not occur when the eye is parallel with the surface followed. But, it will occur and depend on the angle between the plane defined by the eye's equator and the slope of the nearest surface θ_Eye/slope and the direction of the velocity vector of the aerial robot Ψ in the frame of reference associated with the eye.

Hypothesis:. The velocity vector is always parallel to the nearest surface

$$\begin{eqnarray}&&\Psi ={{\theta }_{{\rm Eye}/{\rm slope}}}.\end{eqnarray} \tag{ 20 }$$

As explained in section 3.2, the OF generated by the nearest surface was used to always keep a safe distance from this surface therefore leading to a surface following behavior, and the velocity vector will then automatically line up with the nearest horizontal surface.

In this case, equation (19) becomes:

$$\begin{eqnarray}&&{{\hat{\theta }}_{{\rm Eye}/{\rm slope}}}={{\theta }_{{\rm Eye}/{\rm slope}}}.\end{eqnarray} \tag{ 21 }$$

The estimated reorientation angle ${{\hat{\theta }}_{{\rm Eye}/{\rm slope}}}$ can therefore be used to reorient the eye in order to keep it in parallel with the surface followed, as ${{\theta }_{{\rm EiR}}}={{\theta }_{{\rm EiR}}}+{{\hat{\theta }}_{{\rm Eye}/{\rm slope}}}$ results in θ_Eye/slope = 0.

In the neighborhood of the peak in the OF according to equation (17), the OF will vary according to:

$$\begin{eqnarray}\begin{array}{rcl} \omega (\Phi )\simeq f(\Phi ^{\prime} ) & = & \frac{\parallel \vec{V}\parallel {{{\rm cos} }^{2}}(\Phi ^{\prime} )}{{{D}_{{\rm Down}/{\rm slope}}}.{\rm cos} (\alpha )}. \\ \end{array}\end{eqnarray} \tag{ 22 }$$

The idea was therefore to use the measurements originating from all the LMSs, which are all separated by a given angle, to identify the angle between the eye's equator and the orientation of the surface followed. We had to seek for the coefficients $\frac{\parallel \vec{V}\parallel }{{{D}_{{\rm Down}/{\rm slope}}}.{\rm cos} (\alpha )}$ and ${{\hat{\theta }}_{{\rm Eye}/{\rm slope}}}$ in the function $f(\Phi ^{\prime} )$ = $\frac{\parallel \vec{V}\parallel }{{{D}_{{\rm Down}/{\rm slope}}}.{\rm cos} (\alpha )}.{{{\rm cos} }^{2}}(\Phi ^{\prime} -{{\hat{\theta }}_{{\rm Eye}/{\rm slope}}})$ giving the best approximation in the least squares sense between the OF measurements ω(Φ) and the cosine square function. This procedure could be easily implemented in the microcontroller at almost no cost, as the cosine square function could be estimated by a polynomial second order function using a Taylor expansion around zero:

$$\begin{eqnarray}\begin{array}{rcl} f(\Phi ^{\prime} ) & \simeq & \frac{\parallel \vec{V}\parallel }{{{D}_{{\rm Down}/{\rm slope}}}.{\rm cos} (\alpha )} \\ {} & {} & .(1-{{(\Phi ^{\prime} -{{{\hat{\theta }}}_{{\rm Eye}/{\rm slope}}})}^{2}}) \\ {} & \simeq & {\rm cst}.\left[ -\Phi {{^{\prime} }^{2}}+2.{{{\hat{\theta }}}_{{\rm Eye}/{\rm slope}}}\Phi ^{\prime} \right. \\ {} & {} & +\left. (1-\hat{\theta }_{{\rm Eye}/{\rm slope}}^{2}) \right], \\ \end{array}\end{eqnarray} \tag{ 23 }$$

$$\begin{eqnarray}&&\Longleftrightarrow \;\omega (\Phi )\simeq a.\Phi {{^{\prime} }^{2}}+b.\Phi ^{\prime} +c.\end{eqnarray} \tag{ 24 }$$

We define $X=[\Phi {{^{\prime} }^{2}},\Phi ^{\prime} ,1]$ and from our set of measurements Γ, we determine the coefficients (a, b, c) using the least-squares method:

$$\begin{eqnarray}&&[a,b,c]={\rm inv}(X*X^{\prime} )*X*\Gamma ^{\prime} .\end{eqnarray} \tag{ 25 }$$

In the latter expression, only Γ depends on the OF measurements, whereas all the other parameters are constant and depend only on the fixed orientation of each OF measurement in the reference frame of the eye. Finding the coefficients (a, b, c) requires just one matrix multiplication, and equations (23) and (24) can be used to show that the estimated angle between the eye's equator and the slope of the surface followed can be computed as follows:

$$\begin{eqnarray}&&{{\hat{\theta }}_{{\rm Eye}/{\rm slope}}}=\frac{-b}{2.a}.\end{eqnarray} \tag{ 26 }$$

In our computer simulations, we observed that when noise was present in the OF measurements, we could end up with poorly estimated reorientation angles, resulting in the closed loop in oscillations of the eye. To eliminate these errors, we defined the confidence index:

$$\begin{eqnarray}&&{\rm Confidence}=\frac{\sum \left| X^{\prime} \times b-\omega ({{\Phi }_{i}}) \right|}{{\rm median}(\omega ({{\Phi }_{i}}))}.\end{eqnarray} \tag{ 27 }$$

This coefficient directly reflects how close the OF measurements ω(Φ_i) are from describing a cosinus square function which is the theoretical evolution of the OF measurement with Φ in an obstacle free environment. The lower the confidence index is, the greater the similarity between the OF measurements and the approximated cosine square function will be. The reorientation angle is therefore only updated when this confidence index is below a fixed threshold value that has been selected empirically. This index strongly increases the robustness of this 3rd feedback loop as the reorientation angle will be only updated with reliable OF values.

Although we reorient the eye parallel to the closest surface, the information we use to act on the altitude of the robot is still the forward OF. Indeed, the main advantage of the eye reorientation is that it allows us to perform naturally all the OF measurements in the frame of reference defined by the slope of the closest surface.

Figure 8 gives the angle of reorientation ${{\hat{\theta }}_{{\rm Eye}/{\rm slope}}}$ estimated thanks to the OF measurements in blue superimposed on the theoretical reorientation angle θ_Eye/slope in red, when the BeeRotor II robot was flying at a constant speed at a constant altitude, and the stepper motor regularly changed the angle of the eye relative to the body θ_EiR. The reorientation angle was estimated based on 10 LMSs, as described in section 4.1. Imposing a rotation of α on the eye is equivalent to the situation where the robot is following a surface tilted at this same angle α. As we can see, the accuracy of the estimated reorientation angle was highly satisfactory (the mean error was less than 0.2° and the dispersion was less than 2°), and the response to each of the pitch eye steps was almost instantaneous. As we can see, the reorientation angle was slightly underestimated when the theoretical value departed from zero due to the difference between the cosine square function and the value approximated by a polynomial second order function. However, as the robot uses the measured reorientation angle to constanly align its eye with the closest surface, this underestimation will not unhinge the autopilot as the feedback loop system will always converge in order to make θ_Eye/slope equals to 0. The same argument can also explains why the autopilot will not be strongly disturbed even if the velocity vector is not perfectly parallel to the closest surface as we hypothesized.

Zoom In Zoom Out Reset image size

As described above, the aerial robot equipped with suitable means of performing OF measurements defined in the reference frame associated with its eye, can:

• (i)  
  adjust the aerial robot's altitude to regulate (keep constant) the OF generated by the nearest horizontal surface,
• (ii)  
  adjust its forward speed to regulate (keep constant) the sum of the ventral and dorsal OFs,
• (iii)  
  keep the robot's eye's equator parallel to the nearest surface.

The BeeRotor II's autopilot (see figure 9) is therefore composed of three direct feedback control loops determining: (i) the mean speed of the propellers Ω_Rotors, (ii) the differential thrust of the propellers ΔΩ_Rotors and (iii) the pitch orientation of the eye θ_Eye/slope.

Zoom In Zoom Out Reset image size

The first and second OF feedback control loops adjusting the altitude and the speed implemented onboard BeeRotor II, which are shown here in green and blue, respectively, are similar to the feedback loops implemented in the BeeRotor I autopilot described in section 3.2.

Third OF feedback: based on the least squares estimation method presented in section 4.1 giving the angle between the eye's equator and the slope of the nearest surface followed ${{\hat{\theta }}_{{\rm Eye}/{\rm slope}}}$ (see equation (21)), the red feedback loop (see figure 9) reorients the eye, keeping it parallel with the nearest surface. This eye reorientation will generate an additional small rotational OF that is known and subtracted from every LMS measurement.

At the same time, this method of eye reorientation makes it possible to perform all the OF measurements in the reference frame associated with the aerial robot's eye, which therefore depends on the local slope of the tunnel.

If we take the nearest surface to be the ground and assume that the eye is kept parallel with the local ground orientation (in which case, D_Down/slope = D(Φ).cos(Φ)), we can use equation (3) to express the sum of the ventral OFs measured in the forward and backward oriented ROIs as follows:

$$\begin{eqnarray}\begin{array}{rcl} {{\omega }_{{\rm MaxOF}}} & = & \omega (\Phi )+\omega (-\Phi ) \\ {} & = & \frac{2\parallel \vec{V}\parallel .{{{\rm cos} }^{2}}(\Phi ).{\rm cos} (\Psi )}{{{D}_{{\rm Down}/{\rm slope}}}}. \\ \end{array}\end{eqnarray} \tag{ 28 }$$

If we decompose the velocity vector $\vec{V}$ into its two components V_x/slope = V.cos(Ψ) and V_z/slope = V.sin(Ψ), namely the horizontal and vertical velocity, respectively, we can write equation (28) as follows in the reference frame defined by the local slope of the nearest surface:

$$\begin{eqnarray}\begin{array}{rcl} {{\omega }_{{\rm MaxOF}}} & = & \omega (\Phi )+\omega (-\Phi ) \\ {} & = & 2.{{{\rm cos} }^{2}}(\Phi ).\frac{{{V}_{x/{\rm slope}}}}{{{D}_{{\rm Down}/{\rm slope}}}}. \\ \end{array}\end{eqnarray} \tag{ 29 }$$

In addition, assuming for the sake of simplification that the gaze of the aerial robot is kept parallel with the ground and the ceiling, we can write:

$$\begin{eqnarray}\begin{array}{rcl} {{\omega }_{{\rm SumOF}}} & = & \omega (\Phi )+\omega (-\Phi ) \\ {} & {} & +\omega (180{}^\circ +\Phi )+\omega (180{}^\circ -\Phi ) \\ {} & = & \frac{2{{V}_{x/{\rm slope}}}.{{{\rm cos} }^{2}}(\Phi )}{{{D}_{{\rm Down}/{\rm slope}}}}+\frac{2{{V}_{x/{\rm slope}}}.{{{\rm cos} }^{2}}(\Phi )}{{{D}_{{\rm Up}/{\rm slope}}}} \\ {} & = & 2{{V}_{x/{\rm slope}}}.{{{\rm cos} }^{2}}(\Phi ) \\ {} & {} & \times (\frac{1}{{{D}_{{\rm Down}/{\rm slope}}}}+\frac{1}{{{D}_{{\rm Up}/{\rm slope}}}}). \\ \end{array}\end{eqnarray} \tag{ 30 }$$

It is impossible for the eye to be parallel to both surfaces in a corridor with a contrasting obstacle on the ground. But, if the eye is always parallel to the nearest surface, the OF generated by the other surface will be smaller and the error due to this simplification will not affect the result very greatly.

It can be seen from equations (29) and (30) that thanks to the eye-reorientation principle, the first and second OF feedback control loops will adjust the aerial robot's horizontal speed V_x/slope in the frame of reference defined by the local ground and the distance D_Down/slope between the robot and the ground in the direction normal to the ground. The possibility of detecting the obstacle in advance improves the robustness of the BeeRotor autopilot's performances, and hence its ability to steer a collision-free course in a moving or cluttered environment, as we will see in the following sections.

The controllers implemented onboard BeeRotor are described in detail in appendix D.

To test the performances of the BeeRotor II's improved autopilot endowed with the novel eye reorientation system, an experiment was performed in which an additional feature was included on the ground of the environment (see extension 2) by replacing the 1 m long flat part of the obstacle by a 30 cm high supplementary steep osbtacle (see figure 10(b)). Figure 10(a) shows the trajectory of the aerial robot equipped with a fixed eye versus a decoupled eye which was automatically kept parallel to the ground thanks to the eye-reorientation principle presented in section 4.1. In both cases, the robot's forward speed and altitude were automatically adjusted based on the OF measurements via the first and second OF feedback control loops described in section 4.3. The fixed eye was kept at an angle of 0°relative to the body. As we can see, the reorientation of the eye made it possible for the robot to avoid the additional steep obstacle without crashing because the OF measurements reflected the distance to the ground more accurately. When the aerial robot was flying over the obstacles, we observed that the eye pitched upward in front of the ascending ramp and downward when BeeRotor reached the descending ramp, keeping its body parallel with the surface below. When flying over the flat part of the environment, the eye was oriented at an angle which compensated for the body pitch angle. However, as we can see, the duration of the eye reorientation process was quite long as the eye required several meters to return to the steady state again. This was mainly due to the large time constant chosen for the low-pass filter applied to the signals driving the stepper motor rotating the eye. When this time constant was reduced, the eye reorientation process was faster, generating a large rotational OF superimposed on the translational OF and disturbing the estimation of the required reorientation angle ${{\hat{\theta }}_{{\rm Eye}/{\rm slope}}}$, which often resulted in eye oscillations. With a fixed panoramic eye measuring the OF in all directions, the angle between the eye's equator and the surface could be determined faster, and it could reflect the OF generated by the surface followed more accurately, thus enhancing the robot's ability to follow a terrain at short range while avoiding prominent features.

Zoom In Zoom Out Reset image size

Figures 11(a)–(c) show the altitude of the BeeRotor II in the case of a stationary floor, a floor moving at 50 cm s⁻¹ in the same direction as the robot and a floor moving at 50 cm s⁻¹ in the opposite direction. Each curve gives the robot's altitude during a 40 m long flight with ω_setMaxOF = 150° s⁻¹ and ω_setSumOF = 250° s⁻¹. As we can see, the robot flew at a lower altitude when the floor was rotating in the same direction (figure 11(b)). The OF perceived decreased due to the motion of the ground, and the altitude feedback loop therefore decreased the mean thrust of the propellers in order to compensate for this decrease. On the other hand, when the floor was rotating in the opposite direction (figure 11(c)), the OF perceived was greater and the aerial robot's altitude increased so that the OF was kept at the setpoint value.

Zoom In Zoom Out Reset image size

The aerial robot is shown here every 6 s during each trajectory. Due to the changes in the OF caused by the movement of the ground, the sum of the ventral and dorsal OFs varied. This led to an increase in the robot's forward speed when the ground was moving in the same direction because the sum of the OFs decreased, and vice versa.

Figure 11(d) shows the altitude of the aerial robot flying over a floor moving up and down: the robot's position and orientation are shown every 6 s with ω_setMaxOF = 180° s⁻¹ and ω_setSumOF = 220° s⁻¹. The height of the ground was oscillating between 0 and 64 cm at a frequency of 0.05 Hz, which strongly disturbed the perceived ventral OF. Despite this considerable disturbance, the aerial robot still followed the ground and avoided the obstacle autonomously. In addition, the robot's forward speed was constantly changing due to the changes in the size of the tunnel. In particular, the aerial robot flew at a greater speed when the ground was minimally elevated, which explains why the rotorcraft's altitude did not decrease as much as the height of the ground: the altitude feedback loop adjusted the propellers' speed in order to maintain the ventral OF equal to the maximum OF setpoint value.

Like the BeeRotor I, the BeeRotor II was able to fly autonomously based on OF measurements and adjust its forward speed and its clearance from the ground and the ceiling without any need for an accelerometer or any measurements in the inertial frame of reference. However, thanks to the addition of the eye-reorientation principle, the BeeRotor II's autopilot can reject stronger perturbations and therefore navigate without crashing, even in a moving environment or in the presence of a sudden rise of the terrain.

Figure 12 shows the altitude of the BeeRotor II in a very unstable environment, where the ground was rotating in the forward or backward direction while its elevation was oscillating (see extension 2). The robot's altitude is plotted in figures 12(a) and (b) when the ground was oscillating up and down and moving at 50 cm s⁻¹ in the same direction as the robot with ω_setMaxOF = 125° s⁻¹ and ω_setSumOF = 280° s⁻¹ and with the ground oscillating up and down and rotating at 50 cm s⁻¹ in the opposite direction with ω_setMaxOF = 175° s⁻¹ and ω_setSumOF = 250° s⁻¹. The aerial robot was able to consistently adjust its altitude and its forward speed based on the OF measurements and avoid the obstacle. As observed previously with BeeRotor I, the robot flew at a lower altitude and a higher speed when the ground was rotating in the same direction because the OF perceived in this case was lower. On the other hand, the robot flew at a higher altitude and a lower speed when the ground was rotating in the opposite direction.

Zoom In Zoom Out Reset image size

To assess the robustness of the autopilot mounted onboard the BeeRotor II robot, strong perturbations were applied manually to the robot's pitch angle while it was automatically following the stationary ground with ω_setMaxOF = 150° s⁻¹ and ω_setSumOF = 250° s⁻¹ (see extension 2). Figure 13(a) shows the aerial robot's trajectory during a 25 m-long flight, where a strong negative step was imposed on the robot's pitch angle at a distance of about 8 m and a positive step on the pitch angle after it had travelled 20 m. The robot's pitch angle is presented in figure 13(c), and the negative pitch step of almost −20° and the positive pitch step of more than 10° can be clearly distinguished, whereas the robot's pitch angle during autonomous flight is generally about 5°. Although these perturbations strongly affected the rotorcraft's trajectory, it was still able to avoid the sudden rise of the terrain encountered and regain its initial altitude after travelling a few meters. As the pitch angle directly affects the robot's forward speed, the negative step imposed on the pitch angle strongly reduced its forward speed (see figure 13(b)), whereas the positive step was immediately rejected by the feedback control loops and hardly affected the robot's altitude or its forward speed at all.

Zoom In Zoom Out Reset image size

Insects' antennae are thought to contribute importantly to controlling their airspeed. Data obtained on free-flying insects deprived of their antennae provided direct evidence that these organs are involved in flight control: a decrease in the forward speed was observed in Dipterans deprived of their antennae, although they were still able to fly (Campan 1964, Burkhardt and Gewecke 1965). It has been suggested recently in flies that the position of the antennae is actively controlled and adjusted during flight by the 'antennal positioning reaction' which could allow the animal to measure absolute airspeed when including efference copies sent to the antennae muscles and the signals from the Johnston's organ that measure changes in airspeed (Taylor and Krapp 2007). The airspeed sensor implemented in our robot could then be compared with insects' antennae. Other studies have hypothesized that the 'antennal positioning reaction' enhances the insects' ability to sense changes in the airspeed (Mamiya et al 2011). This finding suggests that the antennae are sensitive only to the acceleration of the air (Fuller et al 2014), which may be processed in a nested feedback loop. With our custom-made airspeed sensor, we tried to use the changes in airspeed as a control parameter, but the signals turned out to be too noisy for this parameter to be suitable for use onboard the aerial robot.

However, it seemed to be worth testing the ability of the BeeRotor robot to fly in its environment without the inner feedback loop regulating the airspeed (see figure 9), since the flight performances of insects deprived of their antennae were degraded. In this configuration, the aerial robot relies solely on the measurements performed by the OF sensors and the rate gyro to adjust its forward speed and its clearance from the ground and the ceiling and automatically orient its eye relative to the nearest surface. Only the second OF feedback loop adjusting the robot's forward speed (in blue in the figure) was modified, and the output from the forward controller (which then became a proportional derivative controller) was then used directly as a setpoint by the nested feedback loop regulating the pitch rate. The conditions of our experiment can not be compared with the ones we could have outdoors due to the wind and the turbulences that would affect the stability of the rotorcraft which would therefore greatly benefit from the inner feedback loop regulating the airspeed.

Despite the removal of the airspeed feedback loop, it can be seen from figure 14(a) (extension 2) that BeeRotor II could still follow the terrain and adjust its forward speed based on the ventral and dorsal OFs, robustly avoiding all the obstacles encountered. The robot's altitude is presented here while it was following the ground (green curve) and the ceiling (cyan curve) with the same setpoint values ω_setMaxOF = 150° s⁻¹ and ω_setSumOF = 250° s⁻¹. The aerial robot kept on following the nearest surface detected by adjusting its two rotors' rotational speeds. The robot is shown here every 5 s: it can be seen from this figure that the robot adjusted its pitch angle and eventually its forward speed depending on the size of the tunnel, even when it was no longer equipped with an airspeed sensor.

Zoom In Zoom Out Reset image size

In order to make the robot land automatically after travelling a little more than 20 m (dotted line), the sum OF setpoint ω_setSumOF was decreased rampwise from 250° s⁻¹ to 100^o s⁻¹ in a 10° s⁻¹ ramp (see figure 14(b)) so as to gradually decrease its forward speed. In order to keep the maximum OF constant, the first feedback control loop gradually decreased the robot distance from the closest horizontal surface detected, so that it eventually landed or docked with a negligible forward speed at touchdown. The duration of the landing phase depended directly on the slope of the ramp imposed on the sum OF setpoint.

The results obtained in the present experiments (see figure 14 and extension 2) show that OF based visual guidance is possible both with and without absolute airspeed sensors: this may explain how insects improve their flight performances using the airspeed or the changes in the air speed (the air acceleration) sensed by their antennae.

Based on three main feedback loops regulating the maximum value between the ventral and dorsal OFs and the sum of these OFs and orienting the robot's eye, BeeRotor II was able to follow terrain autonomously, adjust its forward speed to the size of the tunnel and avoid obstacles. The BeeRotor II robot is equipped for this purpose with a quasi-panoramic eye processing the OF generated by the ground and the ceiling. The eye is automatically kept parallel to the nearest surface by performing a least squares approximation on the OF pattern formed by the array of LMSs. This reorientation method enhances the BeeRotor aerial robot's ability to avoid steep obstacles even in a moving environment (see extension 2) without any need for absolute reference cues indicating the vertical, based, for example, on an accelerometer such as those commonly used these days in flying robots.