Development of collision avoidance system for useful UAV applications using image sensors with laser transmitter

The main goal of this study is to demonstrate the approach of achieving collision avoidance on Quadrotor Unmanned Aerial Vehicle (QUAV) using image sensors with colour- based tracking method. A pair of high definition (HD) stereo cameras were chosen as the stereo vision sensor to obtain depth data from flat object surfaces. Laser transmitter was utilized to project high contrast tracking spot for depth calculation using common triangulation. Stereo vision algorithm was developed to acquire the distance from tracked point to QUAV and the control algorithm was designed to manipulate QUAV's response based on depth calculated. Attitude and position controller were designed using the non-linear model with the help of Optitrack motion tracking system. A number of collision avoidance flight tests were carried out to validate the performance of the stereo vision and control algorithm based on image sensors. In the results, the UAV was able to hover with fairly good accuracy in both static and dynamic collision avoidance for short range collision avoidance. Collision avoidance performance of the UAV was better with obstacle of dull surfaces in comparison to shiny surfaces. The minimum collision avoidance distance achievable was 0.4 m. The approach was suitable to be applied in short range collision avoidance.


Introduction
Unmanned aerial vehicles (UAVs) hold good promises for performance in both civilian and military operations. A broad range of meaningful tasks such as aerial surveillance, simultaneous arrival and attack, search and rescue, flight formation, etc. can be accomplished using the application of UAVs. However, for UAVs to perform successfully in their respective missions and operations, collision avoidance system is necessary to be imparted into them. Without collision avoidance system, an UAV during operation would collide into unexpected objects such as wall, flying objects, flying animals, humans and others, resulting in mission failure, destruction of UAV, damage or injury to the collided objects or a combination of all. Collision avoidance for unmanned ground vehicles (UGVs) can be based on multiple choices including infrared sensors [1], image-based sensors [2], sound sensors [3] and laser scanner [4]. Image-based sensor collision avoidance on UAVs is a relatively good strategy compared to others due to the combined functions for aerial surveillance and obstacle avoidance. In other words, UAVs with other type of sensors do not provide the added ability of aerial surveillance.
For the few past years, researches on the topic of stereo-vision based collision avoidance have been carried out extensively. Most of the researches and projects were focused on determining the distance between obstacle and cameras by using the conventional method of stereo correspondence from pixelto-pixel [5,6,7,8] to obtain the disparity map, which is also known as the depth map. Apart from that, works on determining the depth information using the conventional and improved version of Scale-Invariant Feature Transform (SIFT) are also accomplished [9,1010]. With the aids of image filtering and computation, researchers were able to filter out the object closest to the cameras and determine its depth. However, the conventional method required massive amount of computational power and time [11], thus it is unsuitable to be applied on UAVs where high real-time processing speed is mandatory. Apart from that, high accuracy is also an important criterion that critically determines the collision avoidance performance of UAVs [12].
In this work, a strategy of depth calculation using colour-based tracking instead of the conventional method of pixel-to-pixel correspondence to obtain disparity map is proposed. A laser transmitter is used to project a laser beam onto an obstacle to create a colour-distinct point from the obstacle. The colour-distinct point acts as the reference point for image tracking and depth calculation. The depth value obtained via stereo triangulation is then integrated into the control algorithm based on non-linear model by Kendoul [13] to achieve relative-position collision avoidance. The quadrotor used in this study is as shown in Figure 1.

Camera calibration
Camera calibration is one of the most important procedures in image processing for the purpose of obtaining accurate data from image sensors [14]. Two types of camera calibration carried out in this project were single camera calibration and stereo camera calibration, which was done to obtain the intrinsic and extrinsic parameters for the cameras. The cameras were also redefined by calibration with Optitrack motion capture system to improve the accuracy of the depth estimation algorithm.

Colour-based filtering
Colour-based filtering is used to differentiate the laser beam projection from the obstacle. The colour of each pixel on the images is defined in the HSV colour scheme where each pixel has its own HSV values. HSV is chosen over RGB colour scheme because HSV separates the image intensity from the color information, which is important. By using OpenCV, the pixels containing the laser projection point can be singled out from the rest of the image by adjusting the HSV values. Gaussian blurring method is applied to reduce the noise of the images. Morphological operators, Erosion and Dilation are also used to refine the filtered image and reduce noise. Since the orientation of the laser transmitter was fixed with the camera, the location of the laser projection was constant around a certain region of the image. The original image (640 x 480 pixels) was cropped based this region to limit the area of search to a rectangular box of size (330 x 80 pixels). Figure 2

Stereo Triangulation
The algorithm for depth calculation was based on simple Linear Triangulation method that relates the baseline, focal length and image coordinates as shown in Figure where Z = distance between cameras' baseline to image point, b = length of baseline, f = focal length of cameras, x 1 = x coordinate of image projected on left camera and projected on right camera.

Relative position control algorithm
The collision avoidance of QUAV is achieved mainly by using relative attitude and position control with cascade PID controller.

Controller design and transfer function
In this section, the vision-based controller used for target object difficult to design a reliable autopilot for highly nonlinear and coupled in inner strategy, provides the quadrotor position and velocity with respect to the target. Therefore, the vision based control for object tracking can be formulated as a stabilization problem. controller is then proposed, which is based on attitude and position controller by Kendoul [15] controller is the position ( , , control is the inner loops controller expressed as in Equation 2 and Equation 3.
The desired attitudes are then generated as follows: The algorithm for depth calculation was based on simple Linear Triangulation method that relates the baseline, focal length and image coordinates as shown in Figure 3. This is expressed as Equation 1.
where Z = distance between cameras' baseline to image point, b = length of baseline, f = focal length = x coordinate of image projected on left camera and x r = x coordinate

Relative position control algorithm
The collision avoidance of QUAV is achieved mainly by using relative attitude and position control ontroller design and transfer function based controller used for target object tracking is described difficult to design a reliable autopilot for QUAV since the quadrotor dynamics is inhere coupled in inner-axis. The vision system, with the proposed image algorithm the quadrotor position and velocity with respect to the target. Therefore, the vision based control for object tracking can be formulated as a stabilization problem. A hierarchical nonlinear which is based on the nonlinear model of the multiby Kendoul [15] uses the non-linear controller. The outer loop of the controller, which runs at slow dynamics. However, the orientation control is the inner loops controller that is running at fast dynamics. The PID structure can be in Equation 2 and Equation 3.
then generated as follows: The basis of linear stereo triangulation The algorithm for depth calculation was based on simple Linear Triangulation method that relates the 3. This is expressed as Equation 1. (1) where Z = distance between cameras' baseline to image point, b = length of baseline, f = focal length = x coordinate of image The collision avoidance of QUAV is achieved mainly by using relative attitude and position control is described. In general, it is dynamics is inherently unstable, proposed image algorithm the quadrotor position and velocity with respect to the target. Therefore, the visionhierarchical nonlinear -rotor dynamics. The The outer loop of the at slow dynamics. However, the orientation The PID structure can be Illustration on linear collision avoidance scenario Yaw, # is also controlled by PID controller. The output thrust of the quadcopter is shown as follow: A simple control strategy was established to maintain the distance between QUAV and obstacle as shown in Figure 5.

Hardware set up and implementation
A pair of HD cameras, Logitech® HD Webcam C310 were calibrated before being mounted on the UAV to serve as image sensors. The purpose of the project was to demonstrate the approach for using image sensors for collision avoidance, hence the project was proceeded with on-ground processing in which the cameras were connected physically with the extended USB cables to ground computer. An Arduino compatible laser transmitter, Keyes® KY-008 was then installed at the centre between the stereo cameras to project a distinct colour spot on the obstacle for the purpose of colour-based image filtering. Evaluation of collision avoidance using image sensors was performed at indoor environment with the assistance of Optitrack motion capture system. The UAV's dynamics and attitude response was controlled based on the UAV's and the obstacle's coordinates provided by the Optitrack indoor positioning system. Overall setup of the hardware is shown in Figure 6.

Result and discussion
Two experiments were conducted in an indoor environment assisted by Optitrack Motion Capture system to validate performance of the UAV in static and dynamic collision avoidance. The Optitrack motion capture cameras were chosen as indoor positioning system and testing environment to replace the global positioning system (GPS) because it gives the advantage of easy conduct of repetitive flight simulations. Before conducting the flight experiments, three Optitrack markers were fixed on both the UAV and the obstacle to provide their real-time respective coordinates. The UAV was integrated with the depth estimation algorithm combined with control algorithm to achieve desired collision avoidance using image sensors. Table 1 shows the PID gain values used in the experiment. The results for static and dynamic collision avoidance performance are presented in Figure 7.

Static collision avoidance
The first test flight measures the UAV's collision avoidance performance in static hovering. The UAV was positioned at (0, 0) coordinate and the obstacle was positioned at (1 m, 0) coordinate. Positions of the UAV and the obstacle were tracked by Optitrack system. UAV was lifted up 1.5 m above ground by Optitrack control. The camera sensors were turned on to obtain the real distance value. Desired distance between UAV and obstacle was set to 2.0 m and response of the UAV was then recorded and comparisons were made.
As referred to Figure 7(a), the UAV was able to hover statically at the desired distance (final) with small perturbations but unacceptable error, which was greater than 10%. However, the error obtained when hovering at a shorter distance (final) was acceptable, which was less than 10%. Since the short distance is the important factor for collision avoidance, the performance of the UAV was satisfactory. The results obtained as shown in Table 2 satisfy the work by Waqar [14] where the accuracy of linear triangulation deteriorates as the distance goes higher. The reason behind the accuracy deterioration at far distance may be due to the image point not within and far away from the depth resolution at longer range [16]. Table 2. Distance values between obstacle and UAV in static flight test

Dynamic collision avoidance
The second test flight measured the UAV's collision avoidance performance in dynamic hovering, in which the UAV was moved from a location to another location in non-linear dynamics. The UAV was initially positioned at (0, 0) coordinate and obstacle was positioned at (2 m, 0) coordinate. Positions of the UAV and the obstacle were tracked by Optitrack system. UAV was lifted up 1.5 m above ground by Optitrack control. The camera sensors were turned on to obtain the real distance value. Desired distance between the UAV and the obstacle was set to 1 m. The UAV was controlled by Optitrack to hover towards the obstacle until the desired distance was achieved. The response of the UAV was then recorded and comparisons were made. As referred to Figure 7(b), the UAV was able to hover dynamically and stop at the desired distance (final) but with small perturbations and acceptable error, which was less than 10%. However, the error obtained when hovering from the longer distance (initial) was unacceptable, which was higher than 10%. Again, the collision avoidance performance was satisfactory because collision often occurs at small distance. The results obtained as shown in Table 3 once again satisfy that the accuracy of linear triangulation deteriorate as the distance goes higher. The depth estimation from stereo cameras is also affected by calibration errors and alignment of stereo camera parameters [17]. Table 3. Distance values between obstacle and UAV in dynamic flight test

Colour and reflectivity test flight
Several additional test flights were conducted to test the UAV's collision avoidance performance in response towards obstacles of different colours and reflectivity to validate whether different types of obstacle's surface may degrade the quality of collision avoidance system based on image sensors. In Table 4, it is shown that the UAV's performance was degraded when a surface of high reflectivity was used. It has also proven that the background colour of the obstacle does not influence image sensing of the UAV. Table 4. UAV's performance in colour and reflectivity flight test

Minimum distance test flight
The final section of the UAV test flight included the determination of threshold for minimum desired distance, D between the obstacle and the UAV. A few test flights were conducted to identify minimum distance that could be achieved before collision from a fixed position of 1.5 m. The minimum distance based on the dimension and geometry of UAV was measured to be 0.40 m. Hence, flying smaller than 0.4 m was undesirable for the UAV. The results were tabulated in Table 5 and it can be observed that no collision occurred when the desired distance was reduced to 0.40 m. Thus, the minimum distance that the UAV can achieved when hovering is taken as 0.40 m.

Conclusion and future work
This paper proposes a new approach of calculating distance using image sensor with laser transmitter to achieve collision avoidance. The laser transmitter projects a laser beam on the surface of obstacle to create a colour-distinct spot on the obstacles. OpenCV functions can be used to filter out the colourdistance spot from the rest of the images. This produces a reference point and area for colour-based tracking without the need to compare pixel by pixel as in the conventional depth map method. Control algorithm based on non-linear model developed by Kendoul [15] was applied in this work to achieve relative-positioned collision avoidance assisted by Optitrack Motion Capture System. Experiments and evaluations for the collision avoidance performance were carried out at indoor environment in range of Optitrack cameras, which provided position coordinates and velocities of the UAV and the obstacle. The results obtained showed that linear triangulation with image sensors provide satisfactory results in both static and dynamic collision avoidance. The threshold for minimum desired distance between UAV and obstacle obtained using this approach is 0.4 m. Lastly, the distance values calculated by the linear triangulation with image sensors were good when the distance was small but deteriorate when the distance increases.
The recommended future work for this project is to improve the accuracy for depth calculation by applying Kalman filter to remove highly inaccurate values. A real-time collision avoidance UAV can be developed by upgrading the UAV from on-ground processing to on-board processing by installing OpenCV compatible microcomputers such as Raspberry® Pi. The UAV can also be integrated with others sensors such as infrared (IR) and ultrasound (sonar) to collision avoidance in all four directions. Final recommendation for future work is development of control algorithm for UAV's autonomous or manual hovering at outdoor environment to achieve meaningful task, for instance high altitude surface inspections on tall buildings and structures.