Intelligent robotic arm for human pose recognition based on teleoperation system

With the rapid development of skeleton recognition, machine learning and other technologies, we will find that there are great drawbacks in the control of manipulators. Robot teleoperation refers to the inclusion of human operation in the control loop of robot control. When robots deal with complex perception and a large number of tasks, teleoperation is far superior to intelligent programming when making decisions quickly and dealing with extreme situations. The goal of this paper is to build a robotic arm teleoperation system for human motion capture, so as to solve the problems that the control accuracy of the end of the robotic arm is not high and the motion of the robotic arm is greatly affected by the difference between the human arm in the current related research, the master-slave human motion mapping algorithm is designed and extended with machine learning algorithms. We use inertial motion capture to realize teleoperation, so as to avoid the use of the terminal position and orientation control method of the hand controller to form the control command of the remote robot after tedious calculation, and it is convenient for the operator to complete the attitude tracking task in real time. The obtained attitude information has a larger range, higher sensitivity and better dynamic performance.


Introduction
With the rapid development of science and technology, there are many shortcomings in the control of manipulators.Due to the limitation of robot performance structure, human and robot action scales are different.However, the manipulator is controlled by joint angles, and the key is to convert the human motion capture data into the joint Angle data of the robot motion to drive the robot action.At present, many teams have published their results on this problem.Moulard et al. proposed a motion retargeting method based on optimal motion, in which the robot can accurately imitate the motion capture and record the human motion.Huang Qiang et al. proposed a similarity function for robot to imitate human motion, which is used to describe the similarity degree between robot and human motion.However, the above studies are based on off-line robot control, and there is lack of further research on real-time robot teleoperation based on motion capture.Stanton et al. proposed a particle swarm optimization algorithm to train a neural network to learn the mapping relationship between the human motion capture data and the robot joint Angle to determine the joint Angle position of the robot, but the end control accuracy of this method is not high.In order to improve the control accuracy of the robot arm end and avoid complex kinematic operations, and achieve real-time and reliable teleoperation control, this paper improves the master-slave human motion mapping method for the above goals.Based on the traditional knowledge of human kinematics, the human arm is normalized to overcome the error caused by the different lengths of the human arm.The quaternion obtained by the motion capture system is used to realize the nonsingular mapping of the joint space, and the motion information of the human arm is converted into joint Angle information to directly control the action of the manipulator system.The arm combines the posture recognition technology, the control board processes the data collected by the camera, extracts the characteristics of the human body posture, recognizes the human body posture, and then drives the steering engine to complete the corresponding work to realize the imitation of the human action.

Materials and methods
This system is mainly composed of a motion capture system that can collect human motion posture at the main end and a robot arm that can perform operations at the secondary end, and can control the robot arm to realize human motion following movement. [1]Figure 1 shows the structure of a typical master/slave teleoperation system.The specific research of this system includes three parts: the use of motion capture system and robotic arm system, the research of master-slave human motion mapping method, experimental verification and application analysis.

Motion capture equipment
Motion capture technology is a technology that records human movements and converts them into digital patterns.In recent years, motion capture equipment based on wearable inertial sensors is becoming more and more mature, and it makes up for the lack of optical motion capture technology, in addition to the application and the most mature field of movie special effects, has gradually expanded to ergonomics, robot control and other fields.
In this paper, the "Neuron" sensor developed by Nuoyiteng Company is used to realize motion capture.By fixing the combination of the sensor nodes in the specific parts of the human body, the wireless sensor network is formed between the nodes through wireless communication, forming a  After the user wears the device, the working movement is carried out.The sensor device will capture the movement data of the target object, including the attitude and orientation of the body part, and transmit it to the data processing equipment through the data transmission equipment.After data correction and processing, the three-dimensional model is finally established, and the three-dimensional model is truly and naturally moving with the moving object. [2]In this project, the Axis Neuron software developed by Noytium will be used to export the skeletal motion data -intermediate formula.
Intermediate data is the output method that is closest to the original data of the IMU sensor.In fullbody wear mode, the intermediate data in each frame contains all the sensor data and motion data of the 21 bones and the contact status of the feet.For intermediate data, each bone contains 16 float data, which are: Three position data, three velocity data , four quaternion data , three acceleration data , and three data for the gyroscope. [3]

Robot arm system
We take the UR3 robotic arm as the controlled object.UR3 has a 6-axis joint, can be mounted on a table, and is easy to program.With its excellent characteristics, a number of scientific studies can be carried out in the laboratory, including the teleoperation technology research based on inertial motion capture in this project. [4]By connecting the UR3 arm to the motion capture device, we programmed the arm to operate remotely using PolyScope, a graphical user interface (GUI).Figure 3 shows the UR3 joint.

Research on master-slave human motion mapping method
In this project, due to the adoption of mature motion capture and robotic arm system, we no longer need to carry out complex filtering and other processing on the data collected by the sensor in the process of motion capture, and can directly export the required data by using the corresponding software provided above.At the same time, we do not need to carry out complicated forward and inverse kinematics analysis and solution for the robot arm without simulation.On the basis of saving time cost and reducing project complexity, the focus of this project is on the research of master-slave human motion mapping method.
Due to the limitations of the performance structure of the robot, such as the limitation of the motion range and speed of the joint Angle of the robot, the joint configuration of the robot and the length of each component are not consistent with the human body, which will lead to different motion scales between the human and the robot.In this project, it is concluded that the motion capture data cannot be directly applied to the robot arm system.Because the manipulator is controlled by the joint Angle, the research of master-slave human motion mapping method is to convert the human motion capture data into the joint Angle data of the robot motion to drive the robot motion.In order to improve the control accuracy of the end of the robot arm, avoid complex kinematic operations, and realize real-time and reliable teleoperation control, our team improved the master-slave human motion mapping method for the above goals.The research method of this paper is to normalize the human arm based on traditional human kinematics knowledge.To overcome the errors caused by the different lengths of the human arm, the quaternion obtained by the motion capture system is used to realize joint space non-singular mapping and convert the motion information of the human arm into joint Angle information.Directly control the action of the robot arm system.

Data processing
The system uses two components, gyroscope and accelerometer, to obtain the pose information of the arm.The system mainly uses Kalman filter to reduce the influence of gyroscope drift, so as to ensure the accuracy of the angular velocity data measured by the gyroscope.
In order to obtain the position of the operator's wrist relative to the base coordinate of the shoulder when the operator's arm moves, the system fixes the attitude sensor on the operator's arm's wrist, elbow and shoulder.Due to the roughness of the joint, the system fixes the sensor horizontally on the front part of each joint in order to obtain more accurate data.The Y axis of the sensor is forward along the arm, the Z axis is up, and the X axis points to the right side of the body.In order to represent the relative pose of the shoulder, elbow, and wrist, the coordinate systems of the shoulder, elbow, and wrist in the same direction as the sensor coordinate system are set as {os}, {oe}, and {ow} respectively. [5]fter using position control to get the robot arm near the target position in a few control times, the position of the robot arm needs to be fine-tuned.Since the space coordinate position of the robot arm at this time is very close to the desired position, in order to facilitate operation, the six actions of moving up, moving down, moving left, moving right, rotating clockwise and rotating counterclockwise are specified as basic actions.When the robot arm reaches the desired destination coordinates, the position of the robot arm is adjusted based on the six actions to make it reach the destination coordinates.The six movements can be identified based on the attitude information obtained by the attitude sensor fixed to the wrist. [6]

Results & Discussion
After the realization of the robot arm teleoperation system based on human motion capture, we will carry out experimental verification and application analysis of the designed mapping method. [7]In order to prove the effectiveness of the designed mapping method and the high precision of the end control, this paper uses the Nuoyiten motion capture system to remotely operate the UR3 robot arm, and proves the reliability and effectiveness of the system by completing a series of actions during the process of lifting the water cup and pouring water into the water cup at another position.
The software architecture of the demo interface.The development of demo interface is completed in C++ environment, including application layer, model layer, control layer and support layer.Application layer completes man-machine communication, which mainly includes display module, parameter setting module, simulation module and control instruction module.The display module is mainly used to display the model and motion trajectory of the manipulator.The parameter setting module is mainly used to set the serial port number for communication and the initial position of the robot arm.The simulation module is used to realize the motion simulation of the robot arm and display the pose of the end of the robot arm and the rotation Angle of each joint.In the model layer, OpenGL is used to complete the modeling of the robot arm.The control layer is mainly used for motion simulation and trajectory planning.The support layer is used to complete the inverse solution and forward solution algorithm of the virtual robot arm kinematics, the communication between the operator and the demonstration interface, the communication between the demonstration interface and the robot arm, and the generation of the control instructions of the robot arm. [8]he presentation interface is divided into view window and display window.The view window is used to display the virtual robot arm, and the display window is used to display the initial and destination coordinates of the virtual robot arm.Rotation Angle of each joint; Terminal attitude; Select a serial port; Control instructions.The display results of joint rotation Angle and terminal attitude of the virtual robot arm are completed by the kinematics simulation module. [9]

Conclusion
Human pose estimation using deep learning: review, methodologies, progress and future research directions [10] The core of "intelligent robot arm based on attitude recognition" is a robot arm that can imitate human hand movements 1:1. [11]Because we use 3D reconstruction algorithms, the robot arm has the characteristics of simple manipulation, high sensitivity, high accuracy and low cost.This product is a good combination of theory and application, with the robot arm as the core, to create diversified products, for different age groups, to adopt different application scenarios, so as to achieve a broad market, diverse applications, and make the product more competitive.The master-slave mapping algorithm can solve the difference of human arm well.This system realizes objective and accurate statistics. [12]Through the robot teleoperation system, the operator can achieve the expectation that robots can replace people to complete tasks in complex scenarios and extreme situations, bringing more possibilities to solve some difficult problems in the past, such as: deep sea, outer space exploration, high-risk experiment environment.Human pose estimation and action recognition are fundamental tasks in computer vision [13]

Figure 1 .
Figure 1.Structure of a typical master/slave teleoperation system.The specific research of this system includes three parts: the use of motion capture system and robotic arm system, the research of master-slave human motion mapping method, experimental verification and application analysis.

Figure 2 .
Figure 2. Wear mode.After the user wears the device, the working movement is carried out.The sensor device will capture the movement data of the target object, including the attitude and orientation of the body part, and transmit it to the data processing equipment through the data transmission equipment.After data correction and processing, the three-dimensional model is finally established, and the three-dimensional model is truly and naturally moving with the moving object.[2] In this project, the Axis Neuron software developed by Noytium will be used to export the skeletal motion data -intermediate formula.Intermediate data is the output method that is closest to the original data of the IMU sensor.In fullbody wear mode, the intermediate data in each frame contains all the sensor data and motion data of the 21 bones and the contact status of the feet.For intermediate data, each bone contains 16 float data, which are: Three position data, three velocity data , four quaternion data , three acceleration data , and three data for the gyroscope.[3]

Figure 3 .
Figure 3. Joint and structure of the UR3 robot arm.