Research on 3D Visualization Simulation System of Deck Crew Based on OpenGL Technology

Based on OpenGL technology, a SGR simulation system which including virtual human operation was designed. The motion control of the virtual human was realized by using local finite state machine under the action of the control, and obstacle avoidance method for the virtual human was realized. Compared with the simulation programs developed with Vega Prime, this simulation speed obviously has advantages and the requirements on the computer are also lower.


Introduction
Modeling and simulation (M&S) technology is an important method to study aircraft sorties generation rate(SGR) and is being used to verify the design specifications in carrier design [1] . Some scholars have studied the calculating methods of SGR for a long time, however, most of those research focus on the optimization of operation sequences [2][3] . Because almost all aircraft operations on the deck, including the aircraft movement, aircraft maintenance and ordnance handing, etc., need man to join in, crew play a crucial role to the SGR [4] . There are hundreds of men working on flight deck at the same time. How the crew work efficiently and safely is a question for us to study. The scholars of MIT，University of Montreal and University of Pennsylvania use M&S to study the question of flight deck operation optimization and developed a simulation system for flight deck monitoring and action scheduling, Deck operations Course of Action Planner (DCAP) [5] . In this simulation system, flight deck management, aircraft task planning, aircraft maintenance management, and crew management were integrated together. In DCAP various resources on the flight deck were displayed in two-dimensional shapes, and each man is represented by one circle and is categorized in different colours according to the man's work type.
2D method for display crew on flight deck can only show the position of crew without detail action. In order to display detail action of crews on flight deck in the simulation system, including crew's hands action, body action etc., a 3D simulation system is necessary. In recent decades, virtual human technology is widely used in product design and manufacture, Human Modeling and Simulation Center (HMS) of University of Pennsylvania developed a virtual human dynamic simulation system that provides a solution of simulating the behaviours of human beings [6] . The LIG lab in University of Montreal focuses on body modeling of deformation, walking and grasping [7] .
The technology used widely recent to develop 3D simulations includes OpenGL, OSG, Unity3D and Vega Prime, etc... The 3D virtual human simulation system can be developed based on Di-Guy plug-in in Vega Prime. However, this simulation system requires high performance workstation and cannot run quickly. In our simulation system of calculating SGR we need to simulate 4 days SGR in less one hour, OpenGL is adopted to realize the 3D virtual human-deck simulation system for SGR.

Design of the virtual human-deck simulation system
On the aircraft carrier, there is a series of operations to complete for the aircraft before taking off. Fig.  1 shows the workflow of a single aircraft on flight deck [8] .
Before carrier-based aircraft taking off, operations show on above must be completed orderly and timely. For example, the aircraft needs to unhook the tail hooks by crew when it enters the blocking cable area; it needs to unload ordnance when taxing into the ordnance unloading; during towing the aircraft guide crews need to guide directions and so on. Men are involved in almost all operations processes. Almost 400 men were involved in flight deck operations in the Nimitz aircraft carrier [9] . Usually the crew is divided into the several different categories: aircraft commander, maintenance worker, ordnance handler, oiler, helicopter commander, pilots and others. According to the requirements of aircraft carrier, the men of each categories of human are divided into several groups to meet the maintenance requirements of each aircraft.

FIGURE 1. SINGLE AIRCRAFT'S WORKFLOW ON FLIGHT DECK
In order to study the influence to the SGR of the crew, a sub-simulation system for crew is added to the SGR simulation system. The frame work of simulation system is shown in Fig. 2. And the section shown in dashed box is for virtual human simulation. The virtual human sub-simulation system interacts with other modules in the SGR through the High-Level Architecture (HLA).
Database management model is mainly responsible for receiving and dispatching related data of the system. HLA/pRti provides a basic environment for models exchanging information and data. Take-off and landing control module is used to execute the processes of aircraft launching and landing and assignment allocation of aircrafts. Aircraft transfer module is mainly responsible for aircraft operations planning and dispatching. Tractor motion control module based on the aircraft operations to assist the aircraft scheduling. Assurance operations planning module is mainly responsible for fuel, missile and other security operations planning and implementation. Operation information two-dimensional display module displays of resources and job-related information.  Virtual human deck simulation system mainly includes task information interaction module, task assignment module, virtual human operation setting module, virtual human control module, human task storage and status update module, and visualization module, visualization module Based on OpenGL, etc.

Modeling and gesture control
Complete human skeleton is a very complex system consisting of more than 200 rotating joints that are tightly and complexly connected. However, since the aircraft carrier has a relatively large deck and there are a large number of crew, this paper aims to develop a 3D virtual environment for simulate the crew's work on flight deck. Therefore, cuboids and cylinders are used to represent virtual human bones and joints. The skeleton of virtual human is abstracted as a rigid body, its length and shape are fixed. And pivots are used to represent the joints that with one rotational degree of freedom, such as elbows and knees. Spheres are used to represent joints with two or three rotational degrees of freedom, such as the femoral and shoulder joints. By changing the angle between each adjacent pair of bones, the virtual human shows different actions. Each human in our system consists of 6 cuboids and 12 cylinders. The structure of virtual human using the OpenGL library to draw the virtual human geometry is shown in Fig. 3. Virtual human's action is controlled by finite state machine. The basic action can be produced easier. Firstly, multiple of actions for the virtual human is created and each action include two sub-action: "starting" and "ending ". During each action, the virtual human is given a limited number of key conditions. Each key state records the rotation angle of each joint, performance is the state changes for each bone. When the virtual human executes the action or ends the action, it changes its own state among the key states according to the specified state change direction.

FIGURE 3. THE STRUCTURE OF VIRTUAL HUMAN
For example, the action finite state machine of the commander for aircraft taking-off includes the limited status as shown in Fig. 4 Initial state   5 shows a limited status example of a command taking-off action. As shown, the takeoff action can be divided into five key sequence states. When the action is "start ", the attitude is changed according to the direction of the solid arrow, and when the virtual human changes to the fourth state, the loop changes between the fourth and the fifth states. The direction of the dotted line arrow is the state change process when the action is "ending the command action", and in this direction, the virtual human finally returns to the initial state.

The organization of virtual human
In virtual environment, crew consists of multiple groups, and each group has its own behavior patterns. The group consists of a number of individuals with similar characteristics and similar ability. The behavior patterns among different individuals are generally similar, but maybe not same. The methods to controlling the virtual crowd behavior have three kinds, particle system control, cluster behavior control, hierarchical group control and agent-based control.
The behavior of a group in a virtual crowd is generally determined by the task of the group. As aircraft carrier deck case, the typical behaviors of the virtual group include: focusing on, perceiving, avoiding collision, responding and interacting. Because of take-off and landing frequently, aircraft carrier deck is noisy, and staff cannot use language to communication with each other. The staff conveys information through gestures. In reference of American aircraft carrier, this paper uses colors Each type of human in the table is divided into a number of sub-groups, and the relevant operations are completed according to the instructions from the SGR simulation system.

Obstacle avoidance
During simulation, the virtual human need to arrive at the target station and to complete the corresponding task on time. However, during walking, the virtual human may encounter a series of obstacles such as the aircraft, tractors, other virtual people, etc.，and its movement should be limited to a reasonable range. It is important to calculate a path to avoid obstacles. The main aspects of the virtual human path planning are following three ones: 1) the virtual human can reach the target point from the starting point within a specified time; 2) the virtual human cannot collide with other objects; 3) the path length should be optimized as short as possible.
Therefore, the virtual human path planning in this system uses the algorithm based on behavior. First, according to obstacles that the virtual human may encounter during walking, the behavior in our research is divided into four small units of sub-behavior: going to the target, avoiding the aircrafts, avoiding the tractors, and avoiding other virtual human. Based on the distance between the obstacle and the virtual person, the distance between the virtual person and the target point, the angle between the direction of the virtual human and the target point, an algorithm based on fuzzy logics is designed to produce walking angle and step of the virtual person an present position.   Fig. 6 shows the take-off operation. The relevant human in Fig.6(a) is guiding the aircraft moving to taking-off position, and Fig.6(b) is commanding the plane to take-off.

Examples of simulation
Landing operations is shown in Fig. 7. There is an aircraft just stopped and is in the process of unblocking the rope.

FIGURE 7. LANDING CASE
The aircraft on the deck move from one operating point to another, requiring tractor assistance to complete the towing work. The towing process required a tractor driver and a transfer conductor.

Conclusions
Compared with the software and platform that can be used for virtual human simulation, this paper selected OpenGL technology to develop a virtual human simulation system for SGR process. Based on OpenGL technology, a simulation system which including virtual human operation was designed. The motion control of the virtual human was realized by local finite state machine under the action of the control, and obstacle avoidance method to the virtual human was realized. Compared with the simulation programs developed with Vega Prime, this simulation speed obviously has advantages and the requirements on the computer are also less.