Role of Operator Training Simulator (OTS) in Capability Building towards the Fourth Industrial Revolution (IR 4.0)

The manufacturing industry is now shifting towards Industrial Revolution 4.0 (IR 4.0), in which digitization of manufacturing industry has led the industry towards autonomous system driven by big data as well as machine learning. Despite shifting towards fully automated systems, human intervention is required every now and then especially during unforeseen circumstances. Hence the capability building of the plant personnel as well as technical specialist is inevitable, especially when related to plant safety. The use of Operator Training Simulator (OTS) has been in the industry for quite some time for plant personnel capability development. OTS is a system which combines chemical engineering design and knowledge (embedded in a dynamics simulation model) and process control and safety instrumented system (embedded in Distributed Control System (DCS) database), which can be used for plant personnel’s capability development which serves as the Digital Plant. The usage of OTS system is a new way of learning with the realistic feeling of operating the real plant via the Digital Plant without endangering the plant and personnel. By using OTS system, the plant personnel will be able to understand the process control system installed in the plant as well as process interaction between units in the plant since the Digital Plant was developed as close as the real plant to provide the near-real operation experiences. The OTS system is equipped with many functionalities such as malfunction tests, scenario tests, etc. to assess the plant personnel capability in handling unforeseen circumstances such as critical equipment failures and abnormal plant operation. The system also provides capability assessment for each personnel involves in the training which can be used to evaluate the personnel technical and operating capability, and to re-evaluate the steps taken during the training to improve teaching and learning approaches. By integrating chemical engineering knowledge, plant safety instrumented system and capability development program, OTS system will be able to bring capability development program towards achieving safe and reliable IR 4.0. implementation.


Introduction
Capability building for plant personnel and technical specialist is inevitable even though the industry in now moving towards fully automated systems. This is due to fast and accurate human intervention is required every now and then during unforeseen circumstances. Research shows human errors normally occur due to time constraint and performance demands which results in cognitive overload [1]. Plant personnel are expected to respond and restore the plant condition to normal operating condition, at the same time they are also required to minimize the risk and hazard exposure to people, environment, assets and company's reputation. To address these pressing issues, the Oil and Gas Industry traditionally has 26th Regional Symposium on Chemical Engineering (RSCE 2019) IOP Conf. Series: Materials Science and Engineering 778 (2020) 012044 IOP Publishing doi:10.1088/1757-899X/778/1/012044 2 been utilizing Operator Training Simulator (OTS) systems to prepare the operation personnel with adequate capabilities to handle planned and unplanned process conditions.
OTS system (also known as Digital Plant) is divided into two (2) types which are i) Emulated system (partial stimulation), and ii) Direct Connect (full stimulation). The purpose for both systems are the same but the system architecture for Human Machine Interface (HMI) and control function which makes the two systems different with each other. Both systems use dynamics simulation model to represent the real plant process behavior by incorporating equipment configuration, process routing, instrumentation equipment placement, and etc. Emulated OTS HMI is a replica of the actual Distributed Control System (DCS) graphics, DCS controller configuration and algorithm. Safety Instrumented System (SIS) for emulated system is also a replica of the actual SIS installed and the configuration might be simplified depending on the project requirement. On the other hand, direct connect OTS integrates the dynamic simulation model with the actual DCS and SIS database which includes all the logics permissive which requires more signal from the model to be exposed for the direct connect OTS HMI. Depending on End User requirement, the OTS Project Team could opt for any one of the OTS system as long as it meets the purpose of plant personnel capability development.
OTS has been in the industry for quite some time for many types of industries such as electrical sector [1], power plant sector [2], and this paper will present the application of OTS in oil and gas industry. Where risk and hazard exposure is more severe, the application of OTS as one of capability development tool is more prominent as it enables plant personnel to interact via the Digital Plant without jeopardizing the actual plant operation and performance. The usage of OTS system is a new way of learning with the real feeling of operating the real plant i.e. the Digital Plant without really endangering the plant and the personnel. Some of the applications for OTS are new process familiarization, testing and validating plant startup and shutdown procedures, equipment and instrumentation malfunction training, as well as "what-if" scenario training. The system is also able to capture the plant personnel performance during the training such as the time taken to startup/shutdown the plant and the steps taken to restore the plant to normal condition. This can be utilized to select only competent personnel to operate the plant to ensure safety and reliability. This paper is structured as follows. Section 2 presents brief introduction to OTS system architecture. Section 3 presents the OTS system development, which will describe OTS system development project phases whereas Section 4 discussed the application of OTS system as one of the tools for plant personnel capability development. Section 5 will describe detail on OTS system development lesson learnt captured from a recent development for oil and gas industry, which OTS is developed for Plant A using both emulated and direct connect OTS system to expedite plant personnel capability development. Finally, Section 6 will present the conclusions and recommendation for future work related to OTS development and application in supporting 4 th Industrial Revolution (IR 4.0)

OTS Architecture
Operator Training Simulator (OTS) system consists of several components that serve specific functionalities. Typical components that are installed in each OTS system are: a. Plant Model -this component serves to model the plant equipment and processes in the virtual world, which is constructed using dynamic simulation engine. This is considered the "Heart" of OTS as the quality of training sessions is highly dependent on the accuracy of the Plant Model replicating the actual plant process behaviors. "High Fidelity" Plant Model is developed using dynamic simulation engines that use First Principle equations coupled with extensive Thermodynamic Property Package database to allow realistic reproduction of plant process behaviors. b. Control System -this component serves to replicate the plant control and safety system configuration which will interact with the Plant Model component. The control system covers both the regulatory controls from simple Single-Loop control to complicated Advance Regulatory Control and Sequence Control. In addition, the control system also covers the safety system logics as part of Instrumented Protective Function replication. OTS system architecture can be classified as Direct Connect/Stimulated System or Emulated System. Direct Connect System architecture utilizes the actual plant control system and safety system hardware and software which are coupled with the OTS dynamic simulation engine. The trainees will be exposed to the actual plant Distributed Control System (DCS) and Safety Instrumented System (SIS) configuration, look and feel for more effective transition to the actual plant system. While the Direct Connect system provides the most complete training system for the operators, the OTS system requires a working DCS and SIS databases to enable complete OTS system configuration. This is an issue for Greenfield projects in which the DCS and SIS databases are still under development during the Detail Engineering phase, hence will not be available for the OTS project to utilize.
Emulated System is an alternative to the Direct Connect System in which the DCS and SIS functionalities are replicated in the Plant Model dynamic simulation engine itself. This would allow the OTS project to proceed without the availability of completed DCS and SIS databases, hence would allow faster OTS system completion for training usage. Emulated System is useful to provide training on the plant process configuration and equipment behavior, as well as general overview on the plant control system. Typically, Emulated System is used as a "Bridging" system until the Direct Connect system project is completed.
Both the Direct Connect and Emulated OTS systems are configured in "Island Mode", which means that all associated OTS components are connected within a dedicated Local Area Network (LAN). Therefore, actions by the Trainees on the OTS System will not have any unwanted impact to the actual plant DCS. Thus, the training syllabus can be designed to cover all possible situations that can happen in the actual plant.

OTS Development
The OTS System development typically last from 6 months to 2 years depending on the size and complexity of the plant process and control system. The typical development phases are as per Figure  User Training

Project Scope Definition and Data Gathering
Project Scope Definition and Data Gathering covers the activity of defining the OTS system architecture that will cover both hardware and software (i.e. plant model) configuration. Hardware configuration will depend on OTS system platform chosen, number of Operator Stations required and Direct Connect or Emulated System selected. Software configuration will depend on the plant process technology, equipment configuration and the training features required. Data gathering will be done on the plant process technology, equipment and control system design as well as operating procedures to define the software configuration especially the dynamic plant model. This is crucial to ensure the plant model will replicate the actual plant transient behavior as accurate as possible as this will determine the effectiveness of the training session for the trainees.

Functional Design Specification Development
The Functional Design Specification is a main reference document that represents the OTS System basis of design. The document contains mainly the finalized hardware and software configuration, plant process description, major equipment modeled, dynamic simulation model boundary (Battery Limit), model accuracy level and key parameters to be matched. This document will serve as the main reference throughout the OTS project execution phases to ensure the OTS System developed will meet all the required objectives.

System Engineering
System Engineering is where the actual development of the OTS system takes place. Majority of the time required for this phase will be spent on the OTS Software. This is where the heart of the OTS system (i.e. the plant dynamic process model) will be developed. The development of the dynamic model will require combined knowledge of Chemical Engineering fundamentals (e.g. Heat and Mass Balance, Thermodynamic, Heat and Mass Transfer, Separation Process etc.), equipment configuration and performance (e.g. how a pump works and design specification applied etc.), control system configuration and performance (e.g. how a cascade control is configured and the expected control response etc.), and plant operation (e.g. how the process/equipment is started up, emergency condition procedures etc.).
For the OTS Hardware, required workstations, servers, monitors and keyboards will be procured and hooked up using LAN. Communication between all stations will be tested to ensure connectivity. Once both hardware and software development are completed, both portions will be integrated into a single system and testing will be performed to ensure all scope and agreed functionalities as per the Functional Design Specification document are met.

Acceptance Testing
Acceptance Testing phase covers Model Acceptance Test (MAT), Factory Acceptance Test (FAT) and Site Acceptance Test (SAT) activities. MAT purposes are to evaluate the completeness of the plant dynamic model against the agreed model simulation boundaries, the accuracy of the simulated process parameters against reference data and the acceptability of the model transient response towards any process input changes. FAT purposes are to evaluate how the plant dynamic model interacts with the plant control system and safety interlock system as well as the model stability and accuracy during startup and shutdown operations. SAT purposes are to verify the hardware and software setup at site is as per design configuration and all FAT defective items have been rectified.

Initial Acceptance and User Training
Once the SAT has been successfully completed, the OTS System is handed over to the users as part of Initial Acceptance and the Warranty Period starts. Throughout the Warranty Period (normally it will last for one (1) year), any defect on the OTS System hardware and software will be rectified by the OTS System provider with no cost. In addition, to ensure effective system usability, user trainings will be conducted on several focus areas namely OTS Instructor, OTS Maintenance and OTS Engineering. OTS Instructor Training covers all the OTS System Instructor functionalities that are crucial for the Instructor/Trainer to conduct a successful training session. Among the crucial areas are how to set up training scenarios, how to set up student scoring system and how to activate certain process/equipment malfunctions. OTS Maintenance Training covers the aspect on maintaining the OTS System which includes system backup and cleanup of unnecessary working files. OTS Engineering Training provides technical know-how on the OTS Software functionalities to enable the OTS System Custodian to perform modification on the process dynamic models as well as how to update the DCS/SIS databases.

OTS Application
Traditionally, OTS is primarily used to train Panel Operators on how to operate the plant. However, with the latest advancement in the OTS software platform, DCS and SIS algorithms availability in software format and better computing hardware power, the application of OTS has been expanded to cover additional areas. Among the new areas where OTS system is being applied are:

Control System Function Check
•This activity enables the preliminary verification of the control system design effectiveness in controlling the plant process. Typically, the control system functionality can only be tested during plant commissioning and start up. Any control system functionality that does not work as intended need to be re-engineered, in which will cause delays in plant start up. By using the OTS System to perform the preliminary verification, any control system re-engineering work can be performed earlier before the plant commissioning phase, thus plant start up delays can be avoided.

Operating Procedures Validation
•

Benefits of Having an OTS in a Greenfield Project
An OTS system was developed for a greenfield project Plant A, a Sulfur Recovery Plant which consists of an Amine Regeneration Unit, a Sour Water Stripper Unit, and a Sulfur Recovery Unit. The Sulfur Recovery Unit is a highly automated plant which startup and shutdown sequence are programmed in DCS functionalities with minimal plant personnel intervention. The purpose of having a highly automated system is to ensure all the steps required during starting up and shutting down the plant are according to operating manual. It is also to reduce human error during the process which will leads to safety issues and plant incidents. The OTS system development was executed concurrent with the engineering phase of the overall project development, with the intention of using the system for plant personnel's capability building to be ready for plant startup. The project was executed in two (2) phases, which are Phase 1: Emulated OTS system, and Phase II: Direct Connect OTS system. The purpose of having both systems is to ensure the plant personnel is able to undergo training while waiting for the DCS/SIS system completion. Both Emulated and Direct Connect OTS system dynamic models were developed using Visual Modeler software by Omega Simulation Co. Ltd. to provide rigorous and high speed simulation [3]. DCS and SIS functionalities were emulated in Omegaland for Emulated system while for Direct Connect system, the DCS and SIS functionalities were using the actual database from CentumVP and Prosafe-RS.

Overall Process Understanding
Since Plant A is a greenfield project, most of the plant personnel are new to the overall process configuration and equipment sequencing. Hence, overall process familiarization is vital to ensure the newly hired plant personnel understand the purpose of each process unit as well as the equipment sequencing. By using OTS, plant personnel are able to familiarize with the process sequencing while waiting for the actual plant constructed. They will be able to understand the intention of the process unit sequencing and make sense of overall process objective. Prior to OTS system, plant personnel only have PFD, P&ID, and operating manual as reference document. OTS has merged all the documents into a system and become the digital plant for the plant personnel for further understanding of process behavior. With the assistance of Training Instructor, the plant personnel will have the opportunities to operate the digital plant for startup and shutdown process without having to worry about affecting the actual system. They can afford to make mistake and correct the mistake accordingly using OTS system and this will definitely increase their knowledge about the plant itself.

Operating Procedure Development and Validation
During Engineering phase in the project, plant personnel only have the operating manual on how to operate the equipment in each process unit. This document will provide overall operating philosophy of the process unit for high level understanding. To enable plant personnel to operate the plant to achieve normal conditions, they need to have operating procedure as the document will provide detail step-bystep procedure to operate a specific equipment. Plant A plant personnel took the opportunities during operator training session to develop operating procedure with the assistance of Training Instructor, who has more experiences about Plant A operations. At the end of the training session, standard operating procedures were developed and operating manual provided by contractor was validated.

Abnormal Process Troubleshooting
Many industrial accidents occur due to lack of training on automation and process equipment, instrumentation, safety procedures for operations and maintenance. Plant personnel need to be trained for emergency situations so that they will be able to respond to the process abnormalities and recover from the abnormal situation as fast as they could. One of the functionalities embedded in OTS is malfunction tests and scenario simulation. Some of the critical equipment malfunctions are programmed in OTS system to enable the plant personnel to experience the abnormalities during OTS training. The malfunctions will be activated by Training Instructor and the plant personnel will have to identify the issues as well as rectify the issues as soon as possible. As an example, a control valve is programmed as malfunctioned, could not be opened due to mechanical issues. If this program is activated, it will cause process haywire since the controller could not instruct the valve to open or close since it is stuck. Plant personnel need to identify the issue and open a bypass valve to prevent overall process trip. Other example of typical abnormality introduced in OTS is rotating equipment malfunction. Typically, process unit will have a standby rotating equipment in the event of the normally running equipment is malfunction. If this program is activated in OTS, plant personnel will have to start the standby equipment to ensure the overall process is not interrupted during the equipment malfunction. Using OTS, plant personnel will be trained on how to react during abnormal situations and subsequently bring the plant operation to normal operation.

Process Control and Safety System
Understanding IR 4.0 has shifted the industry towards autonomous system which reduces the human intervention during normal operations. Sulfur recovery unit recover H2S as elemental sulfur through combustion process which consists of high-performance burner, mixing chamber, and heat removing boiler (to recover the energy produced by high temperature combustion) [4]. The operation of high-performance burner need to be managed closely and accurately since it involves combustion which could lead to explosive conditions should the source of fire is not managed properly. Hence, Burner Management System (BMS) is introduced to assure safe startup, normal operation, and shutdown process of burners [5]. The system can monitor flames, manage ignitors, burners, as well as actuators. BMS also have the following functions such as inhibiting startup if the safe conditions are not met, monitor burner to detect unsafe operating conditions, as well as interlocks. Due to complicated setup of advanced controllers for a high-performance burner, the startup and shutdown phases are pre-configured in DCS/SIS system as "Startup sequence" and "shutdown sequence" so that the plant personnel does not have to memorize all the steps and related instrumentations involved in each sequence. By having a direct connect OTS system, the plant personnel will be able to test the sequence developed in the DCS/SIS for BMS startup and shutdown. From the process, the will be able to validate the procedure and check if the procedure is adequate to bring the plant to normal condition. The sequences are different between vendor/product licensor, even experienced plant personnel from similar process need to be trained adequately before operating the actual process plant. After a number of training sessions of starting up the equipment, plant personnel were able to identify what are the minimum requirement before each sequence is able to be activated. They were also able to identify why certain sequences does not behave according to the initial intent.

DCS and SIS System Validation
Phase II: Direct Connect OTS system was executed according to the following steps described in  As mentioned earlier, the OTS system development was for greenfield project and concurrent with actual project execution which its deliverables such as P&ID, DCS and SIS database directly affect the 10 timeline and schedule for OTS system development project. OTS system need to be integrated to the most accurate and latest DCS and SIS database hence the minimum requirement put was to use post DCS and SIS Factory Acceptance Test (FAT). Using an outdated database will result inaccuracies during FAT and more time required to re-integrate the DCS and SIS database to OTS system once the latest database is received. The most challenging step during OTS system development is Factory Acceptance Test (FAT) where the plant personnel need to test all startup procedure to bring the system to steady state condition prior to proceed with the next steps which are plant shutdown and malfunction test. During the activities, plant personnel will be able to validate the SOP provided by EPCC contractor and test if the SOP is adequate to bring the plant to steady state condition. One of the findings from the test are some of the controllers were configured in wrong direction which hinder the process of starting up the plant. There are also missing connections between DCS and SIS database itself which also prevent some equipment from starting up or valves could not be opened/closed. There are also some logics configured in the DCS system preventing some of the equipment to be starting up. Even though there many discrepancies found during the testing, the findings can be used to rectify the actual system at site as this may cause the same problem when the actual plant undergo startup process. This is one of the advantages of OTS system development being executed concurrent with the actual plant design phase where OTS can be used as a validation tool for DCS and SIS design configurations.

Conclusion
OTS is proven to be used to reduce the occurrence of human errors during industrial system operation based on the lesson learnt from Plant A OTS system development and application. It brings benefit for greenfield project during the system development which the OTS system is used as a validation tool for DCS and SIS design and configuration. As for the OTS application, with a proper capability development program, OTS also is able to be used as the training tool for new plant personnel as well as experienced plant personnel. By using OTS, the project team is able to identify tasks bottleneck, which slow down the personnel's performance and subsequently reduce the recovery time after incidents.