Use Case Remote Pilotage – Technology Overview

Starting from a short study on pilotage transfers in terms of safety and sustainability as a motivation, this paper will draft trust requirements from pilots and navigators to remote based systems, before the capabilities of Extended Reality (XR) for these requirements are highlighted. This is followed by outlining the technical approach used within Sea4Value Fairway (S4VF) project and Remote Pilotage – Operational, innovative and Manageable Alternatives for Navigation routines (RePO MAN). The technical implementation within the context of the the use case method is explained and a consolidation of these use cases within the framework of RePO MAN and the possibility of adaptation in further field tests up to implementation is considered. Finally, limitations and deficits of remote pilot concepts will be outlined using the example of RePO MAN merged with the results of the S4VF project.


Introduction
Pilotage transfer is a dangerous task, especially in harsh weather.However, pilotage is also needed during this critical phase of navigation to ensure safe approaches and transit in restricted waters.In general, there are four ways for maritime pilots to enter a vessel: (i) They can use the pilot ladder, which is fixed to the vessels' deck and goes along the hull.(ii) They use a combination of accommodation ladder and pilot ladder.The two of which are regulated in the Pilot Transfer Arrangements [1].(iii) They can further enter through a side door located close to the water line, or (iv) They board by flying in with a helicopter, if a landing place is available [2].
Of the four the helicopter is the safest choice for the pilot but not always available [3].A variety of incidents and accidents, especially related to improper pilotage ladder equipment have been reported over the last years, including severe injuries and death of pilots.Furthermore, pilot transfer -regardless of transfer by pilot vessel or helicopter -is generally fossil-fueled and therefore has an impact on maritime transport sustainability.
Meanwhile, modern technology for shore-based situational awareness is progressing quickly, parallel to improved ship-shore-communication links.While the need to support the onboard crew by pilot expertise is indisputable, it can be discussed if a personnel presence of the pilot onboard is still needed to execute this service, given the personnel risk each pilot is encountering during the transfer process.Thus, modern technology might open a fifth and safe way for a maritime pilot to enter the vessel: Digital entering by Remote Pilotage Technologies.
Based on a brief analysis of the societal rationale for remote pilotage in terms of safety and sustainability in Section 2, Section 3 will introduce the use case of remote pilotage and compare it with existing methods and operations.This is followed by a technical insight into two remote pilotage approaches in Sections 4 and 5, which are Mixed Reality (MR) based and desktop based.MR represents a specific sub-type that fuses real and digital objects to create a hybrid environment where the objects interact in real time.From the comparison of the two systems and the efforts in the projects to locate the onboard sensors as well as the situation awareness in the different concepts, a possible future approach and realistic considerations for necessary further work will finally be discussed in Section 8.

Societal Rationale of Remote Pilotage
The effects of remote pilotage on safety have not yet been studied intensively.However, based on a few published studies, remote pilotage is expected to have benefits in addition to the improvement of the pilots' work safety [4], [5].These include, among others, improved navigation safety, improved pilot utilization rate and reduced costs of pilotage support services [4].In addition, fuel savings and thus the improvement in sustainability as well as the efficiency of the logistic chain must be viewed separately, e.g., the ship no longer has to wait for the pilot to arrive and the pilot boarding the ship at sea no longer slows down the ship's voyage.This might even result in efficiency gains on port call costs.According to Danish Maritime Authority (DMA) remote pilotage has a potential of 55 -60 % reduction to direct costs of maritime pilotage [6], but this will not be further investigated in this study.
As the remote pilotage service requires more advanced technology [7], safety of navigation has been seen to benefit from that development [5].The technology can reduce human error by the pilot [4], [5].Also, capability of the vessel's crew to detect risks and maintain their situational awareness with the help of more advanced systems can contribute to improving the safety of fairway navigation.On the other hand, another desktop study expects that since the pilot has a significant controlling responsibility of the socio-technical system, removing the pilot from the vessel might cause fundamental problems [8].Change in roles and working practices are expected to be extensively, and this needs to be taken into consideration when developing new remote pilotage systems [7].

Pilotage Safety
The first year of 2023 already showed several severe accidents during pilot transfer in the Europe, with two fatalities of a British Pilot in Humber and a Dutch Pilot in IJmuiden as well as a seriously injured German pilot in the Ems mouth.International safety statistics are not directly accessible, but for the EU a total of 124 persons were killed in maritime accidents within the years 2017 to 2021 of which five include maritime pilots [9].This shows the significant risk of pilot transfer arrangements for worker's safety.The pilot ladder is the most common way for a pilot to go onboard of a vessel and often subject to a non-compliant boarding of the pilot [10].In a report the International Maritime Pilots' Association (IMPA) evaluated a total of 4664 pilot returns regarding their compliance.Table 1 groups the numbers by geographical area and IMPA members from Europe claim almost twenty per cent of all returns were noncompliant.Thus, if pilotage and support to the master could be executed in a safe and efficient manner remotely, would this pose a significant effect for improved safety of workers with the risky transfer arrangement taken out of the equation.

Fuel Oil Consumption Estimation Methodology
A list of pilot vessel models, which seem to be the most common pilot vessel shape in the geographical area of interest, and their principal dimensions, displacement and maximum velocity was created.Fuel Oil consumption (FOC) has been estimated on displacement (average value 28.25 tons), lengths, (average value 17.28 m), Froude numbers (average value 1.03), and slenderness ratio of 5.0 for each pilot vessels.In the next step ratios of minimum bare hull resistance to vessel displacement from [11] were scaled from the notional 500 t size and reference length 43.3 m to the size of each individual vessel.Total resistance was determined by multiplying the displacement of the vessel with the provided resistance to displacement ratios for Froude numbers between 0.4 and 1.0.The total resistance values were then divided by the density halved and velocity squared to arrive at a list containing the products of resistance coefficient and reference area, called resistance area, for each Froude number.
The total resistance values were then multiplied with the velocity and divided by an estimated efficiency of 0.65 to determine the required engine power from which the FOC was determined through multiplication with a specific fuel oil consumption of 220g/kWh.Given the speed profile of those vessels presented in Table A1, which shows the speed distribution of the pilot vessels in the corresponding area for year 2019, as well as an estimated average fuel consumption as a function of speed, this results in an estimated FOC for pilot vessels transfers of more than 2,500 t per year.

Pilotage Sustainability
Commonly, pilot boats and ladders are used to bring pilots onboard within Europe.Besides modern pilot boats, those are usually diesel-fueled and operate at high speeds to bring the pilots to and away from the vessels.This comes with respective emissions, which are shortly quantified here based on Automatic Identification System (AIS) data for the waters around Denmark, including the German coastlines, southern Norway and the Swedish west coast.
To conduct the assessment, the following data acquired from Danish Maritime Authority [6] has been used: • All AIS data available at DMA for 2019 • All AIS data related to ship type "Pilot Vessel" • All AIS data from ships related to a Danish, Swedish, Norwegian or German Maritime Identification Digits (MID) Overall, this sample covered approximately 150 vessels with around 60 AIS reports, as depicted in Figure 1 and according values presented in Table 2. Using 3,11 t CO 2 per t diesel, this is equivalent to 8,051 t of CO 2 emission annually in that area (which corresponds to approximately 5,000 cars).Even though safety being the main motivator for remote pilotage, its potential effects on reducing emissions as a side effect should be noted.

Pilotage in Comparison to Remote Pilotage
In the context of the aforementioned difficulties of the pilot to enter the ship, the purpose of pilotage, the accompaniment of a local navigator in special sea areas, should be explained.In difficult to access and crucial sea areas but increasingly in the vicinity of harbors and rivers, a local nautical expert is placed at the service of the captain in charge.This navigator, as the pilot of the ship, works out a voyage plan for handling the situation ahead, in close cooperation with the captain as part of the Master Pilot Exchange (MPX) [12].The general procedure leads from an inventory analysis of the existing infrastructure and its functionality to a joint assessment of the external influences and the targeted positioning of the ship in the various portions of the voyage.Depending on the sea area, this includes the use of external support but also upcoming communication with shore-based infrastructure.Within the framework of this initial discussion, the corresponding mutual trust is formed on both sides to bring the ship, persons on board and cargo safely to the destination by means of the previously negotiated necessary maneuvers.This describes the core process of piloting the ship.However, it is only a part of the pilot's work process.After a prior notification of the vessel to be piloted in an adequate period of time (usually 1-2 days), the static data of the vessel and the planned final berthing location are known.As soon as the ship is close to the pilot's boarding position, the pilot can begin boarding the ship with the help of a selected and available boarding instrument.Once on board, with the MPX, the main part of the procedure begins.Upon completion of the activity, the pilot will leave the ship again.The period of the work process and the boarding procedure, which is not directly related to the pilotage of the ship on board, therefore usually takes the larger part.
This already describes the essential difference of a remote operation.The way to and away from the place of the operation would be saved.Furthermore, a connection to shore-based data would be facilitated, necessary support for the pilot in terms of decision making could be ensured.Since the pilot is physically not on board, the personal contact with the captain the impression and the use of the onboard infrastructure would be omitted.However, the data provided onboard (position data, radar data, engine data, voice communications, and visual sensory data) would be made available for shore-based assessment.

Use Case Description for Remote Pilotage Procedures
During the act of a remote pilotage a trusting relationship between pilot and captain, as well as other psychological elements are crucial.In addition to the reliability of the technical equipment, the seamless connectivity and the meaningful presentation of the navigational data are key.
The analysis in the form of the use cases method [13] for the various actors and functionalities in the pilotage process as well as their communication relationship serves the necessary creation of goals and, in its entirety, the transparent representation of the process.
The information necessary for this was collected in interviews with pilots and captains.Within the Remote Pilotage -Operational, innovative and Manageable Alternatives for Navigation routines (RePO MAN) project, the most basic use cases were realized.These use purely the key players captain and pilot, including their communication process about the positioning of the ship and the related necessary functionalities.Processes involving other parties such as tug, Vessel Traffic Services (VTS), port captain or the crew on board the vessel were not considered as part of the Minimum Viable Product (MVP).
Among the basic functionalities and essentials for an understandable communication between pilot and captain is the matching of dynamic and static vessel data based on an Automatic Radar Plotting Aids (ARPA), AIS and Global Positioning System (GPS) data.Furthermore, the future route of the vessel is shared on the known medium of Electronic Navigational Chart (ENC), as well as the movement of the own ship and the visualization of all participating ships in the traffic situation.The processes of consultation are thus tied to known systems.A direct access to sharing upcoming movement data is the unified understanding of the future route.Via the modification of this functionality (changing way points, determining speeds, rudder angles and starting times), the mutual acceptance of the change and the communicated and controlled implementation of the proposal, the future route becomes an important part of the process.
In the course of the project, the Use Case 2.0 method [14] proved to be the more effective method for a holistic view of the pilotage process with all its actors and communication relationships.Further analysis will be prepared and made available to the project in a later step.

Trust Requirements from Pilots and Navigators
Trust between at least two people is considered to be a combination of goodwill, competence, integrity and predictability.Furthermore, it is a reliable cooperation of the people involved in the process in relation to a risky situation.[15] The necessary trust with the pilot as well as with the captain is to be built up in the context of the MPX.Since this step is obviously missing in the remote pilotage maneuver, interactions and processes have to be created that provide confidence-building measures.During the implementation of the RePO MAN project interviews were conducted in close cooperation with maritime experts to discuss the necessary conditions for a trust-building MPX and the basic steps to build a robust professional base for piloting the vessel.As a result of these efforts, the terms experience, transparency and honesty as well as verbal communication and non-verbal communication have repeatedly been mentioned.
The RePO MAN project therefore tries to integrate these features into a remote control system.The aspects of experience, transparency and honesty can also be built up in the implementation in a good planning phase, visual representation of the quality of the connectivity and extensive communication in advance.The change in communication structure during the maneuvers due to the physical separation of the two operators was also seen as an important component of a trusting relationship in the future.

Sensory loss during remote pilotage
Sensory loss in the case of Remotely Operated Vehicles (ROVs) has been well documented, particularly for unmanned aerial and ground vehicles [16].This is specifically true for kinetic and tactile perception which are not available when the pilot is not present on an operated vessel.This includes also senses for heave, pitch and vibrations of the vessel.This impedes the ability to maneuver, as those senses have been attributed to gain the "ship sense" required to safely handle a ship [17].Further, the communication and shared situational awareness between the pilot and the masters is more challenging.
Thus, remote pilotage solutions should aim to maximize this situational awareness by keeping the experienced sensory loss as low as reasonable possible and eventually compensating parts of that loss by other technological means, as e.g., MR technologies, which have shown immersive shore-based experiences for remote control beforehand.[18], [19] Achieving this has thus been the focus of the two technology demonstrations Sea4Value Fairway (S4VF) and RePO MAN, which addressed that topic by desktop and MR-based human machine interfaces, respectively.

Sea4Value Fairway
The Finnish S4VF project brought together several companies as well as universities and public actors to study the development of intelligent fairway concepts and the digitization of fairway services such as remote pilotage.Concepts, technological solutions, and several other aspects related to remote pilotage were studied and technological experiments were conducted.This public-private partnership project was implemented during the years 2020-2022.The main aim of the S4VF project was to take steps towards future autonomous maritime transport system.This was achieved by creating a model for safe, sustainable, and customer-centric of fairway navigation and a decision-making environment that benefits existing fleets as well as future autonomous vessels.The program searched for combining suitable technologies, practices, and operations to enable future fairway services, such as increased accuracy and cyber-security to positioning, as well as enabling services like remote pilotage.[

Technical Overview
The demonstrated remote pilotage system consists of a workstation where the remote pilot can monitor the vessel from various displays and communicate with the vessel or the personnel on board.Data and videos are streamed directly from the vessel to the workstation.The remote pilot communicates with the vessel's crew over an industry standard meeting application or Voice-Over-Internet Protocol (VoIP).
On the ship side, navigational data from the ship being piloted was collected via International Electrotechnical Commission (IEC) 61162-450 interface.The test vessel was hosting Furuno integrated navigation system, which Furuno operates using the IEC 450 Local Area Network (LAN).Radar data was collected by taking screen shots from the radar computer video output at a slow frame rate and streaming it as a video to the remote pilotage workstation.[21]

Ship Data Connection
The ship data connection was arranged with a special device named "Smartbox" that was connected to the bridge's LAN hub, see Figure 2. To ensure cyber security, a data diode, D, was employed to establish a read-only connection.The "Smartbox" was responsible for collecting, converting, and streaming all data to the cloud using the cellular network.The device reads the information exchanged between the bridge equipment and the Voyage Data Recorder (VDR).
The information is transmitted in standardized ASCII text format using National Marine Electronics Association (NMEA) messages, which are structured in sentences that convey specific details such as GPS position, speed, course, depth, and other relevant data.
The devices linked to the bridge LAN generate data in the form of NMEA sentences, which are plain text encapsulated messages.These sentences are then processed by the "Smartbox" data processing unit, which converts them into Java Script Object Notation (JSON) files.JSON is structured human readable data file used to exchange data between web clients and servers.The files are encrypted and transmitted over a 4G network to a cloud storage system.To ensure redundancy and maintain a reliable connection, two 4G routers were employed, each using a different network operator.
The available bandwidth was shared among various data streams, including navigation equipment data, two audio connections, radar video, and a live stream from a bridge video camera.[ Figure 3 illustrates the data connections between test vessel and remote pilotage center.Once the data was transmitted from the vessel, it was stored in JSON format on the cloud server.This stored data can then be accessed and forwarded to various terminal devices as needed.The remote pilotage workstation itself was connected to the internet via an access point, enabling communication and data exchange between the center and the vessel.

System Overview Shore-side
At the remote pilotage workstation, a computer was equipped with a set of displays handling data received from both the ship and web contents from the internet, as depicted in Figure 4.
In the central upper section of the computer, there are three large displays dedicated to the map view.This map view showcases various information related to vessel movements.On the right side of the map view, a list of target vessels is displayed, providing additional details and relevant information.The map display specifically presents all the AIS targets on the map, allowing for a comprehensive overview of vessel positions and movements.
On the left-hand side, there are two displays.The upper display provides detailed information about the ship, including its specifications and other relevant data.The lower display features a web browser with multiple tabs open, among others a port app, which generated dynamic safety contours based on ship and environmental data [21].
Additionally, there were tabs providing weather information and ship schedules of the port.Moving to the right side, the three displays from top to bottom are dedicated to fairway cameras.During the demonstration, one of the displays showcased a live forward view from the bridge of the vessel.
In the middle section, below the large displays, there are three displays.The top display shows radar video streamed from the cloud, using a video player.The next display features the Electronic Chart Display (ECD) that receives data directly from the ship.On the bottom display there is a tablet that provides pilot data specific to the vessel.The Electronic Chart Display and Information System (ECDIS) on the larger displays also shows AIS information of other vessels, which is received via the vessel's AIS receiver.

RePo MAN
In the project RePO MAN, a MVP is being developed and tested using Extended Reality (XR) technologies to fulfill pilotage requirements.The approach presented here differs from previous systems for shore side support of vessels and their navigators.The shore-based and ship-based XR technologies are further described in detail and their interaction is explained using two User Interface (UI).The Shore UI supports the pilot with a Virtual Reality (VR) system combined with hand tracking, provided with a live video feed originating from a 360degree camera installed on the ship.To enhance the immersive and interactive experience, the video information is combined with other sensory data, e.g., vessel information, traffic symbols, environmental information, or additional information on the expected route.
Onboard of the ship the Ship UI, an Augumented Reality Head-Mounted Display (AR HMD) is used to present the pilot's advice.The Ship UI allows to see a variety of visualizations, including the route ahead, pilot's instructions, information about the ship's position and movement, or environmental information visualized by the glasses in combination with what is visible to the navigator's eyes.For RePO MAN, the focus is on human-centered XR to create a support system that is user-friendly and intuitive.Without neglecting information necessary for the interaction between pilot and navigator.

Technical Overview
The technical system primarily integrates two applications, one on the ship and the other on shore.Aiding these applications are cutting-edge technologies, such as Augmented Reality (AR) and VR, and modern communication protocols like Message Queuing Telemetry Transport (MQTT).By integrating a myriad of technologies, the system enables pilots shore-side to access real-time ship data, facilitating informed decisions and guidance.The principle technical system is a derivative from the one used for MR based remote control of tugs beforehand.[18], [19]

Data Acquisition
Data acquisition onboard the ship is facilitated by a network of sensors communicating via the NMEA 0183 protocol, also see Figure 5. NMEA 0183 is a standard for interfacing marine electronic devices, such as GPS receivers, sonar, and radar, to facilitate data transfer between them.The protocol transmits data in human-readable ASCII sentences, which include information about the device's current state and measurements.
The different devices onboard the ship continuously emit these NMEA 0183 sentences, representing real-time data from the various sensors.A device called "multiplexer" is used to gather all these sentences.The "multiplexer's" function is to amalgamate the various data streams into a single stream that is then broadcast onto the local network.
The backend PC, integrated into this network, is set to listen to this stream through a oneway communication channel.Upon receiving the NMEA 0183 sentences, the application on the backend PC decodes or decrypts them into their actual form -converting the human-readable ASCII text into structured data that the software can interpret.
Finally, this interpreted data is stored in a database for real-time or later use.This method of data acquisition provides a continuous flow of up-to-date information from the ship's sensors, allowing for accurate monitoring and decision-making.

System Overview Ship-side
The ship-side application, right-hand side in Fig. 6, functioning within the environment of Microsoft's Hololens 2 AR glasses, provides an immersive and interactive interface to the ship's captain [23].The application's primary role is to collect data from a myriad of sensors and systems onboard the ship and feed it to the MQTT broker while also displaying essential information in the captain's field of vision, enhancing situational awareness.The AR glasses overlay crucial information, such as the ship's AIS data, onto the real-world view, creating an augmented perception for the captain.This feature is particularly advantageous in unfavorable conditions like fog or disturbed visibility, enabling the captain to see beyond what the naked eye can perceive.In addition to the environmental view, the AR application can also present dynamic route information in the captain's field of view.This interactive route planning mechanism facilitates real-time adaptations to changing maritime conditions.When the shoreside pilot updates the ship's route on the MQTT broker, the change is immediately conveyed to the ship-side application and visually represented in the AR environment.This not only saves time but also reduces the chance of miscommunication or misunderstanding of new route instructions.
In essence, the ship-side AR application forms a critical component of the remote pilotage system.By providing a comprehensive and enriched visual field and facilitating realtime information transfer, it significantly enhances the captain's navigational capabilities, contributing to safer and more efficient maritime operations.

System Overview Shore-side
The shore-side, left-hand side in Figure 6, application employs the power of VR technology to provide a rich, immersive, and data-intensive environment for the shore-based pilot.This application is designed to work with a VR headset and VR-capable hardware, creating a virtual replica of the ship's environment and facilitating remote navigation.The primary function of the shore-side application is to receive, process, and visually present the ship's data and environmental information in a way that closely mimics the actual ship environment.It does so by rendering a live 360-degree video stream, sourced from two 360-degree cameras installed on the ship's deck.This feature places the pilot virtually on the ship, despite being physically situated at the shore station.However, the shore-side application's capabilities extend beyond merely presenting a visual reconstruction.It integrates this visual data with essential ship and environmental data onto a ENC within the VR environment.This chart includes live AIS data, indicating the positions, directions, and speeds of other vessels in the vicinity, thereby providing the pilot with a comprehensive understanding of the situation at sea.Additionally, the shoreside application provides various VR panels, displaying essential ship information such as speed, direction, engine status, and other information like wind speed.This consolidation of visual and informational data within the VR environment facilitates informed decision-making by the pilot.Moreover, the shore-side application allows the pilot to adjust the ship's route.When a new route is published onto the MQTT broker, it's reflected immediately on the ship-side application, enabling swift and accurate route adjustments.
In summary, the shore-side VR application forms an integral part of the remote pilotage system, enabling remote navigation while fostering safety and operational efficiency.Through its immersive environment and data-rich interface, it ensures that the shore-based pilot can make informed decisions based on a comprehensive and accurate understanding of the ship's situation.

Unity as Foundation for Virtual and Augmented Reality
Unity is a cross-platform game engine widely used in the development of immersive digital experiences such as AR, VR, and MR. 1 A variety of tools and features, enables both the shipside and shore-side applications in the remote pilotage system.The Application Programming Interface (API) allows to create complex interactive behaviors, enabling real-time reactions to user actions and system events.[24] On the ship-side application, Unity in combination with Microsoft's Mixed Reality Toolkit [25], is used to design and manage the AR overlays, displayed on the Hololens 2 glasses [23], see Figure 7.The MR toolkit provides a collection of scripts and components intended to accelerate the development of AR applications.It facilitates the integration of spatial awareness, input, and other AR-specific features, enhancing the usability and performance of the ship-side application.
Similarly, on the shore-side, Unity is used in conjunction with the additional plugins to create the VR environment.These plugins are designed to take advantage of the high-resolution capabilities of Varjo-VR headsets.
5.6.360-degree Cameras and Latency 360-degree cameras, capable of recording 4K footage, play a significant role in the system by providing a comprehensive visual feed from the ship's deck.The video captured by these cameras is streamed in real time to the back end PC using the Real-Time Streaming Protocol (RTSP), a standard protocol for controlling multimedia streaming.Given the high data volume associated with 4K video, it becomes necessary to compress the video feed on the backend PC before transmitting it to the shore-side application via a mobile connection.The process of video compression reduces the size of the video data, allowing for more efficient use of the limited bandwidth of the mobile connection.
However, optimizing the balance between video quality, frame rate, and latency is a complex task [26].High-resolution video, while offering detailed imagery, requires more bandwidth and processing power.A higher frame rate, though leading to smoother video playback, further increases the data rate and bandwidth accordingly.Reducing latency is critical for maintaining real-time communication, but this can compromise video quality under bandwidth-limited conditions.One potential solution to these challenges is the use of adaptive bit rate streaming [27].This technique dynamically adjusts the quality of the video in real time according to the network conditions.It ensures that the system uses the available network resources effectively, maximizing video quality without exceeding the bandwidth capacity.
Overall, handling the streaming and compression of 4K video from the 360-degree cameras is a technically challenging yet vital aspect of the remote pilotage system.It allows the shoreside pilot to gain a comprehensive and accurate visual understanding of the ship's environment, aiding effective decision-making.

MQTT Broker
Message Queuing Telemetry Transport (MQTT) is a lightweight messaging protocol designed for constrained devices and networks with low-bandwidth, high-latency, or unreliable connections.It is based on a publisher-subscriber model, with a central node called the broker.The MQTT broker is essentially the nerve center of the MQTT network, facilitating the smooth transfer of messages between the sender (publisher) and receiver (subscriber).
The primary role of an MQTT broker in any system, including our remote pilotage scenario, is to receive all messages, filter them, decide who is interested in them, and then send the message to all subscribed clients.The broker is responsible for the distribution of messages to the clients based on their subscription to specific topics.It maintains session information, handles the authentication and authorization of clients, and ensures the reliability and orderly delivery of messages [28].In the context of the remote pilotage system, the MQTT broker is of paramount importance for several reasons.First, it enables efficient communication of critical data from the ship-side application to the shore-side application, acting as a reliable intermediary.The ship's data, AIS data, and other crucial information from the ship's systems are sent to the MQTT broker and subsequently made available to the shore-side application.The lightweight nature of MQTT makes it particularly suitable for this application.Given the potentially high latency and limited bandwidth of maritime communication networks, MQTT's minimal bandwidth usage is a significant advantage.It allows the efficient use of the available mobile data connection, ensuring that critical ship data is transmitted in real-time, despite the constraints.
Moreover, MQTT is designed with reliability in mind, offering features such as three Quality of Service (QOS) levels, persistent sessions, and a Last Will and Testament mechanism.These features ensure that vital information is not lost in transit, even in the event of temporary connectivity disruptions or device failures.In addition to these benefits, MQTT also provides a flexible and scalable solution.The topic-based filtering mechanism allows the system to easily accommodate additional data sources or subscribers, without requiring significant changes to the existing setup.This could prove beneficial in future enhancements of the remote pilotage system, for example, if additional sensors are installed on the ship or if multiple shore-side stations need to receive the ship data.
In conclusion, the MQTT broker is a cornerstone of the remote pilotage system, ensuring the efficient, reliable, and scalable transfer of crucial ship data to the shore-side application, thereby facilitating informed and timely decision-making by the pilot.

Technical Characteristic for the Consolidation of the Use Cases
The implementation of the different essential use cases to create a possible shared situational awareness was realized by means of immersive technology, the use of standardized messaging protocols and the provision of information from known systems.

AR and VR Advantages
AR and VR technologies can play a significant role in improving the effectiveness of the remote pilotage system by providing adequate data connectivity and ensuring appropriate implementation.Onboard the ship, AR glasses overlay critical information such as AIS data and route updates onto the captain's view.This augmented view can prove especially useful in adverse conditions like fog or other scenarios where visibility is limited.
The VR technology on the shore-side enhances the pilot's situational awareness by offering a 360-degree live video stream from the ship, along with crucial ship data and AIS data integrated onto a sea chart.This has been highlighted as a potential mean to compensate for sensory losses.Thus, the immersive VR environment creates a sense of presence for the pilot, thus allowing them to make decisions as if they were physically on board.

Interactive Route Management
Another important feature of the system is its interactive route management.The shore-side pilot can manipulate the ship's route and update it on the MQTT broker.The new route information is then conveyed to the ship-side application, which displays the updated route to the captain through the AR glasses.This real-time, interactive route planning mechanism enhances the system's responsiveness to changing maritime conditions.

Communication
A VoIP software, facilitates real-time communication between the shore-side pilot and the shipside captain.It supplements the visual aids provided by AR and VR technologies, ensuring effective, immediate, and clear exchanges between the two parties

Comparison of Remote Pilotage Implementations
The S4VF and RePO MAN projects both aim to improve maritime navigation by providing enhanced situational awareness to shore-based pilots.However, their approaches in achieving this goal vary significantly, primarily in terms of user interaction, technological adoption, and information presentation.
The procedure for obtaining data on the ship side and passing it on to the shore side is similar in both projects.Besides the prepared interface from the S4VF project, RePO MAN also uses multiplexer technology to read NMEA sentences directly from the sensors.The main differences in the two projects are the creation of shared situational awareness of a location by two people present at different locations.Furthermore, RePO MAN of the ship side also provides visualization in the form of AR HMD glasses.Moreover, the interaction between both parties is not only focused on verbal communication, but also on sharing data (modifying routes), highlighting situations to support a question or comment, and allowing discussion on specific decisions or suggestions.
S4VF primarily depended on a desktop setup at the shore-side, with information communication to the ship's pilot being central to its operation.The focus was on technical arrangements and data acquisition in the first place.The development of situational awareness functions could not be completed within the scope of the S4VF project.It lacks an interactive dimension, particularly in the areas of route planning and decision making.The approach was primarily information-based, with the shore-side pilot having access to critical ship and system data, but no way to manipulate this data or affect real-time decision-making.
On the other hand, RePO MAN leverages advanced VR and AR technologies to offer a immersive and interactive solution.The use of VR for the shore-based application provides the pilot with an enhanced 360-degree view of the ship's environment from the onboard cameras.Further, AR technology used in the ship-side application of RePO MAN facilitates dynamic route planning, enabling real-time adjustments in response to changing maritime conditions.This interactivity is a user-centric approach to promote shared situational awareness between the shore and ship sides compared to Sea4Value, where the pilot was not directly involved in route planning.
Moreover, RePO MAN incorporates comprehensive data from the ship's systems, AIS, and environmental sensors into the VR and AR interfaces.This integrated data presentation contrasts with Sea4Value's system, which provided the pilot with system information but lacked the immersive, context-rich display of data that RePO MAN achieves.
Integrating new systems into long-standing workflows is a challenging process and requires a high level of user acceptance and the gradual familiarization of both sides with the technologies and the changed workflows.Far-reaching acceptance and the possible modification or adaptation of processes can only be realized through extensive on-site test campaigns that are as realistic as possible.Due to the lack of test capacities, but also the difficulties in test execution due to the increased number of pilots required (one on board, one on shore), developments will progress more slowly.

Conclusion
Remote Pilotage is not about making pilotage obsolete, rather providing pilots modern virtual means to execute their tasks to assist the onboard crew in a safer way.This is expected to have significant effects on workers safety of pilots, but as a side-effect also on emissions of pilotage transfer arrangements.However, to achieve those effects, systems and concepts must be developed, tested and implemented to keep pilotage as a safety backbone for navigating in challenging waters.Within this paper, two approaches have thus been investigated more in detail.
In summary, while both projects share the common objective of enhancing maritime navigation, RePO MAN, with its VR and AR technologies, offers a more immersive, interactive, and context-rich environment for the shore-based pilot.In comparison, Sea4Value, while being information-rich, lacks the interactivity and immersive perspective of the ship's environment provided by RePO MAN.
These technological demonstrations provided a lot of information about advantages and disadvantages of different technical solutions.
It is important to have these kinds of demonstrations in order to find in practice the areas what works and what still needs more development.XR tools are interesting opportunity for enriching the immersion and experience of interaction of remote operations.In any case, a common and standardized interface with respective ship-shore connectivity to ship board systems -a so called Remote Pilot Plug -is needed irrepspectively of the chosen shore-side visualisations to facilitate uptake of Remote Pilotage Technologies and thus offering a fifth way of pilot boarding in future.

Figure 1 .
Figure 1.AIS Tracks of Pilot Transfer Vessels Color Coded by Region

Figure 2 .
Figure 2. Connection between Ship and Shore within the S4V project was enabled via "Smartbox"-connection at the vessel.[21]

Figure 7 .
Figure 7. System overview: Communication between the applications

Table 1 .
[10] returns from participating IMPA members which have been grouped into six geographical areas.[10]

Table 2 .
Fuel 20] [21]onnections from the ship to the remote pilotage center over the internet[21]