Context-attentive robot reconfiguration for human-machine space missions

With humankind’s aspiration of extraterrestrial inhabitation, the need for robotic support is eminent. To master the emerging challenge of providing astronauts with assistive space robots possessing the required adaptability to new situations and tasks under the constraints of severely limited supply chains in extraterrestrial missions, the utilization of existing hardware to its fullest is required. We propose the combination of a concept with an architecture to jointly support context-aware collaboration between humans and robotic systems for implementing effective resource utilization in space missions. The underlying framework is an implementation of a flexible architecture that enables context awareness for robotic systems. It supports processing nodes for retrieval of context information from the raw sensor values as well as further inferences using the output of nodes. Context information spans across three entities and describes the current state of the environment, the astronaut, and the robot itself. This information then can be used to infer the human’s current intention as well as influence the behaviour to be executed next. Additionally, dynamic changes in the data processing chains are handled by the framework to facilitate an adequate adaptation of the system to situational events. We combine this architecture with the concept of space robots as a composition from building blocks, which are supported and made accessible by a software toolkit. The modular design approach enables online self-reconfiguration of robotic hardware and software components. In combination with dynamic mission planning, based on ontological descriptions of available resources and functionalities, robots are able to adapt their physical and computational appearance dynamically during a mission according to different tasks and goals. By incorporating these two developments in a joint deployment we envision raising the efficiency of robotic systems in human-machine interaction through the usage of self-reconfiguration as a reactive behavior in order to adapt to a specific task, recognized or derived from a human’s intention. Transfer of the proposed idea back to Earth may help to abate resource dissipation caused by deploying specialized monolithic systems with a narrow range of capabilities through utilizing the adaptivity of reconfigurable robots.


Motivation
The future is extraterrestrial.Humans and robots need to work together in order to build habitats or other structures on extraterrestrial bodies.To coordinate this collaboration, a strategy of communication is needed.Currently, this is mostly done by a hierarchical approach, in which humans program a robot's behaviour statically before a mission.However, working together in space will require a dynamic relationship between humans and robots, in order to form an effective work team.Therefore the principle of communication needs to be adjusted to allow faster, dynamic, and less complex exchange of tasks, needs, and information in general.Furthermore, the robots used must also be able to react accordingly in this dynamic communication and to actively participate in the collaboration.Classical, monolithic systems are mainly designed to perform one, mostly very specific task.This limits their capability of collaboration with humans strictly and makes maintenance elaborate.The supply with specialized robots as well as resources in general for extraterrestrial missions will be very limited, so that the assignment of new tasks, which may arise from the process of colonization, to already available robotic systems will require an immense workload, if possible at all.Our approach aims to tackle both mentioned limitations of the current state of the art by incorporating the topics in a new way.We shortly present the latest research of two projects within the space robotics domain and suggest the combination thereof as our proposed approach.The goal of the presented concept work is to give impetus to future research and show the importance and potential of current scientific efforts.

Background
In the recent past, research towards both, alternative human-robot communication and the modularisation of robotic systems, has made progress.In the following, the state of the art towards modularity in space context and human-robot interaction (HRI) is shortly presented.The other subsections introduce the two core projects from which the idea for this work stems.With their focus on the main concepts, context-interpretation, and modularisation, they form the scientific basis for the desired goal of intuitive and collaborative work between humans and robots within space missions.

State of the Art
As the field of application of robotic systems extends into the space domain, a variety of complex tasks have to be accomplished.Especially when humans are exposed to a lifethreatening environment, good communication between man and machine is of great importance.Interaction between the members of a human-robot team needs to be productive for a successful collaboration [1].In order to achieve this the robot needs to be capable of perceiving its surroundings, itself and the human it is interacting with.We will call the aggregation of the states of those entities a context.For different situations throughout the interaction, different contexts can be perceived.By recognizing the context and adapting the behaviour accordingly the robot can achieve more suitable support for the human which then could lead to better attainability of the mission goal.The usage of context is presented in [2] along with a proposed framework that facilitates context-aware applications.In [3] the authors present a toolbox for multi-modal context recognition.This context can then be used to derive the human intention.As context will change dynamically during the interaction it is equally important that the used framework can react to changes.Robotic frameworks, such as the Robot Construction Kit (Rock) [4] and the Robot Operating System (ROS2) [5], provide this feature with modular designed components.Rock utilizes the Orocos Real-Time Toolkit as its underlying component model and offers a diverse range of modules, drivers, and tools for robotic applications in both research and industry.Rock adopts a connection-based approach, facilitating easier control of information flow.The formal description of modules enables higher-level components to manage states and connections seamlessly.ROS2, on the other hand, employs a computational graph model, where each running process represents a node.Nodes are interconnected via topics, where they can either listen to or write data.Through the use of a real-time data distribution service, ROS2 achieves real-time communication.Additionally, ROS2 includes a comprehensive set of built-in drivers for robotic applications, along with tools for data visualization, logging, and data playback.Other works focus more on the data fusion of different modalities [6].While many existing frameworks represent partial solutions, there is still the demand for a framework that unifies dynamic adaptability with a context-aware intention recognition using multiple modalities to influence the robot behaviour that supports the interaction with a human adequately.
Dynamic adaptions of robot behaviour can be complemented by modularisation.The modularisation of space robots has many advantages: easy maintenance, lower transport and manufacturing costs, and of course a higher flexibility regarding physical capabilities.When a robotic system is divided into subsystems or components, different granularity and other design features arise [7].Various modular space-systems exist.Modular space robots will be crucial for the success of future space missions [8].The modules or components, of which a robot consists are, inter-connectable and thereby also rearrangeable.This enables one of the major advantages of modularisation: reconfiguration -the change of structural or functional capabilities of the system by adding, dropping or rearranging physical or computational parts.Utilizing robotic reconfiguration to adapt to the environment or specific tasks autonomously has been performed successfully in several works, e.g., [9].However, using reconfiguration as reactive behavior in order to adapt to tasks, which are self-derived by the robot from the observed context, remains an unexplored possibility.Additionally, an advanced modularisation with heterogeneous, macromalistic modules has not been investigated much in the past.

Project KiMMI SF
The goal of KiMMI SF [10] is to develop a software framework for integrating intuitive contextdependent HMI.This framework enables robotic systems to adapt their behaviour depending on the current situation.The perception thereby focuses on three sources: the system itself, the human interaction partner, and the environment.We call the perception and the linked piece of information context.Contexts can have different values therefore we coined the term Context Variable.The collectivity of values of the perceived Context Variables describes the current situation throughout the interaction with a human.The framework uses this information and maps it to predefined intentions [11].An intention thereby can be something the human wants to do next or wants the robotic system to do next.A dynamic data structure with reconfigurable processing chains enables the system to adapt its behaviour quickly and be suitable to the current intention of the human.An additional layer, the so-called Consistency Management, strengthens the operability of the framework and implements the possibility to give feedback to the human which creates transparency during man-machine interaction and can increase human trust into the system [12].
Fundamentally, the architecture of the software framework comprises four pivotal components.The foundation is formed by the "Agents", representing the systems directly or indirectly involved in interactions and steering the system's functionality.Above this lies the "Dataflow Structure", functioning as middleware for orderly data processing chains, optimizing data management and processing.Building upon this structure is the "Dataflow Manager", ensuring precise control over adjustments within the data flow for efficient processing.At the top of the architecture is the "Interaction Manager", a critical component that identifies context and intention.It initiates necessary adjustments based on the context, facilitating smooth and adaptive interactions.The Interaction Manager is central to the framework, orchestrating the interaction process and ensuring the system responds appropriately to different contexts and intentions.

Project MODKOM
Project MODKOM [13] aims to modularise space robots in a macromalistic and heterogeneous way.The focus lies on the technological, mechatronic, and software development of modular functional units for reconfigurable robot systems that can be used in various space missions.These types of modules are of particular benefit and interest, for example, for performing service operations on satellites in orbit or exploring the surfaces of extraterrestrial planets as using robotic systems is less risky and relatively inexpensive compared to manned spaceflight.In MODKOM, existing functional units that are indispensable for robotic space missions are identified and further developed in such a way that a kind of plug-and-play solution can be provided via an interface in terms of both software and hardware.With the additional help of norms, standards, and modules, solutions can be flexibly configured in the future and adapted to new or changing requirements for the overall system with little effort, without having to carry out a completely new development every time.Robotic systems can be reconfigured according to the task based on the modular system.The latest achievements, as well as further explanations on principles and implementations, can be found in [14].The planning of reconfigurations during a mission is done using a mission planning tool, developed for modular space robots and based on an ontological modeling of the same [15].

Approach
In the field of human-machine interaction, the KiMMI SF framework provides a solution for understanding and acting upon inferred intentions.However, the framework still suffers from limitations due to resource constraints: A human's intention can be derived from context data, which describes the current state of the scene in which the robot and the human are interacting.If the current configuration is not sufficient to execute a behaviour to support the human's intention the robot can initiate a reconfiguration.If no reconfiguration is needed, then the robot can directly assist the human.
To continuously recognize intentions, the robot has to be permanently attentive to context changes, which can be described as context-attentive.By recognizing the human's intention, a robot is able to act reactive and respond to the human's desires with services or independently carried out activities.However, the capabilities of the robot, and thereby its reactive capacities are constrained by its physical and computational composition.A robot, consisting of a certain composition of modules, can subsequently only respond to a limited set of human's intentions with helpful behaviour.This set can be enlarged by changing the robot's composition and adding new components -either physical or computational by performing reconfiguration.Reconfiguration can be performed for software modules and hardware modules.Thereby not only the software is easily extensible, but also the structural change of the robot's hardware gets enabled.The usage of self-reconfiguration in this scenario can be declared as reactive behaviour in order to adapt to a specific task, derived from a human's intention, which brings considerable flexibility to the robotic assistance of the human astronaut.Thereby the collaboration between humans and machines is expected to be more effective, smooth, and more widely applicable.

Intention Recognition
What is the identified intended future state?

Dataflow Management Advisor
How should the dataflow be adapted ?

Dataflow Structure
Manage the topology of the dataflow graph, i.e. add/remove vertices or connections between vertices Behaviour Selector be ha vi ou r The dataflow structure supplies context information which is aggregated and presented together with static context information to the intention recognition.Depending on the derived intention the corresponding behaviour is looked up and executed.Figure 1 illustrates an example use case where a robot needs to deliver a specific tool to a human.Consider a situation where a human indicates a need for a tool through gestures, prompting the robot to interpret this intention.The robot evaluates its capabilities, particularly the compatibility of its gripper for grasping the tool.Smooth interaction occurs when the gripper matches the tool's requirements.However, mismatches, caused by an unsuitable or absent gripper, challenge a seamless task execution.In response to this challenge, a solution is proposed in detail in the following.Before initiating the task, the system proactively assesses its capabilities, verifying if the gripper aligns with the task's demands.This pre-task verification ensures swift system reconfiguration in line with the specific task requirements.
At the framework level, this scenario can be realized by an adaption of the framework developed in KiMMI SF incorporating components from MODKOM.All data-generating processes as well as the data flow into the framework for context observation remain untouched.After an intention is successfully recognized, instead of a static mapping between intention and the corresponding behaviour as displayed in figure 2, a reasoning step is performed (see figure 3).The reasoning is based on the ontological organisation-model for modular robots as used in MODKOM.With this change, in comparison with the original, static mapping, the intention is translated into a task for the robot.Afterward, the robot is able to determine, if its current configuration allows the fulfillment of the task.If not, an algorithm determines the most suitable configuration, which includes the knowledge of available resources and modules that can be queried from the knowledge database.In case the robot needs another configuration, reconfiguration as a behavioural building block is included in the execution.Our proposed framework architecture is illustrated in figure 3.
The usability of reconfiguration is a key-factor for the approach of context-attentive robot reconfiguration and will be realizable through the development of heterogeneous modules, with • Enhancing the knowledge reasoner in order to enable reasoning for a larger amount of modules and configurations • Intelligent and autonomous mapping of unknown context to intentions learned from experience This research perspective for collaborative space missions emphasizes a dynamic approach to comprehending human-machine interaction.Through this innovative approach, the aim is to advance the field of human-machine interaction, enabling a higher level of contextual awareness and intelligent responsiveness, as demonstrated by the system's ability to dynamically adapt to evolving contexts.

Figure 1 :
Figure1: Sketch of Case: The robot derives an intention from its observation of the human.If the current configuration is not sufficient to execute a behaviour to support the human's intention the robot can initiate a reconfiguration.If no reconfiguration is needed, then the robot can directly assist the human.

Figure 2 :
Figure 2: KiMMI SF framework architecture: The dataflow structure supplies context information which is aggregated and presented together with static context information to the intention recognition.Depending on the derived intention the corresponding behaviour is looked up and executed.