Task-Mediated Design: A latent space for dexterous soft robotic design

Soft robotics has created a paradigm shift within robotic grasping and manipulation. Rather than using complex control policies and path planning methods to undertake grasp objects, soft robots use their embodied intelligence to deform around an object and securely hold it. Yet despite major advances in the field in recent years, there remains an absence of practical methods for designing new high performing soft manipulators. In this perspective, we propose a novel task-mediated approach to soft manipulator design and discuss its application to robust and dexterous manipulation. The task-mediated approach first abstracts the manipulation task into a characteristics latent space, then designs a soft robot to minimise the distance between the actual performance and characteristic latent variables. In conjunction with state-of-the-art generative design tools and simulators, this approach can accelerate the development of bespoke and general purpose manipulators


Introduction
Soft robots are prized for their ability to adapt their shape and gait to perform numerous tasks in diverse environments, especially deformable and fragile.The potential capabilities of mobile soft robots are frequently compared to natural invertebrates like octopuses and worms, which can reconfigure their bodies to move through cavities and adapt their locomotion to their terrain [Majidi(2014), Laschi et al.(2012), Rus and Tolley(2015)].However, despite the rapid growth in soft robotics research, artificial analogues of these animals remain a distant vision.A similar trend has emerged in robotic grasping and manipulation, where inspiration has regularly been drawn from natural grasping methods such as geckos' adhesive feet and chameleons' tongues [Pang et al.(2022), Hao and Visell(2021)].Yet soft end-effectors remain largely based on anthropomorphic fingers [Hao and Visell(2021)], and despite major technological improvements neither soft nor rigid robots currently come close to matching the gold standard dexterity of human hands.Consider common human manipulation tasks: picking up a soft fruit, unscrewing a bottle-cap, buttoning a shirt, eating with chopsticks, pulling a trigger.Each of these tasks require both a different hand configuration and different force magnitudes and directions, and most require a coordinated sequence of actions to successfully perform the task.Although there are numerous antropomorphic robotic hands, soft grippers, and grasp planners on the market, these routine tasks remain a challenge for robots.
In industries such as healthcare and agriculture there is an acute need for dexterous, soft grasping and manipulation.So what's the holdup and what needs to be done to bring us closer to natural capabilities?In this paper we discuss the hierarchy of design goals for robotic end-effectors, with an emphasis on soft robotic grasping, and propose a task-mediated design approach to bridge the gap between robotic and natural grasping and manipulation capabilities.Here manipulation can be taken to mean a task involving any combination of grasping, reorienting and reconfiguring an object or environment.

Embodied Design or Manipulation Policies
A philosophical divide has emerged in recent years between the communities researching conventional (rigid) robotics and embodied/soft robotics.The rigid robotics community sees the problem of robotic grasping primarily as a control policy problem.That is, if a human hand the gold standard in manipulation, then the research problem is how to teach robots to perform human manipulation strategies.Embodied and soft robotics takes a different perspective, which emphasises the connection between the robot's design and control, and the task.Rather than assuming that a well designed controller can overcome design flaws, it sees the manipulation problem as finding the best combination of tool design and controller (i.e body and brain) for a task.By embodying intelligence in the tool design, simple control strategies can robustly perform complex tasks, reducing the burden on high-level control policies and task planning.For example: monkey tails and eagle talons are both high performing natural grippers, which are simpler to operate than human hands.
Consider the example of harvesting a ripe strawberry, which we will refer to throughout the remainder of the paper.The process for harvesting a strawberry is: (i) Grasping the strawberry's stem (ii) Gently pulling the berry away from the plant and neighbouring berries (iii) Rotating the berry with the stem held stationary (iv) Placing it in a storage container This process must be completed without damaging any berries or the plant, and must be able to be performed in the cluttered plant environment.Hence at a minimum, a candidate gripper must be able to exert the required grasping forces/moments whilst minimising the forces exerted on non-grasped fruit.Ideally it should also minimise the amount of infrastructure required to perform the task (sensors, actuators, computation, etc) Rigid robotics assumes that with sufficiently precise controller, an actuated end-effector can grasp the berry without damaging it or neighbouring berries.In contrast, the communities of embodied intelligence and soft robotics would instead consider the tool's morphology and materials first, to see whether its design can improve task performance and reduce the controller complexity.For example soft fingertips increase grasp robustness and reduce the risk of damage.
To be clear, the concepts are not mutually exclusive, but the two approaches naturally lead to very different research priorities.One focuses on training control policies to create generic, multi-functional end-effectors, whilst the other optimises the tool's design and control for highperforming application specific designs.

Design Challenges
Embodied intelligence design is not a simple task.To design high performing robots, the entire robot and its task must be considered holistically and cooptimised.The resulting problem has a very-high dimensional feature space, making the process of encoding features and identifying high-performing regions difficult.
Hence, the conventional robotics design approach has been to dramatically reduce the feature space by assuming stiff linkages with discrete joints.Whilst convenient, this vastly reduces the quality of results.It is now well understood that distributed actuation and flexible materials is essential for interacting with deformable objects and unstable terrain [Zhu et al.(2022)].However, despite the rapid growth of the field, creating new high-performing soft robots remains a slow and laborious exercise.Our ability to generate new designs is impeded by the complex and non-linear design space, lack of high-quality simulators, complex manufacturing process, and limited sensing capabilities.
A more in-depth discussion of these challenges is beyond the scope of this work, but detailed reviews of the state of the art and current challenges in soft robotic design can be found at [Pinskier and Howard(2022), Chen and Wang(2020), Xavier et al.(2022)].
Here, we propose an overarching framework to develop new soft grippers, which is compatible with any of these design methods.
We hope it can be used not just to design novel grippers, but also to answer major open research questions in the field, such as: (i) Can we make multi-functional soft manipulators with human-like dexterity?(ii) How can we efficiently design soft robots/end-effectors for a specific task?(iii) What strategy is best to perform a complex series of tasks?Is it better to use a single multi-functional tool, or a set of bespoke ones?How many?(iv) How can we efficiently simulate tasks involving interactions between deformable objects and robots whilst capturing their material properties and actuation?Can the simulation to reality gap be closed by incorporating experimental data?

Task-Mediated Design
In most design optimisation/evolution methods, the design problem is specified by first considering the selection of material(s), actuation (real or approximated) and a basic description of task (terrestrial or aquatic locomotion, grasping, etc).With the exception of a small number of targeted studies [Collins et al.(2018),Howard et al.(2022a)], the detailed task requirements and environment in which the task is being performed is only considered superficially (smooth or rough ground, hard or soft object).This simplificiation is a necessary requirement for tractible optimisation in a high-dimensional space, but leads to design spaces far removed from the actual design problem.
We invert this design paradigm and instead propose an design approach which considers the task as the primary optimisation feature, transforms that into a latent task space, then designs around the latent variables.
That is rather than considering manufacturing/design capabilities first (or their approximations) then working forwards to a design, we first consider the task and environment and work backwards to a design.
Hence, we propose a two-stage design process for task-mediated design.In the first stage the task is quantitatively defined and characterised; then in the second, a design is formulated which best approximates the quantitative task definition.
Whilst superficially, this may appear synonymous with inverse design, task-mediated design is far more general.It is not simply designing to reach a set of desired properties, but instead captures the essence of a task or operating environment and find an optimal design for that goal.It determines which forces and displacements are desirable and which are tolerable, and use these as a basis for automated design.Whilst we focus on manipulators in this work, we believe the reduced dimension framework will enable robots to evolve for a broad array of tasks.
An example of the task definition process is illustrated in Figure 1.The act of picking a berry is abstracted to an optimal 4D force array (x,y,z,t) representing the set of forces with picks the fruit with minimal disturbance.The design task is then to find a manipulator which best generates these forces within the required geometric constraints.
This 4D task definintion is suitable for design optimisation, but additional features may also be added to capture higher-level design goals like robustness and dexterity.In the remainder of this work we discuss the tiers of design goals, and task-mediated design can aid in designing up the hierarchy.

A) B) C)
Figure 1.Quantifying the strawberry picking task.(A) The objective of the task is to pick the middle strawberry by pulling it away from the plant with minimal disturbance to the fruit and surrounding foliage.(B) The ideal way to do this is to rotate the berry as it is pulled away from the plant, gently breaking the stem.This requires a set of forces and torques to be applied to firmly hold the berry and generate the motion.(C) Using the berry's material properties (stiffness, coefficient of friction, etc), an optimal 3D force vector can be determined which generates the required motion whilst minimising the deformation of the berry

Task-Mediated Optimisation
The hierarchy of design is illustrated in Figure 2. The most basic design objective is to optimise a gripper for a single object.Whilst ideal for standardised and repetitive pick and place tasks, it is inherently fragile.Small perturbations in the object or the tool position lead to grasp failure.Consider an 'optimal' gripper for a strawberry, it may look something like a 2 part mould of the strawberry, which perfectly matches its geometry and is able to surround the berry to perfectly distribute forces.Whilst this 'imprint' gripping is a good strategy for a uniform object, it will fail to pick berries with even small differences in shape or orientation.For example, the simulated grasp success of imprint grippers fell from 78% to 70% under a 10 deg rotation [Ha et al.(2020)].
The task embodied design process to optimise a single design is a relatively straightforward 2-step process to first identify the task's characteristic load array, then design a gripper to approximate the array.
(i) Configure the task scene in a simulation environment (e.g.finite element or physics simulator), such that all relevant objects are present and their stiffness and friction, and .Hierarchy of design goals: optimisation is useful, but being robust to shape and positioning uncertainly is better, and being able to robustly perform multiple movements is ideal stress and limits are captured.Then evolve the 4D load array to find a grasp is found which meets the requirements.(ii) Generate and evaluate designs using a generative tool to find a valid approximation of the load array.That is, minimise the norm of distance between the applied and desired loads.

Task-Mediated Robust Design
Robustness design aims to make the design insensitive to these small changes, such that the manipulator can grasp a whole class of objects (e.g any strawberry) and tolerate positioning errors.
Designing for robustness is inherantly more challenging than optimality, as the entire operating envelope must be considered in the design.This can be achieved either by simulating across a large number of conditions, or by giving the design sufficient flexibility to be insensitive to small errors.Fit2form design is an example of the former approach, it used a generative design algorithm to optimise a bespoke (rigid) end-effector to robustly grasp a specified object using a parallel plate style gripper [Ha et al.(2020)].The cost function contained terms which maximised grasp stability and robustness, allowing it to generate several robust grasp strategies even without a deformable end-effector.In soft robotics, robust embodied grasping is not even considered explicitly, but rather assumed as a byproduct of their inherent conformability [Ellis et al.(2022)].
To progress the field, robustness must be considered in the design of all soft manipulators.In practice, this means identifying the bounds of performance as well as the ideal grasp pose.
Incorporating this into our task-mediated design framework requires an additional dimension to be added to the cost function, representing the grasped object.The feature array is then a 4D force, plus the object index (x,y,z,t,n).Grasp sensitivity is captured by perturbing the optimised loads.To be valid a grasp must be able to be varied within specified bounds and still complete the task.
Here we see the benefit of the task-mediated approach: the actual design process does not change to accommodate robust design, it is only the task definition which changes.Hence, the method can be layered on top of any existing design strategy.It is fully compatible with sophisticated design strategies like multi-level evolution and adaptive optimisation [?, Pinskier Howard(2022)]

Dexterous design
Atop the design hierarchy is dexterous design, in which the tool is able to perform a stable grasp with sufficient flexibility in its motion to accommodate significant shape and positioning uncertainty [Ciocarlie and Allen(2008)], and has the ability able to perform multiple or complex tasks.For example, a dexterous gripper may be able to perform the entire strawberry harvesting sequence, or pick and pack strawberries with a single tool.
This top tier approaches the goals of rigid robotics, to have single multi-functional devices.Given a fixed design it poses dexterous manipulation as a task to identify the required set of forces to reorient and object and translate that to manipulator links [Okamura et al.(2000)].However, it is still build around the embodied design goals.It is the combination of material, morphology, and control which achieve the task rather than relying on control.
Task-mediated design provides a framework to generate new dexterous designs by simply adding one more dimension to the feature space, representing the task being performed.The task extraction procedure is then repeated across the set of tasks to find the characteristic array across the set of tasks, and objects.

Conclusion
In this perspective we introduce the notion of task-mediated design, a 2-step design process.It augments the traditional generate-design-test-regenerate methodology with the latent taskspace.This will enable higher quality designs to be found more efficiently by capturing critical features and reducing the search dimension.We hope task-mediated design will help accelerate the design and development of novel high-performing soft manipulators and mobile soft robots.
Figure2.Hierarchy of design goals: optimisation is useful, but being robust to shape and positioning uncertainly is better, and being able to robustly perform multiple movements is ideal