Effects of caudal fin stiffness on optimized forward swimming and turning maneuver in a robotic swimmer

In animal and robot swimmers of body and caudal fin (BCF) form, hydrodynamic thrust is mainly produced by their caudal fins, the stiffness of which has profound effects on both thrust and efficiency of swimming. Caudal fin stiffness also affects the motor control and resulting swimming gaits that correspond to optimal swimming performance; however, their relationship remains scarcely explored. Here using magnetic, modular, undulatory robots (μBots), we tested the effects of caudal fin stiffness on both forward swimming and turning maneuver. We developed six caudal fins with stiffness of more than three orders of difference. For a μBot equipped with each caudal fin (and μBot absent of caudal fin), we applied reinforcement learning in experiments to optimize the motor control for maximizing forward swimming speed or final heading change. The motor control of μBot was generated by a central pattern generator for forward swimming or by a series of parameterized square waves for turning maneuver. In forward swimming, the variations in caudal fin stiffness gave rise to three modes of optimized motor frequencies and swimming gaits including no caudal fin (4.6 Hz), stiffness <10−4 Pa m4 (∼10.6 Hz) and stiffness >10−4 Pa m4 (∼8.4 Hz). Swimming speed, however, varied independently with the modes of swimming gaits, and reached maximal at stiffness of 0.23 × 10−4 Pa m4, with the μBot without caudal fin achieving the lowest speed. In turning maneuver, caudal fin stiffness had considerable effects on the amplitudes of both initial head steering and subsequent recoil, as well as the final heading change. It had relatively minor effect on the turning motor program except for the μBots without caudal fin. Optimized forward swimming and turning maneuver shared an identical caudal fin stiffness and similar patterns of peduncle and caudal fin motion, suggesting simplicity in the form and function relationship in μBot swimming.


Introduction
Fish are exceptional swimmers, as they elegantly navigate a variety of aquatic environments, negotiate challenging hydrodynamic conditions, and execute an array of rapid maneuvers [1][2][3].Central to their swimming capabilities is the caudal fin which serves as their primary hydrodynamic surface for both propulsion and maneuvers [4][5][6].The role of the caudal fin is most pronounced in species of the body and/or caudal fin (BCF) form which represents more than 85% of fish species [7].The effects of fish's caudal fin on swimming performance have been studied extensively in living fish or using robotic models.In living fish, Nauen and Lauder measured the flow patterns and calculated thrust of caudal fin of chub mackerel based on digital particle image velocimetry [8]; Borazjani revealed the critical role of the caudal fin in generating hydrodynamic forces during C-start turning maneuver (i.e. a rapid escape response where the fish body bends into a C-shape and then bends out quickly to generate thrust to flee predators or capture preys [9]) of a bluegill sunfish using computational fluid dynamics (CFD) simulation [10]; Tack and Gemmell compared forked caudal fin of mojarras and truncated caudal fin of pinfish in terms of thrust generation and efficiency during cruising [11].With robotic models, Feilich and Lauder constructed twelve caudal fins using four distinct shapes and three plastic materials with varying flexural stiffness to study how shape interacts with stiffness to produce swimming performance [12]; Low and Chong explored the effects of caudal fin design parameters, such as aspect ratio and stiffness, on the swimming performance of a fish robot [13]; Park et al examined the kinematic condition for maximizing the thrust of a robot fish with four different types of caudal fins [14]; Matta et al studied the effect of caudal fin shape on thrust generation and flow structures using a thunniform swimming robot [15]; Krishnadas et al explored the relations between caudal fin geometric features and propulsive efficiency [16].
Caudal fin stiffness has profound influences on swimming performance, and stands out as a pivotal design factor within the multivariate design space of caudal fins [17,18].Numerous studies have ventured into understanding the design of caudal fin stiffness and its effects on various aspects of swimming performance [12,[19][20][21][22]. Using the Tunabot platform, Zhong et al showed the significance of tunable stiffness in the caudal peduncle, demonstrating its potential role for enabling both rapid and efficient swimming [23].Esposito et al explored the effects of fin ray stiffness on thrust generation using a robotic fish caudal fin with six individually moveable fin rays [24].Park et al designed a variable-stiffness flapping mechanism and showed that optimized stiffness for thrust generation increases with the driving frequency [25].
Despite the knowledge produced by these studies, one major limitation is that they did not explicitly consider or quantify the effects of caudal fin stiffness on the optimized swimming gaits (e.g.motion of the caudal fin) or motor control (e.g.motor signals), which in turn affect the swimming performance.Most of existing studies using robotic models assumed fixed gaits or motor control when studying the effects of caudal fin design (for an exception see Shan et al [26]).However, for swimming or any locomotion type, the physical design and neural control systems are often interdependent, as they jointly evolve and are highly intertwined in the evolution of biological organisms [27,28].In addition, recent work in robotics also underscores the importance of joint design or optimization of robot physical body and controller [29][30][31][32][33][34].Therefore, it can be hypothesized that both the desired gaits and the motor control vary with the caudal fin stiffness, and it is of critical importance to optimize them, if possible, for each of the stiffness investigated.
Compared with studies on steady forward swimming and thrust generation by caudal fins, those on other swimming functions, which are also indispensable to the fish designs, biomechanics, and behaviors, remain understudied.For example, coral reef fishes rely heavily on turning maneuvers to adeptly navigate and engage with their intricate physical habitats [35].Prey fish often employ the C-start motion as an evasive maneuver to escape from predators [36,37].Eel and lamprey are adept at swimming backwards, which is also a frequently used escape maneuver [38].Some fish species also use burst-and-coast gaits to stabilize visual field, enhance sensory capabilities, and reduce energy expenditure [39].How caudal fin stiffness might influence these diverse swimming behaviors remains unknown and warrants more extensive investigation.
The goal of this study is to investigate the effect of caudal fin stiffness on two common swimming functions: forward swimming and turning maneuver.In particular, we aim at addressing the following questions: (1) to what degree does the caudal fin stiffness affect the motor control programs and the corresponding swimming gaits optimized for each swimming function?( 2) is there an optimal caudal fin stiffness for each swimming function?(3) if so, do these two swimming functions share an identical optimal caudal fin stiffness?The fish or fish-inspired swimming is characterized by motor-controlled fluid structure interactions (FSI), which involves the intricate coupling of structural dynamics, hydrodynamics, and rigid body dynamics, rendering it a challenge to model with high precision and low computational cost [40].Therefore, we used a robophysical model named µBot [41,42], which has modular designs and can have its caudal fin easily switched to designs of different stiffness.For µBot installed with each caudal fin designs, we performed policysearch reinforcement learning (RL) in experiments to optimize the motor control programs that maximized a reward defined for each swimming function.For forward swimming, the motor control programs were generated by a central pattern generator (CPG), and the reward was defined as the swimming speed.For turning maneuver, the motor control programs were parameterized as a series of square waves of varied time delays and intensity, and the reward was defined as the final heading change.Upon convergence of the RL, we compared the optimized motor control programs, swimming gaits and swimming performance for all the tested caudal fin stiffness.The results showed that the caudal fin stiffness played a significant role in determining the optimal motor control program and the swimming gaits, and there existed an optimal caudal fin stiffness that maximized both the speed in forward swimming and the final heading change in turning maneuver.

Overview of µBot platform and experimental setup
Magnetic, modular, undulatory robot, i.e. µBot, is an experimental robot model developed to investigate fish-inspired robotic swimming using the BCF propulsion mechanism (figure 1).Notable for its modifiable morphology, compact dimensions, easy assembly, and durability, µBot platform facilitates rapid prototyping across various design configurations such as body length, shape, and stiffness.Another feature of µBot is that that the swimming gaits/kinematics are not prescribed; instead, they emerge from FSI via motor control.More comprehensive description of µBot can be found in our previous work [41][42][43].
µBot uses modular design, comprising a head module, body segment modules, a caudal fin module, and rubber suit modules.Electromagnetic actuators are integrated into both the head and body modules, as illustrated in figure 1(a).In our previous work, we studied the effects of number of actuators or body Degrees-of-Freedoms (µBot-DoFs, DoFs = 2, 4, 6) on the forward swimming speed.The results indicated that the optimized swimming speed increased as the DoFs increased, while the normalized swimming speed in body length per second (BL/s) decreased.Among the tested robots, µBot-4 showed good balance between the speed in mm s −1 and the normalized speed [42].Compared with µBot-2, µBot-4 generates more diverse gaits and motor behaviors [42].Therefore, we used µBot-4 to study the effect of caudal fin stiffness on forward swimming.In terms of turning maneuver, µBot-6 was utilized as its higher DoFs is more advantageous for studying turning maneuver that often requires large bending of the body (e.g.C-start [44]).A fully assembled µBot measures 2.5 cm in width and 3.5 cm in depth, with its length and weight being dependent on the body DoFs.Specifically, the µBot-4 spans 170 mm and weighs 59 g, whereas the µBot-6 spans 224 mm and weighs 78 g (figure 1(b)).Both µBots were made nearly neutrally buoyant and swum below the water surface.
The RL was performed with µBot swimming in a water tank (58 cm width × 56 cm height × 305 cm  Three carbon fiber rods as skeleton 5 Carbon fiber rod with rubber 3.81 × 10 −4 Seven carbon fiber rods as skeleton 6 Acrylic board 6.55 × 10 −3 length) (figure 2).µBot motion was captured by an overhead monochrome camera (acA2000-165umNIR, Basler AG Inc, Ahrensburg, Germany) equipped with a 760 nm filter and IR light sources.
The µBot motor control system was implemented using a laptop, an Arduino Mega board, three L293B motor controllers, and a DC power supply.
To track the µBot motion, reflective markers were positioned on µBot's dorsal side at every joint connecting the adjacent segments (the markers' placements can be viewed in videos S1 and S2).The positions of the markers were captured by the overhead camera for swimming gaits extraction.The overhead camera was calibrated with MATLAB camera calibration toolbox (MATLAB R2020a, MathWorks, Natick, MA, USA) with an accuracy of ±1.5 mm when measuring 100 mm distance.At each time step in the experiment, the laptop relayed voltage signals to the robot via the Arduino and the motor controllers, while simultaneously receiving µBot markers' data from the camera.Serial communication for robot operation and motion capture was facilitated within MATLAB (Serial Port Interface and Image Acquisition Toolbox), running at a frequency of 50 Hz.
In the robot's design, the caudal fin was affixed to the peduncle segment as a cantilever beam (see figure 1(a)), allowing it to deform during swimming.In the swimming gaits visualization, we abstracted the caudal fin's movement to a straight line delineated by reflective markers located at the peduncle and the caudal fin tip.The caudal fin deformation is indirectly captured by the lumped peduncle angle, as shown in figure S1.

Caudal fin design and fabrication
We fabricated six caudal fins of identical shape (rectangle of 3.0 cm length × 2.5 cm depth) but with varied stiffness, which corresponded to total seven testing cases (including the case without a caudal fin, table 1).Case-0 represents the bare robot absent of a caudal fin.Case-1 to case-6 represents the robot with increasing flexural stiffness ranging from 1.58 × 10 −6 -6.55 × 10 −3 Pa m 4 , encompassing a wide range of stiffnesses observed in biological fishes [12].The thickness of the fins was less than 1 mm.Note that the flexural stiffness of the body rubber suits is 2.33 × 10 −5 Pa•m 4 , which is slightly higher than that of case-3.For case-4 and case-5, the caudal fin design was inspired by the caudal fin of bluegill sunfish where the internal skeleton supports the musculature [45].For these two cases, the caudal fins included carbon fiber rods (3 rods for case-4 and 7 rods for case-5; analogy to the internal skeleton of biological fins) connected by a thin layer of silicone rubber (Ecoflex 00-30, Smooth-On Inc, Macungie, PA, USA; analogy to the musculature of biological fins), and both sides of the fin were covered by polyester films (figure S2).However, unlike biological fins that can actively control the fin's conformation, µBot's fins were passively deformed due to the FSI.Also note that fish can control caudal fin musculature to modify the stiffness [46], while the stiffness of µBot's fins was changed by the number of carbon fiber rods.For the remaining cases, the chosen materials were simply cut into the appropriate caudal fin shape.Notably, the design of the µBots incorporates a designated slot within the peduncle segment, streamlining the process of fin installation and interchangeability, as highlighted in figure 1(a).

Motor control for forward swimming
For forward swimming, the motor control program was generated by a CPG.A CPG can produce rhythmic outputs in the absence of rhythmic inputs and has been widely utilized in the control of robots demanding oscillatory movements [47][48][49].In this study, we employed a dual-neuron model of CPG as conceived by Matsuoka [50].Each actuator was controlled by a CPG module, including two neurons inhibiting each other.This mechanism is visualized in figure 3. Correspondingly, the mathematical representation of each neuron is encapsulated within two ordinary differential equations (ODEs), which are, where U and V are states of the ODEs; n represents the number of actuators (or CPG modules); τ is the time constant; E i and β i are external stimulus and adaption coefficient of ith module; α represents the mutual inhibition weight; ω is the inter-module connection weight; y i,out is the output from the ith CPG module [51].
The rhythmic outputs produced by the CPG can be distinguished by their frequency, inter-module phases, and intensities.As the CPG outputs are directly used as the voltage signal of the robot actuators, the frequency of the rhythmic outputs is also the motor frequency, as well as the undulation frequency of the robot.Within the CPG network, τ is instrumental in determining the frequency of the outputs.β i predominantly influences the inter-module phases and E i adjusts the intensities of the outputs.To simplify the learning process, all parameters except for those mentioned above were held constant, as shown in table 2. As a result, the parameter vector set for learning was (highlighted in figure 3).For forward swimming, the voltage limit was set as ±20 V. Any CPG outputs that surpassed this voltage threshold were truncated to the limit.

Motor control for turning maneuver
Fish exhibit a variety of turning maneuvers for navigation, exploration of underwater environments, predation, escape response, etc. Notably, the C-start and S-start are rapid escape mechanisms that can be executed in the absence of forward swimming, where the fish quickly contorts its body into distinct shapes and sprint away with a near zero radius turning and linear acceleration [44,52,53].On the other hand, the J-turn is a milder change in direction, primarily observed when a cruising fish decides to alter its course.During this maneuver, the fish forms a Jshaped curve with its body, then propelling itself in the direction of the bend [54].Fish also engage in yawing turns, which allow them to adjust their orientation without significant linear motion [55].Notably, regardless of the specific turning maneuvers and their associated swimming gaits, all these turns share two fundamental stages.In the bending stage (stage 1), the fish bends its body into a curve while changing its heading.In the restoring stage (stage 2), it then straightens its body while propagating a backward wave exiting from its caudal fin.Inspired by the swimming gait patterns and muscle activation sequences of biological fishes and robots (e.g.see [53,[56][57][58]), we designed the motor control program for turning maneuver of µBot as a series of square waves of varied time delays and intensity (figure 4).During stage 1, a uniform voltage (V 1 , bending voltage) was applied across all actuators, however, with incremental time delays of initiation (D i ) from the head of the robot to its caudal fin.This pattern of motor program drove the robot body to form a curve of varying shape (depending on the delays) while initiating the turning.The bending voltage was added for a specific duration specific duration (T 1 ), which is the same among all the actuators.Advancing to stage 2, a restoring voltage (V 2 ) was introduced to all actuators for a period (T 2 ) before returning to zero voltage, allowing the robot to straighten its body.While the V 1 and V 2 for the two stages were learned individually, to expediate the learning process in experiments, uniform voltage was applied across all actuators.The time delay, as experienced by the body actuators, in relation to the head actuator, is described as, where i = 2,…,n; n is the number of actuators with n = 1 at the head; K 1 , K 2 and K 3 are the coefficients of the time delay to be learned experimentally.We designed the time delay function as equation ( 2) so that the time delay could be increasing linearly (when K 1 is dominant), with a faster rate (when K 2 is dominant), or with a slower rate (when K 3 is dominant) towards the posterior actuators of the robot.Therefore, the parameter vector that was optimized via policy-search RL (i.e. the policy parameters) is . For the turning maneuver, the voltage limit is set as ±30 V.

Experimental motor learning via EPHE
In our previous work, we used parameter exploring policy gradient (PGPE) RL method for experimental motor learning, which requires empirical hyperparameters' tuning [41].To reduce the number of empirically-tuned hyperparameters, we employed the EM-based policy hyper parameter exploration (EPHE) algorithm for RL of the µBots motor control.EPHE integrates PGPE with EM-based updates to maximize a lower bound of the expected return during parameters' update [59].A standout feature of EPHE is that it eliminates the need for a learning rate, thus reducing the number of hyperparameters that require empirical tuning.Within this framework, the policy parameters vector ν is sampled from the normal distribution N (ν | h) of the hyperparameter vector h which governs the policy parameters' distribution.The hyperparameter vector h = [µ σ] T contains µ and σ for the mean and standard deviation of the normal distribution of ν.During each episode, M policy parameters vectors v m were sampled from N (ν | h) and the top K parameters vectors were selected based on their corresponding reward R(v m ).Thereafter, the hyperparameter vector h was updated based on the sampled parameters and rewards as, where v k denotes kth sampled policy parameters and R(v k ) represents the reward for v k .In our study, the reward was quantified as either the steady swimming speed (for forward swimming) or the final heading change (for turning maneuver).
In EPHE, the number of sampled rollouts (M) and the number of selected best samples (K) were the hyperparameters that were determined empirically.During our experiments, we sampled 20 rollouts in each episode and selected the seven best samples (M = 20 and K = 7).With this hyperparameter configuration, the learning experiments typically reached convergence in fewer than 20 episodes.A learning experiment was concluded once fluctuations in the average swimming speed remained below 2 mm s −1 (in the case of forward swimming) or when the variations in the average turning angle stayed within 2 • (for turning maneuver) across three consecutive episodes.

Forward swimming 3.1.1. Swimming speed
The convergence of reward (i.e.forward swimming speed) during the learning process is shown in figure 5(a) (an example of the learning process is also provided in video S1).The maximized speed and converged motor frequency corresponding to the seven cases of caudal fin stiffness are shown in figures 5(b) and (c), respectively.The highest speed was achieved in case-3 at 1.3 BL/s (215 mm s −1 ), with stiffness at 2.30 × 10 −5 Pa m 4 .Our previous work showed that swimming speed in BL/s of µBots decreased with the body length [42], and the similar trend is also observed in biology [60].However, here we showed that case-0, with shortest body length due to the lack of a caudal fin, can only achieve 0.36 body length per second (BL/s) or 51 mm s −1 , which is the slowest among all the cases, confirming the importance of caudal fin in BCF swimming.

Motor frequency
In prior research, we have shown that the swimming gaits and speed of µBot are highly sensitive to the frequency of motor input [41,42].Here, our results showed that the optimized frequencies can be separated into three groups (figure 5(c)).In the absence of a caudal fin, the optimized frequency was around 4.6 Hz (with standard derivation (SD) = 0.09 Hz), significantly lower than those of other cases.For cases 1-3 with low to medium stiffness, the optimized motor frequency is around 10.6 Hz (case-1: 10.9 Hz with SD = 0.51 Hz; case-2: 10.4 Hz with SD = 0.47 Hz; case-3: 10.6 Hz with SD = 0.49 Hz).For cases 4-6 with medium to high stiffness, the frequency is about 8.4 Hz (case-4: 8.3 Hz with SD = 0.30 Hz; case-5: 8.2 Hz with SD = 0.32 Hz; case-6: 8.7 Hz with SD = 0.35 Hz).

Swimming gaits
The three groups of the optimized frequencies were associated with three types of swimming gaits emerged from the FSI (figure 6, also see video S1), primarily differentiated by the number of wave(s) (N) along the body: N = 1 for µBot without caudal fin, N = 2 for µBot with low-medium caudal fin stiffness and high motor frequency, and N = 1.5 for µBots with medium-high caudal fin stiffness and low motor frequency.
Without caudal fin: in the absence of a caudal fin, the whole body of the robot bended like a 'C' shape oscillating from side to side as a single standing wave (N = 1, see the cyan curve for case-0 in figure 6), leaving a large lateral displacement in the middle of the body (at joints 2 and 3) and two points with small oscillatory amplitude (i.e.nodal points): one slightly posterior to joint-1 and the other slightly anterior to joint-4.This gait was unique to the µBot absent of caudal fin and not observed in µBots with a caudal fin.
With caudal fins: for cases 1-6, a consistent pattern emerged in the anterior portion of the robot, i.e. they shared identical nodal points anterior to joint-1 and in the middle of joint-2 and joint-3.However, the overall lateral displacements of medium-high stiffness (cases 4-6) were higher than those observed in low-medium stiffness (cases 1-3).The most noticeable differences were at the posterior part of the body (joint-3, joint-4, peduncle, and caudal fin), where the joints 3 and 4 are the last two active joints that include actuators, and the peduncle is the passive joint between the caudal fin and joint-4.For cases 1-3, the reduced stiffness allowed pronounced lateral movements of the peduncle and the large deformation of the caudal fin (as further illustrated in figure 7(a)).This resulted in the robot exhibiting a dual-wave undulation along its body (N = 2, see the cyan curve for case-2 in figure 6).In contrast, the high stiffness in cases 4-6 curtailed the lateral motion of the peduncle and deformation of the caudal fin (refer to figure 7(a)), causing joint-4, the peduncle, and the caudal fin tip to move as nearly a single segment (therefore elongating the effective length of the caudal fin).Consequently, the robot's undulatory pattern transitioned to a 1.5 wave cycle (N = 1.5, cyan curve for case-5 in figure 6).Note that the bending angles of the body segments are usually between 10 • and 20 • [42].This range falls between the caudal fin deformation angles of case-3 and case-4 (figure 7(a)).

Strouhal number
We also calculated the average lateral undulatory velocity of the peduncle and the caudal fin tip, Strouhal number (St), and body-and-caudal-fin Strouhal number (St BCF ), as presented in figure 7(b).St is a dimensionless fluid number characterizing the wake structures and hydrodynamics of undulatory swimmers [61,62].In µBot swimming, St is defined as, where f is the undulating frequency; A is the lateral total excursion of the caudal fin tip (twice the caudal fin displacement); U is the swimming speed; V cf = 2fA is the caudal-fin tip velocity.In our previous work, we found that St BCF better separates the different swimming modes compared with traditional St [42].BCF Strouhal number St BCF is defined as in a similar way with St, where V BCF is the weighted sum of the lateral velocities of the entire BCF (except head) normalized by the total body length L, where n is the number of body segments; V k and L k are the lateral velocity (V k =2fA k , where A k is the lateral total excursion of kth segment) and length of kth segment, respectively.The lateral velocity of the peduncle peaked at case-1 and consistently declined through case-6.Intriguingly, the highest lateral velocity of the caudal fin tip was observed in case-3, aligning perfectly with the peak swimming speed (figure 7(b)).Focusing on the Strouhal number (St), its minimum value of 0.55 was registered at case-6.Meanwhile, the caudal fin stiffness that produced the topmost swimming speed (case-4) had a St of approximately 0.8.This St value for µBots is notably higher than the 0.2-0.4range, conventionally considered optimal for biological swimmers [61].As for the St BCF , it remained nearly constant from case-1 to case-5 at an average of 0.21.However, both case-0 and case-6 deviated from this trend, recording higher values of 0.37 and 0.32, respectively.

Turning maneuver 3.2.1. Final heading change
The convergence of reward (i.e.final heading change or turning angle) during the learning process is shown in figure 8. Without a caudal fin, the µBot could barely turn, as its final heading change was less than 5 • , indicating the critical role of caudal fin in turning.For the low-medium caudal fin stiffness (cases 1-3), the final heading change increased with the stiffness, while largest head turning angle was nearly 46 • (at case-3, same as the forward swimming).As the caudal fin further stiffened, the final heading change declined slightly and became nearly constant at approximately 34 • for cases 5 and 6.

Motor control program and turning gaits
Without caudal fin: the optimized motor program and turning gaits for case-0 without the caudal fin were presented in figures 9(a) and 10(a), respectively (video S2).In stage 1 (bending stage), there were relatively large delays in propagating actuation voltage V 1 (or initiation of bending) along body the segments, which had led to discrete segment-by-segment bending along the body and little net body rotation (note that smooth body bending resulted from smaller delays would have led to even smaller final heading change).In stage 2 (restoring stage), the voltage V 2 has a nearly equal magnitude to V 1 but at opposite polarity, indicating a symmetric bending and restoring torque applied by the actuators; this led to a near symmetric bending and restoring body motion, therefore resulting in near zero head turning angle (5 • ).With caudal fin: for cases with a caudal fin (case 1-6), the motor control and turning gaits were presented in figures 9 and 10 (also see video S2) and a detailed view of case-3 gaits was presented in figure 11.In stage 1 (bending stage), there were relatively smaller delays in propagating actuation voltage V 1 along body the segments, which resulted in a phase lag between the bending of the anterior and posterior of the body (figures 9(b)-(g)).The learned V 1 in stage 1 (figure 4) reached the maximum voltage limit for all the cases except case-2 (figure 9(c)).In the initial period, a rapid 'I → S' shape change was generated by the anterior body bending (the anterior bend of 'S' , figures 10(b)-(g) and 11(b)) and the differential action of the anterior bending and the resistance provided by the caudal fin (leading to the posterior bend of 'S').The resistance of the caudal fin likely came from the fluid added mass (or inertia) and damping [9].Next in stage 1, the actuation voltage V 1 arrived at the posterior part of body, extending the anterior body bending to the entire body, and therefore leading to a 'S → C' shape change (figure 11(c)).More importantly, 'S → C' shape change led to a rapid stroke of the caudal fin, which hypothetically generated a large hydrodynamic force and a body torque that led to a net body rotation (figure 11(d) and video S2).This body rotation continues in a coasting period when the V 1 was still applied and the body remained at the 'C' shape (lasted approximately for 2 s, figures 10 and 11(d)-(f)).In stage 2, the body returned from 'C' to 'I' shape, while accompanied by a moderate amount of recoil that was lower than the initial turning angle (figures 11(g)-(h)).Their difference resulted in a final heading change at the end of the entire maneuver.The voltage magnitude (V 1 in figure 4) was either nearly zero (figures 9(b), (c) and (g)) or very small and opposite to the returning motion of the body (figures 9(d), (e) and (f)), so that the 'C → I' shape change was likely driven by the body stiffness.The restoring stage was also relatively slow compared to the bending stage.
The most noticeable effect of caudal fin stiffness was shown by the exact location of the posterior bend of the 'S' shape during 'S → C' shape change (orange curves, figures 10(b)-(g)).The posterior bend occurred at the peduncle for more compliant caudal fins but moved anteriorly when stiffness increased.With more compliant caudal fins, the caudal fin experienced apparent large deformation when the robot assumed 'S' shape and then bent to 'C' shape (case-1 and case-2).However, for stiffer caudal fins (case-3 to case-6), the caudal fin deformation (figure 10(h)) became significantly lower (the motion of the caudal fin and the last body segment became more synced, therefore extended the effective caudal fin length).The posterior bending, however, moved towards the body segment anterior to the peduncle (joint-4), resulting in an 'S' shape of relatively larger posterior proportion than those of more compliant caudal fins.During the 'C → I' shape change, the posterior bend also occurred at the peduncle for more compliant caudal fins and move to joint-4 for stiffer caudal fins (purple curves, figures 10(b)-(g)), and the caudal fin deformation decreased with the stiffness (figure 10(h)).
The final heading change, the initial head steering, and head recoil for all the cases were also shown in figure 10(h).The initial head steering increased from case-0 to case-3 and stayed relatively constant from case-3 to case-6.Case-3 also head the smallest head recoil, therefore achieving the largest final heading change.

Discussion
In this work, we systematically examined the effects of the caudal fin stiffness on both forward swimming (using µBot-4) and turning maneuver (using µBot-6) via experimental robot motor learning.Our results confirmed the indispensable role of caudal fins in both swimming functions-the µBot absent of a caudal fin swims the slowest (figure 5(b)) or can barely turn (figure 8(b)), even with an optimized motor program.We also quantified how caudal fin stiffness affects the optimal motor program, the swimming gaits and the swimming performance.For forward swimming, variations in caudal fin stiffness led to three modes of optimized swimming gaits with distinct motor frequency (figures 5(c) and 6); for turning maneuver, the caudal fin stiffness had considerable effects on the 'I → S → C → I' shapechanging sequence (figure 10) with relatively minor effect on optimized motor program (figures 9(b)-(g)).For both functions, the stiffness affected the amplitudes of caudal fin deformation, therefore changing the effective caudal fin length, as illustrated in figure 12. Notably, among the six tested stiffness, case-3 with the medium stiffness at 2.30 × 10 −5 Pa m 4 emerged as the optimal one for both swimming functions.

Combination of caudal fin stiffness and motor frequency introduces two modes of forward swimming in terms of effective caudal fin length or caudal fin deformation (N = 1.5 vs N = 2)
Our results show that there were two swimming modes (N = 1.5 and N = 2) emerged from two combinations of the caudal fin stiffness and the motor frequency (figure 12).When µBot caudal fin had low-medium stiffness and operated at high frequency (∼10.6 Hz, cases 1-3), the caudal fin experienced considerable deformation (as measured by lumped peduncle angle, 26.1 • -33.0 • ), and functioned as a separate segment from peduncle (rendering number of body wave N = 2).On the other hand, with medium to high stiffness and low frequency (∼8.4 Hz, cases 4-6), the caudal fin experienced negligible deformation (1.6 • -6.5 • ) and effectively joined with the peduncle as a single caudal fin segment with increased length (rendering number of body wave N = 1.5).Notably, despite the monotonic increase of caudal fin stiffness, both the learned motor frequency and the degree of caudal fin deformation (figure 12) remained quite constant within each mode.The transition between the two modes occurred quite abruptly following the stiffness corresponding to the highest swimming speed (cases-3), which corresponds to the highest caudal-fin tip velocity (figure 7(b)).

µBots' turning gaits share similarity with the C-start in fish
Our results show that the optimized turning gaits of µBots resemble the C-start turning maneuver in biological fish with notable differences.C-start often involves the formation of an 'S' shape before transitioning into a complete 'C' shape [36,44], a pattern also observed in µBot' turning gaits (figure 11).Additionally, µBot exhibited a learned short time delay in body actuation, similar with the muscle activation pattern of fish during C-start turning [53].However, it is crucial to distinguish the primary functions of these maneuver: C-start in fish functions as an escape maneuver, facilitating rapid changes in both heading and position [10,36,63].To accelerate the process, fish generate large contralateral muscle force to kick out of a transient 'C' shape [52,53], the duration of which is usually tens of milliseconds [64].In contrast, µBot in our experiments seeks to only maximize the final heading change, the optimized motor program therefore leads to a coasting motion by maintaining the 'C' shape to extend the rotation period.Subsequently, it gradually transitions from 'C' back to 'I,' presumably reducing head recoil.To fully comprehend and replicate the complete C-start gaits in fish, future efforts should consider designing learning rewards that account for escape acceleration, distance, and angular turning speed.

Comparison of swimming performance between µBot and biological fish
Among all the stiffness investigated for µBot swimming, we found that the stiffness for the fastest forward swimming (1.3 BL s −1 ) and the largest turning angle (46 • ) were identical.These performance measures are comparable with many biological fish.For example, Axolotls generally swim at 0.5-3 BL s −1 [65]; Green jacks have a common swimming speed between 0.8 and 5 BL s −1 [66]; Red seabreams often have a turning angle between 25 • and 100 • [67].However, fish can reach a top speed at more than 20 BL s −1 [68] and a maximum turning angle at more than 150 • [69], indicating that there is still a discrepancy between µBot and fish.
In our previous work, we found that St BCF reveals two fundamental classes of undulatory swimming in both biological and robotic swimmers, i.e. slow speed-to-undulation swimmers (St BCF = 0.186) and fast speed-to-undulation swimmers (St BCF = 0.066), and all the robots belong to slow speed-to-undulation swimmers, including µBots [42].Our work here suggests that tuning caudal fin stiffness alone is insufficient to enable the robot to swim in fast speed-toundulation class (figure 7(b)).To achieve faster speed and fast speed-to-undulation class, further investigations on caudal fin design, body shape, and body stiffness would be required for the robot to exploit the liftbased (vorticity) method, a mechanism that is exploit by many fish species [70].
Regarding the turning maneuver, fish can generate a burst of muscle force for rapid movements.For example, Amia calva, comparable in size to the µBot, can exert muscle forces amounting to several Newtons for body bending [71].In contrast, µBot actuators are limited to a force output of 0.25 N.For achieving a larger turning angle, actuation mechanisms that are capable of generating a burst of force should be considered in the future work.

Optimized forward swimming and turning maneuver share an identical caudal fin stiffness
It is widely believed that fish's form and function are closely related, for example, their body morphological traits are used to categorize them into specialists or generalists for cruising, accelerating and maneuvering [2].In this work, among all the stiffness investigated for µBot swimming, we found that the stiffness for the fastest forward swimming (cruising) and the largest turning angle (maneuvering) were identical.This result is related to the shared effect of the caudal fin stiffness on caudal fin deformation and its effective length when joined with peduncle (figure 12).Although our results are limited to µBot forward swimming and turning, this result points to the possibility that a morphological feature or mechanical property can promote multiple swimming functions, which could potentially simplify the form-function relationship and the corresponding robot design.Nonetheless, more comprehensive future work with more extensive robot configurations and swimming functions needs to be tested for generating more concrete design principles.

Figure 1 .
Figure 1.Overview of µBot platform.(a) Design of µBots (µBot-6 used as an example) with the important segments labeled.Caudal fin was designed to be easily installed or removed through the sliding slot, which simplified the caudal fin replacement.A cross-section view of the body segment is presented with actuator's components labeled.(b) Photos of top view of µBot-4 and µBot-6 with rubber suit removed.

Figure 2 .
Figure 2. Experimental setup for robot learning.The motor control of µBots were generated by a CPG (forward swimming) or square waves (turning maneuver).The parameters of the CPG and the square waves were updated with policy gradient RL method based on the swimming performance.

Figure 3 .
Figure 3. CPG network for µBot-4 in forward swimming.The CPG parameters labeled with colors were learned experimentally, while those in black were fixed.The values or description of the parameters are shown in table 2. The dashed box represents a CPG module corresponding to a single actuator.

Figure 4 .
Figure 4. Motor control for turning maneuver of µBot-6.Dot and curve with the same color in the figure denote the position and motor signal for the same actuator, respectively.

Figure 5 .
Figure 5. Experimental learning results for µBot-4 in forward swimming.(a) Convergence of forward swimming speed in experimental learning.(b) Dependency of swimming speed on caudal fin stiffness.(c) Dependency of motor frequency on caudal fin stiffness.The colored shadows represent three different groups based on the motor frequency.They also corresponded to three types of swimming gaits presented in figure 6. Error bars represent the standard deviation, which are obscured by the symbols for some points.Note that the body length of case-0 µBot is shorter than other cases due to the lack of a caudal fin.

Figure 6 .
Figure 6.Examples of optimized motor inputs and midline kinematics for µBot-4 in forward swimming.Dot on robot sketch and motor input curve with the same color in the figure denote the position and motor signal for the same actuator, respectively.The lateral displacements of all the midline kinematics were extracted from the experiments, which were scaled up two times for clearer illustrations.The colored shadows correspond to three types of swimming gaits.The cyan curves mark the body waves for the three types of swimming gaits.

Figure 7 .
Figure 7. Details caudal fin kinematics in forward swimming.(a) Dependency of caudal-fin tip displacement (black dot), peduncle displacement (green square), and lumped bending angle of peduncle (red dot) on the caudal fin stiffness.(b) Dependency of caudal-fin tip velocity (black triangle), peduncle velocity (black star), Strouhal number (brown dot), and BCF Strouhal number (gray square) on the caudal fin stiffness.

Figure 8 .
Figure 8. Experimental learning results for µBot-6 in turning maneuver.(a) Convergence of final heading change in experimental learning.(b) Dependency of final heading change on caudal fin stiffness.Error bars represent the standard deviation.

Figure 9 .
Figure 9. Learned motor inputs for turning maneuver.(a)-(g) represent case-0 to case-6.Dot on robot sketch and motor input curve with the same color in the figure denote the position and motor signal for the same actuator, respectively.

Figure 10 .
Figure10.Learned body kinematics for turning maneuver.(a)-(g) represent case-0 to case-6.Red, green, and blue lines represent the initial heading, the initial head steering, and the heading after the turning maneuver, respectively.The final heading change (red), head recoil (light green), and initial head steering (black) are presented for each case.The orange curves and purple curves present the body shape when the caudal fin had the largest deformation in stage 1 and 2, respectively.(h) Dependency of heading and caudal fin deformation on the caudal fin stiffness.

Figure 11 .
Figure 11.Example of turning gait of case-3.(a)-(h) represent different moments in time sequence.Gray dotted lines indicate the initial position of the robot.Red arrows represent the instant movement of different parts of the robot body.Blue curve arrows in (b) and (c) demonstrate the instant bending direction of the caudal fin.

Figure 12 .
Figure 12.Summary of the effects of caudal fin stiffness.Stiffer caudal fin syncs the caudal fin motion with the peduncle segment so that the effective caudal fin becomes longer.

Table 1 .
Numbering, material, flexural stiffness, and description of the caudal fins.