Self-organization in Piano Playing: Why Pattern Transition?

Over billions of years, natural lives and organs have evolved with essential self-organized life-sustaining activities such as locomotion and respiration. Biological studies have shown that the Central Pattern Generator (CPG) is the essential mechanism for such pattern transitions. Embodied intelligence is a paradigm for investigating how humans employ the decentralized controller and embodiment of the body to interact with their environment (piano). Previously piano playing robots were hard-coded to implement mechanical contact and musical generation, resulting in clumsy non-anthropomorphic keystrokes. Instead, we revisit the piano challenge by studying the biomechanical physics of the body, the self-organization of reflexive behavior, and the neuro-muscular synergy in terms of coordinated behavioral diversity and energy minimization.


Introduction
Biology is full of examples where organisms perceive their bodies, control their bodies, and interact with the environment in a meaningful way.The paradigm, embodied cognition [2], shows that physical bodies developed to solve task-specific difficulties with the ecological assembly of perceptions, such as army ants finding the best route with their polarized light-sensitive eyes.The same cognitive design principles can be used in robotics.Exploring how biological systems make use of the physical bodies can give us insights into how the robotic system can have the same embodied intelligence among the brain (controller), body and interactions with the environment.The dead fish swimming experiment [1], for example, revealed that the body of a biological system can interact with its surroundings efficiently even without the assistance of the brain, which inspired the energy-free passive walker robot [3].Soft robots have been conceived, constructed, and applied to various dexterous manipulation tasks using the embodied intelligence emphasized between bio-inspired robotics and the environment to investigate how bodies work to minimize the load on the brain, how the sensorized robot can mechanically self-regulate its behaviors and how the "body shapes the way we think" [20].
Music, one elegant form of art, acts as a barrier-free language that bridges people from varying cultural backgrounds to communicate and share their emotions.For years of training, people have been trying to master musical instruments with the purpose of expressive playing, in which embodied intelligence plays a significant role in human-machine interaction.Piano playing, different from other instrumental performances, is a complex task emerging through the reciprocal interaction between the environment and the player's hands.Recent developments in robotics demonstrate an increasing interest in music and entertainment robots.The production 1292 (2023) 012015 IOP Publishing doi:10.1088/1757-899X/1292/1/012015 2 of high-quality expressive performance, particularly, requires nuanced coupling between the dexterity, adaptability, and mechanical compliance of the fingers and the dynamics of the piano itself.Traditional research on piano playing have centered on the joint biodynamics [5,13] of piano strikes and their robot replica [22,27,24,26,4,17,12,21,27,16,21,25].The piano challenge is narrowly understood either without considering the biomechanical constraint or merely via mechanically engineering a robot reproduction.The way how the body makes use of embodied intelligence over dexterous manipulation -piano playing in this case, needs to be further investigated.The study of fundamental pattern transitions in piano playing provides an entry point into the self-organization of the organ's essential life-sustaining behaviors.We will primarily consider: What is the definition of pattern transition?What kind of pattern transition can be observed?In what ways does embodied intelligence play a significant role in expressive piano playing?
There is debate regarding what the true pattern transition is and why it is significant in biological systems.The gait transition in stance-swing [14], walking-running [7,6,23], and swimming-crawling [11,10] are typical problems that are very interesting to investigate because they depict a natural and spontaneous shift from one state to another that obviously differentiates, without the need for complex additional system input.Researchers attempt to explain the transitional pattern, however, these are difficult to generalize to complex biological systems [18].We explore the pattern transition by investigating the coordinated upper-limb arm-swing behavior of a robot pianist with neuro-muscular synergy.Conventional understanding holds that CPG, which is derived from the spinal cord, is effective for coordinated rhythmic behaviors, such as natural vertebrate locomotion.However, it has yet to be demonstrated how this distributed/decentralized controller benefits non-locomotion cases.Pattern transition is ubiquitous in those repetitive coordinated rhythmic behaviors.One purpose of this study is to investigate whether pattern transition occurs in non-locomotion circumstances and how it affects expressive piano playing.
We exploit embodied intelligence to allow a robot pianist to perform expressive piano playing.The piano-playing scenario is replicated with anthropomorphic features including the anatomically-correct musculoskeletal upper-limb system, neuro-modulatory brain stem circuits and neuro-muscular synergy.The pianist interacts with the environment in a reciprocal way, during which we analyze how the body is making use of the self-organization, and embodiment of the body to address the piano challenge with a brainless controller.
The rest of this paper is organized as follows.Section 2 covered the pattern transition and the criteria for defining it.Section 3 depicts an anthropomorphic piano player's embodied intelligence and how the robotic multi-joint system exploits the pattern transition to achieve expressive piano playing.Section 4 provides a case study, while the subsequent sections provide a conclusion and discussion.

Spontaneous Pattern Transition
Patterns in nature are observable regularities, and the outcome of transitional action is obviously recognizable.Smooth pattern transitions are ubiquitous in biological systems; they are essential for natural bodies because they demonstrate how the system reacts to external stimuli at different levels of synchronization.This could be due to optimization considerations, such as minimizing energy expenditure.However, there is no definition of pattern transition and no explanation of how it may be generalized to other natural lives and organs.We put forth an effort to define the pattern transition, which must meet the following requirements: • Criteria 1: The transitional activity occurs in response to specific stimulus injection.
• Criteria 2: It is a spontaneous rather than manually imposed procedure.
• Criteria 3: The system's fundamental output exhibits nonlinear variation.
Traditionally, the research for pattern transitions centered on biological locomotion.The biped walking-to-running [23,7] study shows a very good example that how the gait of the biped walker smoothly changes to another form of pattern with only energy injection into the system.There is an evident measure to define the transition from a double-hump to a singlehump gait trajectory.The patterns differ in that when walking, at least one foot makes touch with the ground, whereas when running, both feet are off the ground for a certain amount of time.Note that the rise in speed caused by the energy injection does not count as a transitional activity (violating Criteria 3).Importantly, the transitional gait is achieved spontaneously by the body without any kind of manually-enforced programming.
There are numerous different forms of pattern transitions in nature that follow the aforementioned criteria 1.For instance, consider the flow pattern transition for boiling water.The water molecules are synchronized with distinct patterns as a result of the energy input, enabling the liquid-gas transition.It has been observed that a solitary firefly exhibits an uneven illumination pattern, however automatic synchronization is accomplished when they are placed in a swarm.We do not fully understand how they achieve the pattern transition but the stimuli input for the system could be the communication among the agents.When a predator is nearby, the same pattern explains the fish swarming behavior.Other natural locomotion examples include the swimming-to-crawling and gliding-to-flying transitions, with both neural modulation and energy as the transitional stimuli.
Interestingly, the falling paper displays an unusual non-locomotion pattern transition, since the interactions between the body and the environment can always be categorized into four patterns for a free-falling paper [9].Similarly, the neuronal cells of the brain are synchronized with distinct patterns for a relaxed and a focused state, respectively.The input of this transitional activity is still unclear, but it may be fluctuations in synaptic stimulation, such as the firing rate of neurons.Although it is difficult to determine a concrete measure to quantify the "nonlinear variation of the pattern," phase difference and offset appear to be viable candidates because they characterize the repetitive rhythmic system.For the walking-to-running gait transition, the phase difference, not the rise in speed, is considered a pattern transition.

Synchronized Arm-Swing in Piano Playing
Dexterous manipulation is a research realm in which many fingers or manipulators of the bespoke end-effectors collaborate with each other to manipulate objects.Dexterous soft manipulation is distinguished by the fact that it is object-oriented and can be accomplished with a variety of cooperative solutions among soft bodies [19].In piano playing, the multiple subsystems work cooperatively to perform keystroke actions, and the flow of keystrokes through time produces musical articulation.The entire piano-playing body is a synchronized neuro-modulation-based entity, with musculoskeletal tissues and bones serving as constraints.
Embodied intelligence's role has the potential to overcome the limitations of expressive robotic piano playing.It is believed to be a grand challenge for a robot to manipulate musical instruments and produce music with its own "emotional variety."Traditional musical robots are accused of being simply sound-generating machines rather than true musicians.Before we get into the technical details of robotic instrument playing.We must investigate the underlying mechanism by which humans use embodied intelligence for musical events.Several human piano playing studies have shown that for different-tempo piano keystrokes, the multi-joint system (wrist, elbow, and shoulder) exhibits varied patterns [5,13].For example, in a slow-frequency keystroke, the wrist joints contribute more, allowing the end-effector to work as a bouncing behavior, whereas in a high-speed piano strike, the motion in the elbow is the primary energy consumption.This is a natural technique in which the body synchronizes several groupings of musculoskeletal parts.Humans employ the primate corticospinal system to control individual finger and arm motions in order to master playing the piano [15].A distributed and decentralized controller is utilized in this process to exploit and prioritize the "intelligence of the embodiment."Self-organization in piano playing occurs as a result of neuro-muscular coupling, in addition to the decentralized controller (see Figure 2).The spinal cord-inspired controller (CPG) can synchronize the entire upper-limb system for pattern transition.In order to study how selforganization occurs in piano playing, we tune the CPG parameters, such as the firing rate, membrane potential, and body parameters, such as the spring, damping coefficients for the joints, center of mass, the density of the body, the gradient of the mass, and so on.

A Case Study: Robot Pianist
To investigate anthropomorphic piano playing, a high-fidelity replication of a pianist was computationally duplicated using a state-of-the-art multi-body dynamics platform.As shown in Figure 5, a three-neuron CPG oscillatory network has been configured as the basic neural input.An anthropomorphic hand and upper limbs have been duplicated based on the morphology of real anatomical elements.A full-scale modern piano is also modeled in the simulation with a row of 88 keys that includes 7 C-major scale octaves.The finger-key interaction is replicated as a collision between the hand's distal joints and the piano blocks, with friction defined to provide resistance to prevent relative motion.

Experimental Setup
We modeled the passive piano playing by mechanically replicating the geometric features of an anatomically correct hand.The hand is attached to a muscle-actuated upper-limb system.Note there is no individual actuation for finger joints and the spatial motions of the hand result from the sole actuation on the wrist.The compliant behaviors of fingers (output) are caused by the contact between the fingertip and piano note and are an indirect outcome of the wrist actuation (input).In addition, joint sensor blocks are used to allow the measurement of joint angle and torque.The monitored data is then written to the Workspace for further iteration in parameter optimization.The connecting joint between two geometric bodies is modeled as a mass-spring damping system which is constrained by a revolute joint with certain equilibrium positions.
The key-strike action is represented as the collision between two geometries.In the simulator, we defined the contact between the fingertip and piano key by adding the Spatial Contact Force block.This block prevents the penetration between two objects and is capable of sensing the normal and frictional forces on the contact point, which enables the dynamic analyses of the finger-key interactions.
The simulation platform is able to imitate the pipeline of the entire piano playing scenario.Anthropomorphic upper limbs including the shoulder, elbow, wrist and hand, as well as an 88-key piano are built with detailed mechanical, geometric and auditory characteristics (Figure 4a and b).Two MIDI events, which resulted from the reciprocal interaction between the two primitives, are interpreted to assess the produced musical phrases(Figure 4c).The entire handpiano interaction is depicted in Figure 4d.For this simulator, the exclusive input is the torques exerted on the three joints.Figure 3 illustrates the block diagram of the detailed construction of the simulator in Simscape.
The keystroke action is the result of rhythmic coordination of the entire upper limbs, which includes the shoulder, upper arm, forearm, and hand.Figure 5 depicts the upper limb motions involved in piano playing.The practical actuation of the upper limbs is a complex synergy of multiple muscle groups such as deltoid, latissimus dorsi, biceps brachii, brachioradialis and so on.The upper limbs in the simulation are modeled from genuine bones with a humanoid morphology.In order to simulate the musculoskeletal motion in keystrokes, a computational forward-dynamic model [8] has been exploited to replicate the aforementioned three muscles in a biomechanical way.This model is based on a Hill-type muscle structure and provides a high-fidelity simulation of the concentric and eccentric contractions.The output is a one-dimensional force, while the model's inputs include muscle-tendon-complex (MTC) length, contraction velocity, and neural excitation.

Coordinated Rhythmic Behaviors
The robot player is instructed to actively change the articulation styles during the performance to showcase the robot's ability to handle different expressive patterns.We have analyzed the motions of the entire multi-motor upper limbs and the corresponding MIDI output for a tempotransition piano playing case.With different levels of biologically inspired neural synaptic excitation, the pattern transition between slow and fast styles is performed dynamically.The robot pianist manipulates speedchanging musical flows via neuro-muscular synergy, during which the body and reflexive neurons exhibit spontaneous and self-organized shifts.It can be seen from Figure 6a and 6b that the CPG-controlled multi-muscle system reaches a natural higher arm-swing frequency for the transition from Andante, 86BP M to M oderato, 113BP M , and vice versa.The resultant tempo  transition is achieved by tuning the CPG neuron's excitation to the musculoskeletal multi-joint system, which replicates the neuro-modulatory mechanism in real human limb muscles.The endeffector achieves steady repetitive keystroke actions despite the variations in limb displacement.
The coordinated arm-swing behaviors ensure repetitive keystroke actions for the finger-key interaction.The piano notes have been pressed down multiple times and the corresponding MIDI events were created for each keystroke.The MIDI protocol has been employed to parameterize the keystroke actions.In the simulation, the mechanical contact of finger-key interaction is converted into a MIDI event specifying the note's pitch, timing, and loudness.Figure 6c illustrates the resultant MIDI output for the Andante-Moderato transition patterns in line with the multi-joint coordination.
It can be seen from Figure 6c that the piano keyboard was manipulated successfully with an accurate switching between two expressive tempi, implying that the robot could accomplish a variety of expressive styles in terms of the articulation metric.For both the slow-to-fast and fast-to-slow transitions, the first keystroke pattern deviates slightly from the expected one, but it quickly stabilizes around the desired tempo with the supplied coordinated musculoskeletal system.A case study showing how embodied intelligence is the potential to solve the challenges of expressive piano playing is given.Based on this we conclude that the CPG is able to be exploited in non-locomotion cases.The CPG matters in the way that gives the neural excitation and stimuli to coordinate the repetitive behaviors in natural lives or organs.The embodiment of the muscular model, on the other hand, provides limitations and limits how the body interacts with

Figure 1 .
Figure 1.Various pattern transitions in nature.(a) The phase of water molecules performs a state transition from liquid to gaseous phase within water boiling.(b) A salamander robot[11] achieves transitional locomotion from swimming to crawling.Inspired by the aquatic-toterrestrial locomotion in vertebrate locomotion, the robot uses a primitive spinal cord model to modulate the complex locomotion dynamics including velocity, direction, and gait type for tetrapods.(c) The human's biped walking to running is a natural ubiquitous pattern transition[23, 7].(d) The gliding-to-flying transition.(e) Paper disk shows various falling patterns with varying morphological features[9].Based on automatic experimentation and unsupervised learning technique, the falling paper problem has been investigated, where the paper disk exhibits four falling patterns: tumbling, chaotic, steady and periodic.(f) Marine organism shows various patterns of swarm behavior and (g) The neural cells of the brain show self-organized transitional activity.

Figure 2 .
Figure 2. Neuro-muscular synergy in piano playing.(a) The entire upper limb is a complex biological system with a good embodiment, allowing for self-organized reflexive behaviors for pattern transitions.(b) A decentralized controller achieves the synchronization of the multimotor upper-limb system.(c) Coordinated piano playing depicts that the body is able to interact with the environment with reduced brain control.

Figure 3 .
Figure 3. Simscape block diagram of the piano playing simulator.

Figure 4 .
Figure 4. Different components of the simulation platform.(a) The anthropomorphic hand consists of anatomically correct objects placed in a natural bending pose.The revolute joints between two phalanges are characterized by spring stiffness and damping coefficient.(b) An octave starting from note C is simulated with 7 white keys and 5 black keys.The angular displacement of the key can be detected as a result of the finger-key contact.(c) Musical phrase produced from a single-note key-strike action is parameterized by two MIDI events.A triggering threshold is used to determine the key-press and key-release motions.(d) Illustration of the pipeline work of the simulator.First, actuation signals are sent to the upper-limb muscles to enable the 6-DoF spatial motions of the hand.Then, the contact with the piano denotes the hand-piano interaction, during which the finger joints demonstrate various passive dynamics.Finally, the angular displacement of the pressed key, together with the chosen threshold, is used to produce the musical signals that characterize the quality of the piano playing.

Figure 5 .
Figure 5.Control architecture of the CPG-based keystroke coordination for a virtual piano player with anthropomorphic musculoskeletal properties.

Figure 6 .
Figure 6.Tempo transition whilst piano playing.(a) The angular displacement of three upperlimb joints.(b) Z-axis displacement of the thumb and litter finger.(c) MIDI output for the Andante-Moderato transition.