Optimal active particle navigation meets machine learning^(a)

Besides macroscale agents, even microorganisms can perceive information from their environment and use it for navigation. For example, the ability of sperm cells to sense gradients in the concentration of those chemicals which are emitted by the egg cell [10,11] is crucial for the survival of many species. Similarly, bacteria possess a remarkable spectrum of biochemical sensors which allow them, e.g., to measure gradients in oxygen, nutrient, or autoinducer concentration [12–14] and to use them for navigation.

Besides biological microswimmers, since almost two decades [15] also synthetic microswimmers have become available, and are currently studied together in the research field of active matter [16–35]. Synthetic microswimmers can be steered by external fields [36–53] (or even with feedback control systems [54–56]) and can react to their environment through various forms of taxis [57,58], which may be used in the future to help them navigate through our blood vessels to detect and perhaps repair mutated cells [59,60], transport drugs to cancer cells [61–63] or perform microsurgery [64].

While optimal navigation problems at the macroscale have been studied for decades [65,66], based on methods such as optimal control theory and dynamic programming [67–70], and more recently reinforcement learning [71,72], at the microscale corresponding explorations have started only recently. Here, the smallness of the particles leads to various new challenges: i) Microswimmers are subject to significant fluctuations due to Brownian motion (or errors and delays in the steering protocol), hence they cannot accurately predict the outcome of their navigational maneuvers. ii) Microswimmers interact hydrodynamically with walls, obstacles, and other microswimmers, which can qualitatively change the required navigation strategy to reach a target fastest [73]. iii) The displacement rate of microswimmers due to their environment can exceed their self-propulsion speed (typically $\sim \mu\,\text{m/s}$) by orders of magnitude, e.g., in the blood vessels which is opposite, e.g., to airplanes in the wind. iv) For many prospective applications, microswimmers will face unknown environments with only local information about their surroundings available and will require transferable navigation strategies (fig. 1).

Zoom In Zoom Out Reset image size

Consider an overdamped dry active particle (no hydrodynamic interactions) in a time-independent and two-dimensional complex environment. The equation of motion for the particle position $\vec{r}(t) = (x(t), y(t))$ can be compactly written as [86]

$$\begin{equation} \dot{\vec{r}}\,(t) = v_0(\vec r) \hat{n}(t) + \vec{f}(\vec{r}) + \sqrt{2D} \vec{\eta}(t). \end{equation} \tag{ 1 }$$

Here $\vec f(\vec r)$ represents a general force, flow, and viscosity field and D, $\vec \eta$ represent the translational diffusion coefficient and Gaussian white noise of zero mean and unit variance. Let us first focus on the idealized situation where the agent can freely and instantaneously control its self-propulsion direction $\hat{n}(t) = (\text{cos}\psi(t), \text{sin}\psi(t))$ but not its speed $v_0(\vec r)$, which may depend on space [53,87]. The goal is to find, for a given starting and end point, the connecting path $\vec r(t)$ (equivalently the steering angle $\psi(t)$), allowing the active particle to reach the target fastest. To solve this problem for vanishing noise $(D=0)$, we now write the traveling time as a functional of the path y(x) and of $y^{\prime}(x) = \mathrm{d} y(x)/\mathrm{d} x$ as

$$\begin{equation} T\left[y(x), y^{\prime}(x), x\right] = \int\limits_{0}^{T}\mathrm{d} t = \int_{x_{A}}^{x_{B}} \frac{\mathrm{d}x}{|\dot x(y(x),y'(x),x)|} \end{equation} \tag{ 2 }$$

and minimize it by solving the Euler-Lagrange equation $\frac{\mathrm{d}}{\mathrm{d} x} \frac{\partial L}{\partial y^{\prime}}-\frac{\partial L}{\partial y}=0$ (boundary value problem) for $L\left(y(x), y^{\prime}(x), x\right)=\frac{1}{|\dot x(y(x),y^{\prime}(x),x)|}$, where $\dot{x}$ denotes the velocity in the x-direction. Using (1), one obtains the Lagrange function [86]

$$\begin{equation} L(x, y(x),y^{\prime}(x))\!=\!\frac{\left(1\!+\!y^{\prime 2}\right)}{\left|f_{x}\!+\!y^{\prime} f_{y} \!\pm\! \sqrt{v_{0}^{2}\left(1\!+\!y^{\prime 2}\right)\!-\!\left(f_{y}\!-\!y^{\prime} f_{x}\right)^{2}}\right|}\!, \end{equation} \tag{ 3 }$$

where $\vec f = (f_x,f_y)$ and v₀ may depend on x, y.

Active particles and ray-optics: The Euler-Lagrange equation, together with eq. (3) can be readily solved for constant $\vec f$ and constant v₀, yielding $y^{\prime}(x)=\text{const}$, showing that the shortest path is the fastest in any constant field. That is, the active particle steers such that it exactly compensates for the drift due to the environment. In piecewise constant environments, the optimal trajectory is also piecewise constant, yielding Snell's law for active particles which involves a generalized refractive index that can also be negative as for light in meta-materials [88,89]. (See also [90].) Other exact solutions for the Euler-Lagrange equation can be obtained by exploiting conservation laws (symmetries) showing that the shortest path is typically not the fastest in complex environments. In rotating flow fields, active particles sometimes even have to initially swim away from the target to reach it fastest. Note that optimizing other quantities, such as the dissipated power along the path, leads to a different Lagrange function and hence in general also to a different navigation strategy [86].

Instead of dry active particles, ref. [73] considers microswimmers which hydrodynamically interact with walls and obstacles. One key result was to show that the optimal navigation strategy which microswimmers require can qualitatively differ from the one which leads to optimal trajectories for a dry active particle or a macroscopic vehicle in the same environment (fig. 3(a)).

In very complex environments (e.g., chaotic or unknown cases), optimal navigation problems can typically not be solved exactly, but methods based on reinforcement learning [83] can still be applied. Some works have used tabular Q-learning, for efficient real-time control of self-thermophoretic active particles (fig. 2(b)) [91], or for learning to navigate optimally inside an environment hosting a Mexican hat potential without brim (fig. 2(c)) [92]. Other recent works have used actor-critic methods [93] and also deep reinforcement learning [94,95] to study optimal navigation within increasingly complex environments. In ref. [95] in particular a deep reinforcement learning-based method has been developed to determine asymptotically optimal trajectories. Here, the key challenge was to develop an approach that is capable of finding the global optimum rather than some locally optimal path. The key "trick" to meet this challenge was to use a policy gradient-based method that "understands" and directly focuses on optimizing the expected total reward (as opposed to the off-policy methods such as Q-learning). To benchmark this method, results were compared to exactly known optimal trajectories (fig. 3).

Reference [96] treats the microswimmer navigation problem as a Markov decision problem and minimizes a cost function to solve mazes, assuming global knowledge of the environment (see also [97] for maze solving with droplets). Very recently, swarms of intelligent colloidal microrobots were also trained for capturing Brownian cargo particles within mazes [98]. Apart from maze solving, recently, a series of studies [98–100] have explored microswimmer navigation also in an unknown environment containing obstacles that are locally explored by the microswimmer, using (deep) reinforcement learning. It was found that smart colloids receiving local sensory input were able to navigate around obstacles to reach a target using deep reinforcement learning [99] and to accomplish complex navigation and localization tasks under time constraints [100].

As opposed to unknown environments, for the problem of optimal navigation in chaotic (turbulent) environments, one can in principle use optimal control theory (Pontryagin's principle) to determine the exact optimal trajectory. However, this problem is numerically not easily solvable with shooting methods since, in chaotic environments, a tiny variation of the initial condition commonly results in a completely different endpoint, and a systematic variation of the initial conditions is not useful and one needs to work with an extended target domain. This difficulty in evaluating the equation representing the exact optimal solution makes the usage of reinforcement learning particularly valuable and accordingly, ref. [93] has recently used an actor-critic–based reinforcement learning approach for Zermelo's problem in turbulent flow fields (fig. 2(d)).

Reference [94] in turn has explored the effects of accounting for environmental cues (such as vorticity, flow velocity, etc.) within the input features of a deep reinforcement learning method and the resulting strategies for a point-to-point navigation task within a turbulent flow. Reference [101] in turn used an adversarial reinforcement learning method to train microswimmers for time-efficient point-to-point navigation within statistically homogeneous and isotropic turbulent fluid flows which were able to outperform the naive strategy of always moving in the direction of the target.

While the swimming direction of synthetic microswimmers can be typically controlled with external fields [33], many biological microswimmers autonomously change their swimming direction through suitable shape deformations. In line with that, several recent works have used reinforcement learning to explore the swimming mechanism of deformable agents [102–105]. Related works have used learning approaches to understand how a swimmer needs to deform to swim as fast as possible [106], to follow a predetermined path [107], to exhibit chemotaxis [108] or to achieve optimal point-to-point navigation [109–111]. For example, ref. [110] considers three-link models of (bionic) fish which receive only their orientation and distance to a (moving) target as input data. They learn generic strategies which are then explored in situations that the swimmer has not encountered throughout training.

In another line of work, smart active particles have been trained to exploit underlying turbulent flows to escape local fluid traps [112], reach target regions with high-vorticity [113], or navigate towards the highest altitude achievable [114]. Very recently, smart microswimmers equipped with tabular Q-learning were also able to demonstrate efficient navigation strategies (while only having access to local information) within environments hosting various motility fields [115].

When learning the result of a navigation problem (or of another complex control problem), it often remains unclear how close the resulting trajectory or navigation strategy is to the real optimum. Of course, convergence of the reward does not guarantee optimality: The reward can converge to an arbitrary local optimum [83] and even if it converges to the global optimum it is often unclear if this optimum is representative for the (asymptotic) physical optimum, or just optimal within the given reward definition, discretization, hyperparameter choice, and the chosen learning algorithm. Accordingly, one major open challenge is to develop reinforcement learning approaches that can reproduce exact results and which can be used to go beyond those in [95].

While we expect that fluctuations can qualitatively change (fig. 1(b)) the required navigation strategy to optimally reach a target (fig. 2(a)) [79], as, e.g., in the cliff walking problem [147], they have little effect on the required navigation strategies in other problems. Accordingly, it would be important to systematically understand and formulate criteria for when fluctuations lead to strong quantitative or even qualitative changes in the required navigation strategy. As for the development of reinforcement learning algorithms, fluctuations lead to environments that are only partially observable, hence making the outcome of decisions (actions) made by the agents not accurately predictable. In certain problems, this unpredictability can challenge the robustness of training, which highlights the need for novel methods based on deep reinforcement learning (such as trust region methods [148,149]) which are capable of maintaining robust learning in volatile setups.

While microorganisms require strategies to navigate and find food in environments that they have never encountered before and which may change over time in an unpredictable way, many navigation problems for active particles so far hinge on a fixed (or deterministic dynamic) environment. An early work that addresses optimal navigation of "colloidal robots" in unknown environments based on deep reinforcement learning is [99]. Developing powerful methods to allow determining transferable navigation strategies in the future will likely require methods from model-based reinforcement learning [150,151] and more importantly world models [152] where the agents strive to learn a model (representation) of the environment (which here can even be translated to learning the physics of the setup) and use this representation of the environment for planning its future actions.

We are currently witnessing a rapid advancement in the development of new reinforcement learning methods. Accordingly, it is not surprising that some of the most powerful methods have not yet been applied to active matter and related optimal navigation problems. A very promising line of study that has been recently applied to famous games such as Chess, Go, and Shogi is the introduction of reinforcement learning algorithms with integrated planning such as the AlphaGo and AlphaZero [153–155]. We believe, given the robustness of their planning phase (thanks to a built-in Monte Carlo tree search of possible future outcomes), these methods can be very useful in tasks requiring high degrees of accuracy and confidence in the optimality of the learned strategies.

Recently, several works have focused on motile agents learning collective behaviors [156] such as flocking from simple low-level principles and incentive designs with reinforcement learning techniques [157,158]. A related line of study concerns the application of multi-agent reinforcement learning [159] to microswimmer problems. One can imagine complex tasks such as localizing cancer cells or collecting microplastics while having low environmental awareness (fig. 1(f)), which would require multiple smart microswimmers to cooperate and share their gathered knowledge of the host environment to guarantee success.

Data availability statement: No new data were created or analysed in this study.