Non-perturbative renormalization for the neural network-QFT correspondence

We consider a fully connected NN $f_{\theta,N}(x): f_{\theta,N}: \mathbb{R}^{d_{\textrm{in}}} \rightarrow \mathbb{R}^{d_{\textrm{out}}}$ with learnable parameters (weights and biases) $\theta = (W_0, b_0, W_1, b_1)$, a single hidden layer of width N, and an activation function σ:

$$\begin{equation} f_{\theta,N}(x) = W_1(\sigma(W_0x+b_0))+b_1, \end{equation} \tag{ 1 }$$

where the weights W_i and biases b_i characterize the affine transformation of each layer and σ acts element-wise. The weights W₀ and W₁ follow centered Gaussian distributions $\mathcal{N}(0, \sigma_W^2 / d_{\textrm{in}})$ and $\mathcal{N}(0, \sigma_W^2/N)$ respectively, and both biases b₀ and b₁ are drawn from centered Gaussian distributions $\mathcal{N}(0, \sigma_b)$. The input data x is a d_in-dimensional vector, while we take $d_{\textrm{out}} = 1$ for the output data for simplicity. As a consequence, W₀ is a $(d_{\textrm{in}}, N)$-matrix, W₁ a $(N, 1)$-matrix, b₀ a N-vector, and b₁ a scalar. The Gauss-net activation is slightly peculiar because it acts as an exponential of the layer output normalized by the data of the previous layer:

$$\begin{equation} x_1~\equiv \sigma(W_0 x + b_0) = \frac{e^{W_0 x+b_0}}{\sqrt{\exp \bigg[2(\sigma_b^2+\frac{\sigma_W^2}{d_{\textrm{in}}} x^{\,2})\bigg]}}. \end{equation} \tag{ 2 }$$

Finally, we stress that the NN is randomly initialized and that we will not consider the effect of training.

Information on the NN can be extracted by considering correlations of the outputs: they are encoded by the 'experimental' correlation (or Green) functions $G^{(n)}_{\textrm{exp}}$ [32]:

$$\begin{equation} G^{(n)}_{\textrm{exp}}(x_1, \ldots, x_n) = \langle\, f_{\theta,N}(x_1) \cdots f_{\theta,N}(x_n) \rangle, \end{equation} \tag{ 3 }$$

where the statistical average ⁹ is taken over a large number of NNs with identical N and parameter distributions. The numerical evaluation of these quantities is explained in appendix A.

NNs $f_{\theta,N}: \mathbb{R}^{d_{\textrm{in}}} \rightarrow \mathbb{R}^{d_{\textrm{out}}}$ with $N \to \infty$ are well-described statistically by a Gaussian distribution:

$$\begin{equation} P[\,f\,] = \frac{1}{Z}\, \exp \left(-\frac{1}{2}\, \int_{(\mathbb{R}^{d_{\textrm{in}}})^2}\, dx dy \, f(x) \Xi (x,y) f(y) \right), \end{equation} \tag{ 4 }$$

where the factor Z ensures that the expression is normalized when integrating over the full functional space:

$$\begin{equation} \int [df\,]\, P[\,f\,] = 1, \end{equation} \tag{ 5 }$$

$[df\,]$ denoting the path integral measure in functional space and $\Xi(x, y)$ the kinetic operator (Gaussian kernel). In general, we will omit the subscripts $(\theta, N)$ on NN function samples and write simply f.

The origin of this Gaussian behavior in the limit $N\to \infty$ can be traced from central limit theorem: since $f_\theta(x)$ is formally a sum of N identically distributed random terms which self-average. The function f(x) splits into two contributions:

$$\begin{equation} f(x) = f_{W}(x)+f_b(x), \end{equation} \tag{ 6 }$$

where $f_b(x)\equiv b_1$ being essentially N independent variables and following the Gaussian law $\mathcal{N}(0,\sigma_b)$, whereas $f_W(x)$ goes toward a Gaussian distribution only for large N. Formally, it reads as:

$$\begin{equation} f_W(x): = W_1 x_1, \end{equation} \tag{ 7 }$$

where x₁ is given by (2) (such that f_W depends on $W_0, W_1$ and b₀). As stated above, for large N, one expects that such a quantity self-averages around its mean, and thus that fluctuations are small:

$$\begin{equation} \langle\, f_W\, f_W \rangle \underset{N\to \infty}{\sim} \langle\, f_W \rangle \langle\, f_W \rangle = 0, \end{equation} \tag{ 8 }$$

where the last equality follows from the assumptions that initial distributions for θ are centered and non-correlated. Hence, the statistical properties of f_W are essentially given by a centered Gaussian distribution, up to $1/N$ corrections. Obviously, the random nature of f_W is inherited from the initial parameter distribution, however the asymptotic Gaussian behavior arises from the law of large numbers.

The 2-point correlation (or Green) function:

$$\begin{equation} K(x, y) \equiv \int [df\,]\, P[\,f\,] f(x) f(y), \end{equation} \tag{ 9 }$$

is the inverse of the Gaussian kernel $\Xi(x,y)$ which appears in the free action (4):

$$\begin{equation} \int dz \, \Xi(x,z) K(z,y) = \delta(x-y). \end{equation} \tag{ 10 }$$

However, according to (6), it is also possible to decompose K as:

$$\begin{equation} K(x, y) = K_W(x, y) + K_b(x, y), \qquad K_b(x, y) = \sigma_b^2, \end{equation} \tag{ 11 }$$

where K_W is the 2-point function associated to f_W . It corresponds to the Fisher information metric [27, 28] in the information geometry language, and is fixed from the choice of the activation function.

In this paper, we essentially focus on translation invariant kernels $K_W(x,y)\equiv K_W(\vert x-y\vert)$, which is achieved by the Gauss-net architecture (2), corresponding to the kernel:

$$\begin{equation} K_W(\vert x-y \vert) = \sigma_W^2 \,e^{- \frac{\sigma_W^2}{2d_{\textrm{in}}} \vert x-y\vert^2}, \end{equation} \tag{ 12 }$$

where $\vert x-y\vert: = \sqrt{\sum_i(x-y)_i^2}$ denotes the ordinary Euclidean distance between x and y.

In field theory language, the kernel enters in the definition of the classical kinetic action ¹⁰ (i.e. the log-likelihood in probability theory):

$$\begin{equation} S_{\textrm{kin}}[\,f\,]: = \frac{1}{2}\, \int_{(\mathbb{R}^{d_{\textrm{in}}})^2}\, dx dy \, f(x) \Xi (x,y) f(y), \end{equation} \tag{ 13 }$$

the corresponding probability distribution being given by the exponential law $P[\,f\,]\propto e^{-S_{\textrm{kin}}[\,f\,]}$ in (4). The n-point correlation (or Green) functions are defined as:

$$\begin{equation} G_0^{(n)}(x_1, \ldots, x_n) \equiv Z^{-1} \int [df\,]\, e^{-S_{\textrm{kin}}[\,f\,]} f(x_1) \cdots f(x_n). \end{equation} \tag{ 14 }$$

In the free theory, $G_0^{(n)}$ is completely determined in terms of $G_2(x, y) = K(x, y)$ through Wick's theorem, and vanishes for n odd [32]. Hence, this implies that:

$$\begin{equation} G^{(n)}_{\textrm{exp}}(x_1, \ldots, x_n) \underset{N\to \infty}{\sim} G_0^{(n)}(x_1, \ldots, x_n). \end{equation} \tag{ 15 }$$

In this subsection, we discuss some definitions related to the data-space ¹¹ corresponding to the NN input x, and how they differ from [32].

Continuity and infinity in computer science exist only as idealizations. First, any real number $x \in \mathbb{R}$ is represented numerically by a decimal number, bounded in precision by the number of bits used to encode it. For instance, if the maximal number of decimals is n₀, two numbers x and $x+10^{-m}$ cannot be absolutely distinguished for $m\gt n_0$. To be more realistic, we should see the data-space $\mathbb{R}^{d_{\textrm{in}}}$ as a lattice of step $a_0 = 10^{- n_0}$ rather than a continuum manifold. The lattice spacing a₀ provides what physicists call an UV cut-off. Varying this parameter amounts to changing the data resolution, in full similarity with spacetime resolution in usual QFT. Second, computers cannot store an infinite amount of information. For this reason, it cannot handle infinite numbers (except as special data types and formal rules) and it is necessary to restrict the data to a finite interval $x \in [- L/2, L/2]$ ¹² . Hence, we consider the data-space to be a $(2~N_0)^{d_{\textrm{in}}}$ square lattice with spacing a₀, $N_0~\in \mathbb{N}$, and total hypervolume:

$$\begin{equation} \mathcal{V} = L^{d_{\textrm{in}}}, \qquad L : = 2a_0N_0. \end{equation} \tag{ 16 }$$

It is generally more convenient to work in Fourier (or momentum) space. The allowed momenta $p = (p_1,\ldots, p_{d_{\textrm{in}}})$ lie in the first Brillouin region:

$$\begin{equation} p_i = \frac{2\pi n_i}{L}\,, \qquad n_i \in \mathbb{Z}_{N_0}. \end{equation} \tag{ 17 }$$

Note that we assume periodic boundary conditions. This is not a problem for L large enough; for small L ¹³ , we can simply repeat the data set a large number of times to obtain a large enough effective volume to make the boundary conditions irrelevant.

In the rest of this paper, we use the following definitions:

Definition 1. We call $(2N_0)^{d_{\textrm{in}}}$ the discrete volume and a₀ the working precision.

Note that $(2N_0)^{d_{\textrm{in}}}$ also counts the number of states in the first Brillouin region. In this discrete setting, we can write the Fourier series of the network f(x) (for $x\in \left(a\mathbb{Z}_{N_0}\right)^{d_{\textrm{in}}}$) as:

$$\begin{equation} f(x) = :\frac{1}{N_0^{d_{\textrm{in}}}}\sum_p g(p) e^{ip x}, \end{equation} \tag{ 18 }$$

the basis functions e^ipx being normalized such that $\sum_{x} e^{i(p_1-p_2) x} = N_0\delta_{p_1p_2}$. Note that (18) holds for any discrete function on the lattice. In the continuum limit, for small a₀ and N₀ large such that L remains fixed, discrete sums can be replaced by integrals. Moreover, for volume large enough, integrals becomes standard Fourier transform. We call this limit the thermodynamic limit following the standard terminology in physics, and we focus on this regime in our investigations. Taking the Fourier transform of the 2-point function:

$$\begin{equation} \tilde{K}(p) = \frac{1}{(2\pi)^d} \int dx \, K(x) e^{-ipx}, \end{equation} \tag{ 19 }$$

where $px: = \sum_{i = 1}^{d_{\textrm{in}}}\, p_i x_i$, we get for (12) ¹⁴ :

$$\begin{equation} \tilde{K}(p) = (\sigma_W^2)^{1-d_{\textrm{in}}/2} \left(\frac{d_{\textrm{in}}}{2\pi} \right)^{d_{\textrm{in}}/2} e^{-\frac{d_{\textrm{in}}}{2\sigma_W^2}p^2}. \end{equation} \tag{ 20 }$$

Note that in this continuum approximation, the Dirac delta $\delta(p)$ has to be understood as a shorthand notation for a Kronecker delta $(2\pi)^{-d_{\textrm{in}}} \mathcal{V} \delta_{p0}$. Translation invariance is crucial to obtain a kernel (20) which depends on a single momentum, and reflection invariance implies that it must be a function of p². It would be interesting to understand how to generalize our computations for kernels which are not translation invariant [32].

For small p², we may expand $\tilde{K}(p)$ in power of p²,

$$\begin{equation} \tilde{K}(p) = (\sigma_W^2)^{1-d_{\textrm{in}}/2} \left(\frac{d_{\textrm{in}}}{2\pi} \right)^{d_{\textrm{in}}/2}-\frac{d_{\textrm{in}}}{2}(\sigma_W^2)^{-d_{\textrm{in}}/2} \left(\frac{d_{\textrm{in}}}{2\pi} \right)^{d_{\textrm{in}}/2} \,p^2+\mathcal{O}(p^4). \end{equation} \tag{ 21 }$$

Up to $\mathcal{O}(p^4)$ corrections, the propagator looks like the canonical propagator of a free scalar field theory:

$$\begin{equation} \tilde{K}(p) = \frac{1}{m^2_0+Z_0p^2+\mathcal{O}(p^4)}, \end{equation} \tag{ 22 }$$

where:

$$\begin{align} m^2_0 = (\sigma_W^2)^{d_{\textrm{in}}/2-1} \left(\frac{d_{\textrm{in}}}{2\pi} \right)^{-d_{\textrm{in}}/2}\,, \quad Z_0 = \frac{d_{\textrm{in}}}{2}(\sigma_W^2)^{d_{\textrm{in}}/2-2} \left(\frac{d_{\textrm{in}}}{2\pi} \right)^{-d_{\textrm{in}}/2}. \end{align} \tag{ 23 }$$

In the QFT terminology, Z₀ and $m^2_0$ are respectively the wave function renormalization and bare mass. One can rescale the field to to set $Z_0 = 1$, in which case the mass becomes:

$$\begin{equation} \bar{m}_0^2 = Z_0^{-1} m_0^2 = \frac{2\sigma_W^2}{d_{\textrm{in}}}. \end{equation} \tag{ 24 }$$

We adopt the following definition:

Definition 2. The mass $\bar{m}_0^2$ defines the typical mass scale and its inverse defines the (IR) correlation length ξ, or the typical observation scale:

$$\begin{equation} \xi^2 : = d_{\textrm{in}}/(2\sigma_W^2). \end{equation} \tag{ 25 }$$

Note that the large volume limit is defined with respect to this correlation length, i.e. $L\gg \xi$. Beside the existence of an intrinsic length scale, the system the propagator at large distance behaves like $(\bar{m}_0^2+p^2)^{-1}$.

Assigning the label x (position space) to the original data and p (momentum space) to the Fourier conjugate may seem arbitrary. Indeed, while the machine precision provides a natural UV cut-off and an associated identification of the data-space as position space (since UV corresponds to small distances in that space), signals in Fourier space are also represented in the computer up to the machine precision. However, in that case, using the machine precision as a UV cut-off would not match the usual intuition in QFT. Given a translation-invariant kernel, another possibility is to identify the momentum space as the space where the propagator is diagonal, such that the propagator in position space depends on the distance $\vert x-x^{\prime} \vert$.

For a GP, as we have seen earlier, correlation functions $G^{(2n)}$ for n > 1 can be decomposed as a sum of product of 2-point functions thanks to the Wick theorem [23]. For N large but finite, the distribution is not exactly Gaussian, and the correlation functions do not match with Gaussian predictions.

The deviation of the QFT and experimental correlation functions from the Gaussian case are denoted as:

$$\begin{equation} \Delta G^{(n)}_{\textrm{exp}} : = G^{(n)}_{\textrm{exp}} - G_0^{(n)}, \qquad \Delta G^{(n)} : = G^{(n)} - G_0^{(n)}. \end{equation} \tag{ 26 }$$

Note that $G_0^{(n)}$ are still the large N Green functions defined in (14). Importantly, we identify the exact 2-point function $G^{(2)}$ with the kernel K which equals $G_0^{(2)}$. Since it contains (quantum) corrections due to the interactions, the free 2-point Green function computed from the kinetic term only is not known (in standard QFT, the converse is true, see section 2.3.2 for a discussion). We will see in section 2.2.3 that connected functions $\Delta G^{(2n)}_c$ behaves as:

$$\begin{equation} \Delta G^{(2n)}_c = \mathcal{O}(1/N^{\,n-1}) \,,\qquad \forall\, n, \end{equation} \tag{ 27 }$$

which has also been investigated analytically and numerically in [32] (see also appendix A) ¹⁵ . This scaling is consistent with the fact that the exact 2-point function is independent of N.

Qualitatively, this is reminiscent of what happens for the Ising model in large dimension. The local magnetization self-averages because the number of closest neighbors is large, and the statistical properties remain (quasi)-Gaussian. For space dimension d large but finite, thermodynamical quantities can be computed as power series in $1/d$, which do not affect universal quantities, as soon as d > 4. For d < 4 however, the decoupling of physical scales breaks down and the Gaussian approximation is not suitable [24].

The same scenario is expected to be true for NNs. For finite N, the distribution does not obey Wick theorem, and correlations functions receive contributions which do not reduce to products of 2-point functions, and the classical action must include non-Gaussian contributions, i.e. products of f of degrees higher than 2. However, as soon as N remains large enough, deviations from the Gaussian behavior are expected to remain small. In the classical action, these corrections materialize as product of m fields (m > 2), that we call interactions:

$$\begin{equation} S[\,f\,] = S^{^{\prime}}_{\textrm{kin}}[\,f\,] + S_{\textrm{int}}[\,f\,], \end{equation} \tag{ 28 }$$

where $S^{^{\prime}}_{\textrm{kin}}[\,f\,]$ is a new free action of the form (13). We follow the orthodox assumption in field theory that S is polynomial, and we denote generally as couplings the monomials. The correlation functions are computed using (14) by replacing $S_{\textrm{kin}}[\,f\,]$ with $S[\,f\,]$. But, since the interacting action $S_{\textrm{int}}[\,f\,]$ is built from cubic and higher powers of f, this generally prevents from computing the path integral exactly, and one has to resort to a perturbative expansion encoded in terms of Feynman graphs [32]. The form of the interactions is discussed in the next subsection. For the rest of this paper, we discard the contribution f_b in (6) from our analysis, and omit the subscript W.

The set of allowed couplings defines the theory space. They are generally guided by physical arguments, the symmetries of the system, and fundamental assumptions about the physical laws. This is especially the case for fundamental physics, where the expected properties of the space-time background play a key role. In turn, the structure of space-time is itself a consequence of the interactions between physical matter ¹⁶ . Indeed, if we are able to say that something is 'here', this is because we can interact with this thing. In words, a statement such that 'the field must interact locally' is physically equivalent to 'locality is defined by the interactions of fields'. In other words, space-time in physics is more that a set of d_in coordinates $x\in \mathbb{R}^{d_{\textrm{in}}}$. It is equipped with a group structure, the Poincaré group, which dictates how the coordinates can be transformed from one to the other. As the history of the relativity theory shows [66–68], these properties are essentially consequences of interactions between light and matter.

In the QFT framework, the role of the background space is played by $\mathbb{R}^{d_{\textrm{in}}}$. In [32], the authors adopt a conservative approach for most of their analysis, building the couplings as products of fields at the same point $x\in \mathbb{R}^{d_{\textrm{in}}}$:

$$\begin{equation} S_{\textrm{int}} = \sum_n g_n\, \int_{\mathbb{R}^{d_{\textrm{in}}}} dx \, \big( f(x) \big)^n. \end{equation} \tag{ 29 }$$

However, this makes various assumptions which may not be valid for a general NN QFT. For this reason, we will make them explicit and explain how to gradually lift them to consider the most general QFT. Deciding which assumptions to use should be dictated by numerical evidences: in particular, it was found in [32] that (29) is sufficient for the activation functions and range of input parameters considered there (see appendix A for more details). This approach can be considered as NN phenomenology, in the sense that we are writing a model to match observations, but we can also use this model to check theoretical facts such as dualities [33, 69, 70].

The first assumption is locality of the interaction: the fields appearing in the monomial $f(x)^n$ can be taken at different points (for simplicity, we consider a single coupling in S_int):

$$\begin{equation} S_{\textrm{int}} = g \int d x_1~\cdots d x_n \, f(x_1) \cdots f(x_n). \end{equation} \tag{ 30 }$$

This breaks locality because fields at different points in space(time) can interact together. In fact, since g is a constant, this happens for arbitrarily large distances. Note that this preserves translation invariance $x_i \to x_i + a$.

The next natural step is to replace g by a coupling function, i.e. a function of space but independent of the field. Going back to (29) where all fields are at the same points, we can write a local action with a coupling function:

$$\begin{equation} S_{\textrm{int}} = \int dx \, g(x) \, \big(\, f(x) \big)^n. \end{equation} \tag{ 31 }$$

It was argued in [32] from technical naturalness that g(x) must be approximately constant since a coupling function g(x) breaks the translation invariance of the action. However, this is correct only when assuming locality of the action: replacing g by a coupling function in (30) gives the non-local action [26]:

$$\begin{equation} S_{\textrm{int}} = \int d x_1~\cdots d x_n \, g(x_1, \ldots, x_n) f(x_1) \cdots f(x_n). \end{equation} \tag{ 32 }$$

However, translation invariance can be preserved if g depends only on the distances between the points:

$$\begin{equation} g(x_1, \ldots, x_n) = g(x_1 - x_2, \ldots, x_1 - x_n, \ldots x_{n-1} - x_n). \end{equation} \tag{ 33 }$$

Moreover, having a coupling functions gives more control on the interaction region, for example, by restricting the non-locality to a small region. For instance, we can set $g(x_1, \ldots x_n) = 0$ if $|x_i - x_j| \gt \ell$ for any pair (i, j). This allows representing non-locality by derivatives in momentum space and to show that they are subleading in the deep IR. Simple non-local models of this form have been considered in [32].

A special type of such a non-local interaction is obtained by smearing the fields (only in the interactions): in (29), we can replace the field f(x) by another $\tilde f(x)$ given by a convolution with a kernel $\kappa(x, y)$ [71–73]:

$$\begin{equation} \tilde f(x) : = \int d y \, \kappa(x, y) f(y), \end{equation} \tag{ 34 }$$

such that

$$\begin{equation} S_{\textrm{int}} = g \int dx \, \big( \tilde f(x) \big)^n = g \int dx dy_1~\cdots dy_n \, \kappa(x, y_1) \cdots \kappa(x, y_n) \, f(y_1) \cdots f(y_n). \end{equation} \tag{ 35 }$$

In order for this to make sense, the Fourier transform of the kernel $\kappa(x, y)$ must be an entire analytic function (with rapid decay if one wants to ensure UV finiteness). This corresponds to a coupling function:

$$\begin{equation} g(x_1, \ldots, x_n) = g \int dx \, \prod_{i = 1}^n \kappa(x, x_1). \end{equation} \tag{ 36 }$$

Smeared fields naturally appear in string theory and are responsible for its well-behaved UV behavior [74, 75]. In fact, for the Gauss-net, rescaling the field f to remove the exponential from the kinetic term (20) is equivalent to smearing the field (as pointed out earlier by comparing with the p-adic string [64, 65]).

There is a final assumption in all the previous interactions we wrote: that all components of x (which is a d_in-dimensional vector) are homogeneous. First, this means that coordinates can be added/subtracted to each other. Second, this also implies that the role played by the ith coordinate can be played by the jth, or by any linear combination of the coordinates. Physically, this means that the previous interactions have a $O(d_{\textrm{in}})$ symmetry (the Euclidean rotation group, or the Lorentz group in Lorentzian signature). Together with translations (if present ¹⁷ ) $x_i \to x_i + a$, this builds the d_in-dimensional Euclidean group $\mathrm{Is}(d_{\textrm{in}}) = \mathbb{R}^{d_{\textrm{in}}} \rtimes O(d_{\textrm{in}})$ (or Poincaré group in Lorentzian signature), which leaves the Euclidean distance invariant. This is why we often write f(x) instead of $f(x_1, \ldots, x_{d_{\textrm{in}}})$ where $x = (x_1, \ldots, x_{d_{\textrm{in}}})$.

However, it is not clear a priori that the data-space possesses this symmetry: it may not be possible to exchange two data components or even consider linear combinations if the components are not homogeneous. Despite the fact that the free theory supports such a symmetry, the rotational invariance has no meaning for a NN in general. Moreover, symmetries of the free theory can be broken by interactions, which are necessary to fully characterize the system. Conditions under which input and output symmetries can be present has been analyzed in [33]. In general, one can start by assuming no symmetry in order to describe the most general model, and then adapt to what the numerical experiments are indicating.

Hence, we need to consider fields for which each component is independent: this amounts to interpreting f(x) as a field over d_in independent copies of $\mathbb{R}$, meaning that each of the d_in components is independent and cannot be transformed into the others. Given that there are several fields, this means that the ith component of a given point can be inserted only in the ith argument of a field, however, it is not necessary to use all components of a single point in a single field. Obviously, this expression is non-local because the field is evaluated for components corresponding to different points. For example, for $d_{\textrm{in}} = 3$, one can write the following cubic interaction:

$$\begin{equation} S_{\textrm{int}} = g \int d x d y d z \, f(x_1, y_2, z_3) f(y_1, z_2, x_3) f(z_1, x_2, y_3), \end{equation} \tag{ 37 }$$

where x_i , y_i and z_i are the components of the 3-dimensional points x, y and z. Note that nothing prevents to use only two points, for example setting y = z and integrating only over x and y, or more generally to repeat the same component in any number of fields (for an early example, see [76]).

Such general theories are too wild and it is hard to make sense of them. A controllable subclass is provided by random tensor field theories [77]. In this case, the fields are tensors, each component of the positions being seen as a (continuous) index and indices can be contracted pairwise only (which is achieved by integrating over the component, since the index is continuous), such that a given component can appear at most twice. An intuitive way to represent it is to assign a color to each component, and Feynman diagrams can be written in terms of strand graphs (generalizing ribbon graphs from matrix models). For instance, a possible quartic interaction for $d_{\textrm{in}} = 3$ is:

$$\begin{equation} S_{\textrm{int}} = g \int d^3x d^3y f(x_1,x_2,x_3) f(x_1,y_2,y_3) f(y_1,y_2,y_3) f(y_1,x_2,x_3). \end{equation} \tag{ 38 }$$

We will see in the next section that tensor field theories are particularly interesting in the RG approach because, under some additional conditions, they possess a natural background-independent power-counting (see next subsection).

We conclude this section by clarifying a subtlety concerning QFT in curved spaces. In this case, the Euclidean (or Poincaré) group is not a global symmetry (symmetries of the action are given by the isometry group of the background space) and one may ask what is the difference with tensor field theories. The point is that this group is still a local symmetry (general relativity can be seen as gauging the Poincaré group) such that the properties discussed above continue to hold. Indeed, one can always consider the tangent space associated to a point: since it is isomorphic to flat space, it means that the coordinates are still homogeneous.

We are aiming to construct a field theory which admits a well-defined RG flow. In standard QFT, the rigorous construction of such a flow requires essentially three basics ingredients: (1) a scale decomposition, (2) a locality principle, (3) a power-counting.

The scale decomposition is the first ingredient to construct slices, and then to define a partial integration procedure. Power-counting and locality, in turn, are essential to understand the notions of effective couplings, i.e. how Feynman graphs can be replaced by an effective vertex together with a slice-dependent coupling. As long as we are endowed with $\mathbb {R}^{d_ {in}}$ as a background space, all of these notions are obvious. Scale decomposition is intuitively related with the notion of metric distance, locality and non-locality are defined with respect to the background itself, and power-counting is related to dimensionality as well. Indeed, the existence of an extrinsic length scale and the requirement that the classical action S is dimensionless, to give meaning to the exponential e^−S , allow fixing the dimensions (in terms of the length scale unit) of the couplings appearing in the classical action. This is the choice made in [32] done. Assuming $[dx]_x = 1$, where $[Q]_x$ denotes the dimension of the quantity Q in units of x, they were able to fix the dimension of couplings like (29),

$$\begin{equation} [\delta S_{\textrm{in}}^{(n)}]_x = 0 \,\quad \Leftrightarrow \quad [g]_x = -d_{\textrm{in}}-\frac{n [K]_x}{2}. \end{equation} \tag{ 39 }$$

We call it Λ-scaling such a scaling, for some reference scale Λ having (x)-dimension 1. However, from the discussion above, we can be a little puzzled by the fact of assigning a physical dimension to the variable x, and to truly view $\mathbb {R}^{d_{in}}$ as a background space. Rather, we adopt the minimal point of view considering it only as a configuration space, without dimension. Sacrificing the background space then makes the issues of scale, locality and power-counting less intuitive. The discussion in section 2.1.3 shows that the theory has a canonical notion of scale, given by the Fourier modes (spectrum) of the propagator. Regarding the notions of locality and power-counting, the difficulty is quite similar to that encountered in canonical approaches to quantum gravity, where space-time and background metric disappear [61]. In this context, a clever solution was found, which in some sense defines the power-counting from a locality principle, starting from the observation that standard locality in field theory can be algebraically translated as the ability of connected Feynman diagrams to be contracted to a point. Locality can then be defined algebraically from the requirement that, at least for some leading order sector, such a contraction procedure exists. A recent example, arising from quantum gravity models is provided by tensorial field theories [62, 78, 79]. In these theories, interactions are non-local in the usual sense (from the point of view of the configuration space) but, for some of them, the only divergences come from a sub-family of Feynman diagrams (in general the so-called melonic diagrams), which is contractible to an elementary vertex compatible with some internal symmetry defining the tensorial interactions themselves. Interactions having these properties are then said to be local. In turn, graphs admitting such a contraction property have been shown to admit a well-defined power-counting. The reason for this is that, to be well-defined, a power-counting requires that the existence of a family of Feynman graphs having the same behavior with respect to some cut-off Λ. If, order by order in the perturbative series, quantum corrections have different scaling behavior with respect to Λ, no power-counting exists. The existence of a contraction procedure allows defining the relative scaling of the various terms entering in the classical action with respect to Λ, such that there exists non-vanishing leading sectors of the perturbative expansion which have the same behavior with respect to Λ.

Let us illustrate heuristically on a simple example how contractibility and power-counting allows fixing the scaling dimension of couplings. Consider the following classical action:

$$\begin{equation} S[\phi] = \int dx \left[ \frac{1}{2} \phi(x) (-\Delta+m^2)\phi(x)+ g \phi^4(x)\right], \end{equation} \tag{ 40 }$$

describing the scalar field $\phi: \mathbb{R}^d\to \mathbb{R}$, and where Δ denotes the standard Laplacian. It is moreover local in the usual sense. For g small enough, quantum corrections can be computed using standard perturbation theory. The first contribution to the effective mass $\delta^{(1)} m^2$ arise from the following integral in Fourier space (the symmetry factors are irrelevant for our discussion):

$$\begin{equation} \delta^{(1)} m^2~\propto g\int \frac{dp}{p^2+m^2} \propto g \Lambda^{d-2}, \end{equation} \tag{ 41 }$$

for some cut-off Λ for large momenta. In the same way, the first correction for g, say $\delta^{(2)} g$ involves the following integral:

$$\begin{equation} \delta^{(2)} g \propto g^2~\int \frac{dp}{(p^2+m^2)^2}\propto g^2~\Lambda^{d-4}, \end{equation} \tag{ 42 }$$

the upper index referring to the number of vertices involved in the Feynman diagram. Now, to obtain a well-defined power-counting, the correction for g has to scale with Λ in the same way as g itself. This is solved by $g\sim \Lambda^{4-d}$, and we say that the Λ-scaling of g is $[g]_{\Lambda} = 4-d$. This moreover implies $\delta m^2~\sim \Lambda^2$, and thus $[m^2]_{\Lambda} = 2$. Now, we have to check that it is coherent to all orders of the perturbative expansion. To this end, let us consider a Feynman graph $\mathcal{G}_V$, of order V contributing to the perturbative expansion through the amplitude $\mathcal{A}_{\mathcal{G}_V} \sim g^V\Lambda^{\omega(\mathcal{G}_V)}$. Contracting along a spanning tree $\mathcal{T}_V\subset \mathcal{G}_V$, we reduce the original number of propagator edges L to $L-V+1$, and the resulting graph looks like an effective (local) vertex, having $L-V+1$ loops of length one (tadpoles). Each tadpole behaves like $\int dp/(p^2+m^2)$, and thus scales as $\Lambda^{d-2}$. The divergent degree for the contracted graph $\mathcal{G}_V\backslash\mathcal{T}_V$ is therefore:

$$\begin{equation} \omega(\mathcal{G}_V\backslash\mathcal{T}_V) = (d-2)(L-V+1). \end{equation} \tag{ 43 }$$

Because the contraction procedure removes V − 1 propagator edges, it increases $\omega(\mathcal{G}_V)$ by $2(V-1)$: $\omega(\mathcal{G}_V\backslash\mathcal{T}_V) = \omega(\mathcal{G}_V)+2(V-1)$. Finally, because the interaction is quartic, we have the relation $2L = 4V-N$, N being the number of external edges. Finally, we get:

$$\begin{equation} \omega(\mathcal{G}_V) = (d-4)V+2+(d-2)\left(1-\frac{N}{2} \right). \end{equation} \tag{ 44 }$$

Each vertex contributes a factor $\Lambda^{d-4}$, and the scaling $g\sim \Lambda^{4-d}$ ensures that all the quantum corrections have the same scaling. Moreover, setting N = 2, we get ω = 2, in agreement with the one-loop scaling dimension for mass.

Obviously, because this theory is local in the usual sense, the derived scaling dimensions are exactly the same as the one derived from the standard dimensional analysis of the classical action. The two methods however do not coincide for non-local interactions such that (38). We argue that this more abstract way to think about locality, scaling and power-counting is more appropriate in a context where the construction of the theory space is not guided by experimental evidences, such that it seems more appropriate to work from the outset within a sufficiently broad framework to accommodate future developments in formalism. However, the exploration of these aspects for NNs is beyond the scope of this paper, since standard locality seems to hold for the Gauss-net kernel [32].

There exists another scaling dimension, called N-scaling, associated to the behavior of correlation functions with respect to the width N of the hidden layer. The Gaussian universality for large N ensures that the couplings g_n behave as $g_n \sim N^{-\alpha(n)}$ for some positive function $\alpha(n)$.

The computation can be done by returning to the definition (7) and using the fact that W₁ follows a centered Gaussian distribution with variance $\sigma_W^2/N$. For instance, we find:

$$\begin{equation} \langle\, f(x) f(y) \rangle = \sum_{i,j = 1}^N \langle W_1^{\,(ik)} W_1^{\,(jk))} \rangle \langle x_1^{(i)} y_1^{(j)} \rangle = \frac{\sigma_W^2}{N} \sum_i \, \langle x_1^{(i)} y_1^{(i)} \rangle, \end{equation} \tag{ 45 }$$

which is of order 1. The computation of higher correlation functions can be done using a similar strategy, from the assumptions that the $x_1^{(i)}$ with different index i are statistically independent variables. This in particular ensures that:

$$\begin{equation} \langle x_1^{(i)} x_1^{(i)} x_1^{(j)} x_1^{(j)} \rangle \sim \langle x_1^{(i)} x_1^{(i)} \rangle \langle x_1^{(j)} x_1^{(j)} \rangle, \end{equation} \tag{ 46 }$$

from this observation, a tedious calculation which is given in [32] shows that the connected 4-point function $G^{(4)}_{c}(x_1,x_2,x_3,x_4)$ has to scale as $1/N$, and more generally that $\alpha(n) = n/2-1$.

This analytic result also shows the limitations of the approach. Indeed, we expect that a more fundamental method will be able to predict the weights of the interactions. Moreover, the derivation assumes the relation (46), and thus the independence of the $x_1^{(i)}$ having different indices i, but such an assumption seems to be in conflict with an interaction such that (29), which morally must introduce couplings mixing different outputs from the definition (7) of f_W . One may expect that these difficulties could be solved by working with a random vector of size N, with components $\varphi_i(x)$ rather than with the function $f_W(x)$, defining it as an observable $f_W(x): = \langle \varphi_i \rangle$ i.e. the vacuum of the corresponding theory. However, the construction of such a theory is going beyond the scope of this paper, and we plan to investigate it in a forthcoming work.

The RG is probably one of the most important concepts discovered in physics during the last century and forms together with field theory the reference framework of modern physics, from condensed matter to high energies. Pioneered in the works of Wilson and Kadanoff [34, 80–82], RG is based on the idea of organizing the theory according to length scales, integrating out short distance degrees of freedom following a recursive procedure called coarse-graining and providing an effective description for the long distance degrees of freedom, through an effective action where microscopic interactions are hidden in effective interactions. Note that RG is in fact a semi-group, which is non-invertible. Thus at each step, information is lost, and RG can be viewed as a systematic procedure to extract large scale relevant features.

To illustrate the physics underlying the Wilson procedure, and before making contact with NN field theory, let us consider a physical system made of a single real scalar field φ whose configuration probability follows the exponential form $p[\phi] = e^{-S[\phi]}$, for some classical action $S[\phi]$. To have a concrete example in mind, we can take for φ the real field described by the classical action (40). All the statistical properties of the distribution can be derived from the generating functional (partition function):

$$\begin{equation} Z[\,j] = \int [d\phi] \, e^{-S[\phi]+\int dx \, j(x)\phi(x)}. \end{equation} \tag{ 47 }$$

This integral being over all configurations for $\phi(x)$, all the degrees of freedom are integrated out in one step. Equation (47) provides a canonical definition of what is microscopic and what is macroscopic, two limits that we conventionally call UV and IR:

• (a)  
  In the UV limit, no fluctuations are integrated out. The field configurations are therefore fixed from the extrema of the classical action S.
• (b)  
  In the IR limit, all fluctuations are integrated out. The configurations are fixed by a new action Γ, called effective action.

The effective action Γ is in turn defined as the Legendre transform of the free energy $\mathcal{W}[\,j]: = \ln Z[\,j]$,

$$\begin{equation} \mathcal{W}[\,j]+\Gamma[\Psi] = \int dx\, j(x) \Psi(x), \end{equation} \tag{ 48 }$$

the classical field Ψ being defined as $\Psi(x): = \delta \mathcal{W}/\delta j(x)$.

The RG is nothing but a path between these two boundaries. It is constructed by partially integrating out the degrees of freedom building the field φ. Note that such a partial integration procedure is never arbitrary, and the Wilson RG assumes the existence of a canonical slicing $s = \{s_1,s_2,\cdots ,s_{\infty}\}$ ¹⁸ in the configuration space of elementary degrees of freedom, allowing to integrate partially following a preferred order. In general, this slicing is provided by the spectral distribution $\mu(E)$, $E\in \mathbb{R}$ of the UV 2-point function for an exponential family like (47): $s_i \subset \mu(E)$. In fact, the 2-point function can be identified with the Fisher information metric along the constrained space with fixed couplings. This gives a connection between RG and information geometry [31] because of the regularity property of the Fisher metric, and in absence of singular structures, distance between probability distributions has to be reduced with coarse-graining, explaining the power of RG to discuss of universality in physics [24]. Integrating all the degrees of freedom in the first slice s₁ leads to an effective model with classical action S' which defines a new effective physics where effects coming from degrees of freedom in the first slice are hidden in effective interactions. Now, integrating the slice s₂, we obtain a new classical action S'' and so on. Such a partial integration (up to a global rescaling of fields to reach a fixed point) is called a RG transformation, and the chain of RG transformations describes a 'move' in the interior of the theory space:

$$\begin{equation} S\to S^{^{\prime}}\to S^{^{\prime\prime}} \to \cdots, \end{equation} \tag{ 49 }$$

bounded by UV and IR effective physics (figure 1).

Zoom In Zoom Out Reset image size

Let us illustrate how that works on the concrete example of the scalar field φ described by action (40). In that case, $\mu(E)$ corresponds to the spectrum of the Laplacian Δ, whose eigenmodes are Fourier modes, and $E\equiv p$. Assuming continuity of the spectrum, we can consider infinitesimal coarse-graining, integrating out slices of infinitesimal thickness. This leads to a differential equation describing how the couplings change as the reference scale changes. Formally, this can be done as follows. We assume the existence of an upper bound for p, say Λ, and we call $\mu_\Lambda(p)$ the spectrum of the free 2-point function with cut-off Λ, $K_{\Lambda}(p)$. As the cut-off Λ moves, degrees of freedom are added or removed from the spectrum. Thus, let us consider the bare action 'at scale Λ':

$$\begin{equation} S_\Lambda[\phi] = \frac{1}{2} \int \mu(p) dp \, \phi(p) K_{\Lambda}^{-1}(p) \phi(-p)+\mathcal{V}_\Lambda[\phi], \end{equation} \tag{ 50 }$$

where $\mathcal{V}[\phi]$ includes interactions following our definition of section 2.1. Now let us consider the running cut-off $\Lambda(s) = s\Lambda$, for $s\in [0,1]$, which interpolate between the UV scale s = 1, and IR scale s = 0. If $K_{\Lambda(s)}(E)$ is at least $\mathcal{C}^{(1)}$ in s, we can consider the variation at first order from s to $s^{^{\prime}} = s+\delta$:

$$\begin{equation} K_{\Lambda(s)}(p) = K_{\Lambda(s^{\prime})}(p)+ \Delta_{\Lambda(s)}(p) \delta. \end{equation} \tag{ 51 }$$

The following decomposition can be translated as a partial integration from the original partition function using the functional identity:

$$\begin{equation} \int [d\phi d\chi] \, e^{-\tilde{S}_{\Lambda (s^{\prime})}[\phi,\chi]} = \left( \frac{\det{\Delta_{\Lambda(s)}}\det{K_{\Lambda(s^{\prime})}}}{\det{K_{\Lambda(s)}}}\right)^{1/2}\, \int [d\phi] \, e^{-S_{\Lambda (s)}[\phi]}, \end{equation} \tag{ 52 }$$

where:

$$\begin{equation} \tilde{S}_{\Lambda (s^{\prime})}[\phi,\chi] = \frac{1}{2} \int \mu(p) dp \left( \phi(p) K_{\Lambda(s^{\prime})}^{-1}(p) \phi(-p)+\chi(p)\Delta^{-1}_{\Lambda(s)}\delta^{-1} \chi(-p) \right)+\mathcal{V}_{\Lambda(s)}[\phi+\chi], \end{equation} \tag{ 53 }$$

and χ denotes the degrees of freedom integrated out. Indeed, defining:

$$\begin{equation} e^{-\tilde{\mathcal{V}}_{\Lambda(s^{\prime})}[\phi]}: = (\det{\Delta_{\Lambda(s)}})^{-1/2} \int [d\chi] e^{- \frac{1}{2} \int \mu(p) dp \, \chi(p)\Delta^{-1}_{\Lambda(s)}\delta^{-1} \chi(-p)- \mathcal{V}_{\Lambda(s)}[\phi+\chi]}, \end{equation} \tag{ 54 }$$

and:

$$\begin{equation} S_{\Lambda(s^{\prime})}[\phi] = \frac{1}{2} \int \mu(p) dp \, \phi(p) K_{\Lambda(s^{\prime})}^{-1}(p) \phi(-p)+\tilde{\mathcal{V}}_{\Lambda(s^{\prime})}[\phi], \end{equation} \tag{ 55 }$$

we show that the identity (52) can be rewritten as:

$$\begin{equation} \int [d\phi] \, e^{-S_{\Lambda(s^{\prime})}[\phi]} = \left(\frac{\det{K_{\Lambda(s^{\prime})}}}{\det{K_{\Lambda(s)}}}\right)^{1/2} \int [d\phi] \, e^{-S_{\Lambda(s)}[\phi]}. \end{equation} \tag{ 56 }$$

The classical action at scale $\Lambda(s^{\prime})$ formally looks like the action at the scale $\Lambda(s)$. What is different between them is the interaction, which comes at scale $\Lambda(s^{\prime})$ from a partial integration over the field χ. The transformation (54) can be translated in a differential equation for δ small enough. Indeed, in this limit, the modes χ have a large mass, and can be treated perturbatively. Thus, expanding $\mathcal{V}_{\Lambda(s)}[\phi+\chi]$ in powers of χ and keeping only terms of order 2, we get Polchinski's equation [80]:

$$\begin{equation} \Lambda \frac{d \tilde{\mathcal{V}}_{\Lambda}}{d\Lambda}\bigg|_{\Lambda = \Lambda(s)} = -\frac{1}{2} \int dp \mu(p) \, \Delta_{\Lambda(s)}(p) \Big( \frac{\delta^2~\tilde{\mathcal{V}}_{\Lambda(s)} }{\delta \phi(p)\delta\phi(-p)} -\frac{\delta \tilde{\mathcal{V}}_{\Lambda(s)}}{\delta \phi(p)} \frac{\delta \tilde{\mathcal{V}}_{\Lambda(s)}}{\delta \phi(-p)} \Big). \end{equation} \tag{ 57 }$$

This equation is formally 'exact'. However, this has the reputation to be very hard to solve for many reasons. The first one is that it takes place in a functional space of infinite dimension. If we decide to work in a reduced phase space, taking into account only the most relevant interactions, difficulties appear, instabilities with respect to the considered truncation appear as soon as we try to get beyond the perturbative sector, which is precisely what we are aiming at in this paper. For this reason, and as it is the case for the largest part of non-perturbative investigations in the literature [52, 53, 56, 63, 83–85], we will prefer to use the Wetterich formalism, better for dealing with non-perturbative approximations. We will discuss this method in the next section.

The analogy between NNs and RG is evident: both are aiming at extracting relevant features from a massive number of degrees of freedom. RG shows that microscopic details can be ignored to describe long distance physics, and that microscopic theories can be indistinguishable from their common large distance properties. Extracting regularities from large sets of data is exactly what machine learning does; and, as we recalled in the introduction, the question of the relevance of the RG in artificial intelligence is growing in the literature [16–20]. However, the effective field theory that we presented in the first part offers a new framework to discuss aspects related to the RG in the study of the behavior of NNs [32]. As the previous section stressed out, the field theory that we consider exhibits strong similarities with theories usually considered by physicists: the long distance (i.e. large volume, small momenta) limit (22) of the free propagator being the same as for the usual scalar field φ described by the action (40). This formal similarity will serve as a guide in the construction of the RG, and it is very tempting to carry out a coarse-graining in momenta, exactly as for the scalar field φ in the previous section. We will discuss two different coarse-graining strategies which we call respectively passive and active RGs. But before we go into them in detail, let us make a few general remarks about what distinguishes this NN field theory from ordinary theories.

In the standard scenario, what is known is the UV theory i.e. the classical action. This action itself is viewed as an effective description, valid at some fundamental scale and ignoring the details about nature and physics of microscopic degrees of freedom underlying the physical world. The choice of the classical action is constrained by predictivity (which promotes just-renormalizable theories), consistency with quantum effects (compensation of anomalies in gauge theories, for instance), and the effective structures at the scale at which the theory is defined, which generally implies some symmetries (rotation, reflection, gauge invariance, etc). In this respect, the RG aims to provide an approximation of the exact quantum theory, and to compare it with experiments. The case of the field theory that we consider differs from this general picture in its relations between UV and IR scales. The propagator (12) is exact and defined in the deep IR. From a RG point of view, the knowledge of this propagator takes into account all the fluctuations at all scales. But, for finite N, the knowledge of the 2-point functions is not sufficient to reproduce higher correlations functions, and non-Gaussian interactions are required in the classical action to reproduce experimental correlation functions. But, due to these interactions, the flow of the different ingredients entering in the definition of the classical action becomes non-trivial, with the consequence that both S_kin and S_int in the UV are unknown. Thus, in some sense, the situation is the inverse of what we do in ordinary field theory: we have to infer the form of the UV theory (or more likely a class of UV theory) from the knowledge of only a part of the IR theory. By construction, such an inference cannot lead to a single solution, but a class of solutions which have to satisfy the following requirements:

• (a)  
  reproduce the exact 2-point functions up to the experimental precision;
• (b)  
  reproduce the deviations from Wick's theorem, due to interactions, and which are less and less perturbative as N becomes small, once again up to irrelevant corrections with respect to the experimental precision.

Any measurement in physics comes with a finite precision: hence, two effective descriptions are considered to be equivalent and sufficient to describe something if the predictions agree up to the experiment precision. The precision is also finite in numerical simulations, and this explains why that we are able to infer only an equivalent class of models rather than a point in the theory space. In the first section, we showed that the relative relevance of the interactions is not the same such that irrelevant interactions contribute below the machine precision threshold, meaning that we have no way to distinguish between several initial conditions whose trajectories are sufficiently close in the IR (see figure 2). This argument allows working, in a first approximation, within a finite subspace of the full theory space, focusing on interactions having the largest canonical dimension.

Zoom In Zoom Out Reset image size

2.3.2.1. Passive RG

Because of the existence of an intrinsic length scale ξ defined in (25), we can think to partially integrate microscopic degrees of freedom with respect to this length scale to construct a proper RG flow following standard field theory. In this picture, what is playing the role of a microscopic scale is the working precision (see section 2.1.3), which introduces a cut-off in momentum integration $\Lambda = 1/a_0$. We can then construct a coarse graining procedure from grid size dilatation (see figure 3).

Zoom In Zoom Out Reset image size

Note that from such a procedure, one needs to have $\xi \gg a_0$. Because the maximal value for p is $p_\infty = 2\pi/a_0$, we can show that it implies $p_\infty \xi \gg 1$, which invalidates the expansion (21). However, it may happen that such an expansion holds in a sufficiently large domain. A necessary condition is that the expansion (21) holds for the smallest (nonzero) momentum $p_0 = 2\pi/a_0N_0$, implying:

$$\begin{equation} \xi \ll a_0~N_0~\equiv L, \end{equation} \tag{ 58 }$$

which is the condition defining the large volume limit.

A dilatation procedure as described in figure 3 induces a RG by partial integration of momenta into the windows $\sim ]1/a_0^{\prime}, 1/a_0]$. The existence of two complementary limits $\xi \ll L$ and $\xi \gg a_0$ is reminiscent of a crossover scale behavior, between a deep UV limit $p\sim 1/a_0$ and a deep IR limit ($p\sim 1/L$), which we will study separately in the next section. Note that such a crossover scale appears generally in situations where two very different mass scales appear, ensuring decoupling ¹⁹ of some effects associated to the larger one when experiments focus on the first one [86]. Here, what plays the role of a large mass is the inverse of the typical observation scale ξ; in the very large mass limit, $p_\infty \ll (\xi)^{-1}$, and the IR sector recovers all the physics. In the opposite limit, $p_0~\gg (\xi)^{-1}$, everything is UV, and an expansion such that (21) does not hold. In other words, for $p\ll (\xi)^{-1}$, one expects that quantum effects are suppressed with powers of $(\xi)^{-1}$. This observation can be a source of improvement for approximations used to solve the RG flow equation (61) in the next section. In particular, we understand that contributions coming from higher couplings will tend to stay small if they are at the transition scale $(\xi)^{-1}$. Section 3 is devoted to this RG strategy.

2.3.2.2. Active RG

In the process described above, the observation scale ξ is kept fixed and the working precision is changed. Conversely, we can keep the data (i.e. the working precision) fixed and change the observation scale. If the first version is essentially passive with respect to the NNs (i.e. the latter is not changed), this strategy is, in contrast, active (see figure 4). Indeed, remembering the expression (25), ξ is completely determined in terms of σ_W , the standard deviation of the weight distribution. Hence, flowing in the observation scale is equivalent to changing the weight standard deviation, and thus the NN.

Zoom In Zoom Out Reset image size

Physically, if we think of a thermodynamic system like a ferromagnet, such a strategy is equivalent to turning the thermostat's knob to lower the temperature towards the critical regime. This alternative point of view is the subject of the section 4.

Remark 1. The active RG is closer to the RG version considered in the [32] than the passive scheme, as the flow equations derived in section 4 show explicitly. However, despite this formal contact, our approach differs by its very construction. While, from their point of view, the RG is the mathematical explanation of a principle of invariance with respect to a certain volume 'cut-off', our RG is the result of a procedure of partial integration of the degrees of freedom of the field.

Indeed, the RG flow is usually performed with respect to a UV cut-off (spacetime/data-space resolution) and not an IR cut-off (volume). In [32], the large volume cut-off was introduced because the 2-point function diverges at large distance (at least, for the ReLU-net), which reminds the short-distance divergence of the canonical propagator in particle QFT. Moreover, one can ask whether the data-space should be identified with the position or momentum space in usual spacetime QFT, and in principle this could depend on the problem. From our arguments in section 2.1.3, it seems more natural to identify the data-space with the position space (except in the case where the data is already the Fourier transform of a space(time) process). IR divergences are also present in particle QFT, and they are cured following different methods according to their origin. The first case are massless particles, for which a refined definition of amplitudes is needed [25, 87]. An IR cut-off (such as a mass) can be introduced at intermediate stages to regulate the integral, but it is not a renormalization parameter. In practice, the divergences of the ReLU-net arise from a similar origin (singularity in the propagator for large distance/zero-momentum). Second, IR divergences appear for internal on-shell propagators: they translate the fact that quantum effects shift the vacuum and masses of the fields. Resummation of quantum effects through renormalization leads to finite results [25, 74]. Third, some quantities can diverge for infinite volume for example when studying phase transition: in that case, the usual method is to study the theory with different values of a volume cut-off and to extrapolate to infinite volume (thermodynamic limit) [88]. However, this is not a renormalization flow. For these reasons, we take a more conservative approach and identify small resolution in data space with the UV limit, and perform the RG flow for the associated cut-off.

It is also noted that in [32, section 4.3] that the Gauss-net does not require renormalization because the 2-point function is exponentially decaying with the distance, such that all integrals are convergent. In fact, the previous paragraph shows that renormalization is still needed in this case because its role is not only to handle properly (spurious) UV divergences, but also to take into account quantum effects (some of which lead to IR divergences). Said another way, renormalization provides a mapping between the bare and physical parameters (at a given energy scale) there is always a renormalization flow in the space of couplings. Indeed, the bare parameters describe the properties of the fields without interactions: they are not physical because fields do not live in isolation and any measurement implies an interaction. A famous example of a perfectly finite theory but which has an infinite number of finite (such that predictivity is not lost) counter-terms and non-trivial RG flow (with the so-called stub length) is string field theory [75, 89, 90].

To begin, we focus on the simplest truncation around sixtic interactions, discarding from our analysis contributions arising from higher couplings. This is equivalent to setting:

$$\begin{equation} \Gamma_k^{(2n)}[\Psi = 0] = 0 \,, \quad \textrm{for}\, n\gt3, \end{equation} \tag{ 63 }$$

where to avoid confusion with the example given in section 2.3, we denote as Ψ the classical field. Note that such an expansion around $\Psi = 0$ is named symmetric phase expansion, and we call symmetric phase the domain of the full phase space where it remains valid. It may happen that such an expansion breaks down, in cases where $\Psi = 0$ becomes an unstable vacuum. This is the case when phase transitions are encountered. In this section, we focus on the symmetric phase, and discuss more elaborate formalisms in the next section. Approximation (63) ensures that we keep effects up to order $1/N^2$.

To be more concrete, we assume that $\Gamma_k[\Psi]$ can be decomposed as a sum of two contributions:

$$\begin{equation} \Gamma_k[\Psi] = \Gamma_{k,\textrm{kin}}[\Psi]+U_k[\Psi], \end{equation} \tag{ 64 }$$

where:

• (a)  
  The kinetic contribution $\Gamma_{k,\textrm{kin}}[\Psi]$ keeps all the quadratic terms in $\Gamma_k[\Psi]$.
• (b)  
  The effective potential $U_k[\Psi]$ gathers the non-Gaussian contributions in the expansion of $\Gamma_k[\Psi]$.

Without lost of generality, the kinetic contribution can be written as:

$$\begin{equation} \Gamma_{k,\textrm{kin}}[\Psi] = \frac{1}{2}\sum_p \, \Psi(p) \mathcal{K}_k(p^2) \Psi(-p). \end{equation} \tag{ 65 }$$

The kernel $\mathcal{K}_k(p^2)$ is a priori difficult to track. Fortunately, because we are aiming to deal with IR effects, the momentum p is expected to be small, justifying to expand $\mathcal{K}(p)$ in power of p²:

$$\begin{equation} \mathcal{K}_k(p) = \mathcal{K}_k(0)+p^2~\mathcal{K}_k^{\prime}(0)+\mathcal{O}(p^4). \end{equation} \tag{ 66 }$$

The first term of this expansion define the running mass, and we denote it as $m^2(k)$. In the same way the second term of the expansion is called running wave function renormalization, and we denote it as Z(k). Nerveless, it is easy to check that in the symmetric phase Z(k) does not depend on the running scale k (see below). Thus we must have $Z(k) = Z_0 = 1$. This scheme defines the derivative expansion [54, 63, 84, 92], and for this section we focus on the two first terms:

$$\begin{equation} \Gamma_{k,\textrm{kin}}[\Psi] = \frac{1}{2} \sum_{p} \, \Psi(p) (p^2+m^2(k))\Psi(-p). \end{equation} \tag{ 67 }$$

To keep only effects up to order $1/N^2$, we consider the following truncation for the effective potential:

$$\begin{equation} U_k[\Psi] = \frac{u_4}{4!} A_4[\Psi]+\frac{u_6}{6!} A_6[\Psi], \end{equation} \tag{ 68 }$$

where:

$$\begin{equation} A_n[\Psi]: = \sum_{\{p_i\}} \delta_{0,\sum_{i = 1}^n p_i} \prod_{j = 1}^n \Psi(p_j). \end{equation} \tag{ 69 }$$

This form follows from the expression of a local interaction of order n in position space: the Fourier transformation to momentum space introduces one momentum p_j for each field together with a delta function for momentum conservation, and a sum (since the p_j take discrete values) over each value of the momentum. The final piece is the regulator r_k . From the choice of the kinetic truncation (67), it is suitable to use the modified version of the standard optimized Litim's regulator [93]:

$$\begin{equation} r_k(p^2): = (k^{\,2}-p^2)\theta(k^{\,2}-p^2), \end{equation} \tag{ 70 }$$

for which analytic computations are possible. In this equation, θ is the step function such that $\theta(x \gt 0) = 1$ and $\theta(x \lt 0) = 0$. The flow equations can be deduced from the exact RG equation (61) by taking successive derivatives with respect to the classical field Ψ. Taking the second derivative gives the flow equation for $\Gamma_{k}^{(2)}(\vec{p}_1, \vec{p}_2)$.

$$\begin{equation} k \frac{d}{dk}\Gamma_{k}^{(2)}(p_1, p_2) = -\frac{1}{2} \sum_{\{p_\alpha\}, \alpha = 0,3,4}\,\left(k\frac{d}{dk}r_k(p_0^2) \right) G_k(p_0, p_3) \Gamma_k^{(4)}(p_1, p_2,p_3,p_4) G_k(p_4, p_0), \end{equation} \tag{ 71 }$$

where, on the RHS, functions are computed for $\Psi = 0$. From the truncation (67), we must have:

$$\begin{equation} G_k(p,p^{\prime}): = \frac{\delta_{p,-p^{\prime}}}{p^2+m^2(k)+r_k(p^2)}. \end{equation} \tag{ 72 }$$

The fourth derivative $\Gamma_k^{(4)}(p_1, p_2,p_3,p_4)$ can be easily computed from the truncation (68), leading to:

$$\begin{equation} \Gamma_k^{(4)}(p_1, p_2,p_3,p_4) = u_4~\delta_{0,p_1+ p_2+p_3+p_4}, \end{equation} \tag{ 73 }$$

replacing $\Psi = 0$ at the end of the computation. Thus, setting $p_1 = 0$ on both sides of equation (71), we get after some calculations ²⁰ :

$$\begin{equation} k \frac{d}{dk}m_k^{\,2} = - \frac{u_4}{(k^{\,2}+m^2(k))^2} \mathrm{Vol}(k), \end{equation} \tag{ 74 }$$

where

$$\begin{equation} \mathrm{Vol}(k): = \left(k^{\,2}\sum_p \theta(k^{\,2}-p^2) \right). \end{equation} \tag{ 75 }$$

Remark 2. In equation (71), the only dependence on the external momenta p₁ and p₂ on left-hand side is through the conservation delta $\delta_{p_1,-p_2}$ arising from the structure of the four-point function vertex $\Gamma_k^{(4)}$. Thus, the field strength Z(k), whose flow equation could be deduced by taking derivatives on both sides of equation (71) with respect to $p_1^2$, vanish identically.

In the same way, taking the fourth and sixth derivatives with respect to M of the flow equation (61), and from the condition (73), we get schematically:

$$\begin{equation} k\frac{d}{dk}{\Gamma}_k^{(4)} = -\frac{1}{2}\tilde{G}_k \Gamma_k^{(6)} G_k+ 3\tilde{G}_k \Gamma_k^{(4)}G_k\Gamma_k^{(4)}G_k, \end{equation} \tag{ 76 }$$

and:

$$\begin{equation} k\frac{d}{dk}{\Gamma}_k^{(6)} = -\frac{1}{2}\tilde{G}_k \Gamma_k^{(8)} G_k+15\tilde{G}_k \Gamma_k^{(6)}G_k\Gamma_k^{(4)}G_k-45\tilde{G}_k \Gamma_k^{(4)}G_k\Gamma_k^{(4)}G_k\Gamma_k^{(4)}G_k. \end{equation} \tag{ 77 }$$

A tedious calculation leads to:

$$\begin{equation} k\frac{du_4}{dk} = -\frac{u_6}{(k^{\,2}+m^2(k))^2}\mathrm{Vol}(k)+ \frac{6u_4^2}{(k^{\,2}+m^2(k))^3}\mathrm{Vol}(k), \end{equation} \tag{ 78 }$$

and

$$\begin{equation} k\frac{du_6}{dk} = \frac{30~u_4u_6}{(k^{\,2}+m^2(k))^2}\mathrm{Vol}(k)-\frac{90~u_4^3}{(k^{\,2}+m^2(k))^3}\mathrm{Vol}(k). \end{equation} \tag{ 79 }$$

These equations illustrate how the scaling can be fixed without assuming any background dimension, as discussed in section 2.2.2. Indeed, a moment of reflection shows that the argument below equation (42) about the existence of a non-trivial expansion is equivalent to the statement that a global rescaling of all couplings must exist such that the flow equations become an autonomous system. For k large enough, the sum in $\mathrm{Vol}(k)$ can be well approximated by an integral, and ²¹ :

$$\begin{equation} \mathrm{Vol}(k) \underset{k \gg 1}{\sim} \frac{1}{(2\pi)^{d_{\textrm{in}}}}\frac{\pi^{d_{\textrm{in}}/2}k^{d_{\textrm{in}}+2}}{\Gamma(d_{\textrm{in}}/2+1)} = :K_{d_{\textrm{in}}}k^{d_{\textrm{in}}+2}. \end{equation} \tag{ 80 }$$

One expects that such an approximation remains valid for $k^{\,2}~\gg 4\pi^2/L^2$, with L being large (see figure 5). Thus, defining,

$$\begin{equation} \bar{u}_2(k) = k^{-2} m^2(k)\,,\quad \bar{u}_{2n} = k^{-2n+(n-1)d_{\textrm{in}}}u_{2n}, \end{equation} \tag{ 81 }$$

we get the autonomous system ($\beta_{2n}: = k d\bar{u}_{2n}/dk$):

$$\begin{align} \beta_2& = -2\bar{u}_2-\frac{K_{d_{\textrm{in}}}\bar{u}_4}{(1+\bar{u}_2)^2}, \end{align} \tag{ 82 }$$

$$\begin{align} \beta_4& = -(4-d_{\textrm{in}})\bar{u}_4- \frac{K_{d_{\textrm{in}}}\bar{u}_6}{(1+\bar{u}_2)^2}+ \frac{6~K_{d_{\textrm{in}}}\bar{u}_4^2}{(1+\bar{u}_2)^3}, \end{align} \tag{ 83 }$$

$$\begin{align} \beta_6& = -(6-2d_{\textrm{in}})\bar{u}_6+ \frac{30~K_{d_{\textrm{in}}} \bar{u}_4\bar{u}_6}{(1+\bar{u}_2)^3}-\frac{90~K_{d_{\textrm{in}}} \bar{u}_4^3}{(1+\bar{u}_2)^4}. \end{align} \tag{ 84 }$$

Zoom In Zoom Out Reset image size

From these equations, it is obvious that the behavior of the flow depends on the dimension d_in. For instance, for $d_{\textrm{in}}\gt4$, all the couplings are irrelevant and trajectories return toward the Gaussian region, the $\bar{u}_2$ axis being the only direction of instability. In contrast, for $d_{\textrm{in}}\lt4$, some couplings become relevant, and trajectories are repelled from the Gaussian region. u₄ is the first one to become relevant, for $d_{\textrm{in}}\gt3$; for $d_{\textrm{in}}\lt3$, u₆ becomes relevant as well. Figure 6 illustrates the behavior of the RG flow for several dimensions. We have integrated numerically the flow equations for $u_2 = 1$, $u_4 = - 0.5$, $u_6 = 0.01$ in figure 7.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

In this section, we consider another approximation scheme for the effective potential U_k . Focusing on the IR regime, we assume that $\Psi(p)$ essentially reduces to its zero component (the macroscopic field):

$$\begin{equation} \Psi(p) \sim \Psi_0~\delta_{0p}, \end{equation} \tag{ 85 }$$

and, defining $\chi: = \Psi_0^2/2$, we expand the effective potential per unit volume in power series around $\chi = \kappa(k)$:

$$\begin{equation} \mathcal{U}_k[\Psi_0] = \frac{1}{2!} u_4(k) \bigg(\chi-\kappa(k)\bigg)^2+\frac{1}{3!} u_6(k) \bigg(\chi-\kappa(k)\bigg)^3+\cdots, \end{equation} \tag{ 86 }$$

within this parametrization, we identify directly κ with the (non-zero) vacuum, which runs with the scale k. The two-point function $\Gamma^{(2)}_k$ is moreover defined as:

$$\begin{equation} \Gamma^{(2)}_k(p,p^{\prime}) = \left(Z(k)p^2+\frac{\partial^2~\mathcal{U}_k}{\partial \Psi_0^2 }\right) \delta(p+p^{\prime}). \end{equation} \tag{ 87 }$$

Note that we introduced the field strength renormalization Z(k) because its own flow is nonzero as soon as κ ≠ 0, i.e. broken phase effect introduces an anomalous dimension. As a technical device, we move the mass contribution in the effective potential. For a uniform field configuration, we must have:

$$\begin{equation} \Gamma_k[\Psi_{\textrm{uniform}}] = \mathcal{V} \mathcal{U}_k[\Psi_{\textrm{uniform}}]. \end{equation} \tag{ 88 }$$

Therefore, taking the derivative with respect to $t: = \ln(k/\Lambda)$ ($\dot X: = k dX/dk$) and writing $\mathcal{U}_k^{\prime} (\chi) = \partial \mathcal{U}_k / \partial \Psi_0$, we get from (61):

$$\begin{equation} \dot{\mathcal{U}}_k[\Psi_0] = \frac{1}{2} \int \frac{dp}{(2\pi)^{d_{\textrm{in}}}} \, \dot{r}_k(p^2) \left( \frac{1}{\Gamma^{(2)}_k+r_k} \right)(p,-p), \end{equation} \tag{ 89 }$$

or, using the definition (87):

$$\begin{equation}{\dot {\cal U}_k}[\chi ] = \frac{1}{2}\mathop \smallint \nolimits^ \frac{{dp}}{{{{(2\pi )}^{{d_{{\text{in}}}}}}}}{\dot r_k}({p^2})\left( {\frac{1}{{Z(k){p^2} + {r_k}({p^2}) + {\cal U}_k^\prime (\chi ) + 2\chi {\cal U}_k^{\prime \prime }(\chi )}}} \right).\end{equation} \tag{ 90 }$$

As in the previous section, we use the Litim regulator ²² , but modify it to deal with the running field strength Z(k):

$$\begin{equation} r_k(p^2) = Z(k)(k^{\,2}-p^2)\theta(k^{\,2}-p^2), \end{equation} \tag{ 91 }$$

leading straightforwardly to:

$$\begin{equation} \dot{\mathcal{U}}_k[\chi] = \frac{\mathrm{Vol}(k)}{Z(k)k^{\,2}+\mathcal{U}_k^{\prime} (\chi)+ 2\chi \mathcal{U}_k^{{\prime} ^{\prime}}(\chi)}. \end{equation} \tag{ 92 }$$

For k large enough, we may use the same integral approximation as for (80), but taking into account that $K_{d_{\textrm{in}}}$ must now depend on the anomalous dimension η_k because of the factor Z(k) in (91):

$$\begin{equation} \eta_k: = k\frac{d}{dk}\ln(Z(k)), \end{equation} \tag{ 93 }$$

such that:

$$\begin{equation} \dot{\mathcal{U}}_k[\chi] = K_{d_{\textrm{in}}}(\eta_k) k^{d_{\textrm{in}}+2}\, \frac{1}{Z(k)k^{\,2}+\mathcal{U}_k^{\prime} (\chi)+ 2\chi \mathcal{U}_k^{{\prime} ^{\prime}}(\chi)}. \end{equation} \tag{ 94 }$$

As in the previous section, we introduce dimensionless quantities (labeled with overlines) as:

$$\begin{equation} \bar{\mathcal{U}}_k = (k^{2})^{-d_{\textrm{in}}/2} \, \mathcal{U}_k\,,\qquad \bar{\chi} = Z(k)(k^{\,2})^{1-d_{\textrm{in}}/2} \chi. \end{equation} \tag{ 95 }$$

Note that all these changes of variable make sense from the requirement that all the terms in the potential must have the same dimension (the dimensions of g and h have been fixed). The derivative on the RHS in equation (94) is taken at χ fixed. Therefore, we have:

$$\begin{equation} \dot{\mathcal{U}}_k[\chi] = (k^{2})^{d_{\textrm{in}}/2} \left[ \dot{\bar{\mathcal{U}}}_k[\bar{\chi}]+(d_{\textrm{in}}+\eta_k)\bar{\mathcal{U}}_k[\bar{\chi}] -(d_{\textrm{in}}-2) \bar{\chi} \frac{\partial}{\partial \bar{\chi}} \bar{\mathcal{U}}_k[\bar{\chi}] \right], \end{equation} \tag{ 96 }$$

where in the RHS the derivative is taken with $\bar\chi$ fixed. We obtain:

$$\begin{equation} \dot{\bar{\mathcal{U}}}_k[\bar{\chi}] = -(d_{\textrm{in}}+\eta_k)\bar{\mathcal{U}}_k[\bar{\chi}] +(d_{\textrm{in}}-2) \bar{\chi} \frac{\partial}{\partial \bar{\chi}} \bar{\mathcal{U}}_k[\bar{\chi}] + \, \frac{K_{d_{\textrm{in}}}(\eta_k)}{1+\bar{\mathcal{U}}_k^{\prime} (\bar\chi)+ 2\bar{\chi} \bar{\mathcal{U}}_k^{{\prime} ^{\prime}}(\bar{\chi})}. \end{equation} \tag{ 97 }$$

The flow equations can be deduced from the normalization conditions at scale k:

$$\begin{equation} \frac{\partial \mathcal{U}_k}{\partial \chi}\bigg\vert_{\chi = \kappa} = 0\,, \qquad \frac{\partial^n \mathcal{U}_k}{\partial \chi^n}\bigg\vert_{\chi = \kappa} = u_{2n}. \end{equation} \tag{ 98 }$$

Hence, because $\dot{\bar{\mathcal{U}}}_k[\bar{\chi} = \bar{\kappa}] = -\bar{g}\, \dot{\bar{\kappa}}$, we obtain for $\bar{\kappa}$,

$$\begin{align} \beta_\kappa = -(d_{\textrm{in}}-2-\eta_k) \bar{\kappa} +K_{d_{\textrm{in}}}\,\frac{3+2\bar{\kappa} \frac{\bar{u}_6}{\bar{u}_4}}{(1+ 2\bar{\kappa} \bar{u}_4)^2}, \end{align} \tag{ 99 }$$

and after a tedious calculation, we obtain for u₄ and u₆:

$$\begin{align} \beta_4& = -(4-d_{\textrm{in}}+2\eta_k) \bar{u}_4+(d_{\textrm{in}}-2) \bar{\kappa} \bar{u}_6-\,\frac{5K_{d_{\textrm{in}}}\bar{u}_6}{(1+2\bar{\kappa} \bar{u}_4)^2}+2K_{d_{\textrm{in}}}\,\frac{(3\bar{u}_4+2\bar{\kappa} \bar{u}_6)^2}{(1+ 2\bar{\kappa} \bar{u}_4)^3}, \end{align} \tag{ 100 }$$

$$\begin{align} \beta_6& = -(6-2d_{\textrm{in}}+3\eta_k) \bar{u}_6-6K_{d_{\textrm{in}}}\, \frac{(3\bar{u}_4+2\bar{\kappa}\bar{u}_6)^3}{(1+2\bar{\kappa} \bar{u}_4)^4}+20~K_{d_{\textrm{in}}}\bar{u}_6~\frac{3\bar{u}_4+2\bar{\kappa}\bar{u}_6}{(1+ 2\bar{\kappa} \bar{u}_4)^3}. \end{align} \tag{ 101 }$$

The computation of the anomalous dimension is long and provided in appendix B. The result is:

Proposition 1. For the Litim's regulator (91), the anomalous dimension η_k in the LPA with kinetic truncation up to order p² is given by:

$$\begin{equation} \eta_k = -H(d_{\textrm{in}})\frac{(3~\bar{u}_4~\sqrt{2\bar{\kappa}} + \bar{u}_6 (2\bar{\kappa})^{3/2})^2}{(1+2\bar{\kappa} \bar{u}_4)^4}, \end{equation} \tag{ 102 }$$

where:

$$\begin{equation} H(d_{\textrm{in}}): = \frac{\pi^{\frac{d_{\textrm{in}}+1}{2}}}{d_{\textrm{in}}+1}\frac{\Gamma\left(\frac{d_{\textrm{in}}}{2}\right)}{\Gamma \left(\frac{d_{\textrm{in}}+1}{2}\right)\Gamma \left(\frac{d_{\textrm{in}}-1}{2}\right)}. \end{equation} \tag{ 103 }$$