An analytic theory of shallow networks dynamics for hinge loss classification^*

We first consider the limit of infinite number of examples, and later discuss the effects induced by a finite training set.

The explicit solution of the training dynamics is obtained making use of the cylindrical symmetry around ${\hat{\mathbf{w}}}^{\ast }$, which implies that the average in the equations of motion (7) does not depend on w, i.e.

$$\begin{equation}{\left\langle u(\mathbf{x},y ;t)\theta \left(\mathbf{w}\cdot \mathbf{x}\right)y\mathbf{x}\right\rangle }_{\mathbf{x},y}=I(t){\hat{\mathbf{w}}}^{\ast },\end{equation} \tag{ 9 }$$

where $I(t)\equiv {\left\langle u(\mathbf{x},y;t)\theta \left(\mathbf{w}\cdot \mathbf{x}\right)y\mathbf{x}\cdot {\hat{\mathbf{w}}}^{\ast }\right\rangle }_{\mathbf{x},y}$. By plugging the identity (9) into equations (6) and (7) one finds that the hydrodynamic equation (5) can be solved by the method of the characteristic, where ρ( θ , t) is obtained by transporting the initial condition through the equation (7). By decomposing the vector w in its parallel and perpendicular components with respect to ${\hat{\mathbf{w}}}^{\ast }$, i.e. $\mathbf{w}={w}^{{\Vert}}{\hat{\mathbf{w}}}^{\ast }+{\mathbf{w}}_{\perp }$, and using the solution ρ( θ , t), one finds that the parameters θ at time t are distributed in law as:

$$\begin{equation}\begin{cases}a(t) & \stackrel{d}{\sim }\quad a(0)\mathrm{cosh}(\gamma (t))+{w}^{{\Vert}}(0)\mathrm{sinh}(\gamma (t)) \\ {w}^{{\Vert}}(t) & \stackrel{d}{\sim }\quad {w}^{{\Vert}}(0)\mathrm{cosh}(\gamma (t))+a(0)\mathrm{sinh}(\gamma (t)) \\ {\mathbf{w}}_{\perp }(t) & \stackrel{d}{\sim }\quad {\mathbf{w}}_{\perp }(0) \end{cases};\qquad \gamma (t)=\frac{\beta }{\alpha \sqrt{d}}{\int }_{0}^{t}I(t)\mathrm{d}t,\end{equation} \tag{ 10 }$$

where a(0), w^∥(0), w_⊥(0) are given by the initial condition distributions: since all initial components of w were taken as i.i.d. Gaussian, so is w^∥(0) and every component of w_⊥(0) for any choice of basis. Using the distribution of θ at time t, one can then compute ${\left\langle u(\mathbf{x},y;t)\theta \left(\mathbf{w}\cdot \mathbf{x}\right)y\mathbf{x}\cdot {\hat{\mathbf{w}}}^{\ast }\right\rangle }_{\mathbf{x},y}$ and hence obtain a self-consistent equation on I(t), which completes the mean-field solution. Similarly, one can obtain explicitly the output function and the indicator function which acquire a simple form:

$$\begin{equation}f(\mathbf{x};\boldsymbol{\theta })=\frac{\mathrm{sinh}(2\gamma (t))}{2\sqrt{d}}{\hat{\mathbf{w}}}^{\ast }\cdot \mathbf{x},\end{equation} \tag{ 11 }$$

$$\begin{equation}u(\mathbf{x},y;t)=\theta \left(\frac{2h\sqrt{d}}{\alpha \enspace \mathrm{sinh}(2\gamma (t))}-y{\hat{\mathbf{w}}}^{\ast }\cdot \mathbf{x}\right)\end{equation} \tag{ 12 }$$

where we have used that f(x; θ ) = 0 at t = 0. As expected, both functions have cylindrical symmetry around ${\hat{\mathbf{w}}}^{\ast }$. The analytical derivation of these results and the following ones is presented in the SM. Since by definition I(t) ⩾ 0 the function γ(t) is monotonously increasing and starts from zero at t = 0. To be more specific, we consider two cases: normally distributed data with unit variance in each dimension, and uniform data on the d-dimensional unit sphere. The corresponding self-consistent equations on γ(t) read respectively:

$$\begin{equation}\dot{\gamma }(t)=\frac{\beta {I}^{N}(0)}{\alpha \sqrt{d}}\left(1-\mathrm{exp}\left[-\frac{2{h}^{2}d}{{\alpha }^{2}\enspace {\mathrm{sinh}}^{2}(2\gamma (t))}\right]\right),\end{equation} \tag{ 13 }$$

$$\begin{equation}\dot{\gamma }(t)=\frac{\beta {I}^{S}(0)}{\alpha \sqrt{d}}\left(1-\mathrm{max}{\left(0,1-4{h}^{2}d/({\alpha }^{2}\enspace {\mathrm{sinh}}^{2}(2\gamma (t)))\right)}^{\frac{d-1}{2}}\right),\end{equation} \tag{ 14 }$$

where ${I}^{N}(0)=1/\sqrt{2\pi }$ and ${I}^{S}(0)={\Gamma}\left(\frac{d+2}{2}\right)/({\Gamma}\left(\frac{d+1}{2}\right)d\sqrt{\pi })$. Both equations imply that γ(t) ∼ t for small t and γ(t) ∼ ln t for large t.

We have now gained a full analytical description of the training dynamics: the node parameters evolve in time following equation (10). Note that their trajectory is independent of the training parameters and the initial distribution, which only affect the time dependence, i.e. the 'clock' γ(t). The change of the output function is given by equation (11), where one sees that only the amplitude of f(x, θ ) varies with time and is governed by γ(t). The amplitude increases monotonically so that more examples can be classified above the margin h at later times; the more examples are classified the slower becomes the increase of γ(t) and hence the dynamics.

Our theoretical prediction can be directly compared with a simple numerical experiment. Figure 1 shows the training of a network with M = 400 on Gaussian input data. The top panels (a) and (b) compare the analytical evolution of the network parameters a_i and ${w}_{i}^{{\Vert}}$ obtained from equation (10) to the numerical one. In (c) we plot γ(t) (computed numerically) showing that it grows linearly in the beginning and logarithmically at longer times, as expected from theory. In (d) we show a scatter plot illustrating that the time when an example is satisfied is proportional to its projection on the reference vector, following on average our estimate based on equation (12). Overall, the agreement with the analytical solution is very good. The spread around the analytical solution in (d) is a finite-M effect, that we will analyze in section 3.3. The departure from the analytical result (10) happens at large time when the finiteness of the training set starts to matter (the larger is the training set the larger is this time). In fact, for any finite number of examples the empirical average over unsatisfied examples deviates from its population average and the dynamics is modified eventually, and ultimately stops when the whole training set is classified beyond margin. We study this regime in section 3.3.

Zoom In Zoom Out Reset image size

The presence of the factor α in the loss function (8) allows us to explore explicitly the crossover between different learning regimes, in particular the 'lazy learning' regime corresponding to α → ∞ [23]. The dynamical equations can be studied in this limit by introducing $\bar{\gamma }(t)=\alpha \gamma (t)$. For concreteness, let us focus on the case of normally distributed data. Taking the α → ∞ limit of equation (13) one finds the equation for $\bar{\gamma }(t)$:

$$\begin{equation}\dot{\bar{\gamma }}(t)=\frac{\beta {I}^{N}(0)}{\sqrt{d}}\left(1-\mathrm{exp}\left[-\frac{2{h}^{2}d}{4\bar{\gamma }{(t)}^{2}}\right]\right).\end{equation} \tag{ 15 }$$

As for the evolution of the parameters and the output function, we obtain:

$$\begin{equation}\begin{cases}{a}_{i}(t)-{a}_{i}(0) & ={w}_{i}^{{\Vert}}(0)\frac{\bar{\gamma }(t)}{\alpha }+O({\alpha }^{-2}) \\ {w}_{i}^{{\Vert}}(t)-{w}_{i}^{{\Vert}}(0) & ={a}_{i}(0)\frac{\bar{\gamma }(t)}{\alpha }+O({\alpha }^{-2}) \end{cases};\qquad \alpha f(\mathbf{x};\boldsymbol{\theta })=\frac{\bar{\gamma }(t)}{\sqrt{d}}{\hat{\mathbf{w}}}^{\ast }\cdot \mathbf{x}.\end{equation} \tag{ 16 }$$

The equations above provide an explicit solution of lazy learning dynamics and illustrate its main features: the θ _i evolves very little and along a fixed direction, in this case given by $({w}_{i}^{{\Vert}}(0),{a}_{i}(0),0)$. Despite the small changes in the nodes parameters, of the order of 1/α, the network does learn since classification is performed through αf(x; θ ) which has an order one change even for α → ∞. In this regime, the correlation between a and w^∥ only increases slightly, but this is enough for classification, since an infinite amount of displacements in the right direction is sufficient to solve the problem.

On the contrary, when α is of order one or smaller, the dynamics is in the so-called 'rich learning' regime [27]. At the beginning of learning, the initial evolution of the θ _i s follows the same linear trajectories of the lazy-learning regime. However, at later stages, the trajectories are no more linear and the norm of the weights increases exponentially in γ(t), stopping only at very large values of γ when all nodes are almost aligned with ${\hat{\mathbf{w}}}^{\ast }$ (for small α). Note that, as observed in Geiger et al [34], with the standard normalization $1/\sqrt{M}$ it would be the parameter $\alpha \sqrt{M}$ governing the crossover between the two regimes.

We compare the two dynamical evolutions in figure 2. The left panel (a) shows the displacement of parameters between initialization and full classification (zero training loss) for a network with α = 10³. As expected, the displacement is small and linear. A very different evolution takes place for α = 10⁻³ in the right (b) panel. The trajectories are non-linear, and all nodes approach large values close to the a = w^∥ line at the end of the training. Correspondingly, the initially isotropic Gaussian distribution evolves toward one with covariance matrix cosh(2γ) on the diagonal and sinh(2γ) off diagonal.

Zoom In Zoom Out Reset image size

Note that for all values of α, even very large ones, the trajectories of the θ _i s are identical and given by equation (10). What differs is the 'clock' γ(t), in particular for large α the system remains for a much longer time in the lazy regime. This is true as long as the number of training samples is infinite. Instead, if the number of data is finite, the dynamics stops once the whole training set is fitted: for large α this happens before the system is able to leave the lazy regime, whereas for small α a full non-linear (rich) evolution takes place. Hence, the finiteness of the training set leads to very distinct dynamics and profoundly different 'trained' models (having both fitted the training dataset) with possibly different generalization properties [25, 34, 35].

The solution we presented in the previous sections holds in the limit of an infinite number of nodes and of training data. Here we study the corrections to this asymptotic limit, and discuss the new phenomena that they bring about.

Finite number of nodes. In the large M limit the a_i and w_i are Gaussian i.i.d. variables. By the central limit theorem, the function (2) concentrates around its average, and has negligible fluctuations of the order of $1/\sqrt{M}$ when M → ∞. If M is large but finite (keeping an infinite training set), these fluctuations of f(x, θ ) are responsible for the leading corrections to mean-field theory. In the SM we compute explicitly the variance of the output function, ${\mathrm{lim}}_{M\to \infty }\enspace M\enspace \text{Var}[f(x,\boldsymbol{\theta })]={\sigma }_{f}^{2}(t)$, with

$$\begin{equation}{\sigma }_{f}^{2}(t)\equiv ((5\enspace {\mathrm{cosh}}^{2}(2\gamma (t))-2\enspace \mathrm{cosh}(2\gamma (t))-3){({\hat{\mathbf{w}}}^{\ast }\cdot \mathbf{x})}^{2}+2\enspace \mathrm{cosh}(2\gamma (t)){\left\vert \mathbf{x}\right\vert }^{2})/(4d).\end{equation} \tag{ 17 }$$

The main effect of this correction is to induce a spread in the dynamics, e.g. of the data with same satisfaction time. This phenomenon is shown in figure 1(d) for M = 400, where we compare the numerical spread to an estimate of the values of ${\hat{\mathbf{w}}}^{\ast }\cdot \mathbf{x}$ such that the hinge is equal to the average plus or minus one standard deviation (details on this estimate in the SM).

Finite number of data. We now consider a finite but large number of examples N (keeping infinite the number of nodes). In the large N limit the empirical average over the data in ${\left\langle u(\mathbf{x},y;t)\theta \left(\mathbf{w}\cdot \mathbf{x}\right)y\enspace \mathbf{x}\right\rangle }_{\mathbf{x},y}$ converges to its mean $I(t){\hat{\mathbf{w}}}^{\ast }$. The main effect of considering a finite N is that the empirical average fluctuates around this value. Using the central limit theorem we show in the SM that the leading correction to the asymptotic result reads:

$$\begin{equation}{\left\langle u(\mathbf{x},y;t)\theta \left(\mathbf{w}\cdot \mathbf{x}\right)y\mathbf{x}\right\rangle }_{\mathbf{x},y}=I(t){\hat{\mathbf{w}}}^{\ast }+\frac{J(t)}{\sqrt{N}}{\boldsymbol{\delta }\mathbf{w}}_{\perp }+O(1/N),\end{equation} \tag{ 18 }$$

where δw_⊥ is a unitary random vector perpendicular to ${\hat{\mathbf{w}}}^{\ast }$ and $J(t)\equiv \sqrt{(d-1){f}^{U}(t)/2}$. The term ${f}^{U}(t)\equiv {\left\langle u(\mathbf{x},y;t)\right\rangle }_{\mathbf{x},y}$, the fraction of unsatisfied examples at time t, controls the strength of the correction, as expected since only unsatisfied data contribute to the empirical average ⟨⋅⟩_x,y . The vector on the rhs of (18) is the one toward which all the w_i align, see equation (10). Therefore, the main effect of the correction (18) is for the nodes parameters to align along a direction which is slightly different from ${\hat{\mathbf{w}}}^{\ast }$ and dependent on the training set. This naturally induces different accuracies between the training and the test sets, i.e. it leads to overfitting ² . Note that the strength of the signal, I(t), is roughly of the order of the fraction of unsatisfied data f^U (t), whereas the noise due to the finite training set is proportional to the square root of it. The larger the time, the smaller f^U (t) is, hence the stronger are the fluctuations with respect to the signal. In figure 3(b), we compute numerically the components of ${\left\langle u(\mathbf{x},y;t)\theta \left(\mathbf{w}\cdot \mathbf{x}\right)y\mathbf{x}\right\rangle }_{\mathbf{x},y}$ parallel and perpendicular to ${\hat{\mathbf{w}}}^{\ast }$, and compare them to I(t) and $J(t)/\sqrt{N}$. Remarkably, we find a very good agreement even for times when $J(t)/\sqrt{N}$ is no longer a small correction. This suggest that an estimate of the time t_o when overfitting takes place is given by $I({t}_{\text{o}})=J({t}_{\text{o}})/\sqrt{N}$. We test this conjecture in (a): indeed the two contributions are of the same order of magnitude for t_o ∼ 50, which is around the time when training and validation errors diverge.

Zoom In Zoom Out Reset image size

We now briefly address the effects due to noise in the labels, see the SM for detailed results and numerical experiments. Mislabeling is introduced by flipping the label of a small fraction δ of the examples. The main effect is to decrease the strength of the signal, I(t), since the mislabeled data lead to an opposite contribution in (9) with respect to the correctly labeled ones. In the asymptotic limit of infinite N and M, the reduction of the signal slows down the dynamics, which stops when the number of unsatisfied correct examples equals the one of mislabeled ones. For large but finite N, the noise $J(t)/\sqrt{N}$ is enhanced with respect to the signal because its strength is related to the fraction of all unsatisfied examples, and not just the correctly labeled ones. Hence, overfitting is stronger and takes place earlier with respect to the case analyzed before.