Hausdorff dimension, heavy tails, and generalization in neural networks^*

Stable distributions appear as the limiting distribution in the generalized central limit theorem [Lév37] and can be seen as a generalization of the Gaussian distribution. In this paper, we will be interested in symmetric α-stable distributions, denoted by $\mathcal{S}\alpha \mathcal{S}$. In the one-dimensional case, a random variable X is $\mathcal{S}\alpha \mathcal{S}(\sigma )$ distributed, if its characteristic function (chf.) has the following form: $\mathbb{E}[\mathrm{exp}(i\omega X)]=\mathrm{exp}(-\vert \sigma \omega {\vert }^{\alpha })$, where $\alpha \in \left(\right.0,2\left.\right]$ is called the tail-index and $\sigma \in {\mathbb{R}}_{+}$ is called the scale parameter. When α = 2, $\mathcal{S}\alpha \mathcal{S}(\sigma )=\mathcal{N}(0,2{\sigma }^{2})$, where $\mathcal{N}$ denotes the Gaussian distribution in $\mathbb{R}$. As soon as α < 2, the distribution becomes heavy-tailed and $\mathbb{E}[\vert X{\vert }^{q}]$ becomes finite if and only if q < α, indicating that the variance of $\mathcal{S}\alpha \mathcal{S}$ is finite only when α = 2.

There are multiple ways to extend $\mathcal{S}\alpha \mathcal{S}$ to the multivariate case. In our experiments, we will be mainly interested in the elliptically-contoured α-stable distribution [ST94], whose chf. is given by $\mathbb{E}[\mathrm{exp}(i\langle \omega ,X\rangle )]=\mathrm{exp}(-{\Vert}\omega {{\Vert}}^{\alpha })$ for $X,\omega \in {\mathbb{R}}^{d}$, where ⟨⋅, ⋅⟩ denotes the Euclidean inner product. Another common choice is the multivariate α -stable distribution with independent components for a vector $\boldsymbol{\alpha }\in {\mathbb{R}}^{d}$, whose chf. is given by $\mathbb{E}[\mathrm{exp}(i\langle \omega ,X\rangle )]=\mathrm{exp}(-{\sum }_{i=1}^{d}\vert {\omega }_{i}{\vert }^{{\alpha }_{i}})$. Essentially, the ith component of X is distributed with $\mathcal{S}\alpha \mathcal{S}$ with parameters α_i and σ_i = 1. Both of these multivariate distributions reduce to a multivariate Gaussian when their tail indices are 2.

We begin by defining a general Lévy process (also called Lévy motion), which includes Brownian motion B_t and the α-stable motion ${\mathrm{L}}_{t}^{\alpha }$ as special cases ⁷ . A Lévy process ${\left\{{\mathrm{L}}_{t}\right\}}_{t\geqslant 0}$ in ${\mathbb{R}}^{d}$ with the initial point L₀ = 0, is defined by the following properties:

• (a)  
  For $N\in \mathbb{N}$ and t₀ < t₁ <⋯< t_N , the increments $({\mathrm{L}}_{{t}_{i}}-{\mathrm{L}}_{{t}_{i-1}})$ are independent for all i.
• (b)  
  For any t > s > 0, (L_t − L_s ) and L_t−s have the same distribution.
• (c)  
  L_t is continuous in probability, i.e. for all δ > 0 and s ⩾ 0, $\mathbb{P}(\vert {\mathrm{L}}_{t}-{\mathrm{L}}_{s}\vert > \delta )\to 0$ as t → s.

By the Lévy–Khintchine formula [Sat99], the chf. of a Lévy process is given by $\mathbb{E}[\mathrm{exp}(i\langle \xi ,{\mathrm{L}}_{t}\rangle )]=\mathrm{exp}(-t\psi (\xi ))$, where $\psi :{\mathbb{R}}^{d}{\mapsto}\mathbb{C}$ is called the characteristic (or Lévy) exponent, given as:

$$\begin{equation}\psi (\xi )=i\langle b,\xi \rangle +\frac{1}{2}\left\langle \xi ,{\Sigma}\xi \right\rangle +{\int }_{{\mathbb{R}}^{d}}\left[1-{\text{e}}^{\text{i}\langle x,\xi \rangle }+\frac{i\langle x,\xi \rangle }{1+{\Vert}x{{\Vert}}^{2}}\right]\nu (\mathrm{d}x),\quad \forall \xi \in {\mathbb{R}}^{d}.\end{equation} \tag{ 5 }$$

Here, $b\in {\mathbb{R}}^{d}$ denotes a constant drift, ${\Sigma}\in {\mathbb{R}}^{d\times d}$ is a positive semi-definite matrix, and ν is called the Lévy measure, which is a Borel measure on ${\mathbb{R}}^{d}{\backslash}\left\{0\right\}$ satisfying ${\int }_{{\mathbb{R}}^{d}}{\Vert}x{{\Vert}}^{2}/(1+{\Vert}x{{\Vert}}^{2})\nu (\mathrm{d}x)< \infty .$ The choice of (b, Σ, ν) determines the law of L_t−s ; hence, it fully characterizes the process L_t by the properties (a) and (b) above. For instance, from (5), we can easily verify that under the choice b = 0, ${\Sigma}=\frac{1}{2}{\mathrm{I}}_{d}$, and ν(ξ) = 0, with I_d denoting the d × d identity matrix, the function exp(−ψ(ξ)) becomes the chf. of a standard Gaussian in ${\mathbb{R}}^{d}$; hence, L_t reduces to B_t . On the other hand, if we choose b = 0, Σ = 0, and $\nu (\mathrm{d}x)=\frac{\mathrm{d}r}{{r}^{1+\alpha }}\lambda (\mathrm{d}y)$, for all $x=ry,(r,y)\in {\mathbb{R}}_{+}\times {\mathbb{S}}^{d-1}$ where ${\mathbb{S}}^{d-1}$ denotes unit sphere in ${\mathbb{R}}^{d}$ and λ is an arbitrary Borel measure on ${\mathbb{S}}^{d-1}$, we obtain the chf. of a generic multivariate α-stable distribution, hence L_t reduces to ${\mathrm{L}}_{t}^{\alpha }$. Depending on λ, exp(−ψ(ξ)) becomes the chf. of an elliptically contoured α-stable distribution or an α-stable distribution with independent components [Xia03].

Feller processes (also called Lévy-type processes [BSW13]) are a general family of Markov processes, which further extend the scope of Lévy processes. In this study, we consider a class of Feller processes [Cou65], which locally behave like Lévy processes and they additionally allow for state-dependent drifts b(w), diffusion matrices Σ(w), and Lévy measures ν(w, dy) for $w\in {\mathbb{R}}^{d}$. For a fixed state w, a Feller process ${\left\{{\mathrm{W}}_{t}\right\}}_{t\geqslant 0}$ is defined through the chf. of the random variable W_t − w, given as ${\psi }_{t}(w,\xi )=\mathbb{E}\left[\mathrm{exp}(-i\langle \xi ,{\mathrm{W}}_{t}-w\rangle )\right]$. A crucial characteristic of a Feller process related to its chf. is its symbol Ψ, defined as,

$$\begin{equation}{\Psi}(w,\xi )=i\langle b(w),\xi \rangle +\frac{1}{2}\left\langle \xi ,{\Sigma}(w)\xi \right\rangle +{\int }_{{\mathbb{R}}^{d}}\left[1-{\text{e}}^{\text{i}\langle x,\xi \rangle }+\frac{i\langle w,\xi \rangle }{1+{\Vert}x{{\Vert}}^{2}}\right]\nu (w,\mathrm{d}x),\end{equation} \tag{ 6 }$$

for $w,\xi \in {\mathbb{R}}^{d}$ [Jac02, Sch98, Xia03]. Here, for each $w\in {\mathbb{R}}^{d}$, ${\Sigma}(w)\in {\mathbb{R}}^{d\times d}$ is symmetric positive semi-definite, and for all w, ν(w, dx) is a Lévy measure.

Under mild conditions, one can verify that the SDE (4) we use as a proxy for the SGD algorithm indeed corresponds to a Feller process with b(w) = −∇f(w), Σ(w) = 2Σ₁(w), and an appropriate choice of ν (see [HDS18]). We also note that many other popular stochastic optimization algorithms can be accurately represented by a Feller process, which we describe in appendix B. Hence, our results can be useful in a broader context.

In this paper, we will focus on decomposable Feller processes introduced in [Sch98], which will be useful in both our theory and experiments. Let W_t be a Feller process with symbol Ψ. We call the process W_t 'decomposable at w₀', if there exists a point ${w}_{0}\in {\mathbb{R}}^{d}$, such that ${\Psi}(w,\xi )=\psi (\xi )+\tilde{{\Psi}}(w,\xi )$, where ψ(ξ) := Ψ(w₀, ξ) is called the sub-symbol and $\tilde{{\Psi}}(w,\xi ){:=}{\Psi}(w,\xi )-{\Psi}({w}_{0},\xi )$ is the remainder term. Here, $\tilde{{\Psi}}$ is assumed to satisfy certain smoothness and boundedness assumptions, which are provided in appendix C. Essentially, the technical regularity conditions on $\tilde{{\Psi}}$ impose a structure on the triplet (b, Σ, ν) around w₀ which ensures that, around that point, W_t behaves like a Lévy process whose characteristic exponent is given by the sub-symbol ψ.

Due to their recursive nature, Markov processes often generate 'random fractals' [Xia03] and understanding the structure of such fractals has been a major challenge in modern probability theory [BP17, Kho09, KX17, LG19, LY19, Yan18]. In this paper, we are interested in identifying the complexity of the fractals generated by a Feller process that approximates SGD.

The intrinsic complexity of a fractal is typically characterized by a notion called the Hausdorff dimension [Fal04], which extends the usual notion of dimension (e.g. a line segment is one-dimensional, a plane is two-dimensional) to fractional orders. Informally, this notion measures the 'roughness' of an object (i.e. a set) and in the context of Lévy processes, they are deeply connected to the tail properties of the corresponding Lévy measure. [Sch98, Xia03, Yan18].

Before defining the Hausdorff dimension, we need to introduce the Hausdorff measure. Let $G\subset {\mathbb{R}}^{d}$ and δ > 0, and consider all the δ-coverings ${\left\{{A}_{i}\right\}}_{i}$ of G, i.e. each A_i denotes a set with diameter less than δ satisfying G ⊂ ∪_i A_i . For any s ∈ (0, ∞), we then denote:

$$\begin{equation}{\mathcal{H}}_{\delta }^{s}(G){:=}\mathrm{inf}\sum\limits _{i=1}^{\infty }\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{m}{({A}_{i})}^{s},\end{equation} \tag{ 7 }$$

where the infimum is taken over all the δ-coverings. The s-dimensional Hausdorff measure of G is defined as the monotonic limit:

$$\begin{equation}{\mathcal{H}}^{s}(G){:=}\underset{\delta \to 0}{\mathrm{lim}}\enspace {\mathcal{H}}_{\delta }^{s}(G).\end{equation} \tag{ 8 }$$

It can be shown that ${\mathcal{H}}^{s}$ is an outer measure; hence, it can be extended to a complete measure by the Carathéodory extension theorem [Mat99]. When s is an integer, ${\mathcal{H}}^{s}$ is equal to the s-dimensional Lebesgue measure up to a constant factor; thus, it strictly generalizes the notion 'volume' to the fractional orders. We now proceed with the definition of the Hausdorff dimension.

Definition 1. The Hausdorff dimension of $G\subset {\mathbb{R}}^{d}$ is defined as follows.

$$\begin{equation}{\mathrm{dim}}_{\mathrm{H}}\enspace G{:=}\mathrm{sup}\left\{s > 0:{\mathcal{H}}^{s}(G) > 0\right\}=\mathrm{inf}\left\{s > 0:{\mathcal{H}}^{s}(G)< \infty \right\}.\end{equation} \tag{ 9 }$$

One can show that if dim_H  G = s, then ${\mathcal{H}}^{r}(G)=0$ for all r > s and ${\mathcal{H}}^{r}(G)=\infty$ for all r < s [EMG90]. In this sense, the Hausdorff dimension of G is the moment order s when ${\mathcal{H}}^{s}(G)$ drops from ∞ to 0, and we always have 0 ⩽ dim_H  G ⩽ d [Fal04]. Apart from the trivial cases such as ${\mathrm{dim}}_{\mathrm{H}}\enspace {\mathbb{R}}^{d}=d$, a canonical example is the well-known Cantor set, whose Hausdorff dimension is (log 2/log 3) ∈ (0, 1). Besides, the Hausdorff dimension of Riemannian manifolds correspond to their intrinsic dimension, e.g. ${\mathrm{dim}}_{\mathrm{H}}\enspace {\mathbb{S}}^{d-1}=d-1$.

We note that, starting with the seminal work of Assouad [Ass83], tools from fractal geometry have been considered in learning theory [DSD19, MSS19, SHTY13] in different contexts. In this paper, we consider the Hausdorff dimension of the sample paths of Markov processes in a learning theoretical framework, which, to the best of our knowledge, has not yet been investigated in the literature.

In this part, we introduce the 'uniform Hausdorff dimension' property for a training algorithm $\mathcal{A}$, which is a notion of complexity based on the Hausdorff dimension of the trajectories generated by $\mathcal{A}$. By translating [Sch98] into our context, we will then show that decomposable Feller processes possess this property.

Definition 2. An algorithm $\mathcal{A}$ has uniform Hausdorff dimension d_H if for any $n\in {\mathbb{N}}_{+}$ and any training set $S\in {\mathcal{Z}}^{n}$

$$\begin{equation}{\mathrm{dim}}_{\mathrm{H}}\enspace {\mathcal{W}}_{S}={\mathrm{dim}}_{\mathrm{H}}\left\{w\in {\mathbb{R}}^{d}:\exists t\in [0,1],w={[\mathcal{A}(S,U)]}_{t}\right\}\leqslant {d}_{\mathrm{H}},\quad {\mu }_{u}-\text{almost}\;\text{surely.}\end{equation} \tag{ 15 }$$

Since ${\mathcal{W}}_{S}\subset \mathcal{W}\subset {\mathbb{R}}^{d}$, by the definition of Hausdorff dimension, any algorithm $\mathcal{A}$ possesses the uniform Hausdorff dimension property trivially with d_H = d. However, as we will illustrate in the sequel, d_H can be much smaller than d, which is of our interest in this study.

Proposition 1. Let ${\left\{{\mathrm{W}}^{(S)}\right\}}_{S\in {\mathcal{Z}}^{n}}$ be a family of Feller processes. Assume that for each S, W^(S) is decomposable at a point w_S with sub-symbol ψ_S . Consider the algorithm $\mathcal{A}$ that returns ${[\mathcal{A}(S)]}_{t}={\mathrm{W}}_{t}^{(S)}$ for a given $S\in {\mathcal{Z}}^{n}$ and for every t ∈ [0, 1]. Then, we have

$$\begin{equation}{\mathrm{dim}}_{\mathrm{H}}\enspace {\mathcal{W}}_{S}\leqslant {\beta }_{S},\enspace \enspace \;\text{where}\;\enspace \enspace {\beta }_{S}{:=}\mathrm{inf}\left\{\lambda \geqslant 0:\underset{{\Vert}\xi {\Vert}\to \infty }{\mathrm{lim}}\enspace \frac{\vert {\psi }_{S}(\xi )\vert }{{\Vert}\xi {{\Vert}}^{\lambda }}=0\right\},\end{equation} \tag{ 16 }$$

μ_u -almost surely. Furthermore, $\mathcal{A}$ has uniform Hausdorff dimension with

$$\begin{equation}{d}_{\mathrm{H}}=\underset{n}{\mathrm{sup}}\enspace \underset{S\in {\mathcal{Z}}^{n}}{\mathrm{sup}}\enspace {\beta }_{S}.\end{equation} \tag{ 17 }$$

We provide all the proofs in appendix E. Informally, this result can be interpreted as follows. Thanks to the decomposability property, for each S, the process W^(S) behaves like a Lévy motion around w_S , and the characteristic exponent is given by the sub-symbol ψ_S . Because of this locally regular behavior, the Hausdorff dimension of the image of W^(S) can be bounded by β_S , which only depends on tail behavior of the Lévy process whose exponent is the sub-symbol ψ_S .

Example 1. In order to illustrate proposition 1, let us consider a simple example, where ${\mathrm{W}}_{t}^{(S)}$ is taken as the d-dimensional α-stable process with d ⩾ 2, which is independent of the data sample S. More precisely, ${\mathrm{W}}_{t}^{(S)}$ is the solution to the SDE given by $\mathrm{d}{\mathrm{W}}_{t}^{(S)}=\mathrm{d}{\mathrm{L}}_{t}^{\alpha }$ for some $\alpha \in \left(\right.0,2\left.\right]$, where ${\mathrm{L}}_{1}^{\alpha }$ is an elliptically-contoured α-stable random vector. As ${\mathrm{W}}_{t}^{(S)}$ is already a Lévy process, it trivially satisfies the assumptions of proposition 1 with β_S = α for all n and S [BG60], hence

$$\begin{equation}{\mathrm{dim}}_{\mathrm{H}}\enspace {\mathcal{W}}_{S}\leqslant \alpha ,\end{equation} \tag{ 18 }$$

μ_u -almost surely (in fact, one can show that ${\mathrm{dim}}_{\mathrm{H}}\enspace {\mathcal{W}}_{S}=\alpha$, see [BG60], theorem 4.2). Hence, the 'algorithm' ${[\mathcal{A}(S)]}_{t}={\mathrm{W}}_{t}^{(S)}$ has uniform Hausdorff dimension d_H = α. This shows that as the process becomes heavier-tailed (i.e. α decreases), the Hausdorff dimension d_H gets smaller. This behavior is illustrated in figure 1.

Zoom In Zoom Out Reset image size

The term β_S is often termed as the upper Blumenthal–Getoor (BG) index of the Lévy process with an exponent ψ_S [BG60], and it is directly related to the tail-behavior of the corresponding Lévy measure. In general, the value of β_S decreases as the process gets heavier-tailed, which implies that the heavier-tailed processes have smaller Hausdorff dimension; thus, they have smaller complexity.

This part provides the first main contribution of this paper, where we show that the generalization error of a training algorithm can be controlled by the Hausdorff dimension of its trajectories. Even though our interest is still in the case where $\mathcal{A}$ is chosen as a Feller process, the results in this section apply to more general algorithms. To this end, we will be mainly interested in bounding the following object:

$$\begin{equation}\enspace \underset{t\in [0,1]}{\mathrm{sup}}\vert \hat{\mathcal{R}}({[\mathcal{A}(S)]}_{t},S)-\mathcal{R}({[\mathcal{A}(S)]}_{t})\vert =\underset{w\in {\mathcal{W}}_{S}}{\mathrm{sup}}\vert \hat{\mathcal{R}}(w,S)-\mathcal{R}(w)\vert ,\end{equation} \tag{ 19 }$$

with high probability over the choice of S and U. Note that this is an algorithm dependent definition of generalization that is widely used in the literature (see [BE02] for a detailed discussion).

To derive our first result, we will require the following assumptions.

• H 1  
  ℓ is bounded by B and L-Lipschitz continuous in w.
• H 2  
  The diameter of $\mathcal{W}$ is finite μ_u -almost surely. S and U are independent.
• H 3  
  $\mathcal{A}$ has uniform Hausdorff dimension d_H.
• H 4  
  For μ_u -almost every $\mathcal{W}$, there exists a Borel measure μ on ${\mathbb{R}}^{d}$ and positive numbers a, b, r₀ and s such that $0< \mu (\mathcal{W})\leqslant \mu \left({\mathbb{R}}^{d}\right)< \infty$ and 0 < ar^s ⩽ μ(B_d (x, r)) ⩽ br^s < ∞ for $x\in \mathcal{W},0< r\leqslant {r}_{0}$.

Boundedness of the loss can be relaxed at the expense of using sub-Gaussian concentration bounds and introducing more complexity into the expressions [MBM16]. More precisely, H 1 can be replaced with the assumption ∃K > 0, such that ∀p, $\mathbb{E}{[\ell {(w,z)}^{p}]}^{1/p}\leqslant K\sqrt{p}$, and by using sub-Gaussian concentration our bounds will still hold with K in place of B. On the other hand, since we have a finite time-horizon and we fix the initial point of the processes to 0, by using [XZ20] lemma 7.1, we can show that the finite diameter condition on $\mathcal{W}$ holds almost surely, if standard regularity assumptions hold uniformly on the coefficients of W^(S) (i.e. b, Σ, and ν in (6)) for all $S\in {\mathcal{Z}}^{n}$, and a countability condition on $\mathcal{Z}$. Finally H 4 is a common condition in fractal geometry, and ensures that the set $\mathcal{W}$ is regular enough, so that we can relate its Hausdorff dimension to its covering numbers [Mat99] ⁹ . Under these conditions and an additional countability condition on $\mathcal{Z}$ (see [BE02] for similar assumptions), we present our first main result as follows.

Theorem 1. Assume that H 1 to 4 hold, and $\mathcal{Z}$ is countable. Then, for a sufficiently large n, we have

$$\begin{equation}\underset{w\in {\mathcal{W}}_{S}}{\mathrm{sup}}\vert \hat{\mathcal{R}}(w,S)-\mathcal{R}(w)\vert \leqslant B\sqrt{\frac{2{d}_{\mathrm{H}}\enspace \mathrm{log}(n{L}^{2})}{n}+\frac{\mathrm{log}(1/\gamma )}{n}},\end{equation} \tag{ 20 }$$

with probability at least 1 − γ over $S\sim {\mu }_{z}^{\otimes n}$ and U ∼ μ_u .

This theorem shows that the generalization error can be controlled by the uniform Hausdorff dimension of the algorithm $\mathcal{A}$, along with the constants inherited from the regularity conditions. A noteworthy property of this result is that it does not have a direct dependency on the number of parameters d; on the contrary, we observe that d_H plays the role that d plays in standard bounds [AB09], implying that d_H acts as the intrinsic dimension and mimics the role of the VC dimension in binary classification [SSBD14]. Furthermore, in combination with proposition 1 that indicates d_H decreases as the processes W^(S) get heavier-tailed, theorem 1 implies that the generalization error can be controlled by the tail behavior of the process: heavier-tails imply less generalization error.

We note that the countability condition on $\mathcal{Z}$ is crucial for theorem 1. Thanks to this condition, in our proof, we invoke the stability properties of the Hausdorff dimension and we directly obtain a bound on ${\mathrm{dim}}_{\mathrm{H}}\enspace \mathcal{W}$. This bound combined with H 4 allows us to control the covering number of $\mathcal{W}$, and then the desired result can be obtained by using standard covering techniques [AB09, SSBD14].

Next, we show that the log n dependency of d_H is not crucial. In the next theorem, we show that the log n factor can be replaced with any increasing function (e.g. log log n) by using a chaining argument, with the expense of having L as a multiplying factor (instead of log L). Theorem 1 holds for sufficiently large n; however, this threshold is not apriori known, which is a limitation of the result.

Theorem 2. Assume that H 1 to 4 hold, and $\mathcal{Z}$ is countable. Then, for any function $\xi :\mathbb{R}\to \mathbb{R}$ satisfying lim_x→∞  ρ(x) = ∞, and for a sufficiently large n, we have

$$\begin{equation*}\underset{w\in {\mathcal{W}}_{S}}{\mathrm{sup}}\left(\hat{\mathcal{R}}(w,S)-\mathcal{R}(w)\right)\leqslant cLB\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{m}(\mathcal{W})\sqrt{\frac{{d}_{\mathrm{H}}\rho (n)}{n}+\frac{\mathrm{log}(1/\gamma )}{n}},\end{equation*}$$

with probability at least 1 − γ over $S\sim {\mu }_{z}^{\otimes n}$ and U ∼ μ_u , where c is an absolute constant.

In our second main result, we control the generalization error without the countability assumption on $\mathcal{Z}$, and more importantly we will also relax H 3. Our main goal will be to relate the error to the Hausdorff dimension of a single ${\mathcal{W}}_{S}$, as opposed to d_H, which uniformly bounds ${\mathrm{dim}}_{\mathrm{H}}\enspace {\mathcal{W}}_{{S}^{\prime }}$ for every ${S}^{\prime }\in {\mathcal{Z}}^{n}$. In order to achieve this goal, we introduce a technical assumption, which lets us control the statistical dependency between the training set S and the set of parameters ${\mathcal{W}}_{S}$.

For any δ > 0, let us consider a finite δ-cover of $\mathcal{W}$ by closed balls of radius δ, whose centers are on the fixed grid

$$\begin{equation}\left\{\left(\frac{(2{j}_{1}+1)\delta }{2\sqrt{d}},\dots ,\frac{(2{j}_{d}+1)\delta }{2\sqrt{d}}\right):{j}_{i}\in \mathbb{Z},i=1,\dots ,\mathrm{d}\right\},\end{equation} \tag{ 21 }$$

and collect the center of each ball in the set N_δ . Then, for each S, let us define the set ${N}_{\delta }(S){:=}\left\{x\in {N}_{\delta }:{B}_{d}(x,\delta )\cap {\mathcal{W}}_{S}\ne \varnothing \right\}$, where ${B}_{d}(x,\delta )\subset {\mathbb{R}}^{d}$ denotes the closed ball centered around $x\in {\mathbb{R}}^{d}$ with radius δ.

• H 5  
  Let ${\mathcal{Z}}^{\infty }{:=}(\mathcal{Z}\times \mathcal{Z}\times \cdots \enspace )$ denote the countable product endowed with the product topology and let $\mathfrak{B}$ be the Borel σ-algebra generated by ${\mathcal{Z}}^{\infty }$. Let $\mathfrak{F},\mathfrak{G}$ be the sub-σ-algebras of $\mathfrak{B}$ generated by the collections of random variables given by $\left\{\hat{\mathcal{R}}(w,S):w\in \mathcal{W},n\geqslant 1\right\}$ and $\left\{\mathbb{1}\left\{w\in {N}_{\delta }({\mathcal{W}}_{S})\right\}:\delta \in {\mathbb{Q}}_{ > 0},w\in {N}_{\delta },n\geqslant 1\right\}$ respectively. There exists a constant M ⩾ 1 such that for any $A\in \mathfrak{F}$, $B\in \mathfrak{G}$ we have $\mathbb{P}\left[A\cap B\right]\leqslant M\mathbb{P}\left[A\right]\mathbb{P}[B]$.

This assumption is common in statistics and is sometimes referred to as the ψ-mixing condition, a measure of weak dependence often used in proving limit theorems, see e.g. [Bra83]; yet, it is unfortunately hard to verify this condition in practice. In our context H 5 essentially quantifies the dependence between S and the set ${\mathcal{W}}_{S}$, through the constant M > 0: smaller M indicates that the dependence of $\hat{\mathcal{R}}$ on the training sample S is weaker. This concept is also similar to the mutual information used recently in [AAV18, HŞKM21, RZ19, XR17] and to the concept of stability [BE02].

Theorem 3. Assume that H 1, 2 and 5 hold, and H 4 holds with ${\mathcal{W}}_{S}$ in place of $\mathcal{W}$ for all n ⩾ 1 and $S\in {\mathcal{Z}}^{n}$, (with s, a, b, r₀ can potentially depend on n and S). Then, for n sufficiently large, we have

$$\begin{equation}\underset{w\in {\mathcal{W}}_{S}}{\mathrm{sup}}\vert \hat{\mathcal{R}}(w,S)-\mathcal{R}(w)\vert \leqslant 2B\sqrt{\frac{[{\mathrm{dim}}_{\mathrm{H}}\enspace {\mathcal{W}}_{S}+1]{\mathrm{log}}^{2}(n{L}^{2})}{n}+\frac{\mathrm{log}(7M/\gamma )}{n}},\end{equation} \tag{ 22 }$$

with probability at least 1 − γ over $S\sim {\mu }_{z}^{\otimes n}$ and U ∼ μ_u .

This result shows that under H 5, we can replace d_H in theorem 1 with ${\mathrm{dim}}_{\mathrm{H}}\enspace {\mathcal{W}}_{S}$, at the expense of introducing the coupling coefficient M into the bound. We observe that two competing terms are governing the generalization error: in the case where ${\mathrm{dim}}_{\mathrm{H}}\enspace {\mathcal{W}}_{S}$ is small, the error is dominated by the coupling parameter M, and vice versa. On the other hand, in the context of proposition 1, ${\mathrm{dim}}_{\mathrm{H}}\enspace {\mathcal{W}}_{S}\leqslant {\beta }_{S}$, μ_u -almost surely, implying again that a heavy-tailed W^(S) would achieve smaller generalization error as long as the dependency between S and ${\mathcal{W}}_{S}$ is weak.

In this section, we empirically analyze the proposed metric with respect to existing generalization metrics, developed for neural networks. Specifically, we consider the 'flat minima' argument of Jastrzevski et al [JKA+17] and plot the generalization error vs η/B which is the ratio of step size to the batch size. As a second comparison, we use heavy-tailed random matrix theory based metric of Martin and Mahoney [MM19]. We plot the generalization error with respect to each metric in figure A1. As the results suggest, our metric is the one which correlates best with the empirically observed generalization error. The metric proposed by Martin and Mahoney [MM19] fails for the low number of layers and the resulting behavior is not monotonic. Similarly, η/B captures the relationship for very deep networks (for D = 16 & 19), however, it fails for other settings.

Zoom In Zoom Out Reset image size

We also note that the norm-based capacity metrics [NTS15] typically increase with the increasing dimension d, we refer to [NBMS17] for details.

We consider a simple synthetic logistic regression problem, where the data distribution is a Gaussian mixture model with two components. Each data point ${z}_{i}\equiv ({x}_{i},{y}_{i})\in \mathcal{Z}={\mathbb{R}}^{d}\times \left\{-1,1\right\}$ is generated by simulating the model: y_i ∼ Bernoulli(1/2) and ${x}_{i}\vert {y}_{i}\sim \mathcal{N}({m}_{{y}_{i}},100{\mathrm{I}}_{d})$, where the means are drawn from a Gaussian: ${m}_{-1},{m}_{1}\sim \mathcal{N}(0,25{\mathrm{I}}_{d})$. The loss function ℓ is the logistic loss as ℓ(w, z) = log(1 + exp(−yx^⊤ w)).

As for the algorithm, we consider a data-independent multivariate stable process: ${[\mathcal{A}(S)]}_{t}={\mathrm{L}}_{t}^{\alpha }$ for any $S\in {\mathcal{Z}}^{n}$, where ${\mathrm{L}}_{1}^{\alpha }$ is distributed with an elliptically contoured α-stable distribution with $\alpha \in \left(\right.0,2\left.\right]$ (see section 2): when α = 2, ${\mathrm{L}}_{t}^{\alpha }$ is just a Brownian motion, as α gets smaller, the process becomes heavier-tailed. By theorem 4.2 of [BG60], $\mathcal{A}$ has the uniform Hausdorff dimension property with d_H = α independently from d when d ⩾ 2.

We set d = 10 and generate points to represent the whole population, i.e. ${\left\{{z}_{i}\right\}}_{i=1}^{{n}_{\text{tot}}}$ with n_tot = 100 K. Then, for different values of α, we simulate $\mathcal{A}$ for t ∈ [0, 1], by using a small step-size η = 0.001 (the total number of iterations is hence 1/η). We finally draw 20 random sets S with n elements from this population, and we monitor the maximum difference ${\mathrm{sup}}_{w\in {\mathcal{W}}_{S}}\vert \hat{\mathcal{R}}(w,S)-\mathcal{R}(w)\vert$ for different values of n. We repeat the whole procedure 20 times and report the average values in figure A2. We observe that the results support theorems 1 and 3: for every n, the generalization error decreases with decreasing α, hence illustrates the role of the Hausdorff dimension.

Zoom In Zoom Out Reset image size

In this section, we provide the additional details which are skipped in the main text for the sake of space. We use the following VGG style neural networks with various number of layers as

• VGG4: Conv(512)–ReLU–MaxPool–Linear
• VGG6: Conv(256)–ReLU–MaxPool–Conv(512)–ReLU–MaxPool–Conv(512)–ReLU–MaxPool–Linear
• VGG7: Conv(128)–ReLU–MaxPool–Conv(256)–ReLU–MaxPool–Conv(512)–ReLU–MaxPool–Conv(512)–ReLU–MaxPool–Linear
• VGG8: Conv(64)–ReLU–MaxPool–Conv(128)–ReLU–MaxPool–Conv(256)–ReLU–MaxPool–Conv(512)–ReLU–MaxPool–Conv(512)–ReLU–MaxPool–Linear
• VGG11: Conv(64)–ReLU–MaxPool–Conv(128)–ReLU–MaxPool –Conv(256)–ReLU–Conv(256)–ReLU–MaxPool–Conv(512)–ReLU–Conv(512)–ReLU–MaxPool–Conv(512)–ReLU–Conv(512)–ReLU–MaxPool–Linear
• VGG16: Conv(64)–ReLU–Conv(64)–ReLU–MaxPool–Conv(128)–ReLU–Conv(128)–ReLU–MaxPool–Conv(256)–ReLU–Conv(256)–ReLU–Conv(256)–ReLU–MaxPool–Conv(512)–ReLU–Conv(512)–ReLU–Conv(512)–ReLU–MaxPool–Conv(512)–ReLU–Conv(512)–ReLU–Conv(512)–ReLU–MaxPool–Linear
• VGG19: Conv(64)–ReLU–Conv(64)–ReLU–MaxPool–Conv(128)–ReLU–Conv(128)–ReLU–MaxPool–Conv(256)–ReLU–Conv(256)–ReLU–Conv(256)–ReLU–Conv(256)–ReLU–MaxPool–Conv(512)–ReLU–Conv(512)–ReLU–Conv(512)–ReLU–Conv(512)–ReLU–MaxPool–Conv(512)–ReLU–Conv(512)–ReLU–Conv(512)–ReLU–Conv(512)–ReLU–MaxPool–Linear

where all convolutions are noted with the number of filters in the paranthesis. Moreover, we use the following hyperparameter ranges for step size of SGD: {1e–2, 1e–3, 3e–3, 1e–4, 3e–4, 1e–5, 3e–5} with the batch sizes {32, 64, 128, 256}. All networks are learned with cross entropy loss and ReLU activations, and no additional technique like batch normalization or dropout is used. While computing the empirical BG index over the layers, we only consider convolutional layers ignoring the final fully-connected layer. We also release the full source code of the experiments at https://github.com/umutsimsekli/Hausdorff-Dimension-and-Generalization.

In our proofs, in addition to the Hausdorff dimension, we also make use of another notion of dimension, referred to as the Minkowski dimension (also known as the box-counting dimension [Fal04]), which is defined as follows.

Definition 3. Let $G\subset {\mathbb{R}}^{d}$ be a set and let N_δ (G) be a collection of sets that contains either one of the following:

• The smallest number of sets of diameter at most δ which cover G
• The smallest number of closed balls of diameter at most δ which cover G
• The smallest number of cubes of side at most δ which cover G
• The number of δ-mesh cubes that intersect G
• The largest number of disjoint balls of radius δ, whose centers are in G.

Then the lower- and upper-Minkowski dimensions of G are respectively defined as follows:

$$\begin{equation}\underline{{\mathrm{dim}}_{\mathrm{M}}}\enspace G{:=}\underset{\delta \to 0}{\mathrm{liminf}}\enspace \frac{\mathrm{log}\vert {N}_{\delta }(G)\vert }{-\mathrm{log}\delta },\hspace{25.0pt}\bar{{\mathrm{dim}}_{\mathrm{M}}}G{:=}\underset{\delta \to 0}{\mathrm{limsup}}\enspace \frac{\mathrm{log}\vert {N}_{\delta }(G)\vert }{-\mathrm{log}\delta }.\end{equation} \tag{ D.1 }$$

In case the $\underline{{\mathrm{dim}}_{\mathrm{M}}}\enspace G=\bar{{\mathrm{dim}}_{\mathrm{M}}}\enspace G$, the Minkowski dimension dim_M(G) is their common value.

We always have $0\leqslant {\mathrm{dim}}_{\mathrm{H}}\enspace G\leqslant \underline{{\mathrm{dim}}_{\mathrm{M}}}\enspace G\leqslant \bar{{\mathrm{dim}}_{\mathrm{M}}}G\leqslant d$ where the inequalities can be strict [Fal04].

It is possible to construct examples where the Hausdorff and Minkowski dimensions are different from each other. However, in many interesting cases, these two dimensions often match each other [Fal04]. In this paper, we are interested in such a case, i.e. the case when the Hausdorff and Minkowski dimensions match. The following result identifies the conditions for which the two dimensions match each other, which form the basis of H 4:

Theorem 5 ([Mat99] theorem 5.7). Let A be a non-empty bounded subset of ${\mathbb{R}}^{d}$. Suppose there is a Borel measure μ on ${\mathbb{R}}^{d}$ and there are positive numbers a, b, r₀ and s such that $0< \mu (A)\leqslant \mu \left({\mathbb{R}}^{d}\right)< \infty$ and

$$\begin{equation}0< a{r}^{s}\leqslant \mu ({B}_{d}(x,r))\leqslant b{r}^{s}< \infty \enspace \enspace \;\text{for}\;\enspace \enspace x\in A,0< r\leqslant {r}_{0}\end{equation} \tag{ D.2 }$$

Then ${\mathrm{dim}}_{\mathrm{H}}\enspace A={\mathrm{dim}}_{\mathrm{M}}\enspace A=\bar{{\mathrm{dim}}_{\mathrm{M}}}\enspace A=s$.

Egoroff's theorem is an important result in measure theory and establishes a condition for measurable functions to be uniformly continuous in an almost full-measure set.

Theorem 6 (Egoroff's theorem [Bog07] theorem 2.2.1). Let $(X,\mathcal{A},\mu )$ be a space with a finite nonnegative measure μ and let μ-measurable functions f_n be such that μ-almost everywhere there is a finite limit f(x) := lim_n→∞  f_n (x). Then, for every ɛ > 0, there exists a set ${X}_{\varepsilon }\in \mathcal{A}$ such that $\mu \left(X{\backslash}{X}_{\varepsilon }\right)< \varepsilon$ and the functions f_n converge to f uniformly on X_ɛ .

Proof. Let Ψ_S denote the symbol of the process W^(S). Then, the desired result can obtained by directly applying theorem 4 on each Ψ_S . □

We first prove the following more general result which relies on $\bar{{\mathrm{dim}}_{\mathrm{M}}}\enspace \mathcal{W}$.

Lemma 1. Assume that ℓ is bounded by B and L-Lipschitz continuous in w. Let $\mathcal{W}\subset {\mathbb{R}}^{d}$ be a set with finite diameter. Then, for n sufficiently large, we have

$$\begin{equation}\underset{w\in \mathcal{W}}{\mathrm{sup}}\vert \hat{\mathcal{R}}(w,S)-\mathcal{R}(w)\vert \leqslant B\sqrt{\frac{2\bar{{\mathrm{dim}}_{\mathrm{M}}}\enspace \mathcal{W}\enspace \mathrm{log}(n{L}^{2})}{n}+\frac{\mathrm{log}(1/\gamma )}{n}},\end{equation} \tag{ E.1 }$$

with probability at least 1 − γ over $S\sim {\mu }_{z}^{\otimes n}$.

Proof. As ℓ is L-Lipschitz, so are $\mathcal{R}$ and $\hat{\mathcal{R}}$. By using the notation ${\hat{\mathcal{R}}}_{n}(w){:=}\hat{\mathcal{R}}(w,S)$, and by the triangle inequality, for any ${w}^{\prime }\in \mathcal{W}$ we have:

$$\begin{equation}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert =\left\vert {\hat{\mathcal{R}}}_{n}\left({w}^{\prime }\right)-\mathcal{R}\left({w}^{\prime }\right)+{\hat{\mathcal{R}}}_{n}(w)-{\hat{\mathcal{R}}}_{n}\left({w}^{\prime }\right)-\mathcal{R}(w)+\mathcal{R}\left({w}^{\prime }\right)\right\vert ,\end{equation} \tag{ E.2 }$$

$$\begin{equation}\leqslant \left\vert {\hat{\mathcal{R}}}_{n}\left({w}^{\prime }\right)-\mathcal{R}\left({w}^{\prime }\right)\right\vert +2L{\Vert}w-{w}^{\prime }{\Vert}.\end{equation} \tag{ E.3 }$$

Now since $\mathcal{W}$ has finite diameter, let us consider a finite δ-cover of $\mathcal{W}$ by balls and collect the center of each ball in the set ${N}_{\delta }{:=}{N}_{\delta }(\mathcal{W})$. Then, for each $w\in \mathcal{W}$, there exists a w' ∈ N_δ , such that ||w − w'|| ⩽ δ. By choosing this w' in the above inequality, we obtain:

$$\begin{equation}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \leqslant \left\vert {\hat{\mathcal{R}}}_{n}\left({w}^{\prime }\right)-\mathcal{R}\left({w}^{\prime }\right)\right\vert +2L\delta .\end{equation} \tag{ E.4 }$$

Taking the supremum of both sides of the above equation yields:

$$\begin{equation}\underset{w\in \mathcal{W}}{\mathrm{sup}}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \leqslant \underset{w\in {N}_{\delta }}{\mathrm{max}}\left\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\right\vert +2L\delta .\end{equation} \tag{ E.5 }$$

Using the union bound over N_δ , we obtain

$$\begin{equation}{\mathbb{P}}_{S}\left(\underset{w\in {N}_{\delta }}{\mathrm{max}}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \geqslant \varepsilon \right)={\mathbb{P}}_{S}\left(\bigcup\limits _{w\in {N}_{\delta }}\left\{\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \geqslant \varepsilon \right\}\right),\end{equation} \tag{ E.6 }$$

$$\begin{equation}\leqslant \sum\limits _{w\in {N}_{\delta }}{\mathbb{P}}_{S}\left(\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \geqslant \varepsilon \right).\end{equation} \tag{ E.7 }$$

Further, for δ > 0, since |N_δ | has finitely many elements, we can invoke Hoeffding's inequality for each of the summands on the right-hand side and obtain

$$\begin{equation}{\mathbb{P}}_{S}\left(\underset{w\in {N}_{\delta }}{\mathrm{max}}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \geqslant \varepsilon \right)\leqslant 2\vert {N}_{\delta }\vert \mathrm{exp}\left\{-\frac{2n{\varepsilon }^{2}}{{B}^{2}}\right\}=:\gamma .\end{equation} \tag{ E.8 }$$

Notice that N_δ is a random set, and choosing ɛ based on |N_δ |, one can obtain a deterministic γ. Therefore, we can plug this back in (E.5) and obtain that, with probability at least 1 − γ

$$\begin{equation}\underset{w\in \mathcal{W}}{\mathrm{sup}}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \leqslant B\sqrt{\frac{\mathrm{log}(2\vert {N}_{\delta }\vert )}{2n}+\frac{\mathrm{log}(1/\gamma )}{2n}}+2L\delta .\end{equation} \tag{ E.9 }$$

Now since $\mathcal{W}\subset {\mathbb{R}}^{d}$, $\bar{{\mathrm{dim}}_{\mathrm{M}}}\enspace \mathcal{W}$ is finite. Then, for any sequence ${\left\{{\delta }_{n}\right\}}_{n\in \mathbb{N}}$ such that lim_n→∞  δ_n = 0, we have, ∀ > 0, ∃n > 0 such that n ⩾ n implies

$$\begin{equation}\mathrm{log}(\vert {N}_{\delta }\vert )\leqslant (\bar{{\mathrm{dim}}_{\mathrm{M}}}\enspace \mathcal{W}+{\epsilon})\mathrm{log}({\delta }_{n}^{-1}).\end{equation} \tag{ E.10 }$$

Choosing ${\delta }_{n}=1/\sqrt{n{L}^{2}}$ and ${\epsilon}=\bar{{\mathrm{dim}}_{\mathrm{M}}}\enspace \mathcal{W}$, we have for $\forall n\geqslant {n}_{\bar{{\mathrm{dim}}_{\mathrm{M}}}\enspace \mathcal{W}}$,

$$\begin{equation}\mathrm{log}(2\vert {N}_{\delta }\vert )\leqslant \mathrm{log}(2)+\bar{{\mathrm{dim}}_{\mathrm{M}}}\enspace \mathcal{W}\enspace \mathrm{log}(n{L}^{2})\enspace \enspace \;\text{and}\;\enspace \enspace 2L{\delta }_{n}=\frac{2}{\sqrt{n}}.\end{equation} \tag{ E.11 }$$

Therefore, we obtain with probability at least 1 − γ

$$\begin{equation}\underset{w\in \mathcal{W}}{\mathrm{sup}}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \leqslant B\sqrt{\frac{\mathrm{log}(2)+\bar{{\mathrm{dim}}_{\mathrm{M}}}\enspace \mathcal{W}\enspace \mathrm{log}(n{L}^{2})}{2n}+\frac{\mathrm{log}(1/\gamma )}{2n}}+\frac{2}{\sqrt{n}},\end{equation} \tag{ E.12 }$$

$$\begin{equation}\leqslant B\sqrt{\frac{2\bar{{\mathrm{dim}}_{\mathrm{M}}}\enspace \mathcal{W}\enspace \mathrm{log}(n{L}^{2})}{n}+\frac{\mathrm{log}(1/\gamma )}{n}},\end{equation} \tag{ E.13 }$$

for sufficiently large n. This concludes the proof. □

We now proceed to the proof of theorem 1.

Proof of theorem  1. By noticing ${\mathcal{Z}}^{n}$ is countable (since $\mathcal{Z}$ is countable) and using the property that ${\mathrm{dim}}_{\mathrm{H}}{\cup }_{i\in \mathbb{N}}{A}_{i}={\mathrm{sup}}_{i\in \mathbb{N}}\enspace {\mathrm{dim}}_{\mathrm{H}}\enspace {A}_{i}$ (cf [Fal04], section 3.2), we observe that

$$\begin{equation}{\mathrm{dim}}_{\mathrm{H}}\enspace \mathcal{W}={\mathrm{dim}}_{\mathrm{H}}\bigcup\limits _{n\geqslant 1}\;\bigcup\limits _{S\in {\mathcal{Z}}^{n}}{\mathcal{W}}_{S}=\underset{n\geqslant 1}{\mathrm{sup}}\underset{S\in {\mathcal{Z}}^{n}}{\mathrm{sup}}\enspace {\mathrm{dim}}_{\mathrm{H}}\enspace {\mathcal{W}}_{S}\leqslant {\mathrm{d}}_{\mathrm{H}},\end{equation} \tag{ E.14 }$$

μ_u -almost surely. Define the event ${\mathcal{Q}}_{R}=\left\{\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{m}(\mathcal{W})\leqslant R\right\}$. On the event ${\mathcal{Q}}_{R}$, by theorem 5, we have that ${\mathrm{dim}}_{\mathrm{M}}\enspace \mathcal{W}=\bar{{\mathrm{dim}}_{\mathrm{M}}}\enspace \mathcal{W}={\mathrm{dim}}_{\mathrm{H}}\enspace \mathcal{W}\leqslant {d}_{\mathrm{H}}$, μ_u -almost surely.

Now, we observe that

$$\begin{equation}\underset{w\in {\mathcal{W}}_{S}}{\mathrm{sup}}\vert \hat{\mathcal{R}}(w,S)-\mathcal{R}(w)\vert \leqslant \underset{w\in \mathcal{W}}{\mathrm{sup}}\vert \hat{\mathcal{R}}(w,S)-\mathcal{R}(w)\vert .\end{equation} \tag{ E.15 }$$

Hence, by defining $\varepsilon =B\sqrt{\frac{2(\bar{{\mathrm{dim}}_{\mathrm{M}}}\enspace \mathcal{W})\mathrm{log}(n{L}^{2})}{n}+\frac{\mathrm{log}(1/\gamma )}{n}}$, and using the independence of S and U, lemma 1, and (E.14), we write

$$\begin{align*} {\mathbb{P}}_{S,U}& \left(\underset{w\in {\mathcal{W}}_{S}}{\mathrm{sup}}\vert \hat{\mathcal{R}}(w,S)-\mathcal{R}(w)\vert > B\sqrt{\frac{2{d}_{\mathrm{H}}\enspace \mathrm{log}(n{L}^{2})}{n}+\frac{\mathrm{log}(1/\gamma )}{n}};{\mathcal{Q}}_{R}\right) \\ & \leqslant {\mathbb{P}}_{S,U}\left(\underset{w\in \mathcal{W}}{\mathrm{sup}}\vert \hat{\mathcal{R}}(w,S)-\mathcal{R}(w)\vert > B\sqrt{\frac{2{d}_{\mathrm{H}}\mathrm{log}(n{L}^{2})}{n}+\frac{\mathrm{log}(1/\gamma )}{n}};{\mathcal{Q}}_{R}\right) \\ & ={\mathbb{P}}_{S,U}\left(\underset{w\in \mathcal{W}}{\mathrm{sup}}\vert \hat{\mathcal{R}}(w,S)-\mathcal{R}(w)\vert > B\sqrt{\frac{2{d}_{\mathrm{H}}\mathrm{log}(n{L}^{2})}{n}+\frac{\mathrm{log}(1/\gamma )}{n}};\bar{{\mathrm{dim}}_{\mathrm{M}}}\enspace \mathcal{W} \leqslant {d}_{\mathrm{H}};{\mathcal{Q}}_{R}\right) \\ & \leqslant {\mathbb{P}}_{S,U}\left(\underset{w\in \mathcal{W}}{\mathrm{sup}}\vert \hat{\mathcal{R}}(w,S)-\mathcal{R}(w)\vert > \varepsilon ;{\mathcal{Q}}_{R}\right). \end{align*}$$

Finally, we let R → ∞ and use dominated convergence theorem to obtain

$$\begin{align*} & {\mathbb{P}}_{S,U}\left(\underset{w\in {\mathcal{W}}_{S}}{\mathrm{sup}}\vert \hat{\mathcal{R}}(w,S)-\mathcal{R}(w)\vert > B\sqrt{\frac{2{d}_{\mathrm{H}}\enspace \mathrm{log}(n{L}^{2})}{n}+\frac{\mathrm{log}(1/\gamma )}{n}}\right) \\ & \leqslant {\mathbb{P}}_{S,U}\left(\underset{w\in \mathcal{W}}{\mathrm{sup}}\vert \hat{\mathcal{R}}(w,S)-\mathcal{R}(w)\vert > \varepsilon \right) \\ & \leqslant \gamma , \end{align*}$$

which concludes the proof. □

Similar to the proof of theorem 1, we first prove a more general result where $\bar{{\mathrm{dim}}_{\mathrm{M}}}\mathcal{W}$ is fixed.

Lemma 2. Assume that ℓ is bounded by B and L-Lipschitz continuous in w. Let $\mathcal{W}\subset {\mathbb{R}}^{d}$ be a bounded set with $\bar{{\mathrm{dim}}_{\mathrm{M}}}\mathcal{W}\leqslant {d}_{\mathrm{M}}$. For any function $\rho :\mathbb{R}\to \mathbb{R}$ satisfying lim_x→∞  ρ(x) = ∞ and for a sufficiently large n, with probability at least 1 − γ, we have

$$\begin{equation*}\underset{w\in \mathcal{W}}{\mathrm{sup}}\left({\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\right)\leqslant cLB\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{m}(\mathcal{W})\sqrt{\frac{{d}_{\mathrm{M}}\rho (n)+\mathrm{log}(1/\gamma )}{n}},\end{equation*}$$

where c is an absolute constant.

Proof. We define the empirical process

$$\begin{equation*}{\mathcal{G}}_{n}(w){:=}{\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)=\frac{1}{n}\sum\limits _{i=1}^{n}\ell (w,{z}_{i})-{\mathbb{E}}_{z}[\ell (w,z)],\end{equation*}$$

and we notice that

$$\begin{equation*}\mathbb{E}[{\mathcal{G}}_{n}(w)]=0.\end{equation*}$$

Recall that a random process ${\left\{G(w)\right\}}_{w\in \mathcal{W}}$ on a metric space $(\mathcal{W},d)$ is said to have sub-Gaussian increments if there exists K ⩾ 0 such that

$$\begin{equation}{\Vert}G(w)-G({w}^{\prime }){{\Vert}}_{{\psi }_{2}}\leqslant Kd(w,{w}^{\prime }),\end{equation} \tag{ E.16 }$$

where ${\Vert}\cdot {{\Vert}}_{{\psi }_{2}}$ denotes the sub-Gaussian norm [Ver19].

We verify that ${\left\{{\mathcal{G}}_{n}(w)\right\}}_{w}$ has sub-Gaussian increments with $K=2L/\sqrt{n}$ and for the metric being the standard Euclidean metric, d(w, w') = ||w − w'||. To see why this is the case, notice that

$$\begin{equation*}{\mathcal{G}}_{n}(w)-{\mathcal{G}}_{n}({w}^{\prime })=\frac{1}{n}\sum\limits _{i=1}^{n}\left[\ell (w,{z}_{i})-\ell ({w}^{\prime },{z}_{i})-\left({\mathbb{E}}_{z}\ell (w,z)-{\mathbb{E}}_{z}\ell ({w}^{\prime },z)\right)\right],\end{equation*}$$

which is a sum of i.i.d. random variables that are uniformly bounded by

$$\begin{equation*}\left\vert \ell (w,{z}_{i})-\ell ({w}^{\prime },{z}_{i})-\left({\mathbb{E}}_{z}\ell (w,z)-{\mathbb{E}}_{z}\ell ({w}^{\prime },z)\right)\right\vert \leqslant 2L{\Vert}w-{w}^{\prime }{\Vert},\end{equation*}$$

by the Lipschitz continuity of the loss. Therefore, Hoeffding's lemma for bounded and centered random variables easily imply that

$$\begin{equation}\mathbb{E}\left\{\mathrm{exp}\left[\lambda \left({\mathcal{G}}_{n}(w)-{\mathcal{G}}_{n}({w}^{\prime })\right)\right]\right\}\leqslant \mathrm{exp}\left[\frac{2{\lambda }^{2}}{n}{L}^{2}{\Vert}w-{w}^{\prime }{{\Vert}}^{2}\right],\end{equation} \tag{ E.17 }$$

thus, we have ${\Vert}{\mathcal{G}}_{n}(w)-{\mathcal{G}}_{n}({w}^{\prime }){{\Vert}}_{{\psi }_{2}}\leqslant (2L/\sqrt{n}){\Vert}w-{w}^{\prime }{\Vert}$.

Next, define the sequence δ_k = 2^−k and notice that we have δ_k ↓ 0. Dudley's tail bound (see for example theorem 8.16 in [Ver19]) for this empirical process implies that, with probability at least 1 − γ, we have

$$\begin{equation}\underset{w,{w}^{\prime }\in \mathcal{W}}{\mathrm{sup}}\left({\mathcal{G}}_{n}(w)-{\mathcal{G}}_{n}({w}^{\prime })\right)\leqslant C\frac{L}{\sqrt{n}}\left[{S}_{\mathcal{W}}+\sqrt{\mathrm{log}(2/\gamma )}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{m}(\mathcal{W})\right],\end{equation} \tag{ E.18 }$$

where C is an absolute constant and

$$\begin{equation*}{S}_{\mathcal{W}}=\sum\limits _{k\in \mathbb{Z}}{\delta }_{k}\sqrt{\mathrm{log}\vert {N}_{{\delta }_{k}}(\mathcal{W})\vert }.\end{equation*}$$

In order to apply Dudley's lemma, we need to bound the above summation. For that, choose κ₀ such that

$$\begin{equation*}{2}^{{\kappa }_{0}}\geqslant \mathrm{d}\mathrm{i}\mathrm{a}\mathrm{m}(\mathcal{W}) > {2}^{{\kappa }_{0}-1},\end{equation*}$$

and any strictly increasing function $\rho :\mathbb{R}\to \mathbb{R}$.

Now since $\bar{{\mathrm{dim}}_{\mathrm{M}}}\mathcal{W}\leqslant {d}_{\mathrm{M}}$, for the sequence ${\left\{{\delta }_{k}\right\}}_{k\in \mathbb{N}}$, and for a sufficiently large n, whenever k ⩾ ⌊ρ(n)⌋, we have

$$\begin{align*} \mathrm{log}\vert {N}_{{\delta }_{k}}(\mathcal{W})\vert & \leqslant 2{d}_{\mathrm{M}}\mathrm{log}({\delta }_{k}^{-1}) \\ & =\mathrm{log}(4){d}_{\mathrm{M}}k. \end{align*}$$

By splitting the entropy sum in Dudley's tail inequality in two terms, we obtain

$$\begin{align*} {S}_{\mathcal{W}}& =\sum\limits _{k\in \mathbb{Z}}{\delta }_{k}\sqrt{\mathrm{log}\vert {N}_{{\delta }_{k}}(\mathcal{W})\vert } \\ & =\sum\limits _{k=-{\kappa }_{0}}^{\lfloor \rho (n)\rfloor }{\delta }_{k}\sqrt{\mathrm{log}\vert {N}_{{\delta }_{k}}(\mathcal{W})\vert }+\sum\limits _{k=\lfloor \rho (n)\rfloor }^{\infty }{\delta }_{k}\sqrt{\mathrm{log}\vert {N}_{{\delta }_{k}}(\mathcal{W})\vert }. \end{align*}$$

For the first term on the right-hand side, we use the monotonicity of covering numbers, i.e. $\vert {N}_{{\delta }_{k}}\vert \leqslant \vert {N}_{{\delta }_{l}}\vert$ for k ⩽ l, and write

$$\begin{align*} \sum\limits _{k=-{\kappa }_{0}}^{\lfloor \rho (n)\rfloor }{\delta }_{k}\sqrt{\mathrm{log}\vert {N}_{{\delta }_{k}}(\mathcal{W})\vert }& \leqslant \sqrt{\mathrm{log}\vert {N}_{{\delta }_{\lfloor \rho (n)\rfloor }}(\mathcal{W})\vert }\sum\limits _{k=-{\kappa }_{0}}^{\lfloor \rho (n)\rfloor }{\delta }_{k} \\ & \leqslant \sqrt{\mathrm{log}(4){d}_{\mathrm{M}}\lfloor \rho (n)\rfloor }\sum\limits _{k=-{\kappa }_{0}}^{\infty }{\delta }_{k} \\ & \leqslant \sqrt{\mathrm{log}(4){d}_{\mathrm{M}}\rho (n)}{2}^{{\kappa }_{0}+1} \\ & \leqslant 4\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{m}(\mathcal{W})\sqrt{\mathrm{log}(4){d}_{\mathrm{M}}\rho (n)}. \end{align*}$$

For the second term on the right-hand side, we have

$$\begin{align*} \sum\limits _{k=\lfloor \rho (n)\rfloor }^{\infty }{\delta }_{k}\sqrt{\mathrm{log}\vert {N}_{{\delta }_{k}}(\mathcal{W})\vert }& \leqslant \sqrt{\mathrm{log}(4){d}_{\mathrm{M}}}\sum\limits _{k=\lfloor \rho (n)\rfloor }^{\infty }\sqrt{k}{\delta }_{k} \\ & \leqslant \sqrt{\mathrm{log}(4){d}_{\mathrm{M}}}\sum\limits _{k=0}^{\infty }k{\delta }_{k} \\ & =2\sqrt{\mathrm{log}(4){d}_{\mathrm{M}}}. \end{align*}$$

Combining these, we obtain

$$\begin{equation*}{S}_{\mathcal{W}}\leqslant 2\sqrt{\mathrm{log}(4){d}_{\mathrm{M}}}\left\{1+2\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{m}(\mathcal{W})\sqrt{\rho (n)}\right\}.\end{equation*}$$

Plugging this bound back in Dudley's tail bound (E.18), we obtain

$$\begin{equation*}\underset{w,{w}^{\prime }\in \mathcal{W}}{\mathrm{sup}}\left({\mathcal{G}}_{n}(w)-{\mathcal{G}}_{n}({w}^{\prime })\right)\leqslant CL\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{m}(\mathcal{W})\frac{\sqrt{{d}_{\mathrm{M}}\rho (n)}+\sqrt{\mathrm{log}(2/\gamma )}}{\sqrt{n}}.\end{equation*}$$

Now fix ${w}_{0}\in \mathcal{W}$ and write the triangle inequality,

$$\begin{equation*}\underset{w\in \mathcal{W}}{\mathrm{sup}}{\mathcal{G}}_{n}(w)\leqslant \underset{w,{w}^{\prime }\in \mathcal{W}}{\mathrm{sup}}\left({\mathcal{G}}_{n}(w)-{\mathcal{G}}_{n}({w}^{\prime })\right)+{\mathcal{G}}_{n}({w}_{0}).\end{equation*}$$

Clearly for a fixed ${w}_{0}\in \mathcal{W}$, we can apply Hoeffding's inequality and obtain that, with probability at least 1 − γ,

$$\begin{equation*}{\mathcal{G}}_{n}({w}_{0})\leqslant B\sqrt{\frac{\mathrm{log}(2/\gamma )}{n}}.\end{equation*}$$

Combining this with the previous result, we have with probability at least 1 − 2γ

$$\begin{equation*}\underset{w\in \mathcal{W}}{\mathrm{sup}}{\mathcal{G}}_{n}(w)\leqslant CL\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{m}(\mathcal{W})\frac{\sqrt{{d}_{\mathrm{M}}}\sqrt{{K}_{{d}_{\mathrm{M}}}}+\sqrt{\mathrm{log}(2/\gamma )}}{\sqrt{n}}+B\sqrt{\frac{\mathrm{log}(2/\gamma )}{n}}.\end{equation*}$$

Finally replacing and γ with γ/2 and collecting the absolute constants in c, we conclude the proof. □

Proof of theorem  2.The proof follows the same lines of the proof of theorem 1, except that we invoke lemma 2 instead of lemma 1. □

Proof. We start by restating our theoretical framework in an equivalent way for mathematical convenience. In particular, consider the (countable) product measure ${\mu }_{z}^{\infty }={\mu }_{z}\otimes {\mu }_{z}\otimes \dots \enspace$ defined on the cylindrical sigma-algebra. Accordingly, denote $\mathbf{S}\sim {\mu }_{z}^{\infty }$ as an infinite sequence of i.i.d. random vectors, i.e. $\mathbf{S}={({z}_{j})}_{j\geqslant 1}$ with z_j ∼ _i.i.d. μ_z for all j = 1, 2, .... Furthermore, let S_n := (z₁, ..., z_n ) be the first n elements of S. In this notation, we have $S\stackrel{\mathrm{d}}{=}{\mathbf{S}}_{n}$ and ${\mathcal{W}}_{S}\stackrel{\mathrm{d}}{=}{\mathcal{W}}_{{\mathbf{S}}_{n}}$, where $\stackrel{\mathrm{d}}{=}$ denotes equality in distribution. Similarly, we have ${\hat{\mathcal{R}}}_{n}(w)=\hat{\mathcal{R}}(w,{\mathbf{S}}_{n})$.

Due to the hypotheses and theorem 5, we have $\bar{{\mathrm{dim}}_{\mathrm{M}}}{\mathcal{W}}_{{\mathbf{S}}_{n}}={\mathrm{dim}}_{\mathrm{H}}{\mathcal{W}}_{{\mathbf{S}}_{n}}=:{d}_{\mathrm{H}}(\mathbf{S},n)$, μ_u -almost surely. It is easy to verify that the particular forms of the δ-covers and N_δ in H 5 still yield the same Minkowski dimension in (D.1). Then by definition, we have for all S and n:

$$\begin{equation}\underset{\delta \to 0}{\mathrm{limsup}}\enspace \frac{\mathrm{log}\vert {N}_{\delta }({\mathcal{W}}_{{\mathbf{S}}_{n}})\vert }{\mathrm{log}(1/\delta )}=\underset{\delta \to 0}{\mathrm{lim}}\underset{r< \delta }{\mathrm{sup}}\enspace \frac{\mathrm{log}\vert {N}_{r}({\mathcal{W}}_{{\mathbf{S}}_{n}})\vert }{\mathrm{log}(1/r)}={d}_{\mathrm{H}}(\mathbf{S},n),\end{equation} \tag{ E.19 }$$

μ_u -almost surely. Hence for each n

$$\begin{equation}{f}_{\delta }^{n}(\mathbf{S}){:=}\underset{\mathbb{Q}\ni r< \delta }{\mathrm{sup}}\frac{\mathrm{log}\vert {N}_{r}({\mathcal{W}}_{{\mathbf{S}}_{n}})\vert }{\mathrm{log}(1/r)}\to {d}_{\mathrm{H}}(\mathbf{S},n),\end{equation} \tag{ E.20 }$$

as δ → 0 almost surely, or alternatively, for each n, there exists a set Ω_n of full measure such that

$$\begin{equation}\underset{\delta \to 0}{\mathrm{lim}}\enspace {f}_{\delta }^{n}(\mathbf{S})=\underset{\delta \to 0}{\mathrm{lim}}\underset{\mathbb{Q}\ni r< \delta }{\mathrm{sup}}\enspace \frac{\mathrm{log}\vert {N}_{r}({\mathcal{W}}_{{\mathbf{S}}_{n}})\vert }{\mathrm{log}(1/r)}\to {d}_{\mathrm{H}}(\mathbf{S},n),\end{equation} \tag{ E.21 }$$

for all S ∈ Ω_n . Let Ω* := ∩_n Ω_n . Then for S ∈ Ω* we have that for all n

$$\begin{equation}\underset{\delta \to 0}{\mathrm{lim}}\enspace {f}_{\delta }^{n}(\mathbf{S})=\underset{\delta \to 0}{\mathrm{lim}}\underset{\mathbb{Q}\ni r< \delta }{\mathrm{sup}}\enspace \frac{\mathrm{log}\vert {N}_{r}({\mathcal{W}}_{{\mathbf{S}}_{n}})\vert }{\mathrm{log}(1/r)}\to {d}_{\mathrm{H}}(\mathbf{S},n),\end{equation} \tag{ E.22 }$$

and therefore, on this set we also have

$$\begin{equation*}\underset{\delta \to 0}{\mathrm{lim}}\underset{n}{\mathrm{sup}}\enspace \frac{1}{{\alpha }_{n}}\mathrm{min}\left\{1,\vert {f}_{\delta }^{n}(\mathbf{S})-{d}_{\mathrm{H}}(\mathbf{S},n)\vert \right\}\to 0,\end{equation*}$$

where α_n is a monotone increasing sequence such that α_n ⩾ 1 and α_n → ∞. To see why, suppose that we are given a collection of functions ${\left\{{g}_{n}(r)\right\}}_{n}$ where r > 0, such that lim_r→0  g_n (r) → 0 for each n. We then have that the infinite dimensional vector ${({g}_{n}(r))}_{n}\to (0,0,\dots \enspace )$ in the product topology on ${\mathbb{R}}^{\infty }$. We can metrize the product topology on ${\mathbb{R}}^{\infty }$ using the metric

$$\begin{equation*}\bar{d}(\mathbf{x},\mathbf{y})=\sum\limits _{n}\frac{1}{{\alpha }_{n}}\enspace \mathrm{min}\left\{1,\vert {x}_{n}-{y}_{n}\vert \right\},\end{equation*}$$

where $\mathbf{x}={({x}_{n})}_{n\geqslant 1}$, $\mathbf{y}={({y}_{n})}_{n\geqslant 1}$, and 1 ⩽ α_n → ∞ is monotone increasing. Alternatively, notice that if lim_r→0  g_n (r) → 0 for all n, for any > 0 choose N₀ such that for n ⩾ N₀, 1/α_n < , and choose r₀ such that for r < r₀, we have ${\mathrm{max}}_{n\leqslant {N}_{0}}\vert {g}_{n}(r)\vert < {\epsilon}$. Then for r < r₀ we have

$$\begin{align*} \underset{n}{\mathrm{sup}}\frac{1}{{\alpha }_{n}}\mathrm{min}\left\{1,\vert {g}_{n}(r)\vert \right\}& \leqslant \mathrm{max}\left\{\underset{n\leqslant {N}_{0}}{\mathrm{sup}}\frac{1}{{\alpha }_{n}}\enspace \mathrm{min}\left\{1,\vert {g}_{n}(r)\vert \right\},\underset{n > {N}_{0}}{\mathrm{sup}}\frac{1}{{\alpha }_{n}}\enspace \mathrm{min}\left\{1,\vert {g}_{n}(r)\vert \right\}\right\} \\ & \leqslant \mathrm{max}\left\{\frac{{\epsilon}}{{\alpha }_{n}},\underset{n > {N}_{0}}{\mathrm{sup}}\frac{1}{{\alpha }_{n}}\right\}\leqslant {\epsilon}, \end{align*}$$

where we used that α_n ⩾ 1.

Applying the above reasoning we have that for all S ∈ Ω*

$$\begin{equation*}{F}_{\delta }(\mathbf{S}){:=}\underset{n}{\mathrm{sup}}\frac{1}{{\alpha }_{n}}\enspace \mathrm{min}\left\{1,\vert {f}_{\delta }^{n}(\mathbf{S})-{d}_{\mathrm{H}}(\mathbf{S},n)\vert \right\}\to 0,\end{equation*}$$

as δ → 0. By applying theorem 6 to the collection of random variables {F_δ (S); δ}, for any δ' > 0 we can find a subset $\mathfrak{Z}\subset {\mathcal{Z}}^{\infty }$, with probability at least 1 − δ' under ${\mu }_{z}^{\infty }$, such that on $\mathfrak{Z}$ the convergence is uniform, that is

$$\begin{equation*}\underset{\mathbf{S}\in \mathfrak{Z}}{\mathrm{sup}}\underset{n}{\mathrm{sup}}\frac{1}{{\alpha }_{n}}\enspace \mathrm{min}\left\{1,\vert {f}_{r}^{n}(\mathbf{S})-{d}_{\mathrm{H}}(\mathbf{S},n)\vert \right\}\leqslant c(r),\end{equation*}$$

where c(r) → 0 as r → 0. Notice that c(r) = c(δ', r), that is it depends on the choice of δ' (for any δ', we have lim_r→0  c(r; δ') = 0).

As U and S are assumed to be independent, all the following statements hold μ_u -almost surely, hence we drop the dependence on U to ease the notation. We proceed as in the proof of lemma 1:

$$\begin{equation}\underset{w\in {\mathcal{W}}_{{\mathbf{S}}_{n}}}{\mathrm{sup}}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \leqslant \underset{w\in {N}_{\delta }^{{\mathbf{S}}_{n}}}{\mathrm{max}}\left\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\right\vert +2L\delta .\end{equation} \tag{ E.23 }$$

Notice that on $\mathfrak{Z}$ we have that

$$\begin{equation*}\underset{\mathbf{S}\in \mathfrak{Z}}{\mathrm{sup}}\underset{n}{\mathrm{sup}}\frac{1}{{\alpha }_{n}}\enspace \mathrm{min}\left\{1,\vert {f}_{r}^{n}(\mathbf{S})-{d}_{\mathrm{H}}(\mathbf{S},n)\vert \right\}\leqslant c(r),\quad \text{for}\enspace \text{all}\enspace \enspace \enspace r > 0,\end{equation*}$$

so in particular for any sequence {δ_n ; n ⩾ 0} we have that

$$\begin{equation*}\left\{\mathbf{S}\in \mathfrak{Z}\right\}\subseteq \bigcap\limits _{n}\left\{\frac{1}{{\alpha }_{n}}\enspace \mathrm{min}\left\{1,\vert {f}_{{\delta }_{n}}^{n}(\mathbf{S})-{d}_{\mathrm{H}}(\mathbf{S},n)\vert \right\}\leqslant c({\delta }_{n})\right\},\end{equation*}$$

or

$$\begin{equation*}\left\{\mathbf{S}\in \mathfrak{Z}\right\}\subseteq \bigcap\limits _{n}\left\{\left\vert {N}_{{\delta }_{n}}\left({\mathcal{W}}_{{\mathbf{S}}_{n}}\right)\right\vert \leqslant {(1/{\delta }_{n})}^{{d}_{\mathrm{H}}(\mathbf{S},n)+{\alpha }_{n}c({\delta }_{n})}\right\}.\end{equation*}$$

Let ${({\delta }_{n})}_{n\geqslant 0}$ be a decreasing sequence such that ${\delta }_{n}\in \mathbb{Q}$ for all n and δ_n → 0. We then have

$$\begin{align*} \mathbb{P}& \left(\underset{w\in {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})}{\mathrm{max}}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \geqslant \varepsilon \right) \\ & \leqslant \mathbb{P}\left(\left\{\mathbf{S}\in \mathfrak{Z}\right\}\cap \left\{\underset{w\in {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})}{\mathrm{max}}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \geqslant \varepsilon \right\}\right)+{\delta }^{\prime }. \end{align*}$$

For ρ > 0 and $k\in {\mathbb{N}}_{+}$ let us define ${J}_{k}(\rho ){:=}\left(\right.k\rho ,(k+1)\rho$] and set ρ_n := log(1/δ_n ). Furthermore, for any t > 0 define

$$\begin{equation*}\varepsilon (t){:=}\sqrt{\frac{{B}^{2}}{2n}\left[\mathrm{log}(1/{\delta }_{n})\left(t+{\alpha }_{n}c({\delta }^{\prime },{\delta }_{n})\right)+\mathrm{log}(M/{\delta }^{\prime })\right]}.\end{equation*}$$

Notice that ɛ(t) is increasing in t. Therefore, we have

$$\begin{align*} & \mathbb{P}\left(\underset{w\in {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})}{\mathrm{max}}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \geqslant \varepsilon ({d}_{\mathrm{H}}(\mathbf{S},n))\right) \\ & \hspace{25.0pt}\leqslant {\delta }^{\prime }+\mathbb{P}\left(\left\{\left\vert {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\right\vert \leqslant {\left(\frac{1}{{\delta }_{n}}\right)}^{{d}_{\mathrm{H}}(\mathbf{S},n)+{\alpha }_{n}c({\delta }_{n})}\right\}\right. \\ & \hspace{25.0pt}\quad \left.\cap \left\{\underset{w\in {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})}{\mathrm{max}}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \geqslant \varepsilon ({d}_{\mathrm{H}}(\mathbf{S},n))\right\}\right) \\ & \hspace{25.0pt}={\delta }^{\prime }+\sum\limits _{k=0}^{\lceil \frac{d}{{\rho }_{n}}\rceil }\mathbb{P}\left(\left\{\left\vert {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\right\vert \leqslant {\left(\frac{1}{{\delta }_{n}}\right)}^{{d}_{\mathrm{H}}(\mathbf{S},n)+{\alpha }_{n}c({\delta }_{n})}\right\}\right. \\ & \hspace{25.0pt}\quad \left.\cap \left\{\underset{w\in {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})}{\mathrm{max}}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \geqslant \varepsilon ({d}_{\mathrm{H}}(\mathbf{S},n))\right\}\cap \left\{{d}_{\mathrm{H}}(\mathbf{S},n)\in {J}_{k}({\rho }_{n})\right\}\right) \\ & \hspace{25.0pt}={\delta }^{\prime }+\sum\limits _{k=0}^{\lceil \frac{d}{{\rho }_{n}}\rceil }\mathbb{P}\left(\left\{\left\vert {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\right\vert \leqslant {\left(\frac{1}{{\delta }_{n}}\right)}^{{d}_{\mathrm{H}}(\mathbf{S},n)+{\alpha }_{n}c({\delta }_{n})}\right\}\cap \left\{{d}_{\mathrm{H}}(\mathbf{S},n)\in {J}_{k}({\rho }_{n})\right\}\right. \\ & \hspace{25.0pt}\quad \left.\cap \bigcup\limits _{w\in N({\delta }_{n})}\left(\left\{w\in {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\right\}\cap \left\{\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \geqslant \varepsilon ({d}_{\mathrm{H}}(\mathbf{S},n))\right\}\right)\right) \\ & \hspace{25.0pt}\leqslant {\delta }^{\prime }+\sum\limits _{k=0}^{\lceil \frac{d}{{\rho }_{n}}\rceil }\sum\limits _{w\in {N}_{{\delta }_{n}}}\mathbb{P}\left(\left\{\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \geqslant \varepsilon (k{\rho }_{n})\right\}\right. \\ & \hspace{25.0pt}\quad \left.\cap \left\{w\in {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\right\}\cap \left\{\left\vert {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\right\vert \leqslant {\left(\frac{1}{{\delta }_{n}}\right)}^{{d}_{\mathrm{H}}(\mathbf{S},n)+{\alpha }_{n}c({\delta }_{n})}\right\}\cap \left\{{d}_{\mathrm{H}}(\mathbf{S},n)\in {J}_{k}({\rho }_{n})\right\}\right), \end{align*}$$

where we used the fact d_H(S, n) ⩽ d almost surely, and that on the event d_H(S, n) ∈ J_k (ρ_n ), ɛ(d_H(S, n)) ⩾ ɛ(kρ_n ).

Notice that the events

$$\begin{equation*}\left\{w\in {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\right\},\left\{\vert {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\vert \leqslant {(1/{\delta }_{n})}^{{d}_{\mathrm{H}}(\mathbf{S},n)+{\alpha }_{n}c({\delta }_{n})}\right\},\left\{{d}_{\mathrm{H}}(\mathbf{S},n)\in {J}_{k}({\rho }_{n})\right\},\end{equation*}$$

are in $\mathfrak{G}$. To see why, notice first that for any $0< \delta \in \mathbb{Q}$

$$\begin{equation*}\vert {N}_{\delta }({\mathcal{W}}_{{\mathbf{S}}_{n}})\vert =\sum\limits _{w\in {N}_{\delta }}\mathbb{1}\left\{w\in {N}_{\delta }({\mathcal{W}}_{{\mathbf{S}}_{n}})\right\},\end{equation*}$$

so $\vert {N}_{\delta }({\mathcal{W}}_{{\mathbf{S}}_{n}})\vert$ is $\mathfrak{G}$-measurable as a finite sum of $\mathfrak{G}$-measurable variables. From (E.20) it can also be seen that d_H(S, n) is also $\mathfrak{G}$-measurable as a countable supremum of $\mathfrak{G}$-measurable random variables. On the other hand, the event $\left\{\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \geqslant \varepsilon (k{\rho }_{n})\right\}$ is clearly in $\mathfrak{F}$ (see H 5 for definitions).

Therefore,

$$\begin{align*} & \mathbb{P}\left.\left(\underset{w\in {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})}{\mathrm{max}}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \geqslant \varepsilon ({d}_{\mathrm{H}}(\mathbf{S},n))\right)\right) \\ & \hspace{25.0pt}\leqslant {\delta }^{\prime }+M\sum\limits _{k=0}^{\lceil \frac{d}{{\rho }_{n}}\rceil }\sum\limits _{w\in {N}_{{\delta }_{n}}}\mathbb{P}\left(\left\{\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \geqslant \varepsilon (k{\rho }_{n})\right\}\right) \\ & \hspace{25.0pt}\quad \times \mathbb{P}\left(\left\{w\in {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\right\}\cap \left\{\left\vert {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\right\vert \leqslant {\left(\frac{1}{{\delta }_{n}}\right)}^{{d}_{\mathrm{H}}(\mathbf{S},n)+{\alpha }_{n}c({\delta }_{n})}\right\}\right. \\ & \hspace{25.0pt}\quad \left.\cap \left\{{d}_{\mathrm{H}}(\mathbf{S},n)\in {J}_{k}({\rho }_{n})\right\}\right), \\ & \hspace{25.0pt}\leqslant {\delta }^{\prime }+2M\sum\limits _{k=0}^{\lceil \frac{d}{{\rho }_{n}}\rceil }{e}^{-\frac{2n{\varepsilon }^{2}(k{\rho }_{n})}{{B}^{2}}}\sum\limits _{w\in {N}_{{\delta }_{n}}}\mathbb{P}\left(\left\{w\in {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\right\}\right. \\ & \hspace{25.0pt}\quad \left.\cap \left\{\left\vert {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\right\vert \leqslant {\left(\frac{1}{{\delta }_{n}}\right)}^{{d}_{\mathrm{H}}(\mathbf{S},n)+{\alpha }_{n}c({\delta }_{n})}\right\}\right. \\ & \hspace{25.0pt}\left.\quad \cap \left\{{d}_{\mathrm{H}}(\mathbf{S},n)\in {J}_{k}({\rho }_{n})\right\}\right) \\ & \hspace{25.0pt}\leqslant {\delta }^{\prime }+2M\sum\limits _{k=0}^{\lceil \frac{d}{{\rho }_{n}}\rceil }{e}^{-\frac{2n{\varepsilon }^{2}(k{\rho }_{n})}{{B}^{2}}}\sum\limits _{w\in {N}_{{\delta }_{n}}}\mathbb{E}\left(\mathbb{1}\left\{w\in {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\right\}\right. \\ & \hspace{25.0pt}\quad \left.\times \mathbb{1}\left\{\left\vert {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\right\vert \leqslant {\left(\frac{1}{{\delta }_{n}}\right)}^{{d}_{\mathrm{H}}(\mathbf{S},n)+{\alpha }_{n}c({\delta }_{n})}\right\}\times \mathbb{1}\left\{{d}_{\mathrm{H}}(\mathbf{S},n)\in {J}_{k}({\rho }_{n})\right\}\right) \\ & \hspace{25.0pt}\leqslant {\delta }^{\prime }+2M\sum\limits _{k=0}^{\lceil \frac{d}{{\rho }_{n}}\rceil }{e}^{-\frac{2n{\varepsilon }^{2}(k{\rho }_{n})}{{B}^{2}}}\mathbb{E}\left(\sum\limits _{w\in {N}_{{\delta }_{n}}}\mathbb{1}\left\{w\in {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\right\}\right. \\ & \hspace{25.0pt}\quad \left.\times \mathbb{1}\left\{\left\vert {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\right\vert \leqslant {\left(\frac{1}{{\delta }_{n}}\right)}^{{d}_{\mathrm{H}}(\mathbf{S},n)+{\alpha }_{n}c({\delta }_{n})}\right\}\times \mathbb{1}\left\{{d}_{\mathrm{H}}(\mathbf{S},n)\in {J}_{k}({\rho }_{n})\right\}\right), \end{align*}$$

$$\begin{align*} & \leqslant {\delta }^{\prime }+2M\sum\limits _{k=0}^{\lceil \frac{d}{{\rho }_{n}}\rceil }{e}^{-\frac{2n{\varepsilon }^{2}(k{\rho }_{n})}{{B}^{2}}}\mathbb{E}\left(\vert {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\vert \times \mathbb{1}\left\{\left\vert {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})\right\vert \leqslant {\left(\frac{1}{{\delta }_{n}}\right)}^{{d}_{\mathrm{H}}(\mathbf{S},n)+{\alpha }_{n}c({\delta }_{n})}\right\}\right. \\ & \quad \left.\times \mathbb{1}\left\{{d}_{\mathrm{H}}(\mathbf{S},n)\in {J}_{k}({\rho }_{n})\right\}\right) \\ & ={\delta }^{\prime }+2M\sum\limits _{k=0}^{\lceil \frac{d}{{\rho }_{n}}\rceil }{e}^{-\frac{2n{\varepsilon }^{2}(k{\rho }_{n})}{{B}^{2}}}\mathbb{E}\left({\left[\frac{1}{{\delta }_{n}}\right]}^{{d}_{\mathrm{H}}(\mathbf{S},n)+{\alpha }_{n}c({\delta }_{n})}\times \mathbb{1}\left\{{d}_{\mathrm{H}}(\mathbf{S},n)\in {J}_{k}({\rho }_{n})\right\}\right). \end{align*}$$

Now, notice that the mapping t ↦ ɛ²(t) is linear, with Lipschitz coefficient

$$\begin{equation*}\frac{\mathrm{d}{\varepsilon }^{2}(t)}{\mathrm{d}t}=\frac{{B}^{2}}{2n}\enspace \mathrm{log}(1/{\delta }_{n}).\end{equation*}$$

Therefore, on the event {d_H(S, n) ∈ J_k (ρ_n )} we have

$$\begin{equation}{\varepsilon }^{2}({d}_{\mathrm{H}}(\mathbf{S},n))-{\varepsilon }^{2}(k{\rho }_{n})\leqslant ({d}_{\mathrm{H}}(\mathbf{S},n)-k{\rho }_{n})\frac{{B}^{2}}{2n}\enspace \mathrm{log}(1/{\delta }_{n})\end{equation} \tag{ E.24 }$$

$$\begin{equation}\leqslant {\rho }_{n}\frac{{B}^{2}}{2n}\enspace \mathrm{log}(1/{\delta }_{n}).\end{equation} \tag{ E.25 }$$

Hence,

$$\begin{equation*}{\varepsilon }^{2}(k{\rho }_{n})\geqslant {\varepsilon }^{2}\left({d}_{\mathrm{H}}(\mathbf{S},n)\right)-\frac{{B}^{2}}{2n},\end{equation*}$$

where we used the fact that ρ_n = log(1/δ_n ). Therefore, we have

$$\begin{align*} & \mathbb{P}\left(\underset{w\in {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})}{\mathrm{max}}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \geqslant \varepsilon ({d}_{\mathrm{H}}(\mathbf{S},n))\right) \\ & \hspace{25.0pt}\leqslant {\delta }^{\prime }+2M\mathbb{E}\left(\sum\limits _{k=0}^{\lceil \frac{d}{{\rho }_{n}}\rceil }{e}^{-\frac{2n{\varepsilon }^{2}(k{\rho }_{n})}{{B}^{2}}}{\left[\frac{1}{{\delta }_{n}}\right]}^{{d}_{\mathrm{H}}(\mathbf{S},n)+{\alpha }_{n}c({\delta }_{n})}\times \mathbb{1}\left\{{d}_{\mathrm{H}}(\mathbf{S},n)\in {J}_{k}({\rho }_{n})\right\}\right) \\ & \hspace{25.0pt}\leqslant {\delta }^{\prime }+2M\mathbb{E}\left(\sum\limits _{k=0}^{\lceil \frac{d}{{\rho }_{n}}\rceil }{e}^{-\frac{2n{\varepsilon }^{2}({d}_{\mathrm{H}}(\mathbf{S},n))}{{B}^{2}}+1}{\left[\frac{1}{{\delta }_{n}}\right]}^{{d}_{\mathrm{H}}(\mathbf{S},n)+{\alpha }_{n}c({\delta }_{n})}\times \mathbb{1}\left\{{d}_{\mathrm{H}}(\mathbf{S},n)\in {J}_{k}({\rho }_{n})\right\}\right) \\ & \hspace{25.0pt}={\delta }^{\prime }+2M\mathbb{E}\left({e}^{-\frac{2n{\varepsilon }^{2}({d}_{\mathrm{H}}(\mathbf{S},n))}{{B}^{2}}+1}{\left[\frac{1}{{\delta }_{n}}\right]}^{{d}_{\mathrm{H}}(\mathbf{S},n)+{\alpha }_{n}c({\delta }_{n})}\right). \end{align*}$$

By the definition of ɛ(t), for any S and n, we have that:

$$\begin{equation*}2M{e}^{-\frac{2n{\varepsilon }^{2}({d}_{\mathrm{H}}(\mathbf{S},n))}{{B}^{2}}+1}{\left[\frac{1}{{\delta }_{n}}\right]}^{{d}_{\mathrm{H}}(\mathbf{S},n)+{\alpha }_{n}c({\delta }_{n})}=2e{\delta }^{\prime }.\end{equation*}$$

Therefore,

$$\begin{equation*}\mathbb{P}\left(\underset{w\in {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})}{\mathrm{max}}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \geqslant \varepsilon ({d}_{\mathrm{H}}(\mathbf{S},n))\right)\leqslant (1+2e){\delta }^{\prime }.\end{equation*}$$

That is, with probability at least 1 − (1 + 2e)δ' we have:

$$\begin{align*} \underset{w\in {N}_{{\delta }_{n}}({\mathcal{W}}_{{\mathbf{S}}_{n}})}{\mathrm{max}}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert & \leqslant \varepsilon ({d}_{\mathrm{H}}(\mathbf{S},n)) \\ & =\sqrt{\frac{{B}^{2}}{2n}\left[\mathrm{log}(\sqrt{n}L)\left(t+{\alpha }_{n}c({\delta }^{\prime },{\delta }_{n})\right)+\mathrm{log}(M/{\delta }^{\prime })\right]}. \end{align*}$$

Choosing ${\delta }_{n}=1/\sqrt{n{L}^{2}}$ and α_n = log(n), for each δ' > 0, with probability at least 1 − (1 + 2e)δ' we have

$$\begin{equation}\underset{w\in {\mathcal{W}}_{{\mathbf{S}}_{n}}}{\mathrm{sup}}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \leqslant B\sqrt{\frac{\mathrm{log}(\sqrt{n}L)\left[{d}_{\mathrm{H}}(\mathbf{S},n)+c({\delta }^{\prime },{\delta }_{n})\mathrm{log}\enspace n\right]+\mathrm{log}\left(\frac{M}{{\delta }^{\prime }}\right)}{2n}}+\frac{2}{\sqrt{n}},\end{equation} \tag{ E.26 }$$

where for each δ' > 0, we have lim_n→∞  c(δ', δ_n ) = 0. Hence, for sufficiently large n, we have with probability at least 1 − (1 + 2e)δ' we have:

$$\begin{equation}\underset{w\in {\mathcal{W}}_{{\mathbf{S}}_{n}}}{\mathrm{sup}}\vert {\hat{\mathcal{R}}}_{n}(w)-\mathcal{R}(w)\vert \leqslant 2B\sqrt{\frac{[{d}_{\mathrm{H}}(\mathbf{S},n)+1]{\mathrm{log}}^{2}(n{L}^{2})+\mathrm{log}\left(\frac{M}{{\delta }^{\prime }}\right)}{n}}.\end{equation} \tag{ E.27 }$$

Setting γ := (1 + 2e)δ' and using 1 + 2e < 7, we obtain the desired result. This concludes the proof. □