The reusability prior: comparing deep learning models without training

In figure 1, there is an informal description of context. We define the set of all contexts for a given parameter as follows: Definition 1.

definition Let f_iw be a function node that at least takes a given parameter node w and an associated input h_i ; let $\mathcal{O}_k$ be any node from the set of all output or leaf nodes $\mathcal{L}$ in graph $\mathcal{G}$, and p a path that connects f_iw to $\mathcal{O}_k$. The set of all contexts for w, $\mathcal{C}_{w}$ is given by

$$\begin{equation} \mathcal{C}_{w}=\bigcup\limits_{f_{iw} \in \mathcal{G}}\{\,p \mid f_{iw}((h_i, w), \ldots) \begin{array}{cl}p\\\leadsto\end{array} \mathcal{O}_k \quad \forall \mathcal{O}_k \in \mathcal{L}\}. \end{equation} \tag{ 1 }$$

Zoom In Zoom Out Reset image size

Note that f_iw can be any function node within the model graph $\mathcal{G}$ that at least takes parameter w and some hidden or input feature h_i . Parameter w can be used in multiple places within the model with separate inputs leading to different contexts (i.e. $f_{jw}(h_j, w, \ldots)$). In other words, $\mathcal{C}_w$ is the union of all contexts associated with w, regardless of where w is used within the model or whether w is explicitly shared.

An important assumption we make is that f_iw is mostly nonlinear in a similar manner to the mainstream deep learning models. If all function nodes were instead equivalent to a given linear function, the model could be collapsed into a single layer where each input is associated with a single learnable parameter. Such a model can not learn functions more complicated than a linear map (i.e. a single matrix multiplication).

Overall, each parameter has its own set of contexts which is the union of all paths contributing to an output node, where h_i is a part of the computational graph rather than an instance of an actual output feature from a previous layer. We use this definition of context for analyzing model architectures and defining other relevant concepts. In figure 1, we demonstrate some of the possible contexts that can arise inside a model graph. In figure 1(b) the computational graph of a model with three layers is described. How can model design change the number of possible contexts? We provide an example in figure 2, comparing two configurations that yield different numbers of contexts for the same number of parameters.

Zoom In Zoom Out Reset image size

Small models often hit an upper bound early on in terms of performance for large-scale benchmarks, yet parameter sharing is ubiquitous in deep learning. For instance, convolutional kernels [20, 21] repeat spatially (i.e. sliding window), while recurrent modules [22–26] (or any autoregressive model such as [27–29]) repeat temporally. Domain-specific benefits and being able to work with different input or output sizes are fair motivations for parameter sharing; ultimately model capacity is recovered by scaling to larger models at the cost of more floating point operations per second (FLOPS).

An explicit form of parameter sharing is cross-layer parameter sharing which ties the weights of architecturally repeating layers together. The literature shows that, at the cost of more FLOPs, cross-layer parameter sharing can sometimes improve parameter efficiency. Once a model with cross-layer parameter sharing is scaled back to a reasonable capacity, it has a reasonable chance to outperform the original baseline model [30–35]. In section 5.6 we confirm a similar outcome with our examples on some of the EfficientNetv2 models [36].

We claim that all mainstream DL models already share parameters directly or indirectly to various degrees, and this affects their parameter efficiency. In section 4.1, we introduce a simple horizontal unrolling approach that can make parameter sharing explicit for all directed acyclic graphs (DAG).

We propose the generalized notion reusability prior to disentangle the intrinsic design and training choices that affect a model's performance in section 4. A consequence of the reusability prior is that for models of similar size and capacity, designs that maximize the expected number of contexts are more likely to improve parameter efficiency. Our experiments on EfficientNetv2 as well as the literature on cross-layer parameter sharing provide supporting evidence for the reusability prior.

We introduce a simple counting approach for model comparison. By treating relative frequencies derived from the number of all possible contexts per learnable parameter as a probability distribution, we define quantities for the comparison of model graphs, including entropy, expected spread, and total surprisal. We give formal proof that when the total number of contexts is held constant, increasing the expected spread reduces the entropy. We also introduce approaches to estimate performance for directly comparing models without training. We compare analysis and experiment results in tables 3 and 4. We scope our work by only focusing on the model design aspect described in section 2.2.3. The data and training aspects can be taken into account as well when considering the number of possible contexts. We leave this for future work.

In summary, our major contributions are twofold:

• We introduce the reusability prior (section 4.2), and provide a methodology based on graph analysis (section 4.3). To our knowledge, we are the first to introduce a generalized notion of reusability that ties training, design, and data aspects together. For the design aspect, we introduce graph analysis based quantities derived from counting the number of contexts for each learnable parameter ¹ . Overall, the quantities we introduce allow us to compare arbitrary DAGs or model architectures without relying on any training. Our graph analysis based quantities aligned well with the empirical results, including at least two important edge cases in section 5 in tables 3 and 4.
• We empirically confirm that it is possible to achieve higher parameter efficiency by aggressively sharing parameters in EfficientNetv2 models. In our experiments, even though the numbers of parameters were at least 60% larger for the original baseline EfficientNetv2-b0 models, we observed that EfficientNetv2-S models with aggressive parameter sharing consistently outperformed the baselines on CIFAR-10 [37], CIFAR-100 [38], and Imagenet-1K [39] image classification benchmarks.

SharesNet [30] demonstrates that for wide residual networks (WRNs) [94], it is possible to surpass the original model's performance with parameter sharing and then scaling (i.e. increasing depth and/or width). Similar to lookup based CNNs [95], Savarese and Maire train a model that learns to take a linear combination of a shared pool of kernels [31]. Then the coefficients for the linear combination of kernels are also used for constructing a similarity matrix so that similar layers can be shared. This approach outperforms the original WRN baseline on Imagenet-1K and CIFAR-10 with a similar or less number of parameters. Atom-coefficient decomposed convolution [32], inspired from [96], first decomposes convolutional kernels into bases and coefficients, and then shares the coefficients across layers. This leads to improved parameter efficiency for very deep convolutional nets [42], ResNet, and WRN baselines. Shapeshifter networks used for neural parameter allocation search do not make architectural assumptions such as repeated layers, but instead, learn to reuse parameters from a limited pool by transforming them into weights for any architecture [69].

Aside from the computer vision domain, a lite BERT, bidirectional encoder representations from transformers [33] uses cross-layer parameter sharing to first reduce the number of parameters and then scale up to a capacity comparable to the original model [97]. This surpasses the original model's performance for language understanding tasks. Other relevant examples share attention weights [34], speed up training by parameter sharing and then unsharing [98], and explore sandwich style weight sharing for generative transformers [35].

The usage of implicit models can be considered a continuous form of cross-layer parameter sharing. Well known examples are neural ordinary differential equation solvers [99, 100], variants of deep equilibrium models (DEQ), and multiscale DEQs [101, 102].

Deep learning literature has various strategies for improving performance. Intrinsically, they often seem to maximize the number of contexts for model components. In section 1.1, and figure 1 we provide more details about what is meant by context. Essentially, the number of contexts is affected from data, augmentation, training, regularization, and model design choices.

2.2.1. Diversity in input affecting the number of contexts

2.2.2. Diversifying the roles of model components during training

2.2.3. Design choices impacting the role and scope of each model component

It is possible to horizontally duplicate parameters to remove multiple output edges in a given model graph so that each node has a single output edge. This would reproduce all node outputs from scratch, eliminating indirect parameter sharing (i.e. replacing feature sharing with direct parameter sharing). The resulting unrolled models would be functionally equivalent. We illustrate our point in figure 3. The nodes in earlier layers are duplicated more as they play more diverse roles. We introduce a simple recursive algorithm for horizontal unrolling in appendix A.

Zoom In Zoom Out Reset image size

Horizontal unrolling can convert any mainstream DL model into a form where parameter sharing is always direct, and each parameter is duplicated as many times as their number of contexts that can arise due to the computational graph. To our knowledge, no mainstream DL model exponentially grows in terms of the number of parameters with depth. Hence their unrolled graphs share parameters. This manner of parameter sharing in the unrolled graph uses exponentially fewer parameters than a more general graph exemplified in figure 3(c) that does not share any parameters. We call such graphs a uniform graph and formally define them as follows: Definition 2.

definition Let $\mathcal{G}$ be a model graph. $\mathcal{G}$ is uniform if and only if $|\mathcal{C}_{w_i}| = 1 \,\,\, \forall{w_i} \in \mathcal{G}$, where $|\mathcal{C}_{w_i}|$ is the cardinality of the set of all contexts for parameter w_i .

Note that for the horizontal unrolling to work, any cyclic graph would need to be vertically unrolled into a DAG first. This already happens in practice during training. Additionally, for differentiable approaches that transform a set of learnable parameters to weights, the parameter transformation and the originally shared parameters would need to be included in the graph (i.e. every w_i would be replaced by the DAG that generates it).

There is a connection between parameter efficiency and parameter sharing. By estimating what the number of contexts in the unrolled form would be for each parameter, we quantify expected spread, total surprisal, and entropy for computational graphs of models.

We conjecture that the expected number of contexts for model components is the major reason for differences in model performance. We introduce the reusability prior as follows:

Model components are forced to function in diverse contexts not only due to the training, data, augmentation, and regularization choices but also due to the model design itself. These aspects explicitly or implicitly impact the expected number of contexts for model components. Until model capacity is reached, maximizing this number improves parameter efficiency for models of similar size and capacity. By relying on the repetition of reusable components, a model can learn to describe an approximation of the desired function more efficiently with fewer parameters.

We provide justifications from the literature in section 2, definitions in sections 1.1, and 4.3, finally supporting evidence from our experiments in section 5.

Based on the reusability prior, and the notion of context, we define the expected spread, entropy, and total surprisal. Then we provide two different ways of estimating model performance based on total surprisal and expected spread. Definition 3.

definition For a given model graph $\mathcal{G}$, the expected spread is given by

$$\begin{equation} E[\![\log_{2}|\mathcal{C}|]\!]= \sum_{i=1}^{N_\mathcal{G}} p(w_{i}) \log_{2}|\mathcal{C}_{w_{i}}| \end{equation} \tag{ 2 }$$

where $|\mathcal{C}_{w_i}|$ is the cardinality of the set of all contexts for w_i , $N_\mathcal{G}$ the number of learnable parameters, and $p(w_{i})$ is the relative frequency:

$$\begin{equation} \frac{|\mathcal{C}_{w_{i}}|} {N_\mathcal{C}} \end{equation} \tag{ 3 }$$

where $N_\mathcal{C} = \sum\limits_{w_{j} \in \mathcal{G}} |\mathcal{C}_{w_{j}}|$ the total number of contexts in $\mathcal{G}$.

In section 4.4.1 we prove that the expected spread is equal to the relative entropy or Kullback–Leibler divergence [110] of P(W) from the discrete uniform distribution.

Overall, this quantity prescribes a design maximizing the expected number of contexts ² . Note that $N_\mathcal{C}$ and $N_\mathcal{G}$ grow differently. In 2, similar to $N_\mathcal{C}$, $|\mathcal{C}_{w_i}|$ grows exponentially with depth for the mainstream deep architectures.Definition 4.

definition Let w_i be a parameter in graph $\mathcal{G}$. The entropy of the discrete probability distribution for the parameters of $\mathcal{G}$ based on the number of contexts is given by

$$\begin{equation} H(W) = -\sum_{i=1}^{N_\mathcal{G}} {p(w_i) \log p(w_i)} \end{equation} \tag{ 4 }$$

where $p(w_{i}) = |\mathcal{C}_{w_{i}}|/ N_\mathcal{C}$ is the relative frequency.

Note that $p(w_{i})$ is also used in the calculation of expected spread in 3. Definition 5.

definition Let w_i be a parameter with relative frequency $p(w_{i})$ in graph $\mathcal{G}$. The total surprisal is given by

$$\begin{equation} \mathcal{S}_\mathcal{G} = -\sum_{i=1}^{N_\mathcal{G}} {\log p(w_i)}. \end{equation} \tag{ 5 }$$

Since the multiplicative $p(w_{i})$ term is removed, this is no longer the expected surprisal, i.e.entropy in 4. Consequently, this quantity gives the surprisal of each parameter equal weight ³ .

If we consider parameters that affect smaller portions of the model (i.e. with a smaller spread) more specific or surprising, then for optimal encoding of the horizontally unrolled graph, expected spread would encourage assigning shorter bit lengths for more repeated components. For instance, parameters of the mainstream deep learning models have an exponential distribution where parameters in the earlier layers have exponentially larger numbers of contexts. For large models with the same number of parameters, this results in a much lower entropy compared to a uniform distribution which has the maximum possible entropy $\log_{2}{N_\mathcal{G}}$.

Overall, when other conditions such as model size, capacity, training data etc are similar, the reusability prior encourages increasing the expected spread. This may lead to an improvement in parameter efficiency as the entropy, i.e. the expected bit length, is reduced. In other words, unrolled graphs have smaller description lengths when there is a lot of repetition in the earlier layers: to reuse is to simplify.

4.4.1. Proofs for the connections between the quantities for model comparison

Lemma 1.

lemma Let P(W) be the discrete probability distribution of parameters based on their number of contexts in graph G. Kullback–Leibler (KL) divergence $D_{\mathrm{KL}}(P(W)||P_{U}(W))$ from the discrete uniform distribution $P_{U}(W)$ is equivalent to the expected spread.

Proof.

proof Let the relative frequency of w_i be $p(w_i) = c_i / N_\mathcal{C}$ where $c_i = |\mathcal{C}_{w_{i}}|$ and $N_\mathcal{C} = \sum\limits_{w_{j} \in \mathcal{G}} |\mathcal{C}_{w_{j}}|$. Then the KL divergence of the probability distribution of parameters based on the number of contexts P(W) from uniform distribution $P_U(W)$ where $p_U(w_i) = 1/N_{\mathcal{C}}$ is given by

$$\begin{align*} D_{\mathrm{KL}}(P(W)||P_{U}(W))& = \sum\limits_{i=1}^n p(w_i) \log_{2} (p(w_i) / p_{U}(w_i)) = \sum\limits_{i=1}^n p(w_i) \log_{2} (c_i / N_\mathcal{C} / (1/N_\mathcal{C}))= \sum\limits_{i=1}^n p(w_i) \log_{2}(c_i)\\ &= E[\![\log_{2}|\mathcal{C}|]\!]. \end{align*}$$

Note that, if P(W) is derived from the graph in figure 3(a) by directly counting the frequencies from its unrolled version in figure 3(b), $P_U(W)$ can correspond to the uniform graph in figure 3(c). In general, the cases where $p(w_j) = 0$ and $p_{U}(w_j) = 1/N_\mathcal{C}$ do not change the summation. Theorem 1.

theorem Let $N_\mathcal{C} = \sum\limits_{w_{j} \in \mathcal{G}} |\mathcal{C}_{w_{j}}|$ be the total number of all contexts. $\forall \mathcal{G}_i$ when $N_{\mathcal{C}}$ is held constant, maximizing the expected spread minimizes the entropy.

Note that $N_{\mathcal{C}}$ is equivalent to the number of parameters in G's horizontally unrolled uniform graph as in figures 3(c) and 2. For graphs with identical unrolled uniform graphs, maximizing the expected spread is equivalent to parameter sharing ⁴ . For different architectures, as long as $N_{\mathcal{C}}$ is held constant, the theorem still holds. The formal proof is as follows:Proof.

proof For any discrete probability distribution, it is already known that:

$$\begin{align*} H_{U}(X) = H(X) + D_{\mathrm{KL}}(P(X)||P_{U}(X)) .\end{align*}$$

Therefore, from lemma 1 we show that KL divergence is always:

$$\begin{align*} D_{\mathrm{KL}}(P(W)||P_{U}(W)) = E[\![\log_{2}|\mathcal{C}|]\!] .\end{align*}$$

Thus, we can write:

$$\begin{align*} H_{U}(W) = H_{\mathcal{G}}(W) + D_{\mathrm{KL}}(P(W)||P_{U}(W)) = H_{\mathcal{G}}(W) + E[\![\log_{2}|\mathcal{C}|]\!] = \log_{2}{N_\mathcal{C}}. \end{align*}$$

Hence, when $N_\mathcal{C}$ is constant, reducing the entropy would increase the expected spread and vice versa.

Model performance is majorly associated with the number of parameters and the size of the training data. For CNNs the training image size is relevant as well. Yet it is often unclear how exactly some model designs consistently outperform others when the number of parameters as well as the training data and conditions are similar. A consequence of the reusability prior is that by estimating the relative frequencies of each parameter, it is possible to quantify how the model design itself impacts the entropy, expected spread, and total surprisal. For the scenario where the training data and strategies such as regularization etc are unchanged, we propose using total surprisal to predict model performance, with the assumption that when other conditions are similar, a model with a higher descriptive ability would perform better. We normalize this quantity for different model sizes and multiply it by the input size as follows: Definition 6.

definition Let $\mathcal{S}_\mathcal{G}$ be the total surprisal of graph $\mathcal{G}$, N_I the total number of input nodes and $|\mathcal{G}|$ the summation of the total number of input, output and weight nodes. The estimated performance is given by

$$\begin{equation} \mathcal{P}_\mathcal{G} = \log_{2}\left(\mathcal{S}_\mathcal{G} \frac{N_{I}}{|\mathcal{G}|}\right). \end{equation} \tag{ 6 }$$

Note that the number of weight nodes can be larger than the number of parameters $N_\mathcal{G}$ when explicitly sharing parameters.

4.5.1. An alternative estimation of model performance

As a potential alternative to the total surprisal in (6), expected spread multiplied by the number of learnable parameters $N_\mathcal{G}$ can be used (i.e. replacing $\mathcal{S}_\mathcal{G}$ with $N_\mathcal{G} E[\![\log_{2}|\mathcal{C}| + 1]\!]$). This alternative estimation is given by:

$$\begin{equation} \mathcal{P^{^{\prime}}}_\mathcal{G} = \log_{2}\left(N_\mathcal{G}E[\![\log_{2}|\mathcal{C}| + 1]\!] \frac{N_{I}}{|\mathcal{G}|}\right). \end{equation} \tag{ 7 }$$

Our main justification for using expected spread is given in section 4.2. For models with a comparable number of parameters and model size, models with a higher expected spread (i.e. with the ability to approximate more complicated functions) would likely perform better after training until convergence. We observe this in table 2 for multiple experiments and datasets.

Table 1. Details on CIFAR and Imagenet datasets.

Dataset	Train	Validation	Classes
CIFAR-10 [38]	50 000	10 000	10
CIFAR-100 [38]	50 000	10 000	100
Imagenet-1K [115]	1.28M	50 000	1000

Table 2. EfficientNetV2 [36] trained from scratch on CIFAR-10, CIFAR-100, and Imagenet-1K. At the cost of more FLOPs, V2-S-shared models that aggressively apply cross-layer parameter sharing (bold) achieve better top-1 accuracy scores than V2-B0 models that have at least 60% more parameters.

Model	Dataset	Params	Score	Augm.	Batch	Epoch	FLOPs
V2-B0-shared	CIFAR-10	1.7 M	95.3	None	512	300	0.7B
V2-B0-original	CIFAR-10	5.9 M	95.6	None	512	300	0.7B
V2-S-shared	CIFAR-10	3.1 M	96.0	None	512	300	8.8B
V2-B0-shared	CIFAR-100	1.8 M	78.8	AutoAug [116]	512	300	0.7B
V2-B0-original	CIFAR-100	5.9 M	80.6	AutoAug	512	300	0.7B
V2-S-shared	CIFAR-100	3.2 M	81.5	AutoAug	510	300	8.8B
V2-B0-shared	Imagenet-1K	3.0 M	73.7	RandAug [117]	400	352	0.7B
V2-B0-original	Imagenet-1K	7.1 M	76.9	RandAug	400	352	0.7B
V2-S-shared	Imagenet-1K	4.4 M	78.3	RandAug	400	352	8.8B

Overall, by relying on our graph-based analysis and counting approach, we proposed two crude ways to estimate the performance of models without any training ⁵ . In practice, both have different strengths and weaknesses that we discuss in section 6.

Table 3. Analysis without training vs our experiment results from Imagenet-1K. Note that a naive prediction would correlate the given top-1 accuracy scores directly with the number of parameters. Our performance estimations based on total surprisal and expected spread (P_G and $P^{^{\prime}}_G$ respectively) instead correctly favor V2-S-shared (bold) compared to V2-B0.

Model	Score	PG	$\text{P}^{^{\prime}}_\text{G}$	Exp.spread	Entropy	Params	${\text{Img.}}$
V2-B0-shared	73.7	25.00	25.44	746.608+	10.2	3.0 M	224
V2-B0	76.9	26.28	26.72	746.608	10.2	7.1 M	224
V2-S-shared	78.3	26.33	26.97	1505.547	9.8	4.4 M	384
V2-S	83.2	28.72	29.28	1505.546	9.8	21.6 M	384

Table 4. Analysis without training vs some of the top-1 accuracy scores for larger models [36] for Imagenet-1K. Note that a naive prediction would correlate the given top-1 accuracy scores directly with the number of parameters. Our graph analysis based estimations instead assign estimated performances based on total surprisal and expected spread (P_G and $P^{^{\prime}}_G$ respectively) to ResNet-50 (bold) lower than V2-B3 and V2-S models which both have a smaller number of parameters.

Model	Score	PG	$\text{P}^{^{\prime}}_\text{G}$	Exp.spread	Entropy	Params	${\text{Img.}}$
V2-B1	79.80	26.75	27.20	913.4	10.2	8.2 M	240
ResNet-50	80.30	27.25	27.60	465.4	13.3	25.6 M	380
V2-B3	82.10	27.71	28.20	1167.2	10.6	14.5 M	300
V2-S	83.6+	28.72	29.28	1505.5	9.8	21.6 M	384
V2-M	85.10	29.9+	30.4+	2171.1	9.8	54.4 M	480
V2-L	85.70	30.5+	30.9+	3095.4	10.3	119.0 M	480

We quantify the predictions from the reusability prior for model graphs as follows:

• (i)  
  We estimate the relative frequencies of the learnable parameters in the horizontally unrolled computational graph of a given model (e.g. for the original graph in figure 3(a) it is possible to estimate the relative frequencies from its unrolled form in figure 3(b). Please see appendix B for the full example.).
• (ii)  
  The resulting probability distribution allows deriving model-level quantities that we described in section 4.3.
• (iii)  
  We use total surprisal and expected spread for estimating model performance as described in section 4.5.

In practice, since the EfficientNetv2 and ResNet-50 models have convolutional layers, we took into account the full computational graph that would depend on aspects such as image size, strided convolutions, batch normalization, and pooling layers. Thus we included all learnable parameters from convolutional kernels, batch normalizations, and final fully connected layers for classification which have bias weights. The Python implementation of our graph analysis relies on imitating the computational graph of EfficientNetv2 and ResNet-50 models with layers that, instead of inference, count the aggregated number of contexts. This does not need to create a horizontally unrolled graph which would have been exponentially more expensive. Yet this approach still allows gathering the total number of contexts for each learnable parameter in a precise manner and calculating model-level quantities for comparison. Our full code release to reproduce the analysis results is available at [113].

EfficientNetv2 models have a relatively high parameter efficiency, and they have competitive performance in image classification benchmarks. Would applying aggressive parameter sharing to an already compact model still improve parameter efficiency? To answer this question, we conducted multiple experiments on EfficientNetv2 models. We focused on comparing EfficientNetv2-B0, and EfficientNev2-S models. For cross-layer parameter sharing, our experiments revealed results in agreement with the literature discussed in section 2.1. To make it easier to minimize confounding variables due to data and training strategies, we limited the models we trained from scratch to only EfficientNetv2. We conducted our training in a controlled setting, spanning three different benchmarks. This helped us eliminate differences due to hardware and hyperparameters for the selected models.

For the EfficientNetv2 experiments, we modified the official Tensorflow implementation [114] to be able to optionally apply aggressive parameter sharing. This strategy relies on roughly treating weight matrices of the same shape as the same. One exception is that we did not share the batch normalization weights to help the training stay stable.

During the initialization of models with parameter sharing, we keep a global scope dictionary. Convolutions of the same scope are only created once. Convolutional layers are represented as four-dimensional matrices where their dimensionality is defined by channel and kernel sizes. More precisely, we map each shared convolution to a scope name that is constructed by combining the number of input channels, number of output channels, kernel size, and strides.

For cross-layer parameter sharing, we used a new hyperparameter which aggressively shares convolutions but uses separate batch normalization layers: model.weight_sharing = all_but_bn. The only change we made to the hyperparameters when comparing an original model with one that shares convolutional layers is we used model.weight_sharing = None for the original one. We closely followed the default hyperparameters for EfficientNetv2 given by Tan and Le [36], except for the following changes:

• For all models trained with CIFAR-10, instead of transfer learning, we trained from scratch. We used no augmentation and trained for 300 epochs without any training stages. We used the following hyperparameters: train_epoch = 300, batch_size = 512, data.ibase = 32, train.lr_warmup_epoch = 5, train.lr_sched = exponential, data.mixup_alpha = 0, data.cutmix_alpha = 0, train.lr_base = 0.016, model.bn_momentum = 0.99.
• For all models trained with CIFAR-100, we followed the same scenario and hyperparameters, except for the data augmentation: train_epoch = 300, batch_size = 512, data.ibase = 32, train.lr_sched = exponential, train.lr_warmup_epoch = 5, data.augname = autoaug, train.lr_base = 0.016, model.bn_momentum = 0.99.
• For all models trained with Imagenet-1K, we used the following hyperparameters: train_epoch = 352, batch_size = 400, train.stages = 4, train.lr_sched = exponential, model.dropout_rate = 0.075, train.lr_warmup_epoch = 5, data.ram = 2, train.ema_decay = 0.9999, train.lr_base = 0.025, model.bn_momentum = 0.99, data.augname = randaug.

The changes in the batch size and learning rate are necessary to be able to train the models with smaller GPUs (i.e. reduce the batch size and learning rate based on the same ratio). Note that, in the described setting, when trained from scratch, we observed lower performance for both the EfficientNetv2-B0 and EfficientNetv2-S models compared to [36]. Larger batch sizes and training longer can sometimes improve performance but we observed a substantial gap in performance between V2-S-shared vs V2-B0-original models regardless. In our previous experiments with Imagenet-1K, we tested different hyperparameters such as gclip, batch size, as well as a larger number of training epochs. Moreover, for CIFAR-100 we tested using no augmentation. None of these changes in the hyperparameters changed the ranking of the models in terms of top-1 accuracy given in table 2.

In our experiments, we trained our models from scratch using the well known object classification benchmarks for visual recognition as given in table 1, which consists of CIFAR-10, CIFAR-100, and Imagenet Large Scale Visual Recognition Challenge (ILSVRC2012, also referred to as Imagenet-1K).

When EfficientNetv2 models are trained from scratch, we observed a consistent increase in the parameter efficiency for the V2-S models with aggressive parameter sharing compared to V2-B0 which has significantly more parameters, as given in table 2.

Overall, at the cost of additional FLOPs, we observed improved parameter efficiency for all three benchmarks. Alongside the research regarding parameter sharing in the literature discussed in section 2.1, and combined with our graph analysis results in table 3, these results provide supporting evidence for the reusability prior.

In our experiments with EfficientNetv2 models, V2-S-shared models consistently performed better than the V2-B0 models, despite having a significantly lower number of parameters. In table 3, we compared the results from our Imagenet-1K experiments with the results from our analysis of the computational graphs of V2-B0 and V2-S models, with and without parameter sharing. Unlike a naive prediction which would directly correlate the number of parameters with performance, our graph analysis assigned a higher score for the V2-S-shared model.

We compared our graph analysis based quantities for larger models that were trained by Tan et al in [36]. ⁶ In table 4 the estimated performance and expected spread of ResNet is lower, while its entropy and number of parameters are significantly higher than V2-B3 and V2-S models. Coincidentally, for the given models, the performance estimations were roughly $1/3$rd of the actual top-1 accuracies, but unlike the top-1 accuracy, our performance estimation scores are not bounded (e.g. P_G can be negative).