An autoencoder for heterotic orbifolds with arbitrary geometry

Considering the eight 16D shift vectors V_A that define them, heterotic orbifold models with a given space group S exhibit 128 defining parameters. However, as mentioned before, there are highly nontrivial symmetries relating different gauge embeddings that may reduce the number of truly free parameters. Noting that the symmetry invariants ${N}_{n}^{(a)}$ encode information about the shift vectors, one might entertain the possibility to replace the original parameters by a set of cardinalities $\{{N}_{n}^{(a)}\}$, which has shown to be a reasonable strategy [47].

Our purpose here is to design an autoencoder capable of analyzing models of various orbifold geometries at once. The challenge here arises from the fact that each orbifold geometry has very different properties. This implies that the defining parameters associated with each S exhibit e.g. different values and statistical characteristics, see our auxiliary material for two sample orbifold geometries [67]. We must find a method to deal with the various geometries simultaneously. For instance, since in general the total number of fixed points n_fp(S) depends on S, we must restrict the selection of fixed points to a subset of size s_fp, such that s_fp ≤ n_fp(S) and is equal for all considered space groups. In addition, we must consider a numerical label L for the space group associated with each model and, in order to capture more features about the 4D spectrum, we shall include the number of U(1) gauge factors N_U(1). Therefore, we choose as characteristic features of a model the subsets

$$\begin{eqnarray}&&X\,:= \,\left\{{N}_{n}^{(a)}\,\left.\right|\,0\leqslant n\leqslant {s}_{\mathrm{fp}},a=1,2\right\}\,\cup \,\left\{L,{N}_{{\rm{U}}(1)}\right\}.\end{eqnarray} \tag{ 8 }$$

Thus, in this scenario, the number of considered features is 2s_fp + 4.

There is some arbitrariness on the choice of s_fp that we can use to our advantage. It is clear that the larger s_fp is, the better X captures the properties of the orbifold. This is because each chosen fixed point can contain information about a single shift vector V_A and we need information about all of them. However, s_fp cannot be too large because there are some space groups S that admit only a small number of fixed points. Further, taking the maximal number of fixed points of a given geometry is not necessary in general because the definition (6) of the local gauge embedding implies that there are V_local,n at different singularities that can lead to the same values ${N}_{n}^{(a)}$. Moreover, we must also take into account that the computation time grows with the dimensionality of X. So, we find it convenient to consider s_fp of order 10, and choose the fixed points from the (first) twisted sectors (k, ℓ) = (1, 0) and (0, 1) in addition to some others from higher twisted sectors, in order to capture at various (preferably geometrically inequivalent) singularities as much information provided by all shift vectors V_A as possible. Note that choosing s_fp of order 10 exhausts the information coming from all eight V_A unless we incorrectly take only geometrically equivalent fixed points.

Once the feature vector X to be used on different orbifold geometries is defined, we must build the datasets that will allow us to train the autoencoder. With this goal, one must first generate a large number (a few hundred thousands) of inequivalent admissible orbifold models with one or more space groups by using the dedicated automatized tool orbifolder [56]. This generates the 'raw data' containing a large number of sets of gauge shift embeddings {V_A }. In a second step, we run our Makedataset code [68] (detailed in section 4.1) that translates these vectors into the sets of gauge momenta ${I}_{n}^{(a)}$ defined in equations (5) and (7) and the resulting gauge group in 4D, and then computes the invariant feature vectors X.

Due to the differences among the various orbifold geometries, the process previously described will, in general, generate very different sets of data for different geometries. This is not surprising since the process leading to the cardinalities N^a _n involves information that is particular to the geometry, namely the centralizers and the orbifold twisted sectors. Putting all these elements together means that dealing with more than one geometry is not just a matter of extending the data to some extra elements of a given set, it is a whole new extension to study a different problem. Including more than one geometry gives a more generic perspective of the 4D models arising and, possibly, independent of the geometry. This allows us to explore the heterotic landscape in a somewhat more general perspective.

A deep autoencoder is a feedforward NN, mainly used to dimensionally reduce or compress some complex high dimensional data. It is built by two components: the encoder and the decoder. The purpose of the encoder is to identify and reduce the redundancies (and noise) of input data defined by a large number of parameters, lowering step-wise in various (hidden) layers the dimensionality of the parameter space. If the encoder is deep enough, i.e. if it has a large number of layers, and the data is adequate, its capability to encode the input data into a small number of parameters tends to improve. The last layer of the encoder is known as the (central) latent layer or latent-space representation, and it contains a 'compressed code' of the input data, represented in a small number of parameters. The decoder operates inversely to the encoder, reconstructing the data from the latent layer back to its original higher dimensional form. ⁵ One can define the accuracy of an autoencoder as the level of likeness between the output resulting from the decoder and the corresponding original information in the input layer.

Given some input data, one must choose hyperparameters that maximize the accuracy of the algorithm. The properties that describe an autoencoder configuration are:

• Topology: overall structure that defines the way the neurons of the NN are connected among different layers. The topology can be symmetric or asymmetric with respect to the latent or bottleneck layer, fully or partly connected, and can include convolutional layers or other types of substructures. We avoid convolutional or other complex layers for simplicity. ⁶
• Architecture: number of layers and number of neurons per layer (layer size). In the case of an autoencoder, it includes the sizes of all hidden layers of the encoder and decoder, the input and output layers, as well as the size of the latent layer.
• Initial weight distribution: the values of the trainable parameters or weights that characterize the neurons must be initialized at random values using a method that may be useful to arrive at the best accuracy; it is customary to take a Gaussian or uniform distribution, but other options (such as Xavier or He initializations [69–73]) are possible.
• Activation function: together with a bias, it defines the output of a neuron given some input information; it adds some non-linearity to the learning process in order to improve it. Some examples that we shall use in this work include Leaky-ReLU, Softplus, ELU, CELU, ReLU, SELU, Hardtanh, Softmax and LogSigmoid (see e.g. [74–78] for details of activation functions). In principle, every layer can have a different activation function, but we apply, as usual, homogeneously the same activation function to all layers for simplicity. Despite the expectation of a poor performance of Hardtanh, Softmax and LogSigmoid, as will be confirmed, they can be used here as a basis for comparison.
• Loss function: evaluation during the training stage that determines the magnitude of the inaccuracy that the NN has achieved before updating the weights of the network. Motivated by the encouraging results of [47], where the L₂ loss function (equivalent to MSE) provides a good accuracy, we decide to test various loss functions besides the most natural choice of Cross Entropy (CE). Some examples of loss functions used in this paper are CE, SmoothL1, MSE, Huber, BCEWL, L1 and Hinge Embedding (see e.g. [79, 80] for details).
• Optimizer: optimization algorithm used to minimize the loss; some examples applied in this work are Adam, AdamW, Adamax, RMSProp, Adagrad, Adadelta, SGD and ASGD (see e.g.[81–84] for details on these functions).
• Number of epochs: number of times that the algorithm is run to improve the learning skills of the algorithm, trying to minimize the error.
• Batch size: for each epoch, it is the number of samples in which the training input data is split in order to have several training subsets. Typically, large batch sizes lead to better statistical characterizations; however, although commonly varied only to adapt it to the availability of memory, we shall see that the choice of batch size can also help to improve the accuracy of the model.
• Shuffling: to optimize the learning process, whether or not the elements contained in each batch per epoch are randomly shuffled.
• Dropout: if applied, it defines the number of dropout layers and the fraction of neurons that are randomly dropped out; this is typically used with the goal of reducing overfitting.

The choice of the best configuration can be either 'handcrafted' by some educated guess or achieved by a systematic study of the properties that define the autoencoder. We mix both methods: we vary systematically three of the configuration parameters (activation and loss functions, and the optimizer) while fixing all others. In detail, based on the results of [47], we first fix the topology to be feedforward fully connected and symmetric with respect to the latent layer, the initial weight distribution to be Gaussian, and we apply shuffling of the data in each epoch.

To fix the architecture, we adapt the feature vectors X so that they can be readily handled by the autoencoder. Since the selected features are all categorical (they admit a limited set of values), it is convenient to perform a one-hot encoding (OHE). To minimize the use of resources as well as arrive at good results, we take s_fp = 8, leading to 18 features which accept various different sets of values ${N}_{n}^{(a)}\in [0,240]$. In addition, the number of U(1) gauge symmetries in 4D is constrained by 0 ≤ N_U(1) ≤ 16, and L is limited by the number of orbifold geometries that one incorporates in the autoencoder. Each feature X_i is then mapped to a subvector X_OHE,i of (different) dimensionality γ_i . The resulting full one-hot encoded feature vector X_OHE has dimensionality ${\sum }_{i}{\gamma }_{i}={ \mathcal O }(1000)$. This is the size of the input and output layers. We shall aim at arriving at a three-dimensional latent layer, in order to better capture the information from various orbifold geometries. We shall include 7 or 9 hidden layers, aiming at a NN deep enough to provide admissible results.

Other configuration parameters are defined in various tests, as we now describe.

3.2.1. Testing configurations

In order to arrive at the best configuration, we have used orbifold models based on two phenomenologically promising and distinct orbifold space groups, which are known as ${{\mathbb{Z}}}_{8}$–I (1,1) and ${{\mathbb{Z}}}_{12}$–I (1,1) in the notation of [54]. For simplicity, we label them as ${{\mathbb{Z}}}_{8}$ and ${{\mathbb{Z}}}_{12}$, respectively. The features and phenomenology of ${{\mathbb{Z}}}_{8}$ orbifold compactifications have been studied in detail in [58, 85], while many interesting properties of ${{\mathbb{Z}}}_{12}$ models have been presented in [59–61, 86–89]. It is known that these space groups correspond to two of the most fertile ${{\mathbb{Z}}}_{N}$ orbifold geometries, as they yield large sets of MSSM-like models [17–19]. We emphasize that, even though our method is trained here for these geometries, it can be generalized to any choice of geometries. While implementing our method for larger sets, one must consider that the larger the set of geometries and models is, the more computational resources are required.

With the help of the orbifolder [56], we have built a database of nearly 270,000 inequivalent, anomaly-free models from each chosen geometry, with several different gauge groups G_4D. The total of 540,000 models is randomly split into a 66.7% for the training set and a 33.3% for the validation set.

In this case, we have considered s_fp = 8 fixed points per geometry, ⁷ which means that the feature vectors X of equation (8) are 20-dimensional. Analyzing the total of our models, the OHE enhances the feature vectors to 810-dimensional one-hot encoded vectors X_OHE. This represents the input layer.

After some quick tests with various libraries, we found optimal the use of PyTorch [90] and some Pandas [91] and Scikit-learn [92] functions, as they are simple and economic in notation and as effective as other options.

Now we are ready to implement and test our proposal. Since we wish to test for several activation, loss and optimizer functions, we define a basic or 'vanilla' configuration, which is given in table 1. As anticipated, we use seven hidden layers, including the 3D latent layer. The dimensions of the layers of our symmetric NN are 810, 200, 26, 13, 3, 13, 26, 200, 810. We run the training stage for 1,000 epochs, with no assumption about overfitting or other undesirable properties that we want to identify in this phase.

Table 1. Parameters of our 'vanilla' configuration. While others are fixed, we vary the parameters with the label 'default value', one at a time, in order to arrive at the best autoencoder configuration. E.g. we test different optimizers while taking the default values specified here for the activation and loss functions.

Configuration parameters	Chosen values
Topology:	Fully connected and symmetric
Architecture:	810, 200, 26, 13, 3, 13, 26, 200, 810
Initial weight distribution:	Gaussian
Activation function:	SELU (default value)
Loss function:	MSE (default value)
Optimizer:	Adam (default value)
Number of epochs:	1, 000
Batch size:	25
Shuffling:	True
Dropout:	False

We test the different functions of our setup as follows: we choose the one function in the vanilla configuration that we want to test and leave the other two fixed with the 'default values' given in table 1; then, we vary systematically the chosen function through the options listed in section 3.2, training the NN during 1,000 epochs for each resulting configuration. We tested various batch sizes ranging from 2³ to 2⁷, which are within the capabilities of any small or large computer, and observed that varying the batch size also allowed us to arrive at a better accuracy. Finally, we arrived at a batch size of 2⁵ = 32 to minimize the hardware demands while aiming at good results. ⁸ The default values are considered taking into account their good performance in NN for similar purposes and their reduced computing time. In order to evaluate the performance of each configuration, we examine the NN accuracy and loss.

While assessing the results, we realized that the typical definition of the loss function, which compares the full predicted and input feature vectors, exhibits some weakness because of the large number of zeroes appearing in the one-hot encoded vectors that are compared. We improve this situation by demanding that the loss function compare the input and output one-hot encoded vectors by feature. In detail, we implement the redefinition of the loss

$$\begin{eqnarray}&&L({X}_{\mathrm{OHE}},{\widehat{X}}_{\mathrm{OHE}})\,=\,\displaystyle \frac{1}{\#\,\mathrm{batches}}\displaystyle \sum _{k}\displaystyle \frac{1}{\mathrm{batch}\,\mathrm{size}}\displaystyle \sum _{j}\displaystyle \frac{1}{2{s}_{\mathrm{fp}}+4}\displaystyle \sum _{i}F({X}_{\mathrm{OHE},i}^{{kj}},{\widehat{X}}_{\mathrm{OHE},i}^{{kj}}),\end{eqnarray} \tag{ 9 }$$

where i runs over all features, j over all elements of a batch, and k over all batches in the training set. As usual, X_OHE denotes the input one-hot encoded feature vector, and ${\widehat{X}}_{\mathrm{OHE}}$ the prediction of the autoencoder. The chunk ${X}_{\mathrm{OHE},i}^{{kj}}$ associated with the ith feature of the jth element of the kth batch has dimensionality γ_i , which is the number of values that the ith categorical feature admits. The loss function F( · , · ) is chosen here to be one of our testing set: Cross Entropy (CE), SmoothL1, MSE, Huber, BCEWL, L1 and Hinge Embedding.

Our results at this stage are presented in figures 1–3. In figure 1 we show a comparison of the evolution over 1,000 epochs of the accuracy and loss of the various activation functions that we consider in our tests. The best accuracy is achieved by Softplus and Leaky-ReLU, which corresponds to NN that predict correctly up to about 14 out of 20 features. Simultaneously, we observe that these activation functions lower their loss very fast, with a slight preference for Leaky-ReLU, as suggested in the literature, see e.g. [93]. It turns out that this preference is stressed by the fact that including Softplus in our NN requires much longer computation time to arrive at an accuracy comparable with that of Leaky-ReLU.

Zoom In Zoom Out Reset image size

In figure 2, we compare the accuracy and loss that our autoencoder achieves for various loss functions along with the default values for activation function and optimizer of our vanilla configuration, see table 1. The loss has been computed using the redefinition of equation (9). We see that Cross-Entropy (CE) leads to the most accurate NN while the loss function keeps decreasing. However, our results suggest that it is convenient to explore the performance of other almost as accurate loss functions. MSE, SmoothL1 and Huber display very similar performance, so that we may consider any of them as equivalent for our purposes.

Zoom In Zoom Out Reset image size

In figure 3 we present the comparison of the performance of the autoencoder with various optimizers. We observe that the best accuracy and smallest loss, with high variation over the first 400 epochs are Adam, AdamW and Adamax. Interestingly though, we note that the maximal accuracy is about 12 out of 20 features, which differs from the previous comparisons. This is most likely the result of the choice for the default values of loss and activation functions, see table 1. This suggests that we must change our selection in the final configuration.

Zoom In Zoom Out Reset image size

3.2.2. Our best autoencoder configuration

The results of the previous section do not suffice to identify the best configuration. We use now a much larger dataset with 1,260,000 ${{\mathbb{Z}}}_{8}$ and ${{\mathbb{Z}}}_{12}$ orbifold models, split as before in a 2/3 training and 1/3 validation set. We consider autoencoder configurations based on our vanilla proposal, table 1, with the following modifications:

• include a pair of 1620-dimensional hidden layers next to the input and output layers as a means to improve the accuracy of the NN [94];
• instead of the 'default values' of the vanilla configuration, we use the selection of the best activation functions (Leaky-RELU and CELU), error functions (CE and MSE), and optimizers (Adam, AdamW and Adamax), and test the performance of the resulting combinations.

The size of the newly included hidden layers has been tested for best performance.

Given these specifications, we let once more our NN run over 1,000 epochs with each of the proposed configurations. In order to better compare the performance, now we compute the accuracy and loss in both the training set and the validation set. Our results are displayed in figures 4 and 5. We first note the stability of our autoencoder configurations as they all perform similarly in both the training and validation sets. Further, from the evolution of the accuracy of our NN shown in figure 4 we see that the most accurate configurations always include Leaky-ReLU [95] as activation function ⁹ and CE as error function. All variations of Adam optimizer perform very well. Including the evaluation of the loss from figure 5, we find that the combination of activation, error and optimizer functions and the architecture presented in table 2, together with the other NN parameters of the vanilla configuration of table 1, lead to the best autoencoder configuration. Further, this configuration takes the most optimal computation time. ¹⁰

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Table 2. Best parameters for the autoencoder configuration. Based on the various performance tests, we replace the 'default values' provided in the vanilla configuration, table 1, by the selection presented here.

Configuration parameters	Best values
Architecture:	810, 1620, 200, 26, 13, 3, 13, 26, 200, 1620, 810
Activation function:	Leaky-ReLU
Loss function:	Cross-Entropy
Optimizer:	Adam

The first tool that will be required is MakeDataset, which is available as a jupyter notebook at [68]. It allows one to translate the input shift and Wilson lines of orbifold models into feature vectors X to be used by the heterotic orbiencoder(see equation (8)), which is a mandatory step prior to train our NN. In its containing folder, you can find a README file with instructions on how to install it. Once installed, you have just to choose five parameters to start the process:

• Use name_models_from_orbifolderto set the path and name of the input file that contains all shifts and Wilson lines of a given orbifold geometry that will be translated into the characteristic features to train the heterotic orbiencoder. For example
  • name_models_from_orbifolder="Z6_models_SP_3.txt"
• Use name_geometryto set the path and name of the orbifold-geometry file that corresponds to the models to be read by the heterotic orbiencoder. Please, observe that this file must be provided in the precise format defined in the orbifolder. For example
  • name_geometry="./Geometry/Geometry_Z6-I_2_1.txt"
• Use name_datasetto set the name of the output file. For example
  • name_dataset="Dataset_Z6_I_2_1_[1,3,5].csv"
• Use list_sectorsto set the twisted sectors of the orbifold geometry that will be used to generate the characteristic features. It is important to match the number of fixed points of the various geometries used. Take into account that the number of fixed points depends on the selected sectors. (You can verify it with the command sector(), which returns the list of sectors and fixed point for each geometry.) For example
  • list_sectors=[1,3,5]
• Once the process starts, MakeDatasetdisplays the number of feature vectors that have been computed and stored. May you need to interrupt this process, you can restart the generation of feature vectors by restarting the program with an advanced starting point. This is set by the parameter start. For example,
  • start=100
  restarts the generation of feature vectors at the 100th model.

It is important to mention that one needs first to run MakeDataset for each one of the orbifold geometries of interest, then label properly the datasets, and finally concatenate them. (This can be easily done in Pandas.) The name of the resulting concatenated file must feed the variable datasetname of the configuration of the heterotic orbiencoder.

The notebook of the heterotic orbiencoder is configured to work directly by loading the pre-trained example (Trained_Machine.pt) used in the present paper. Following the standard ML nomenclature, hereafter we will refer to the NN configuration together with a set of hyperparameter values as NN-model. Hence, changing some parameters and then attempting to run the default NN-model will unavoidably lead to spurious results.

According to our best configuration given in table 2, the default value for the activation function of all layers is set to Leaky-ReLU, and the dimension of the layers, the loss function and the optimizer must be fixed as follows

In addition, we provide a number of training parameters that can be dialed, although they are set to the best values we identified:

• Use epochsto set the number of epochs that the NN will be trained. For example
  • epochs=1000
• If you would like to save the resulting NN-model after a number of training epochs, set this periodicity in the parameter save_each_epoch. For example, to save the NN-model every 100 epochs, set
  • save_each_epoch=100
  As an additional feature, the best NN-model, i.e. the one leading to the maximal accuracy, is automatically saved at the end of every training.
• Use train_setto set the fraction of the total of the input feature vectors to be used for training. For example
  • train_set=0.6
• Use batchsizeto set the size of the batches (number of feature vectors per batch) to be used for the training. For example
  • batchsize=32
• Use labelto set a label for all the output files generated. For example, you may want to indicate the number of epochs and the activation function to label the files of your training:
  • label="1000e_leaky"
• You can use a schedulerto adjust the learning rate after a specified number of epochs. This modification in the learning rate can help prevent getting trapped in a local minimum during the gradient-descent optimization procedure. A possible configuration of a scheduleris given by
  • scheduler=lr_scheduler.StepLR(optimizer, step_size=400, gamma=0.1)
  where step_size sets the periodicity (in the number of epochs) of the change in the learning rate, gamma sets the decay of the learning rate, and optimizer calls the optimization routine that will be changed every step_size epochs.

These parameters, along with some others further detailed in the jupyter notebook, are contained in a Python dictionary called parameters.

Once the training is finished, you can perform a series of tasks devoted to use the trained NN-models. First, you can visualize the files of the saved NN-models with

• !ls savedModels

The NN-model leading to the best accuracy is denoted by 'best' whereas other NN-models in the list are displayed according to the epoch number at which they are saved.

One of the relevant properties of the saved NN-models is their accuracy in reconstructing the feature vectors X. As expected, every NN-model will have an error. To determine it, one can execute the command routines.reconstruction, which generates two separate files stored in the latentSpace and success folders within the heterotic orbiencoderpath. The first file contains the encoded feature vectores X in the 3D latent space, while the success file counts the number of correctly predicted individual features (X_i ). The command is invoked as follows

• routines.reconstruction(NN-model, saved_model_file_name, **parameters)

where parametersrefers to the Pythondictionary defined in the notebook, saved_model_file_namemust be taken from the list generated by !ls savedModelsand NN-modelrefers to the variable used when the NN-model was saved. You can visualize the resulting files by using the commands

• !ls success
• !ls latentSpaces

We also include some functions to visualize useful information regarding the success rate and the latent space produced by the saved NN-models, using the files contained in the success and latentSpaces folders. The commands to invoke those functions are:

• utils.plot_features_statistics(success_file_name): generates bar charts with the success rates by feature and the success rates by number of correctly predicted features. You can get the success_file_namefrom the list generated by the command ls sucess. For example,
  • utils.plot_features_statistics("success_model-700epoch_1010_e_leaky")
• utils.plot_report(label): generates the plots of the evolution by epoch of accuracy and loss for the validation and training sets. For example,
  • utils.plot_report("1000_e_leaky")
• utils.plot_2d_latent_space(latent_geom1_file_name, latent_geom2_file_name): generates the latent space plots for the pair of geometries that were fed in to the heterotic orbiencoder. You can get latent_geom1_file_nameand latent_geom2_file_namefrom the list generated by the command ls latentSpaces. For example,
  • utils.plot_2d_latent_space("latent_z8_model-100epoch_1010_e_leaky", "latent_z12_model-100epoch_1010_e_leaky")

An intriguing question arises once the heterotic orbiencoderis trained: Can the latent space be used to learn properties of known orbifold models, which were not used for training? Can we gain some insight about the properties in the latent space of orbifold models with promising properties? This is particularly relevant when you have a trained NN-model and want to locate MSSM models in the latent space that were not included in the training dataset. In this scenario, you can copy to the data folder the associated input files containing the feature vectors of the promising orbifold models. Without modifying the training parameters, you can select one of the saved NN-models and process the new data, granting quick access to relevant reports. The parameters and commands to evaluate the parametrical details of the newly inserted models in the latent spaces are:

• datasetname_other: specifies the name of the file that contains the promising models that will be encoded into the 3D latent space. For example
  • datasetname_other="./data/Z12_SM_554models.csv"
• routines.reconstruction_other_models(NN-model, saved_model_file_name, datasetname_other, new_label, **parameters): This command performs the same task as routines.reconstruction, but working on models unknown during training. The parameter new labelsets the name of the file containing the data that encodes in the 3D latent space the promising models. For example
  • routines.reconstruction_other_models(NN-model, "model-700epoch_1010_e_leaky", datasetname_other,"z12_MSSM", **parameters)
• One can generate a plot of the latent space containing the pair of analyzed geometries, and additionally the promising models from each geometry analyzed. For this to work properly, it is important that the labels of the plots be set to MSSM. The parameters needed in this function are: latent_geom1_file_name, latent_geom2_file_name, latent_MSSM_geom1_file_nameand latent_MSSM_geom2_file_name, in that specific order. For example
  • utils.plot_2d_latent_mssm('latent_z8_model-100epoch_1010_e_leaky', 'latent_z12_model-100epoch_1010_e_leaky', 'latent_z8_MSSM-model-100epoch_1010_e_leaky', 'latent_z12_MSSM-model-100epoch_1010_e_leaky')

Although we do not recommend to make changes in the configurations, our routines allow the user to test different configurations easily. One of the most important characteristics to consider is the topology of the heterotic orbiencoder, which is defined in the array dimensions as well as the latent space size, defined by the parameter latent. Note that changing the size of the latent space will need a consecuent change in the functions defined in the module utils.py.

Considering the total of 1,260,000 ${{\mathbb{Z}}}_{8}$ and ${{\mathbb{Z}}}_{12}$ orbifold models, we aim now at obtaining the best configuration of the heterotic orbiencoder for those particular geometries that achieves as fast as possible the maximal possible accuracy while avoiding issues such as overfitting. We split once more the dataset into 2/3 training and 1/3 validation set, and train over a number of epochs. Our first step is to train as long as possible to arrive at high accuracy and observe the behavior of the heterotic orbiencoder. Yet in appendix B we show that it is very easy to fall into overfitting. As we discuss in our appendix, we have tested various methods to reduce overfitting, which turn out to be unsuccessful in our case, implying a positive observation though: The chosen configuration parameters, specifically those of table 2, allow one to obtain high accuracy in short time, before overfitting sets in. In order to avoid it while delivering the best results, our autoencoder must be trained for less than 448 epochs. We saved the best trained NN-model at epoch 445; you can find it in the folder saved models in the documentation [96].

After reducing the ${ \mathcal O }(100)$ parameters that define heterotic orbifold models to just 20 features of the kind introduced in equation (8), as before, our choice of autoencoder configuration can further encode them in just three parameters, delivering the final accuracy displayed in figure 6. From a 3D latent layer our decoder can reconstruct in average 14.4 out of 20 features successfully, corresponding to a 72% of accuracy. Moreover, we note from figure 6(b) that in 37% (46%) of ${{\mathbb{Z}}}_{8}$ and ${{\mathbb{Z}}}_{12}$ orbifold models our autoencoder can successfully reconstruct 18 (16) or more out of 20 features from the three parameters of the latent layer. Further, there are models in which only five features are correctly predicted; however, they are a very small fraction of the models.

Zoom In Zoom Out Reset image size

In previous efforts [47], studying just one orbifold geometry with a different NN configuration, an average of 63% accuracy was achieved when using a 2D latent layer. We have made a test to verify whether the extra dimension of the latent layer is responsible for the improvement we see. We found that, in the case of two geometries, considering the activation function and optimizer of that work and a 2D latent layer leads to a worse accuracy than the one reported. Further, insisting in a 2D latent layer, but keeping other functions as in our configuration leads to a 10% improvement. This delivers some interesting observations: our selection of configuration parameters is the key to improve accuracy, and one needs more than two parameters to encode the information of heterotic orbifold models. We must emphasize that our results are obtained over models from two orbifold geometries (and that this can be readily generalized to more geometries with our code), whereas previous results were only on one geometry.

We expect that our results can be improved by making our autoencoder even deeper, which will be studied elsewhere by getting access to richer hardware resources than currently available to us.

In order to learn to draw some information about heterotic orbifold models from the three parameters (say {y₁, y₂, y₃}) of the latent parameter space of our autoencoder, we must explore the behavior of our models in the latent layer. In figure 7 we display the localization in the latent layer of a sample of 2,000 (randomly chosen) ${{\mathbb{Z}}}_{8}$ and ${{\mathbb{Z}}}_{12}$ orbifold models per geometry. We observe an interesting segmentation of the models by geometry (i.e. by space group), which is evident in the plane y₁ − y₃. This property most certainly shows that the information of the feature X_i = L of equation (8) is correctly encoded in the latent space. The boundary of the two regions is described by the curve given by

$$\begin{eqnarray}&&{y}_{3}-0.63{y}_{1}=0.32,\end{eqnarray} \tag{ 10 }$$

that is also plotted in the central panel of figure 7.

Zoom In Zoom Out Reset image size

Another interesting observation is that the distribution of the models in the 3D latent space is not so wide, so that a classification of heterotic orbifold models would just require a scan of a relatively small 3D region, which is much more feasible than the same task in the original ${ \mathcal O }(100)$-dimensional parameter space. This is the motivation of a larger work in progress, which will be reported elsewhere. An important pending task is the translation of the information contained in the outer layer back to the parameters that define an orbifold compactification.

Although this information is already important, we would like to explore whether one can identify some properties of e.g. phenomenologically promising models in this new parameter space. With this purpose, we use the orbifolder to produce the 128 input parameters of a set of promising ${{\mathbb{Z}}}_{8}$ and ${{\mathbb{Z}}}_{12}$ orbifold models with the matter spectra of the MSSM or SU(5) GUTs [97]. It is crucial to emphasize that these models were not part of the original dataset; hence, they were neither included in the training nor validation sets. So, these models can be considered as a test set.

5.2.1. MSSM-like orbifold models

We have used the orbifolder to construct MSSM-like ${{\mathbb{Z}}}_{8}$ and ${{\mathbb{Z}}}_{12}$ orbifold models. (These models had been previously found in [18]). We identify 176 (554) models of this type arising from the ${{\mathbb{Z}}}_{8}$ (${{\mathbb{Z}}}_{12}$) orbifold geometry. In order to present a balanced statistics of the behavior of our models, we display in figure 8 the localization in the latent space of 150 models per geometry.

Zoom In Zoom Out Reset image size

By observing the details of the localization, we find that close to the boundary between the ${{\mathbb{Z}}}_{8}$ and ${{\mathbb{Z}}}_{12}$ orbifold regions, bounded by the line described by equation (10), most of the MSSM-like models are found. To see this in more detail, in the plane y₁ − y₃ (central panel) of figure 8 we depict by the dashed (dotted) lines the boundaries where 50% (75%) of all promising models are found. In the units of these parameters, the distance from the central boundary to the dashed (dotted) lines is 7 (11). In the y₁ − y₂ (left panel) and y₂ − y₃ (right) latent planes, we find that, even though the boundary between ${{\mathbb{Z}}}_{8}$ and ${{\mathbb{Z}}}_{12}$ are not distinguishable from this perspective, on the left of the dashed (dotted) lines 50% (75%) of all MSSM-like models, including both orbifold geometries. This description holds for all MSSM-like models from the inspected orbifold geometries.

Our observations imply that a search for promising models is quite restricted in the latent layer of our autoencoder. This is unexpected since no information about the phenomenological features of the models was provided in the input dataset. Hence, this invites to further explore the question of a better classification method of heterotic orbifolds, independently of the geometry, through the new parameter space of the latent layer.

We note that, in contrast to the results of [47], no islands are built in the latent parameter space. We find that this is a general feature of our autoencoder. It would be interesting to figure out under which conditions the segmentation behavior arises.

5.2.2. SU (5) grand unified models

In figure 9 we plot the localization in the latent space of 100 models per geometry that yield the massless matter spectrum of SU(5) GUTs. As shown explicitly in appendix C for two sample models, we find that most of the models of this type are collected close to the boundary between the ${{\mathbb{Z}}}_{8}$ and ${{\mathbb{Z}}}_{12}$ regions, described by equation (10). This behavior shows that the observations made for MSSM-like models are repeated for other promising compactifications.

Zoom In Zoom Out Reset image size

We verified the robustness and replicability of our observations, thus disentangling them from a mere stochastic result. We tested different trainings, run on different hardware, by varying initialization weights while keeping everything else fixed. All the tests showed that, although the latent space had variations, there was always a clear boundary between models arising from different geometries and most of the promising models appeared located near to it. For the curious reader, some of our tests can be found in our auxiliary material [98].