A Fourier-transformed feature engineering design for predicting ternary perovskite properties by coupling a two-dimensional convolutional neural network with a support vector machine (Conv2D-SVM)

In computational material sciences, Machine Learning (ML) techniques are now competitive alternatives that can be used in determining target properties conventionally resolved by ab initio quantum mechanical simulations or experimental synthesization. The successes realized with ML-based techniques often rely on the quality of the design architecture, in addition to the descriptors used in representing a chemical compound with good target mapping property. With the perovskite crystal structure at the forefront of modern energy materials discovery, accurately estimating related target properties is even of high importance due to the role such properties may have in defining the functionalization. As a result, the present study proposes a new feature engineering approach that takes advantage of both the direct ionic features and the periodic Fourier transformed reciprocal features of a three-dimensional perovskite polyhedral. The study is conducted on about 27,000 ABX3 perovskite structures with the stability energy, the formation energy, and the energy bandgap as targets. For accurate modeling, a feature-extracting two-dimensional convolutional neural network (Conv2D) is coupled with a prediction-enhancing Support Vector Machine (SVM) to form a hybridized Conv2D-SVM architecture. A comparison with previous benchmark evaluations reveals appreciable improvements in modeling accuracy for all target properties, particularly for the energy bandgap, for which the feature extraction approach yields 0.105 eV MAE, 0.301 eV RMSE, and 93.48% R2. Besides, the proposed design is further demonstrated to out-perform other similar periodic feature engineering approaches in the Coulomb matrix, Ewald-sum matrix, and Sine matrix, all in their absolute eigenvalue forms. All preprocessed data, source codes, and relevant sample calculations are openly available at: github.com/chenebuah/high_dim_descriptor.


Introduction
Perovskites have evolved into versatile energy materials with multifunctional properties including superconductivity [1], piezoelectricity [2,3], ferroelectricity [4][5][6][7], optoelectronics [8,9], magnetoresistivity [10], and catalysis [11], among others. Their multifunctionality stems from their diverse stoichiometry and geometrical distortion, as several chemical elements across the periodic table can occupy distinctive ionic sites within the crystal structure. Some examples of common perovskite stoichiometries include the ternary ABX 3 , double A-site, double B-site, and hybrid organic-inorganic, as well as antiperovskites. The ternary ABX 3 structure constitutes the most prevalent of perovskite compounds thanks to their well-defined ionic arrangement of constitutive chemical elements compared to their more complex stoichiometrical counterparts.
In the ideal cubic-ABX 3 configuration (figure 1), the B-site cation is coordinated by six X-site anions in a cornersharing octahedron, whereas the A-site cation is situated at the center of a twelve-fold coordination system, all encompassed in a three-dimensional polyhedral [12]. Other derivative forms emerging from the ABX 3 stoichiometry predominantly exist in the non-idealized form. The possibility of such non-idealized geometries advantageously creates distortional complexities that are accompanied by specially tailored properties in diverse engineering applications.
Considering their great potential for a large variety of applications, estimating the properties of perovskites is currently of great interest to researchers. Conventionally, perovskite target properties are determined by experimental trials/synthetizations or via first principle (ab initio) quantum mechanical deterministic methods, such as Density Functional Theory (DFT) and Molecular Dynamics (MD) [13,14]. In recent years however, Machine Learning (ML) techniques have emerged as suitable and inexpensive alternatives due to their proven predictive reliability. As applied in computational physical sciences, ML methods are generally based on solving a predefined target-forward challenge or generative-inverse problem, which could be in the form of supervised [15][16][17][18][19][20][21], semi-supervised [22,23] or unsupervised learning [24,25]. The successes realized with ML in target property prediction are often related to the quality of the distinguished features used to describe the material, in addition to the ML design architecture for accurate modeling. Among some relevant perovskite target properties of interest to scientists and engineers are the stability energy, formation energy, and energy bandgap. The stability energy (in some cases referred to as the energy above convex hull) indicates the thermodynamic state of a structure with respect to decomposition at the defined phase composition [26]. The formation energy is the energy required in forming a chemical structure from a disintegrated form and is necessary for developing phase diagrams [13,26]. The energy bandgap quantifies the energy region between the valence band and the conduction band, and as such, is a useful indicator in characterizing the electronic state of a material (i.e. insulating, semiconducting or conducting material) [27]. Even though the aforementioned targets have been previously investigated and reported in existing ML literature, there are great needs and potentials for further predictive improvement, given the combined effect of newer descriptor designs and state-of-the-art ML techniques. In this regard, table 4 provides an outline of some recent target modeling results and reports on the respective modeling prediction technique. In a study conducted by Li et al [16], features that describe the bondvalence properties of ABO 3 compounds were used to predict the formation energy and bandgap at 0.087 eV/ atom MAE and 0.384 eV MAE, respectively, as trained on a gradient boosting machine. Moreover, it was demonstrated in our previous work that modeling the formation energy of ABX 3 and A 2 BB'X 6 perovskites using features that include the convex hull energy as an additional parameter can considerably improve the prediction capability up to 0.055 eV/atom MAE [17]. In a different study by Xie et al [18], a crystal graph convolutional neural network (CGCNN) was developed for predicting the formation energy and bandgap at 0.039 eV/atom MAE, and 0.388 eV MAE, respectively, as performed on a general inorganic crystal dataset. These are just a few examples of studies that continuously contribute in pushing the ML predictive boundaries towards first principle or experimental accuracy levels with the possibility for further modeling improvements emerging from newer methodologies.

Proposed modeling methodology
To contribute in further advancing modeling methodology and improving prediction accuracy, the current study proposes an advanced feature engineering method that combines two different machine learning algorithms. Building on the invertible Fourier Transformed Crystal Property (FTCP) representation [22], the developed descriptor design explores the Fourier transformed reciprocal lattice space of a periodic crystal structure and incorporates additional features for improved results. Although the case study used in the analysis focuses on the ABX 3 ternary perovskite structure, the proposed concept can be extrapolated to other forms of stoichiometrical inorganic compounds. The descriptor used for each perovskite compound is uniquely organized into a high-dimensional input image that is reminiscent of a gray-scale picture format in image recognition and object detection. The high-dimensional input image consists of both real/direct and Fourier transformed reciprocal properties that is based on demonstrated solid-state physics principles. Due to the imposition of the Fourier representations, the descriptor is therefore periodic in form and mimics the longrange atomic ordering in a crystal lattice. For accurate target modeling, a new feature engineering approach is architectured that involves a feature-extracting two-dimensional convolutional neural network (Conv2D) [28] and a prediction-enhancing Support Vector (Regression) Machine (SVM) [29,30] model. On final training, the results reveal updated and improved performance scores on the stability energy, formation energy, and bandgap. The results are compared to some benchmark evaluation and to three other commonly used periodic descriptors in Coulomb-matrix [31,32], Ewald-sum matrix [21,33,34], and Sine matrix [21], all represented in their periodic eigenvalue forms. In addition, the simulation exercise is re-performed using the original descriptor design that emerges from the FTCP elemental property matrix, thereby illustrating the pronounced effect of incorporating more features into the FTCP. The achieved results demonstrate the effectiveness of the proposed design for accurate target modeling of inorganic perovskite crystal structures.

Dataset generation and preprocessing
The present study harnesses data from the Open Quantum Materials Database (OQMD). The OQMD platform openly provides over a million entries of proven DFT calculated thermodynamic and structural properties of inorganic crystal structures [13,35]. The OQMD was primarily chosen for core experimentation due to the robust number of trainable ABX 3 compounds available on the platform, which is needful for deep learning models. Approximately 28,000 ABX 3 structures were initially generated and extracted from the database. The dataset consists of both International Crystal Structure Database (ICSD) compounds and DFT generated compounds. The first preprocessing step involves the removal of data samples having incomplete or wrongful entries. As such, samples with missing information as related to the formation energy, bandgap and/or stability energy were ejected. The next screening process removes certain entries which possess the unfavorable potential of obscuring the modeling accuracy. The threshold used to screen such samples was set at formation energy and/ or stability energy with values more than 5 eV/atom. Besides, these entries are highly impossible to synthesize given their critically unstable state. In the final screening process, only compounds with number of atoms in all crystallographic sites no more than twenty (i.e.  n 20 atoms ) are selected. The search space is limited to 20 atoms due to the relatively lower number of available samples beyond this value. Overall, the data cleaning process resulted into 27,587 ABX 3 compounds with a diverse mix of different A-, B-, and X-site chemical elements, all spanning across the periodic table, including lanthanides and actinides. The dataset also contains a good proportion of ionic-swapping inverse-or anti-perovskites [36]. Figures 2(A)-(C) are histogram plots illustrating the occurrences of contributing chemical elements, as it relates to site-1 cations, site-2 cations, and site-3 anions, respectively, from the three distinctive ionic sites in the ABX 3 compound. As can be observed, the cationic chemical elements that are associated with sites-1 and -2 are fairly distributed among the dataset. For anionic site-3 however, oxide-perovskites dominate with about 70% of all samples in the dataset. Other typical anions present in the dataset include halides such as fluorine, chlorine and bromine, and they occur relatively less at 3.8%, 3.

Fourier-transformed reciprocal crystal space
A crystal system that is periodic in real space with periodicity p, is also periodic in the k-momentum space with periodicity / p p 2 .The k-momentum space is also referred to as the reciprocal space, and can be derived from the real lattice by implementing a Fourier transform [37]. Based on the conservation of crystal momentum, the Fourier transformation emerges into a spatial periodicity of atomic arrangements in the reciprocal space, which results in the expression of a defined structure factor given as [38]: G and X are integer points in the reciprocal and real lattice, respectively. ( ) S G is the structure factor as a function of the reciprocal space, and ( ) V X is the periodic potential or electronic density in a unit cell to be summed over all atoms. Moreover, the periodic potential can be approximated by using a periodic property that belongs to the crystal lattice. Emerging from the FTCP representation [22], the periodic scattering potential is addressed by replacing the conventional atomic form factor with the discrete properties of the ionic elements that occupy the A-, B-and X-sites in a perovskite polyhedral setting. On substituting the periodic potential with the elemental properties for a specific hkl crystal plane, the structure factor given in equation 1 can be rewritten in the form: is used to simplify equation 2 by eliminating the imaginary unit in the exponential term [22]. As a result, the structure factor is reduced to the form given in equation 3: The structure factor, as formulated using equation-3, is therefore used as inputs in the reciprocal space. A brief sample calculation based on equation 3 is demonstrated in supplementary (section S2) for KNO 3 perovskite structure with five atoms in the unit cell. Moreover, three other reciprocal features are introduced in the proposed descriptor design: (1) the reciprocal lattice vectors; (2) the magnitude of the reciprocal vector | | G min normal to a crystallographic plane; and (3) the shortest distance between similar planes, d .
hkl The real lattice vectors are transformed into the reciprocal lattice vectors by applying and satisfying the Kronecker delta function ( ) d ij from solid-state physics [38]. Similar crystal structure descriptors that also impose periodic conditions include: Ewald sum matrix, sine matrix, and modified coulomb matrix. In this research, the predictive performance of the developed model is equally compared with the aforementioned matrices.

ML training via convolutional neural network (Conv2D) coupled with an auxiliary model
The convolutional neural network learns the regressive forward problem by performing non-linear operations in convolutions on the high-dimensional input image [28]. In this study, we train a deep Conv2D model to preliminarily predict the target properties from the input image. By doing so, the hidden layers connected within the architecture of the network are effectively optimized using back propagation as the respective weights are continuously updated to minimize the cost function. We therefore extract the low-dimensional features prior to the final target layer and use them as input for further training in a different algorithm. The feature extraction process is effected by customizing a callable layer within a functional keras API [39] on a tensorflow backend [40] of the Conv2D model. In general, a neuron-like processing unit can be described in the form [41]: Where E is the pre-trained target property; w are the updated weights that are associated with each hidden layer i; X represent the input features to the unit; b is a bias; and AE is the activation function. For the concerned hidden neuron (feature extraction layer) in perspective, the activation function is thus non-linearly effected by introducing a tanh function given the equation 5: h represents the feature extracted layer, which is optimized by enabling a gradient descent with respect to a loss function  and learning rate h over several epochs, as the weights are constantly updated by back-propagation.
Considering the learned target property in particular, the extracted unit possesses the high-quality attributes evolving from the high-dimensional parent image. Figure 4(C) reveals an example of the extracted lowdimensional feature representation for NdGaO 3, which was obtained from the pre-training process on the formation energy. The low-dimensional feature is one-dimensional (first-rank tensor) in size with vector length ( )  : 1 10 10 and is bounded with values between 1 and −1 due to the non-linear effect of the hyperbolized tangent function (tanh) [42] used in activating the extracted hidden layer. For better modeling accuracy, the extracted unit is further analysed using five different auxiliary enhancement models that are comparatively studied on the Conv2D. For measuring the accuracy of each evaluation process, regressive metrics are used in the Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and Coefficient of Determination (R 2 ). For more details on ML architecture, see section S3 in supplementary.

High-dimensional input image (descriptor)
The high-dimensional input image used to describe each perovskite material is structured into two separate blocks: (1) Direct Space Features (DSF) and (2) Periodic Reciprocal Features (PRF). Both blocks are adjoined along their columns to form a two-dimensional matrix representation. The DSF block contains both continuous (real-values) and discretized (one-hot encoded) attributes. The DSF block is designed using direct features in the Table 1. All discretized ionic features used in the present study to describe a perovskite sample in the direct space feature (DSF) block. For each one-hot encoded feature vector of an atom in a unit cell, Fourier transformation is used to project the direct/real space property into the periodic reciprocal crystal space. real space that characterizes the complete crystal structure as a whole and the individual atoms occupying the ionic sites. The continuous features, as originally provided by the database include: lattice base vectors , calculated inter-axial angles from the base vectors (α, β, γ), and the fractional atomic coordinates of the constitutive atoms in the unit cell. Based on the present design, the maximum number of rows allocated to the fractional atomic coordinate in the DSF block coincide with the maximum number of atoms considered in the study (i.e.  n 20 atoms ). For the discretized attributes in the DSF, some modifications are made to the original Fourier Transformed Crystal Property (FTCP) representations. Namely, unlike the FTCP that considers only the ionic chemical properties similarly applied in the Crystal Graph Convolutional Neural Network (CGCNN) model [18], the present descriptor design infuses new features into the DSF discrete block that fit the set objective and available resources. The newly introduced features are: average ionic radius, polarizability, specific heat, and thermal conductivity. They essentially address crucial thermo-chemistry qualities in a crystal [43], which are missing in the standard CGNN. Moreover, appending these new properties ensures that functional properties, wholly used to define some specific applications in a perovskite material, are not omitted. Table 1 summarizes all ionic elemental features transformed into discretized one-hot encoded vectors. Considering the PRF block in particular, all aforementioned discrete ionic features are projected into the reciprocal space using equation 3. Given a unit cell, the constitutive ions are Fourier-transformed using their respective fractional atomic coordinates on a specific set of crystallographic hklMiller indices. In addition to the Fourier-transformed ionic features, we further project the reciprocal lattice vectors and angles, the magnitude of the reciprocal vector | | G min normal to a crystal plane, and the shortest distance between similar crystallographic planes d hkl , which are also missing in the standard FTCP representation. By including the reciprocal lattice vectors in the high-dimensional image, we infuse analogous variables used as inputs in first-principle calculations, and also in the analytical determination of the actual atomic form/scattering factor, as approximated by a sum of Gaussians [44][45][46]. Overall, 57 maximum crystallographic planes are considered in the feature projection process (i.e.  n 57 max ) with the absolute summation of all hkl plane integers no more than three. By combining both DSF-PRF block arrays, the high-dimensional input image becomes complete for modeling investigation. The maximum dimension the input image can be organized into is a ( )154 60 1 gray-scale picture format, corresponding to (´ímage height image width number of channels). Figure 3 illustrates the stacking arrangement of each feature in the high-dimensional image (for more details, see sections S1 and S2 in supplementary). A general example is provided in figure 4, as specific to the NdGaO 3 (NGO) perovskite compound. The rare Earth based NGO material is a well-studied paramagnetic insulator and is extensively utilized as substrates in the fabrication of high-temperature superconducting thin-films [47,48]. Figure 4(A) is a ball-and-stick model, displaying the three-dimensional geometry and interatomic bonding of the NGO unit cell. Emerging from the proposed descriptor design, the NGO compound is represented highdimensionally using figure 4(B). Bright intensities, as observed on the input image, represent higher pixel values from the normalization of features.
It should be noted that representing entry samples using the proposed descriptor form is unique to a particular perovskite structure due to the imposition of the distinctive fractional atomic coordinates used in constructing the Fourier-transformed reciprocal space. Given the experimented OQMD dataset for instance, only about 8000 of all considered samples are non-duplicate entries, whereas the remaining are crystal polymorphs. In as much as polymorphic structures may have identical chemical formulas (or stoichiometry) with similar perovskites, their fractional coordinates, lattice vectors, and number of atoms in the unit cell are entirely different. With these key differences, the high-dimensional input image exclusively describes a particular perovskite sample based on their distinctive crystallographic features. Furthermore, the descriptor concept, as applied in this study, can be broadly expanded to other forms of general inorganic crystalline materials that comes with different stoichiometries (e.g. quaternary compounds). Unlike in the investigated case with the DSF column size reflecting the distinctive ionic sites of the ABX 3 setting, the descriptor used in a different stoichiometric case will have to account for the total number of original ionic site positions in a crystal lattice, in addition to the maximum number of constitutive atoms in the unit cell. Based on the strong effect of the PRF space, the target mapping quality, as obtained from the modified descriptor form, is expected to reproduce similar results as presented in the current study.
3.2. Results and discussion from the preliminary training exercise using the Conv2D model In order to extract the high-quality attributes from the input image, the Conv2D model is first trained to predict the respective target property. As such, for all pre-training purposes, the data is split into three sets: 60% training, 20% validation (network optimization), and 20% for testing on a traditional holdout set. Table 2 reports the standardized errors on all targets as evaluated on the testing set. In general, as the number of hkl planes increases in the Fourier-transformed space, the modeling accuracy considerably improves. By comparing the error evaluations between cases without planes and with maximum planes, the accuracy in stability energy improves   On taking a closer look into the regression fittings in figures 5 and 6, the best fitting performance is realized at 97.19% R 2 on the formation energy when the maximum 57 crystal planes are all used in the periodic projection of direct features. Similarly, for the stability energy, the highest accuracy is obtained at 89.25% R 2 . For the bandgap however, higher marginal errors with R 2 at 84.85% can be observed when compared to their other target property counterparts. Upon reproducing the preliminary results based on the same maximum number of projected crystal planes (i.e. 57) and the original ionic property features (from the FTCP descriptor) yields slightly lower accuracy in standardized measurements across all target properties. As previously explained, the original FTCP descriptor design does not include properties such as the average ionic radius, polarizability, specific heat, thermal conductivity, reciprocal lattice vectors and angles, and the magnitude of the reciprocal vector | | G min normal to a crystal plane. As reported using table 2, the FTCP ionic makeup predicts perovskite targets at 0.094 eV/atom, 0.114 eV/atom and 0.202 eV, corresponding to about 12%, 16% and 10.4% in MAE prediction inaccuracies for the stability energy, formation energy and bandgap, respectively. This recognizes the considerable effect of the additional features incorporated into the high-dimensional image, and recommends their usage for better target-modeling result.
Generally, accurately estimating the bandgap has been a major challenge to researchers, which may in part be due to the obstacle of bandgap undervaluation from deterministic DFT simulation [53]. In the present case study, an attempt is made to further improve the bandgap's predictive capability by hybridizing the Conv2D with a coupled auxiliary model in the sequel feature-extraction arm. By doing so, even deeper trends that are associated with bandgap distribution can be further analyzed. In addition, the capability of the used descriptor to accurately classify bandgap functional properties is demonstrated for two popular classes of importance in electronic applications: (1) metallic/infinite ( ) = E 0 g and (2) non-metallic/finite ( > E 0 g ) bandgap perovskites. Such pre-classification tasks could assist in streamlining bandgap-targeted materials for further experimental investigation [16][17][18].

Results and discussion from the coupled feature extraction approach
For further training, the extracted low-dimensional feature attributes of the parent input image is used as input into auxiliary ML models. The dataset is reorganized into a ( ) ( ) =Ḿ N perovskites features 27, 587 10 matrix; ten features corresponding to the high-quality attributes upon successful extraction from the Conv2D dense (hidden) layer. The modeling accuracy is broadly evaluated on a five-fold cross validation exercise. At first, five auxiliary models were selected and compared based on their relative performances. The considered models include: Gradient Boosting Regression (GBR) [54], Light Gradient Boosting Regression (LGB) [55], Random Forest Regression (RFR) [56], Support Vector (Regression) Machine (SVM) [29,30], and eXtreme Gradient Boosting (XGB) [57]. Upon comparison based on standardized accuracy scores among all models, SVM is identified as the preferred option for predicting target variables. As illustrated in table 3 and figure 8(A), hybridizing the two-dimensional convolutional neural network with the auxiliary support vector machine (i.e. Conv2D-SVM) out-performs its peers. On looking into the average cross-validated scores for stability energy, formation energy, and bandgap, the MAE results for Conv2D-SVM are optimized at 0.05 eV/atom, 0.058 eV/ atom and 0.105 eV, respectively. Likewise, RMSE is updated at 0.116 eV/atom, 0.133 eV/atom and 0.301 eV in the same order. Moreover, the Conv2D-SVM is a well-established fusion design as related to image-based recognition [58,59]. In such analysis, the Conv2D model is attributed to provide strong featuristic representation of the embedded pixels, whereas SVM systematically analyzes the shallow structures emerging from the feature-extracted layers. The present study therefore points to the application potential of the Conv2D-SVM architecture in the field of computational materials science, and highlights its advanced predictive capability on target property prediction. Considering the bandgap in particular, all re-evaluated standardized metrics can be seen to greatly improve. As illustrated in figure 7, the updated regression fitting based on feature extraction for the bandgap improves to 93.48% from the previous 84.85%. Table 4 shows some benchmark results for formation energy and bandgap prediction for general inorganic crystalline materials. It can be seen that predicting the band gap (in particular) using the Conv2D-SVM model provides improved accuracies compared to other descriptor forms, highlighting on the present study contribution in the field.

Comparison with other periodic forms of crystal structure representations
Imposing periodicity with crystal structure descriptors has been found important in accounting for the longrange atomic order in crystalline materials [21,64]. The descriptors used in the present study models the periodicity of an inorganic crystal by implementing a Fourier-transform on unit-cell ionic properties in the reciprocal lattice space. However, other forms of periodic representations exist and have been applied in past studies for predicting target properties. Therefore, the modeled performance obtained are compared to other (2) Ewald-sum matrix; and (3) Sine matrix, all with self-imposed periodic boundary conditions. The aforementioned matrices are strictly analogous to first principle Schrödinger equations due to their atomistic structure-property relation, which is typical in DFT computations. For instance, the Coulomb matrix [31] encodes the constitutive atoms and inter-atomic separations of a finite-system into a two-dimensional square array using identical equations that finds root in solving the Coulomb potential [32]. The Coulomb matrix representation can be generalized in its extended periodic form to account for the electrostatic interaction between neighboring unit cells [21,64]. The Ewald-sum representation is an extension of the Coulomb matrix for periodic systems, and models the electrostatic interaction between constitutive atoms of a crystal, by eliminating inter-dependence between interatomic distances [21,33,34]. The Sine matrix rather encodes the properties of a periodic system from the respective coulombic interaction between atoms using a sine function [21]. Therefore, all perovskite samples in the dataset are described in this study using the  eigenvalue representations of the Coulomb, Ewald sum, and Sine matrix forms. Besides, representing crystals in their differentiable eigenvalue form is suggested to be atomically invariant to translation, rotation and symmetry of neighboring atomic positions, at the expense of uniqueness [31,65,66]. Figures 4(D), (E) and (F) show an example of the one-dimensional eigenvalue spectra on the Coulomb-matrix, Ewald-sum matrix, and Sine matrix, respectively, for the NGO compound. Relative to the proposed feature engineering approach, the eigenvalues are now substituted as the input variables for ML training in the hybridized convolutional neural network -SVM setup. However, the convolutional network used for this training purpose is remodeled to be one-dimensional (i.e. Conv1D), given that the eigen-representative inputs are simply first order tensors of real  Table 4. Some examples of benchmark evaluation from past studies as related to formation energy and bandgap predictions. The higher accuracy of the feature extraction approach can be seen, particularly for the bandgap. numbers and not second order. Prior to training moreover, the eigenvalue features are combined with a new set of generalized features. The purpose of combining both feature sets is to improve the predictive performance of the periodic representation. The generalized features build on the real physicochemical properties that are associated with the ABX 3 perovskite. Moreover, they have been previously used in past literature as targetmapping descriptors [16,17]. By combining the generalised features with the eigenvalue representations, the descriptor used to represent a crystal structure is practically periodic. Therefore, the new input dataset for experimentation is organized into:  table 5 and is performed on five-fold cross-validation. As can be observed from the table, representing crystal structures based on the developed feature extraction process produces better results when compared to their periodic eigenvalue counterparts. A comparative display of the mean and standard deviation from the cross-validation result is illustrated in figure 8. On the formation energy for example, MAE scores are estimated at 0.247 eV/atom, 0.232 eV/atom and 0.245 eV/atom for Coulomb, Ewald-sum and Sine periodic forms, respectively. Moreover, in a similar study by F Faber et al [21], MAE generalization error values on the formation energy were reported at 0.64 eV/atom, 0.49 eV/atom and 0.37 eV/ atom for Coulomb-like matrix, Ewald-sum matrix and Sine function matrix, respectively. The better performance achieved in the current study is primarily due to the concatenation with the generalized features. This highlights the considerable capability of the used Fourier-transformed reciprocal features in periodically describing crystal structures, in addition to the novel feature extraction approach. The computations used to determine the eigenvalue periodic matrices were enabled by DScribe [64], which is an open-source library of descriptors for machine learning in materials science. All calculations related to the atomic coordinates were done in Atomic Units. The source codes for reproducing the calculations, in addition to a breakdown of the generalized features, are made openly available in supplementary (see sections S4 and S5).

References
3.5. Pixel-importance based on hkl crystallographic planes in the Fourier-transformed space In this section, the individual crystallographic planes used in the Fourier-transformation and their relative importance in predicting all considered target properties are further analysed, allowing to investigate the crucial role some crystallographic orientations may play with respect to perovskite formability and electrical behavior. For this purpose, specific Miller indices (hkl planes) are assessed with respect to their ability to provide correlations on the stability energy, formation energy and bandgap. Figure 9 shows the pixel-importance inspection on the projected crystal planes for the different targets as conducted on a gradient boosting machine. Hot pixel intensities denote the superiority of that feature scalar in mapping target properties. As expected, all images are similar in terms of their pixel contrasts regardless of target property. This is because all investigated Table 5. Standard target accuracy measurements from the feature extraction process compared to other forms of periodic descriptors for crystal structure representation. All measurements are reported based on the average scores from a 5-fold cross-validation process. properties are physically inter-related. For instance, estimating the stability energy via thermodynamic calculations has been shown to be possible by constructing a convex hull around the formation energy, or by simply relating the convex hull distance to the formation energy [26,67]. Moreover, it has been reported that using the formation energy as an additional parameter for ML training will positively influence bandgap prediction [16,17]. In retrospect, the rationale behind the identified planes emerges from the structure-factor calculations. As previously discussed, the assumed ansatz for ionic feature projection is analogous to the periodic scattering potential, which can be approximated as the atomic form-factor. Based on this analogy, the defining planes identified in this study may considerably share some similar characteristics to physical experiments. For example, the amplified intensities in XRD plots indicate which planes are more coherent with Bragg's law, and inform on the type of crystal system in x-ray diffraction experiments [38]. To bridge the gap between theory and experiment, it is important to further investigate the identified planes with respect to potential valuable practical Figure 9. Pixel-importance on the Fourier-transformed ionic properties in the reciprocal crystal space, as it relates to the prediction of target properties. The y-axis (image-height) describes the transformed ionic features, whereas the x-axis (image width) shows the specific Miller indices, in the same pattern as the high-dimensional image (illustrated in figure 3). Hot intensities on the image suggests the high relevance of that scalar property in predicting a target. connections in the current ABX 3 case study. The results are summarized in table 7 with focus on significance in perovskite growth/formation, stability, failure mechanism and functionality. For example, the growth of epitaxially stable (Ba,Sr)TiO 3 and Pb(Zr,Ti)O 3 thin-film perovskites proves to preferably take place in the (100) plane [2,68]. Moreover, the (001) plane exhibits high photoresponsivity qualities compared to other planes for CsPbBr 3 isotropic single crystals in optoelectronic applications [9]. As outlined in table 5, the pixel-importance examination also identifies both (100) and (001) planes as determinant. Therefore, the current study suggests the newly identified planes as promising research focus for future experimental studies. This is specifically the case for crystal planes of the form | | | | | | + + = h k l 3 (e.g. (¯) 012 image width: 47) that remain so far only scarcely reported in existing literature across multidisciplinary perovskite or inorganic solid-state research.
3.6. Experimental impact of modeling approach and future study The results presented in study clearly demonstrates the appreciable modeling effect of the proposed Fouriertransformed feature engineering approach used in predicting deterministic perovskite target properties. Accurately estimating such targets can be invaluable to material scientists due to the role they play in defining the formability and functionalization. Customarily, determining these properties requires sophisticated first principle calculations or experimental analysis that are computationally laborious and expensive. In contrast, the proposed Conv2D-SVM model offers a reliable and cost-effective alternative that can be used to determine these properties. Moreover, the study highlights the considerable target-modeling effect of using periodically developed descriptors that are based on the reciprocal lattice space of a crystal. Fourier projecting direct features onto the reciprocal space (Brillouin zones) ensures that analogous features used in first-principle quantum mechanical simulations for obtaining crucial ground state properties are likewise introduced in the present descriptor design. For instance, DFT simulates target properties of many-body systems by approximating a solution to the Schrödinger equation in a self-consistent cycle. The basis sets used to store DFT Hamiltonian charge densities are traditionally resolved using plane waves in the Brillouin point mesh of the reciprocal lattice [78]. As such, priori-supplying surrogate ML models with similar DFT inputs can effectively assist in linking first-principle theories with supervisory machine learning. Comparing the presented results to other contemporary descriptor design (table 4) further demonstrates the impact of the present periodic descriptor design.
Finally, the developed descriptor design could potentially serve in applications related to inverse design simulations for accelerating the discovery of unknown perovskites. Considering the addition of labelled atomic numbers into the Direct Space Feature (DSF) block of the input image, novel perovskites with variational Table 7. Significance of some identified crystal planes with overall good mapping qualities to the considered targets, as related to ABX 3 formability, growth, stability, failure mechanism and functionality.
properties can be identified in the latent space of a generative autoencoder. Moreover, the invertible descriptor design can adapt to discover other forms of perovskite (e.g. hybrid organic-inorganic perovskites), consequential to the reorganization of the DSF block to accommodate such stoichiometrical changes. Using a 2D convolutional autoencoder, figure 10 graphs preliminary learning curves on training and validation sets, as it relates to the reconstruction of all normalized feature embedding associated with the present input image. Upon training for 500 epochs, the decoded perovskite images are demonstrated to be recovered within acceptable error range at 0.066 MAE and 0.103 RMSE, based on standardized evaluation. Table S5 in supplementary provides a full breakdown of constitutive feature error on the reconstruction process of the high-dimensional input image.

Conclusion
The present study demonstrates the effectiveness of the high-dimensional feature engineering approach for higher accuracy in target modeling, with primary focus on the stability energy, formation energy and bandgap of perovskites. The descriptor is inspired by the Fourier transformed Crystal Property (FTCP) representation for invertible material discovery, and is modified to incorporate additional features including: the reciprocal lattice vectors, the magnitude of the reciprocal vector normal to a crystallographic plane, the shortest distance between similar planes, the average ionic radius, the static average electric dipole polarizability, the specific heat, and the thermal conductivity. For target modeling, the descriptor is pre-trained using a two-dimensional convolutional neural network (Conv2D) that is optimized on target properties by back-propagation. The extracted feature is then used as input to a coupled Support Vector Machine (SVM) for further analysis. The following main conclusions can be drawn: (1)The greatest regressive accuracy improvement is achieved for bandgap prediction, with scores of 0.105 eV MAE, 0.301 eV RMSE, and 93.48% R 2 , which represent substantial improvement when compared to existing benchmark evaluations.
(2)The best accuracy scores for the stability energy are 0.050 eV/atom MAE, 0.116 eV/atom RMSE, and 88.60% R 2 , and for the formation energy 0.058 eV/atom MAE, 0.133 eV/atom RMSE, and 98.52% R 2 .
(3)Compared to the most precise eigenvalue representation obtained from three commonly used periodic descriptors (i.e. Coulomb, Ewald-sum, and Sine matrices in their eigenvalue representations), the proposed Conv2D-SVM model yields about 70%, 75%, and 66% greater MAE accuracy on the stability energy, formation energy, and bandgap, respectively.
(4)Sixteen crystallographic planes are demonstrated to best correlate with the targets, and therefore, to be potentially key for modeling the growth/formation, stability, failure mechanism and functionality of perovskites.