Structural and topological phase transitions in Se doping-controlled intrinsic magnetic topological material FeBi2Te4

Defects profoundly affect the magnetic, electronic structure and topological behaviour of intrinsic magnetic topological insulators. Here, we investigate the magnetic, structural stability and topological properties of different Te atomic sites in the intrinsic magnetic topological insulator FeBi2Te4 with Se substitution of the FM-z magnetic order. When Se replaces the outermost site of the septuple layer (the Te atomic layer connected by van der Waals bonds), the Se–Bi bond length formed with its nearest neighbour Bi is drastically shortened and the lattice constant and volume contract accordingly. We speculate that this defect-induced chemical bond enhancement is related to the strong electronegativity of Se over Te. In contrast, the system has lower formation energy, a more stable structure, and enhanced magnetic moment when Se replaces the Te atomic layer inside the septuple layer. Also, we reveal a variety of topological phase transitions due to substitution defects. FeBi2Te4 is considered to belong to the z2×z4 topological classification of higher-order topological insulators with z 4 = 2. The system after Se substitution can be transformed into Weyl semimetals, strong 3D topological insulators, and unknown topological materials without symmetry-base indicators, respectively.

Defects are a common and essential factor in the experimental process. Modulation by defects often leads to interesting electronic structures and topological properties. Therefore, we investigated the effect of Te on the system's energy, electronic structure and topological properties at different lattice sites in FBT in the Se element substituted ferromagnetic state. This leads to the prediction of the topological phase transition due to the doping of Se elements at different equivalence sites. Further, the theoretical basis for defect enhancement of FBT stability and modification of FBT topological properties is given.

Methods and structures
The First-principles calculations were performed in Vienna ab initio simulation package [39,40] by using all-electron projected augmented wave [41] method and Perdew-Burke-Ernzerhof type generalized gradient approximation [42] exchange-correlation function. The cutoff energy chosen for the valence layer wave function based on the plane wave expansion is 350 eV. Spin-orbit coupling was taken into account in all our calculations. For Fe atom, the on-site coulombic and magnetic moment were chosen to be U = 4 eV and 4 µ B , respectively. The conjugate gradient algorithm relaxes the structure until the energy on each atom is less than 1 ×10 −5 eV. The Monkhorst-Pack k point grid centred on Γ is chosen to be 9 × 9 × 3, and the force convergence criterion on each atom is set to 1 ×10 −2 .
FBT belongs to the R3m space group, which consists of the Te-Bi-Te-Fe-Te-Bi-Te septuple layer. Site2 and site3 are occupied by Te atoms between the septuple layers, and the two septuple layers are connected by a Te-Te van der Waals bond. Sites 1 and 4 are occupied by Te atoms within the septuple and form ionic bonds with the neighbouring Bi atom. Formation energy is calculated as follows [43][44][45]: Where ∆H f is the formation energy of every system, E total indicates the calculated total energy of six models, µ is the chemical potential of the species (Fe, Bi, Te, and Se), and n denotes the number of atoms of the corresponding element in the system. Figure 1 gives our calculated structures of Se substituted for Te in different ways in FBT (FeBi 2 Te 4 ), M1 (Se substituted for site2 and site3), M2 (Se substituted for site1 and site4), M3 (Se substituted for site1 and site2), M4 (Se substituted for site1 and site3), FBS (FeBi 2 Se 4 ).
We reproduce band structure obtained by constructing a tight-binding model by use maximum local Wannier functions [46,47]. Using surface Green's function approach [48][49][50][51], the Wannier Charge Centre and surface states of the semi-infinite surfaces are performed by WannierTools package [52].

Results and discussion
The optimized lattice parameters, volume and bond lengths of these six models are shown in table 1. Bond lengths between Te(Se) with corresponding sites i = 1-4 and their nearest neighbour Bi atom are labelled as d i (i = 1-4). Moreover, the distance between the outermost Te atoms (sites 2 and 3) of the two septuple layers is denoted as d 5 . The substitution of Se atoms drives down the lattice constants a, b, c and volume of the FBT material, especially after the total substitution into the FBS system. This substitution defect-induced lattice contraction is associated with the stronger electronegativity of Se than that of Te. The stronger electronegativity will facilitate the formation of stronger bonds with Bi, which is also the result of the reduced Se-Bi bond lengths after Se substitution, especially for the Te site 2 and site 3 on the outer septuple layer (as shown in table 1, d 2 and d 3 for M1, d 2 for M3, d 2 and d 3 for MBS). However, for M2, the Se-Bi bond lengths indicated by d 1 and d 4 are slightly larger than the Te-Bi bond lengths indicated by d 2 and d 3 . Thus the Se substitution of sites 1 and 4 within the septuple layer has a weaker effect on the bond length of the system than the substitution of sites 2 and 3 between the septuple layers. For the outermost site 2 and site 3 atomic distance d 5 of the septuple layer, the d 5 decreases more obviously when Se is doped with these two sites. For example, the d 5 of M2 and MBS systems are only 5.495 Å and 5.434 Å, respectively.
As shown in table 2, all systems have a significant magnetic moment in the z-direction for the FM-z configuration. Therefore, compared to the FBT system, the Se substitution of the Te sites in the inner septuple is beneficial to improving the system's magnetic moment and structural stability. In contrast, substituting the outermost Te sites in the septuple will weaken the magnetic moment and reduce the structural stability of M1 (see the information on the magnetic moment and formation energy in table 2). From the point of view of the formation energy, we know that Se prefers to replace the inner Te atomic sites bonded to Fe (sites 1 and 4) and hates to replace the outermost Te sites of the septuple layer bonded by van der Waals forces (sites 2 and 3). This phenomenon suggests that Se, due to its small ionic radius, is sufficient to shuttle into the   interior of the FBT block and bond with Fe ions but does not enjoy staying between layers. Here, we achieve the alternative substitute defects to regulate the system's structural stability and magnetic moment.
To investigate the effect of Se Te defects on the electronic structure of FBT, we calculate the projected energy band structures of these six systems, which are shown in figure 2. The intrinsic FBT has an indirect band gap of 0.0343 eV (see the black font in figure 2(a)), and the system will change to a direct band gap at Γ after Se substitution (green font in figures 2(b)-(f)). Se doping can effectively enhance the electron leap because the electron leap in the Se-doped system only needs to overcome the energy and does not have to overcome the momentum like FBT also needs to overcome. Moreover, the local band gap at Γ is reduced after the substitution, especially M2 is almost zero. As the enlarged plots in figures 2(b)-(f) show, Se-p will replace Te-p to occupy the contribution near Γ. Furthermore, Se doping with different combination sites will have a significant modulation effect on the energy band structure. When the two Te sites (sites 2 and 3) between the septuple layers are entirely replaced, the topological property of energy band inversion disappears (figures 2(b) and (f)). As in figure 2(c), when Se replaces the sites inside the septuple layer, the energy band structure will behave as a Weyl semimetal with a global zero band gap. At the same time, there is a narrow localised band gap of 0.015 eV at Γ, and the Bi-p component is inverted to the second valence band below the Fermi energy level (see figure 2(c)). For the M4 system, the characteristic Bi-p inversion to the second valence band remains. For the M3 system with asymmetric substitution, Bi-p will invert to the third

System
Symmetry-based indicators Chern number No symmetry indicator C = 0 M4 No symmetry indicator valence band, further burying the topological surface state into the bulk band structure. So, Se doping can induce a topological phase transition of the FBT system from an apparent energy band inversion (figure 2(a)) to no inversion (figures 2(b) and (f)). However, the specific topological classification for this family of systems is not clear. The FM-z ferromagnetic sequence will break the time-reversal symmetry of the system and, therefore, cannot be directly characterized by the Z 2 topological index. So we used the irvsp program to calculate the topological classification of these six systems. Among them, the FM-z configurations of FBT, M1, M2 and MBS have the magnetic group of MSG 1331 (space group No. 166), and their topological properties can be classified by z 2 × z 4 . In contrast, the FM-z configuration of M3 and M4 has the magnetic group of MSG1297 without integer and half-integer spin, so there is no symmetry indicator, as shown in table 3. This is because the M3 and M4 systems have a defect-induced break in structural symmetry resulting in a phase change from space group 166 to space group 160 of the FBT. Combining with table 3 and figure 3, the symmetry index of z 4 = 2 for the FBT and M2 systems may imply that they have a non-axial higher order topological insulator (HOTI) phase in the helical hinge state. Meanwhile, the Chern number C = 1 means that M2 has a defective or pressure-regulated Weyl semimetal phase, which will be described in later. Although the energy band inversion is not apparent, the odd number z 4 = 3 obtained from the symmetry indicator calculation indicates that the M1 and FBS systems are typical 3D topological insulators.
Li et al found the presence of a topological axion state in the antiferromagnetic order of the bulk MnBi 2 Te 4 , which is a type-II magnetic Weyl semimetal. To verify that the M2 system is a Weyl semimetal, we give two Weyl points (W and W ′ ) with opposite chirality in momentum space, as shown in figures 4(b)-(e). The Fermi arc well demonstrates the Weyl semimetal character of the system in figure 4(e). Thus, we achieve a topological quantum phase transition from a topological insulator to a Weyl semimetal by replacing the two Te sites inside the septuple with Se.

Conclusion
In short, based on first-principles calculations, we investigated the qualitative modulation of FM-z order FBT materials' magnetic, structural stability and topological properties by a high concentration of Se doping. A total of six configurations are available, namely FBT (pure FeBi 2 Te 4 ), FBS (pure FeBi 2 Se 4 ), and the semi-doped system FeBi 2 Te 2 Se 2 (M1, M2, M3, and M4). We found that Se doping of the outermost layer of the septuple (site2 and site3) would substantially reduce the bond length between the doping sites and their nearby Bi atoms, thus enhancing Se-Bi bonding but reducing the structural stability, corresponding to the application of compressive stress. This defect-induced contraction of the crystal structure is mainly due to the stronger electronegativity of Se than that of Te. The Se doping of the innermost Te sites (site1 and site4) will enhance the magnetic properties of the system and strengthen the structural stability. In addition, different substitution defects will induce structural phase transitions (from space group 166 to space group 160) and multiple quantum topological phase transitions (from z 2 × z 4 3D strong topological insulator to HOTI phase or Weyl semi-metal phase, and also two symmetry-breaking heterogeneous phases with no symmetry indicators). Our study provides a critical theoretical basis for the theoretical and experimental defect modulation of the magnetic, structural stability and topological properties of the intrinsically magnetic topological insulator FBT.

Data availability statement
The data that support the findings of this study are available upon reasonable request from the authors.