Visualizing the in-Gap States in Domain Boundaries of Ultra-Thin Topological Insulator Films

In recent years, the concept of band topology has led to a rich family of topological materials. Among these materials, the quantum spin Hall (QSH) states and quantum anomalous Hall (QAH) states are mostly pursued by the research community. These systems are characterized by the dissipationless one-dimensional (1D) conduction channels at the edge. Experimentally, although there are many proposed QSH and QAH materials, the quantized edge transport has only been realized in a few materials.^[1–9] In addition, the temperature for the edge channels to be dominant is extremely low (<1 K), except for the recently reported QSH transport in cleaved WTe₂ monolayer flakes (∼100 K).^[6] For possible applications, the growth of ultra-thin films of QSH and QAH materials with relatively high working temperatures becomes a critical issue.

The two-dimensional (2D) QSH and QAH materials is signified by the presence of an inverted bulk gap, which ensures the existence of one-dimensional (1D) edge states. The bulk gap normally is in the order of a few tens of milli-electron volts,^[10] suggesting a much higher working temperature compared with the experimental results. Thus, in the thin films of QSH and QAH materials there must be some limiting factors that prevent the edge channels from dominating the transport. Molecular beam epitaxy is a technique to grow films with finely controlled thicknesses, especially ultra-thin films. Most proposed QSH and QAH materials are ultra-thin-film forms of about a few nanometers. For such thin films grown on the substrates, various parameters such as the lattice mismatch and the growth dynamics will inevitably lead to formation of various kinds of structural defects.^[11–16] In this work, we mainly focus on the domain boundary structures. The domain boundaries, unlike other defects, cannot be easily removed by changing growth parameters. In addition, as suggested by the data here, the domain boundaries may introduce 1D conduction channels into the films.

The family of three-dimensional (3D) strong topological insulators (TI) (Bi₂Se₃, Bi₂Te₃ and Sb₂Te₃) are characterized by an inverted bulk gap and a single Dirac-cone like surface state.^[17–20] Various approaches have been proposed to get QSH and QAH states based on the materials in this family or their derivatives. Theoretically, the gap in ultra-thin 3D TIs can be tuned by thickness, electric field or exchange field into the inverted region to realize possible QSH or QAH states.^[1,21–23] Experimentally, by introducing magnetic moments into the ultra-thin 3D TI films, QAH states have been realized in this family with low working temperatures of about 1 K.^[7,8]

In addition, although the growth of high-quality TI films in this family is well developed and now routine,^[24–26] the investigation of film structures in the very vicinity of substrates is rare. By the growth of ultra-thin films and by a close investigation of the structures therein, we may find the key to improve the working temperatures of current QAH or QSH states, or even to realize the other proposed QSH or QAH systems. Thus, we focus on ultra-thin films of 3D TIs. Here we grow ultra-thin topological insulator Bi₂Te₃ and Sb₂Te₃ films and investigate the domain boundary structures and spatially dependent local density of states (DOS) of these structures. We find two dominant types of domain boundaries and they behave differently in DOS. One of them is suggested to be a possible factor that may provide conduction channels in the electronic transport.

We grow two-quintuple-layer (QL) Bi₂Te₃ and Sb₂Te₃ films on SrTiO₃ (111) (STO) substrates by molecular beam epitaxy (MBE). The substrates are annealed in a vacuum prior to the deposition. The surface peak-to-peak roughness is about 200 pA. Figure 1(a) shows the topographic images for the 2-QL Bi₂Te₃ and Sb₂Te₃ films by scanning tunneling microscopy (STM). The films have dense domain structures probably because of the presence of strain in the films. The surface lattice constants of STO(111), Bi₂Te₃ and Sb₂Te₃ are ∼ 3.90 Å, 4.38 Å and 4.26 Å, respectively. A close investigation of these films indicates that there are two kinds of domain boundaries marked as DW1 and DW2. These domain boundaries are formed because of the different crystal orientations of the adjacent domains. DW1 is dash-dotted-like in appearance and the corresponding two domains have a relatively small rotation angle between them. Figure 1(b) shows a domain boundary of type DW1 with a rotation angle α of about 9°. Atomic resolution images [Figs. 1(b) and 2(a)] show that DW1 is characterized by the insert of an additional atom for every n-type lattice sites.^[11] On the contrary, DW2 forms between domains with relatively large mismatch angles in crystal orientations. The lattices do not match along the boundary and the lattice seems strongly distorted. We checked the domain boundary appearance in 2-QL Bi₂Te₃ and Sb₂Te₃ films. It is found that the domain boundaries in 2-QL Bi₂Te₃ films are exclusively of type DW1. In contrast, DW1 and DW2 appear almost evenly in 2-QL Sb₂Te₃ films. A close investigation reveals that the two domains along DW2 mainly have a rotation angle α of about 30° or about 60°. In the latter case the $\bar{\varGamma }$–$\bar{M}$ or $\bar{\varGamma }$–$\bar{K}$ directions of domains are opposite to each other as shown in Fig. 1(c).

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

For DW1, we can relate the period of the dash-dotted-like structure with the rotation angle α and analyze statistically α for DW1 in both 2-QL Bi₂Te₃ and Sb₂Te₃ films. The results are plotted in Figs. 2(b) and 2(c), respectively. Besides the fact that almost half of the domain boundaries are of type DW2 in Sb₂Te₃, the distribution of α for DW1 in Sb₂Te₃ also shifts to higher values by 2° compared with that in Bi₂Te₃. The relatively large α and the presence of DW2 in Sb₂Te₃ may indicate a weaker interaction between Sb₂Te₃ and STO. The possible stronger film-substrate interaction may help to force the Bi₂Te₃ domains to align with the crystal orientations in the substrate, while in Sb₂Te₃ the initial nucleation may tend to be randomly orientated. In the latter case, there appears an equal distribution between DW1 and DW2 for the domain boundaries in Sb₂Te₃.

By spatially resolved scanning tunneling spectroscopy (STS) we find that DW1 and DW2 behave differently in disturbing the original electronic structure of materials. In the 2-QL Bi₂Te₃ film, spatially dependent STS is measured along the line sketched in Fig. 1(b) and plotted in Fig. 3(b). Figure 3(a) shows typical STS for three different positions along DW1. The positions are also indicated by the black dots labeled by the numbers 1, 2 and 3, corresponding to the positions between the bright dots, about 0.5 nm away from the dots and on the dots, respectively. We can see the formation of bound states periodically along DW1 on positions 1 and 3 at different energies (∼–60 meV and –290 meV). These bound states are bounded and separated from each other.

Zoom In Zoom Out Reset image size

For the 2-QL Sb₂Te₃ film, we here show in Fig. 4(a) STS mapping at different energies on a uniform 2-QL region having DW1, DW2 and a step coming from the substrate. Figures 4(b), 4(c) and 4(d) are the STS away from domain boundaries, on a bright spot in DW1 and on DW2, respectively. The gap shown in Fig. 4(b) is the hybridization gap for the 2-QL Sb₂Te₃ film.^[18] The magnitude of the gap is about 210 meV (approximately from –30 to –240 meV) and is uniform across the regions away from domain boundaries. On the bright spot in DW1 and on DW2, we find bound states starting from –170 meV and –60 meV, respectively. By taking STS mapping at energies of –150, –180 and –193 meV [Fig. 4(a)], we show here that the bound states in DW1 are localized near the bright spots along DW1, while those in DW2 are delocalized along DW2. All these three representative energies are within the hybridization gap and also within the energy range of bound states of DW2. Here, –150 meV is within energy range of DW2's bound states but outside that of DW1; –180 and –193 meV are within the energy ranges both of DW1 and DW2 bound states. We can see that for these three energies, the bound states are delocalized along DW2. For DW1, the bound states are localized and may have different energies for different bright spots. Along the step of the substrate, we do not see the bound state formation although the film must be somehow strained because of the existence of curvature across the step.

Zoom In Zoom Out Reset image size

In conclusion, 2-QL Bi₂Te₃ and Sb₂Te₃ films behave differently in the domain structures. There are two kinds of domain boundaries DW1 and DW2 having small and large rotational angles, respectively. While in 2-QL Bi₂Te₃ there is only DW1, there are both DW1 and DW2 in 2-QL Sb₂Te₃. By the spatially dependent STS, we show that DW1 and DW2 disturb the electronic structures of the ultra-thin films differently. Both DW1 and DW2 introduce bound states along the domain boundaries. The difference is that the bound states in DW1 are localized while those in DW2 are delocalized. The delocalization of bound states in DW2 may contribute to the transport in the ultra-thin films of this material. Our work points out that to realize the edge-state dominated transport at levitated temperatures, domain boundaries of type DW2 (if they exist) need to be suppressed. Bi₂Te₃ is not a good candidate of possible QSH states because of its buried Dirac point. For the ultra-thin Sb₂Te₃ film or its derivatives, further work must be performed to minimize the formation of DW2 either by refining the growth parameters or using suitable substrates.