## Abstract

The emergence of quantization at the nanoscale, the quantum size effect (QSE), allows flexible control of matter and is a rich source of advanced functionalities. A QSE-induced transition into an insulating phase in semimetallic nanofilms was predicted for bismuth a half-century ago and has regained new interest with regard to its surface states exhibiting nontrivial electronic topology. Here, we reveal an unexpected mechanism of the transition by high-resolution angle-resolved photoelectron spectroscopy combined with theoretical calculations. Anomalous evolution and degeneracy of quantized energy levels indicate that increased Coulomb repulsion from the surface states deforms a quantum confinement potential with decreasing thickness. The potential deformation strongly modulates spatial distributions of quantized wave functions, which leads to acceleration of the transition beyond the original QSE picture. This discovery establishes a complete picture of the long-discussed transition and highlights a new class of size effects dominating nanoscale transport in systems with metallic surface states.

## INTRODUCTION

Quantized electronic states generated by the quantum size effect (QSE) in nano-confined systems enable unique tunability for a wide range of phenomena such as superconductivity (*1*), light-matter interaction (*2*), and nonequilibrium carrier dynamics (*3*). Modulations of the bandgap and the density of states further improve functionalities in catalysts (*4*) and information devices (*5*). From a technological point of view, quantization inevitably affects any electronic system fabricated at the nanoscale. One of the most well-known examples is a QSE-induced metal-insulator transition, whose essence is illustrated for a film geometry in Fig. 1 (A and B). When semimetallic bulk bands are quantized, the valence band top and the conduction band bottom no longer cross the Fermi level (*E*_{F}), and the system enters an insulating phase. In the case of a system having metallic surface states as typically observed in topological materials, the transition is marked by the disappearance of conducting channels in the film interior, and thereafter, electric current flows only through the surfaces.

The transition was first predicted a half-century ago on bismuth (Bi) (*6*). A Bi single crystal is a typical semimetal with small carrier pockets and three-dimensional (3D) Dirac dispersions (*7*), which generate unusual magneto-transport responses (*8*–*10*). Moreover, because of the large spin-orbit coupling, Bi surfaces host spin-polarized metallic states (Fig. 1, A, B, and E) that have been intensively examined in the context of electronic topology (*11*–*16*). Although theoretical calculations tended to predict trivial band topology (*11*, *12*), angle-resolved photoelectron spectroscopy (ARPES) experiments have detected electronic band structures exhibiting nontrivial topology (*13*, *14*). A very recent experiment with scanning tunneling microscopy (STM) further supported the presence of a strong topological phase (*15*) and revealed, with systematic band structure calculations, that pure Bi lies very close to a phase boundary between the strong topological and the higher-order topological phases (*16*). Furthermore, recent STM experiments under strong magnetic fields identified surface Landau levels (*17*) and even a nematic quantum Hall liquid phase formed by the Bi surface states (*18*, *19*). The QSE-driven metal-insulator transition in Bi nanofilms, historically called a semimetal-semiconductor transition, originally received great attention as a nanoscale pathway for achieving a substantial thermoelectric figure of merit (*20*) and is now of interest for enhancing the surface-state-induced exotic phenomena.

Evidence of the metal-insulator transition on Bi films was obtained only in this decade by transport measurements on epitaxially grown samples (*21*–*23*). Initially, measurements were performed ex situ on samples with a protective capping layer (*21*, *22*), which provoked controversy (*24*, *25*) in view of possible contributions from surface oxidization (*26*). The latest experiment lastly used in situ conditions and concluded that atomically thin Bi films lose conducting channels derived from the quantized bulk states (*23*). Nevertheless, a recent ARPES measurement on Bi films grown exactly under the same conditions detected a bulk-derived envelope crossing *E*_{F} in atomically thin regions (*27*), in clear contrast to the transport results showing only the interior-insulating phase below a threshold thickness (*21*–*23*). Although this strange contradiction between metallic and insulating signatures observed in completely the same system implies the presence of an intriguing mechanism, essential quantization information was lacking in previous experiments.

Here, using high-resolution ARPES on high-quality Bi nanofilms, we report the first direct observation of the metal-insulator transition with all the quantized energy levels resolved on the films exhibiting a macroscopically insulating phase. Visualization of anomalous level evolution contrasted with tight-binding simulations highlights an additional mechanism beyond simple QSE. The high-resolution ARPES also detects unexpected degeneracy of top two quantized energy levels, which completely breaks a standard quantization rule. Furthermore, our systematic density functional theory (DFT) calculations reveal that the level degeneracy is gradually induced by shifting the whole band structure and accompanies transformation of the bulk-derived wave functions into surface-localized ones. This tendency is totally opposite to a well-known hybridization effect between top and bottom surface states and reconciles the contradiction among the previous experiments in an unprecedented manner. These unusual modulations of quantized bulk states can be fully explained only when we consider deformation of a quantum confinement potential, which is triggered by enhanced effects of Coulomb repulsion with decreasing system size, centering on a size-independent contribution from the surface states. The present study not only solves the serious controversy on the transition discussed for half a century but also introduces the novel size effect that can be universally present in a system with metallic surface states, typically topological materials.

## RESULTS

### Direct observation of a QSE-induced metal-insulator transition

Figure 1 (F and G) shows Fermi surfaces and band structures measured for a 14-bilayer (BL) Bi film (1 BL = 3.93 Å). Two spin-polarized surface-state bands and quantized bulk bands are distinctly observed in good agreement with those calculated in Fig. 1E. In Fig. 1 (H and I), we systematically follow the evolution of the band structures around the hole and the electron pockets with increasing thickness. The valence band top at *E*_{F} in atomically thin films. This contrasts with the previous ARPES result (*27*) and is attributed to a finite lattice strain effect, as discussed later. The conduction band bottom at *E*_{F}. To provide more direct evidence, we shifted the whole band structure by in situ electron doping and reconstructed band dispersions above *E*_{F} (see section S1). As far as we know, this is the first direct observation of a metal-insulator transition with all the related electronic states resolved on a thin-film system that exhibits macroscopically insulating transport. The situation is in clear contrast to studies on atomically thin films of bulk-conducting topological insulators (*28*–*30*).

### Anomalous evolution of quantized energy levels observed in the atomically thin regime

Furthermore, we follow the evolution of the quantized energy levels to investigate mechanisms of the transition, as shown with energy distribution curves (EDCs) at *31*)*k _{z}* is a wave number in the surface-normal direction,

*D*is a film thickness, and Φ is the total phase shift ϕ

_{top}+ ϕ

_{bottom}, as illustrated in Fig. 1B. Equation 1 tells us that the wave number is proportional to the inverse thickness

Therefore, a plot of quantized energy levels versus inverse thicknesses depicts a bulk band dispersion perpendicular to the film surface. At *14*). However, at *32*, *33*) in Fig. 2E, where only pure QSE is implemented. In this case, all the peak evolutions in Fig. 2F are consistently described by parabolic functions, which again supports the validity of Eq. 2 with constant Φ. The notable contrast between these two cases indicates that an additional size effect beyond a primitive QSE is driving this anomalous evolution in atomically thin regions. This leads to enhancement of the critical thickness where the top quantized level crosses *E*_{F}, as highlighted by shaded bars in Fig. 2 (D and F).

### Unexpected level degeneracy and bulk-to-surface transformation

Another unusual signature appears in the precise measurement of band dispersions. Figure 3A shows second-derivative ARPES images for atomically thin films, where both of the two surface-state bands connect to the top quantized bulk band. Because there are states localized at the top and bottom surfaces, these surface-state bands are doubly degenerate in the nearly freestanding film (*34*). This band connection thus indicates that the top quantized level must be quadruply degenerate. That is, *n* = 1 and 2 quantized levels are almost degenerate (Fig. 3B), which has never been reported in ARPES experiments of thin films.

Nevertheless, the observations can be reproduced by first-principles calculations that systematically shift the entire band structures, as shown in Fig. 3C. The shifts are manually induced by changing lattice parameters, where the in-plane lattice constant is modulated by −2, −1, 0, and +1% from left to right, respectively, with the total volume conserved. Along with the shift toward a higher binding energy, the energy separation between *n* = 2 and 3 levels increases. In contrast, the separation between *n* = 1 and 2 levels is reduced; their eventual degeneracy is consistent with Fig. 3 (A and B). It is as if the quantized bands split off from the bulk projection and behave independently as surface-state bands. This is supported by examining wave functions for each quantized state in Fig. 3D. In the left panel, the wave functions have envelope shapes expected for *n* = 1,2,3 states confined in a quantum well. However, as the unusual level degeneracy appears, wave functions of *n* = 1 and 2 states are transformed into surface-localized ones. It is well known that electronic states localized on top and bottom surfaces can hybridize and exhibit a bulk-like character in atomically thin films (*29*). In the case of Bi, this hybridization effect was demonstrated only around *14*, *35*). Here, the notably opposite behavior to the hybridization effect, the transformation of bulk-derived states into surface-localized ones, is observed for the quantized bulk states around

### The central mechanism of the present observations

An important question is what is the central mechanism responsible for the anomalous behaviors. The level degeneracy rules out major contributions of standard size effects related to charge transfer, lattice strain, and recently proposed surface size effects (*27*), all of which only uniformly shifts or expands the quantized bands and never generates degeneracy. Note that the lattice strain effect corresponds to a uniform compression/expansion of in-plane/out-of-plane lattice constants due to a lattice mismatch with substrates. We can still think of modulation in the interlayer spacing, which can affect Bi band structures to a relatively large extent (*36*). However, this effect is also excluded as the central mechanism of the present observations because our DFT calculations already exhibit the level degeneracy and the bulk-to-surface transformation using a homogeneous Bi slab without any modulation of the interlayer spacing implemented. (See also section S3 for detailed considerations about the conventional effects.)

Although the transformation of *n* = 1 and 2 confined wave functions into surface-like states can be viewed as a result of hybridization between the surface and bulk states around *V*_{eff} and a total charge density *n*_{total} are determined by a self-consistent cycle reflecting Coulomb interaction via the Hartree functional and the exchange-correlation functional (*37*). Here, the occupation of the surface-state bands increases from left to right in Fig. 3C, which increases their relative contributions to the total charge density *n*_{total} = *n*_{surface} + *n*_{interior}: This can modify the effective potential *V*_{eff} and electronic structures lastly obtained.

We further conceive that Coulomb repulsion among electrons tends to compress the total charge distribution toward the film center when the surface contributions increase. The behavior is observed in the present calculations (see section S4). In the one-body picture of the DFT framework, the compressed total charge distribution, in turn, makes an electron feel a potential barrier around the film center and deforms *V*_{eff} into a double-well–like shape, as illustrated in Fig. 4A. Eigenstates of a double-well potential have nearly degenerate ground states with wave functions localized in each well; this then provides a qualitative explanation of the experimental observations. Whereas the modulation of the electronic states is triggered by an increase of surface-state occupations in the calculation, the positions of the surface-state bands are almost independent of thicknesses in real Bi films (Fig. 1, H and I). Nevertheless, reducing the system size increases relative contributions of the surface states to the total charge density, which has essentially the same effect. We also examine this picture by performing a numerical simulation using a 1D Schrödinger equation with an ideal quantum-well potential that is gradually deformed to a double well. Figure 4B shows that this simple model quantitatively reproduces both the anomalous evolution and the level degeneracy when a double-well modulation and the resulting change in a band dispersion (an effective mass) have an exponential thickness dependence (see section S5). The insets also demonstrate that the wave functions are transformed to surface-localized ones with decreasing thickness.

Thus, the two essential signatures here, the degeneracy between the top two quantized bulk states and their transformation into surface-localized states, rule out scenarios given by any standard mechanism but are naturally explained by considering the deformation of a quantum confinement potential via Coulomb repulsion from the surface states. One of the most interesting points is that a usually weak effect of Coulomb repulsion and hybridization between the surface and quantized bulk states are enhanced with decreasing system size centering on the presence of the surface states.

## DISCUSSION

We revisit the strange contradiction among recent studies on the metal-insulator transition in Bi films. The central problem is a metallic envelope of quantized bulk states captured by the previous ARPES of atomically thin Bi films grown on a Si(111) substrate (*27*), whose film interiors were insulating in the transport experiments (*22*, *23*). Using the same substrate, we reproduced the ARPES data. As experimentally studied in (*38*), the smaller Si lattice constant relative to that of Ge exerts stronger compressive strain. The stronger strain in Bi/Si films shifts the whole band structure upward compared to the Bi/Ge case, so that the top quantized level crosses *E*_{F} (see fig. S3C). Nevertheless, as shown in Fig. 3D and in the insets of Fig. 4B, the wave function of this top quantized level behaves like a surface-conducting channel, and the resulting shift of an effective band edge makes the system interior insulating even in the Bi/Si case, which is consistent with the transport experiments (*21*–*23*). (See also section S6 for careful considerations of surface and bulk transport channels in atomically thin films.) Therefore, in addition to the exponential level evolution pushed by the emergent double-well potential, the bulk-to-surface transformation further accelerates the metal-insulator transition. This is a complete picture of the long-discussed problem in Bi, whose mechanism is unexpectedly extended centering on the surface states (Fig. 4C).

The present conclusion can be generalized to a new class of size effects in any dimension, where increased contributions of 1D and 2D edge states with decreasing system size modify an effective one-body potential via Coulomb repulsion. Here, the first experimental signature on a specific system still calls for more detailed theoretical studies in a wide range of materials. Nevertheless, the comprehensive discussions presented above suggest that the effect is likely to be present in a universal system with metallic edge states. A target of the greatest interest will be nanofilms of recently discovered Dirac (*39*), Weyl (*40*, *41*), and topological nodal-line semimetals (*42*) that inherently have metallic edge states and a small number of bulk carriers near *E*_{F} inside their point-/loop-like semimetallic nodes. 1D edge/hinge states in 2D (*34*, *43*) or higher-order topological insulator phases (*16*) will also be an interesting playground in view of the stronger confinement. Nanofabrication of such topological materials is an indispensable step for their device applications, in which various size effects including the one discovered here will play an essential role and provide new control parameters for advanced functionalities.

## MATERIALS AND METHODS

### Sample preparation

Bi(111) films were grown epitaxially on a medium-doped *p*-type Ge(111) wafer cleaned by cycles of Ar^{+} sputtering and annealing at 900 K. Bi evaporation was performed at room temperature, followed by annealing at 400 K (*38*). To improve the film quality, the Ge substrate was prepared with large domains by fully outgassing the preparation systems and uniformly sputtering and annealing the substrate. The temperature during the Bi deposition was also carefully controlled. The relative accuracy of film thicknesses was precisely controlled using a quartz thickness monitor, and absolute film thicknesses were calibrated by comparing quantized energy levels with previous reports (*34*, *44*). The high film quality was also confirmed by comparing sharpness in the photoemission spectra of fine quantization structures (Fig. 1, H and I) with the previous reports. The thinnest thickness (7 BL) was set close to the critical thickness of the Bi(111) structure on a Ge(111) substrate (*38*).

### ARPES measurements

High-resolution ARPES measurements were performed at BL-9A of the Hiroshima Synchrotron Radiation Center (HSRC) and BL-21B1 of the National Synchrotron Radiation Research Center (NSRRC). At BL-9A, a high-intensity unpolarized Xe plasma discharge lamp (8.437 eV) was used for magnifying small energy ranges, in addition to the synchrotron radiation (21 eV) for wide-range observations. The measurement temperature was 10 K, and the total energy resolution was 7 meV for 8.437-eV photons and 12 meV for 21-eV photons. The pressure during the measurements was 10^{−9} Pa. The measurement direction was precisely adjusted using an automated six-axis rotational controller and was based on Fermi surfaces that were mapped before every band structure scan. To directly determine unoccupied band structures, we also performed alkali-metal adsorption following an approach in (*45*) (see section S1).

### Tight-binding calculations

Tight-binding calculations for bulk electronic structures of Bi were based on a framework introduced in (*46*), which considers a 16 × 16 matrix composed of hopping parameters of the *sp*^{3} orbitals between first, second, and third nearest-neighbor atoms with spin-orbit coupling implemented. Electronic structures in a slab geometry were calculated by extending the bulk matrix to a larger 16*N* × 16*N* matrix, where *N* is the number of bilayers, as demonstrated in (*32*, *33*). The lattice parameters describing a Bi rhombohedral unit cell and the hopping parameters were taken from (*46*). The surface potential term was also introduced using formulations and parameters in (*32*).

### First-principles calculations

DFT calculations (*37*) were performed with the ABINIT code (*47*). The gradient approximation was selected for the exchange-correlation functional, and a norm-conserving pseudopotential was used in which spin-orbit coupling was implemented. A freestanding Bi slab with a 14-BL thickness was used, and the length of a vacuum region was set to 8 BL (31.5 Å). Lattice parameters of the slab were fixed to the experimental values (*46*), where an in-plane lattice constant is 4.55 Å and intrabilayer and interbilayer spacings are 1.59 and 2.34 Å, respectively. A grid for *k*-point sampling as 9 × 9 × 1 was used. Convergences with respect to an energy cutoff, the *k*-point sampling, and the vacuum length were confirmed for obtained band structures. We also cross-checked the results using the QUANTUM ESPRESSO code (*48*) with similar conditions.

### Model calculations

A numerical simulation was performed using a 1D Schrödinger equation with an electron mass extracted from the tight-binding band structure. Eigenvalues were calculated by taking the Fourier transform of a wave function *L* is a maximum length scale of the simulation (*49*). A finite-height single-well potential was set inside this length scale. A double-well deformation of the single well was introduced with an exponential dependence on thickness. To manually simulate modulation of a band dispersion accompanying the potential deformation, an exponential dependence was also installed for the electron effective mass. All the parameters were adjusted to best fit the experimental data. Further details are described in section S5.

## SUPPLEMENTARY MATERIALS

Supplementary material for this article is available at http://advances.sciencemag.org/cgi/content/full/6/12/eaaz5015/DC1

Section S1. Direct observation of the conduction band bottom located above *E*_{F} at

Section S2. Details of fitting functions in Fig. 2D

Section S3. Additional considerations of standard size effects

Section S4. Changes of a total charge distribution in the first-principles calculations

Section S5. Details of model calculations in Fig. 4B

Section S6. The distinction between surface and bulk states in atomically thin Bi films

Fig. S1. Direct observation of the conduction band bottom located above *E*_{F} at

Fig. S2. No substrate doping dependence of quantized energy levels at

Fig. S3. A finite effect caused by lattice strain from substrates.

Fig. S4. Experimental evaluation of minor contributions from all standard size effects.

Fig. S5. Changes of a total charge distribution in the first-principles calculations.

Fig. S6. Details of model calculations in Fig. 4B.

This is an open-access article distributed under the terms of the Creative Commons Attribution license, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

## REFERENCES AND NOTES

**Acknowledgments:**We thank T. Hirahara and M. Horn-von Hoegen for valuable discussions on the growth of Bi films. We also thank T. Kato, Y. Konishi, and T. Miyamachi for fruitful discussions on edge states in a confined geometry. We acknowledge T. Aono, Y. Ohtsubo, and R. Yukawa for advice on theoretical calculations. We thank A. Burns from the Edanz Group for checking a draft of the manuscript.

**Funding:**The ARPES measurements were performed with the approval of the Proposal Assessing Committee of HSRC (project no. 15-A-38, 17 AU025, and 18 AU008) and the Proposal Assessing Committee of NSRRC (project no. 2015-2-090-1). I.M. acknowledges support by JSPS under KAKENHI grant no. 18H03874. T.-C.C. acknowledges support from the U.S. National Science Foundation under grant no. DMR-17-09945. S.I. acknowledges support from JSPS under KAKENHI grant no. 17 J03534. S.I. was also supported by JSPS through the Program for Leading Graduate Schools (ALPS).

**Author contributions:**S.I. and I.M. planned the experimental project. S.I. fabricated thin-film samples, conducted ARPES experiments, and analyzed the data. M.A., B.F., W.-C.C., and C.-M.C. supported the ARPES experiments. S.I. performed tight-binding and model calculations. S.I. also performed first-principles calculations with support by G.B., J.H., and O.S. S.I., O.S., F.K., and I.M. organized discussions on the results, and S.I. wrote the paper. All authors contributed to the discussions and commented on the manuscript.

**Competing interests:**The authors declare that they have no competing interests.

**Data and materials availability:**All data needed to evaluate the conclusions in the paper are present in the paper and/or the Supplementary Materials. Additional data related to this paper may be requested from the authors.

- Copyright © 2020 The Authors, some rights reserved; exclusive licensee American Association for the Advancement of Science. No claim to original U.S. Government Works. Distributed under a Creative Commons Attribution License 4.0 (CC BY).