Atomic-scale visualization of surface-assisted orbital order

See allHide authors and affiliations

Science Advances  22 Sep 2017:
Vol. 3, no. 9, eaao0362
DOI: 10.1126/sciadv.aao0362


Orbital-related physics attracts growing interest in condensed matter research, but direct real-space access of the orbital degree of freedom is challenging. We report a first, real-space, imaging of a surface-assisted orbital ordered structure on a cobalt-terminated surface of the well-studied heavy fermion compound CeCoIn5. Within small tip-sample distances, the cobalt atoms on a cleaved (001) surface take on dumbbell shapes alternatingly aligned in the [100] and [010] directions in scanning tunneling microscopy topographies. First-principles calculations reveal that this structure is a consequence of the staggered dxz-dyz orbital order triggered by enhanced on-site Coulomb interaction at the surface. This so far overlooked surface-assisted orbital ordering may prevail in transition metal oxides, heavy fermion superconductors, and other materials.


The newfangled orbital-mediated quantum phenomena have proved over the past decade to be far-reaching and more complex than before (13), exhibiting exotic orbital orders (OOs) (4, 5), nontrivial orbital fluctuation–mediated superconductivity (69), and orbital Kondo effect (10, 11). To unravel electronic, spin, and orbital correlations in these phenomena, it is crucial to have a direct, real-space, access to orbital texture, but so far, orbital-sensitive probes have shown rather limited functionality. Recently, several groups demonstrated the orbital selectivity of a scanning tunneling microscope (STM) within a fine-tuned tip-sample distance (TSD) (1214). According to the Tersoff-Hamann theory (15), an STM image corresponds to a contour mapping of local density of states (DOS) at the center of the tip apex curvature. When the tip is away from the surface (<1 nm), it probes the electronic states that extend further from the surface, that is, valence states composed of the s and p orbitals of constituent atoms. When the tip is close, it becomes sensitive to inner core orbitals like the d orbitals, which fall off at short distances from the surface. A well-controlled calibration of TSD allows us to image individual orbital states (1214). Here, we exploit the orbital sensitivity of STM to unveil a surface-assisted cobalt d-OO in the heavy fermion compound CeCoIn5. This compound is naturally born at the quantum critical point having the unconventional d-wave superconducting phase below TC = 2.3 K (16). The compound has a tetragonal HoCoGa5 structure with lattice constants a = 0.461 nm and c = 0.756 nm, as shown in Fig. 1A. Along the c axis, three different layers—Co, In, and CeIn—stack in the order of Co-In-CeIn-In-Co (17).

Fig. 1 The crystal structure and topography of cleaved surfaces of CeCoIn5.

(A) Schematic of the tetragonal crystal structure of CeCoIn5 with dxz-dyz OOs at the topmost cobalt plane. (B and C) Overviews of two typical cleaved surfaces in this study. Narrow terraces (10 to 50 nm) are separated by a step of ~0.76 nm. The topographic images are colorized with their derivatives to emphasize the atomically resolved lattice structures. Red arrows in (C) indicate line defects [(A): tunneling current IT = 50 pA, sample bias voltage VS = 50 mV; (B): IT = 15 pA, VS = 10 mV]. (D) Typical atomically resolved STM image taken on the surface of (B) (IT = 1 nA, VS = 50 mV). We observed an almost identical atomic lattice image on the surface of (C), as shown in Fig. 2B.


Figure 1 (B and C) shows overview images of two typical (001) surfaces of CeCoIn5 cleaved in situ. The narrow terraces on both surfaces are separated by a step of ~0.76 nm, which equals the lattice constant along the c axis, indicating the same layers covering the whole surface area. The surface in Fig. 1B has a relatively large number of defects and adsorbates. In contrast, the surface in Fig. 1C is smooth except for occasional line defects, indicated by red arrows, and surface standing waves around the steps. The visible atomically resolved square lattice on the terraces has an interatomic distance of ~0.5 nm equal to the lattice constant of the CeIn or Co plane (see also Fig. 1D). Although the CeIn layer has two atom species, identical atomic arrays in the CeIn and Co layers have been observed in previous studies (1820). The two layers were claimed to be distinguishable from the shape of the tunneling spectrum (18). However, the viability of such correspondence has been questioned (21); in any case, we could not draw any conclusion within our measurements (fig. S1). On the other hand, these two layers can be unambiguously distinguished once we identify the missing atoms in the CeIn layer. According to previous reports on CeCo(In0.9985Hg0.0015)5 [figure S6 of Aynajian et al. (18)], Hg impurities, which can only substitute for In atoms, partially occupy the top sites of the original square lattice in a CeIn plane. This leads us to conclude that the visible atoms within a normal TSD (tunnel resistance > 2 megohms) are In.

We used the orbital sensitivity of STM to visualize the missing Ce atoms. We took STM images on the two surfaces in Fig. 1 for various tunneling currents IT and sample bias voltage VS fixed to 10 mV. Figure 2 (A and B) shows drastic changes in surface topographies as a function of IT. The white box indicates the unit cell. At IT = 100 nA (Fig. 2A), the terraces in Fig. 1B appear new bright protrusions in between the original atomic sites. These new protrusions are most likely Ce atoms of the CeIn plane (images of Ce 5d orbitals) because no extra atoms exist in the Co plane. By contrast, no appearance of additional atoms on the terraces in Fig. 1C evidences that these surfaces are Co layers.

Fig. 2 Current-dependent topographies of the terraces in Fig. 1 (B and C).

The white box in the images indicates the unit cell. (A) Atomically resolved topographies of the terraces from Fig. 1B (VS = 10 mV, IT = 5 and 100 nA at T = 500 mK). By increasing the current set point, new atoms appear in between original atomic sites with an interval of ~0.5 nm. (B) Atomically resolved topographies of terraces in Fig. 1C (VS = 10 mV, IT = 1, 30, and 100 nA at T = 1.7 K and at B = 5 T). An external out-of-plane magnetic field is applied merely to have better stability for STM measurement. With IT ≳ 10 nA, the shape of atoms gradually changes to a dumbbell with two lobes. The angle between the dumbbells at adjacent sites is 90°. (C) Current trace as a function of tip approaching distance Δz, measured at an atomic site in (B) with VS = 10 mV. IT shows an exponential dependence up to ~160 nA, indicating the tunneling regime. We note that Δz is the relative distance between the tip and surface. Δz = 0 corresponds to the position of the tip, where atomic contact occurs in the Iz spectrum.

On the terraces in Fig. 1C, the round shapes of surface Co atoms gradually transform into elongated dumbbells with IT increasing from 1 to 100 nA, as shown in Fig. 2B. The fact that these dumbbells between adjacent atomic sites alternate in the [100] and [010] directions breaks the original periodic symmetry of Co atoms observed at low IT (see fig. S2 and movie S1 for more details of this evolution of Co atom appearances in STM topographies). This symmetry breaking can hardly be interpreted as resulting from local tip-sample interactions. At all times, we operated our STM in the tunneling regime because the tunneling current IT as a function of TSD has an exponential dependence up to ~160 nA (Fig. 2C). To further ensure that this ordered structure is not a tip-induced artifact but truly reflects the surface DOS at a close distance, we reproduced these results by repeating the experiment with three different mechanically sharpened PtIr tips and many microscopic tips and on two independently cleaved surfaces (see Materials and Methods and note S1 for more details). Tip-induced electric fields are negligible for a bias voltage at 10 mV.

Co 4s electrons are not localized, and their wave functions extend to the vacuum, making the round shapes of Co atoms when the tip is far away (Fig. 2B, left). When the tip is close, the electron tunneling to inner Co 3d orbitals becomes possible (see fig. S3). Among them, the dxz and dyz orbitals can cause dumbbell shapes in STM topography, as is the case for a similar dumbbell shape caused by the dxz orbital observed on a single Co atom (22). The two 3d orbitals are degenerate and partially occupied in tetragonal CeCoIn5, their degeneracy being lifted by the on-site Coulomb interactions at the surface, which will be discussed later. We conclude that these dumbbells are a direct visualization of the Co dxz and dyz orbitals, with unbalanced electron occupations alternating at adjacent Co sites, and that the staggered dumbbell pattern is due to antiferro-type OO involving the two orbitals.

A detailed investigation of the line defects on the terraces in Fig. 1C provides evidence indicating that the observed OO occurs only in a single surface layer. We compared three line profiles along atomic rows with an identical length, shown in Fig. 3A. One is taken within a domain (c), and the other two are taken across the line defect (a and b). By comparing these three line profiles, we see that the positions of the protrusions have no offset across the defect (Fig. 3B), thus excluding any structural mismatch as an origin of the defect. When the tip is far away from the surface, the protrusions show no offset across the defect (Fig. 3B), which excludes any structural mismatch as an origin of the line defect. By contrast, when the tip is close, the staggered dumbbell pattern, which appears on both sides of the defect (Fig. 3C), has a phase mismatch across the defect (Fig. 3D). This shows that the line defect must result from the domain formation of the ordered structure. These defects once again support the fact that the OO is not a tip-induced artifact because they are still visible even when the tip is far away. Figure 1C shows that the line defects are not connected across the steps, thus strongly indicating that they are not three-dimensional domain boundaries, and that the OO occurs within a single Co plane. Because no signature of such d-OO has been observed so far with surface-insensitive bulk methods (16, 2326), the OO is most likely confined in the vicinity of the surface.

Fig. 3 Current-dependent topographies of the surface in Fig. 1C with a line defect.

(A) Topography of the terrace at 1.7 K with a line defect along the vertical axis of the image (IT = 10 nA, VS = 10 mV). (B) Line profiles indicated as lines a, b, and c in (A). Atomic positions in profiles a and b match very well with those in profile c in both sides of the line defect, indicating that no lattice distortion is present across the line defect. (C) Topography in the same field of view with (A) but with a shorter TSD (IT = 100 nA, VS = 10 mV). Dumbbell shapes are now visible. (D) Magnified image of the rectangle area indicated with a sky blue rectangle in (C). Ellipses help to visualize the change in the dumbbell arrangement across the line defect.


To gain greater insight, we have performed first-principles calculations within the generalized gradient approximation (GGA and GGA+U) using WIEN2k code (see Materials and Methods) (2729). Figure 4 (A and B) shows the DOS of Co1 atom in the bulk and on the surface of CeCoIn5, obtained respectively from bulk (GGA) and slab [GGA+U with UCo = 4 eV and UCe = 5 eV (30)] calculations. The positive and negative sides of the DOS represent the majority- and minority-spin contributions, respectively. The dxz and dyz orbitals remain degenerate in the bulk without noticeable spin polarization [0.11 μB (Bohr magneton)] in the d orbitals (Fig. 4A). Still, they are antiferromagnetically coupled to each other and ferromagnetically coupled to the Ce spins. In the following slab calculations, we focus on this magnetic pattern. The degeneracy of the dxz and dyz orbitals is lifted with almost fully occupied majority-spin orbitals and partially empty minority-spin orbitals (Fig. 4B), ending with a finite spin polarization (2.00 μB) on the surface. In particular, the dxz orbital is almost fully occupied, whereas the dyz orbital is half-filled. In the neighboring Co2 atom, we obtained opposite electron occupation in the two orbitals, ensuring a staggered dxz-dyz OO pattern. Figure 4 (C to E) illustrates the anisotropy in the charge density around the Fermi energy at a Co-terminated surface. The charge density is plotted in xy planes located at 0.10 nm (Fig. 4D) and 0.25 nm (Fig. 4E) above the Co-terminated plane. In the in-plane charge density, the OO pattern is only visible at a short distance to the surface (Fig. 4D), in agreement with the TSD-dependent STM images in Fig. 2B.

Fig. 4 First-principles electronic structures.

(A and B) DOS of d orbitals of bulk (A) and surface (B) Co atoms of CeCoIn5. Positive (negative) values show the majority-spin (minority-spin) contribution. The inset of (B) shows the DOS of dxz and dyz orbitals around the Fermi energy, demonstrating unbalanced orbital occupations. (C to E) Integrated charge density at the slab surface between −0.2 and 0.0 eV in the xz and xy planes. The xy planes are chosen at 0.10 nm (D) and 0.25 nm (E) above the topmost Co plane. The maximum value to present the color-coded plot is chosen as 10, 5, and 1 e/nm3 for (C), (D), and (E), respectively.

Although other magnetic patterns of Co spins can compete with the one considered here, the OO, which can facilitate virtual hopping processes to gain energy, is found to be more stable than the phase without OO but has the same magnetic order. Other effects of smaller energy scale, such as the Kondo effect, could alter to some degrees the DOS near the Fermi energy. However, the unbalanced occupation of dxz/yz orbitals persists in the entire energy window, of the order of several electron volts (see Fig. 4B), so the OO feature cannot be easily suppressed.

The experimentally observed and numerically obtained mechanism of the OO can be explained as follows. In general, the number of conduction electrons responsible for the screening of Coulomb interactions is reduced significantly at surfaces owing to the reduction of coordination numbers (31). This leads to a considerable reduction of electrostatic screening around the surface atoms, resulting in the increase of the Coulomb repulsion between localized electrons there. With on-site Coulomb repulsion enhanced, a single Co atom in a Co-terminated surface of CeCoIn5 can develop a Mott gap with a tendency to lift the degeneracy between the dxz/yz orbitals. Meanwhile, intersite Coulomb repulsion provides a driving force to select the OO pattern: The occupied orbitals between adjacent lattice sites tend to avoid aligning with each other to lower the Coulomb energy. Consequently, the orbitals are alternatingly occupied into a staggered pattern.

This surface-assisted OO observed just now might not be exclusive in the well-known heavy fermion compound CeCoIn5, because orbital degeneracy breaking may occur at any other surfaces for the same reason, although it has escaped detection by current surface techniques. The use of STM to sense the orbitals can open a new era for understanding many-body orbital correlations and unveiling order parameters of poorly understood phenomena such as a hidden order in URu2Si2 (32).



High-quality single crystals of CeCoIn5 were grown by an indium self-flux method as described by Petrovic et al. (16). Here, we used five single crystals. The sample qualities were tested by magnetization measurements using a SQUID magnetometer (Magnetic Property Measurement System, Quantum Design Inc.) down to 2.0 K before STM measurements. STM/scanning tunneling spectroscopy measurements were performed with a 3He cryostat-based ultrahigh vacuum STM (USM-1300S, Unisoku Co. Ltd.) and an STM controller (Nanonis SPM control system, SPECS Zurich GmbH). Magnetic fields were applied perpendicular to the cleaved surface.

Atomically flat surfaces parallel to the ab plane were obtained by cleaving at room temperatures under ultrahigh vacuum condition of ~10−8 Pa. We reported results on four successfully cleaved surfaces with a total of nine cleaves (successful rate is 44%). After approaching a cleaved surface, we moved around by changing a full scan area with the XY moving stage to look for a flat clean area. On two of four successfully cleaved surfaces, we could find only CeIn plane–dominating areas. On the other two surfaces, we found both CeIn and Co planes. We also observed a pure indium area stuck to surface steps. We performed STM measurements on all successfully cleaved areas by using three mechanically sharpened PtIr tips and many microscopic tips prepared in situ by applying a voltage pulse (>5 V) or soft indentation into surfaces. All of the tips reproduced clear images of the Ce atoms on CeIn planes and/or the evolution of dumbbell structures of Co atoms on Co planes.

To check the stability of the dumbbell structure against temperature and external magnetic fields, we performed STM measurements on a Co plane with TSD in an external magnetic field up to 5.5 T (>HC2) and at temperature from 500 mK to 6 K (>TC). We obtained identical structures within our noise level for all measured field and temperature ranges.

First-principles calculations

The first-principles calculations were performed by WIEN2k code based on the all-electron, full-potential, linearized augmented plane-wave method. In the calculations, experimental lattice parameters were used (17). The radii (R) of Ce, Co, and In atoms were set to 2.5, 2.5, and 2.39 bohr, respectively. The plane-wave expansion parameter was chosen as RKmax = 7. An 8 × 8 × 7 mesh of k-point sampling was used for the Brillouin zone of the bulk containing 2 Ce, 2 Co, and 10 In atoms. A 6 × 6 × 1 mesh was used for the slab calculation containing 8 Ce atoms and 10 Co atoms for the five–Co layer slab. A 10.7 Å vacuum layer was included in the calculation. The effective Ueff = UJ with J = 0 eV was used for both Ce and Co atoms in the GGA+U calculations. The adopted supercell allows nearest neighboring Ce-Ce atoms to couple antiferromagnetically, in consistency with experiments. We have performed the calculations with and without spin-orbit coupling and presented the results without spin-orbit coupling because the same conclusion can be reached. We have examined the forces of atoms with several frozen structures and found that the bond length between the Co and In atoms tends to be shorter at the surface. However, we cannot identify any tendency of surface reconstruction in distinguishing the two surface Co atoms in terms of geometry. We therefore conclude that the OO between Co dxz and dyz orbitals is responsible for the observed dumbbell structure in STM images and focus on the results of slab calculations using the bulk structure without further structural relaxation.


Supplementary material for this article is available at

fig. S1. Tunneling spectra taken on CeIn- and Co-terminated surfaces and their locations.

fig. S2. Current-dependent cross-sectional profiles of Ce-In and Co planes.

fig. S3. Charge density profiles of the Co atom orbitals on the Co termination.

note S1. Excluding the possibility of tip-induced artifacts in dumbbell formation.

note S2. Topographic similarity between our dumbbell ordered structure and the images in the study of Takahashi et al. (14) and their difference in underlying physics.

movie S1. Evolution of dumbbell structure by decreasing the TSD.

This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial license, which permits use, distribution, and reproduction in any medium, so long as the resultant use is not for commercial advantage and provided the original work is properly cited.


Acknowledgments: We thank M.-T. Suzuki and N. Tateiwa for fruitful discussions and collaborations in the initial stage of the work. We also thank T. Hanaguri for technical advices about sample cleaving. We acknowledge discussions with S.-Y. Shiau, R. Peters, S. Kim, T. Miyamachi, F. Komori, M. Haze, S. Yamamoto, T. Kawae, K. Fujita, C. K. Kim, E. Minamitani, Y. Okada, H. Sakai, and A. Miyake. T.-R.C. and H.-T.J. thank the National Center for Theoretical Sciences, Taiwan, for technical support. Funding: This work was partially supported by Grants-in-Aid for Scientific Research from the Japan Society for the Promotion of Science (nos. 25707025, 26110507, 26120508, and 16K17744). H.L. acknowledges the Singapore National Research Foundation (NRF) for the support under NRF award no. NRF-NRFF2013-03. T.-R.C. was supported by the Ministry of Science and Technology and National Cheng Kung University, Taiwan. Author contributions: Y.Y. conceived and led the project. H.K. planned STM measurements and carried out the measurements and data analysis with Y.Y. Y. Haga and Z.F. synthesized and characterized the single crystals. C.-C.L., T.-R.C., H.-T.J., and H.L. performed the theoretical calculations. Y.Y., C.-C.L., and H.L. wrote the manuscript. Y. Hasegawa supervised the project. All authors commented on the manuscript. Competing interests: The authors declare that they have no competing interests. Data and materials availability: All data needed to evaluate the conclusions in the paper are present in the paper and/or the Supplementary Materials. Additional data related to this paper may be requested from the authors.

Stay Connected to Science Advances

Navigate This Article