Competing instabilities, orbital ordering, and splitting of band degeneracies from a parquet renormalization group analysis of a four-pocket model for iron-based superconductors: Application to FeSe

We report the results of a parquet renormalization group (RG) study of competing instabilities in the full 2D four-pocket, three-orbital low-energy model for iron-based superconductors. We derive and analyze the RG flow of the couplings, which describes all symmetry-allowed interactions between low-energy fermions. Despite that the number of the couplings is large, we argue that there are only two stable fixed trajectories of the RG flow and one weakly unstable fixed trajectory with a single unstable direction. Each fixed trajectory has a finite basin of attraction in the space of initial system parameters. On the stable trajectories, either interactions involving only dxz and dyz or only dxy orbital components on electron pockets dominate, while on the weakly unstable trajectory interactions involving dxz (dyz) and dxy orbital states on electron pockets remain comparable. The behavior along the two stable fixed trajectories has been analyzed earlier [Chubukov, Khodas, and Fernandes, Phys. Rev. X 6, 041045 (2016)10.1103/PhysRevX.6.041045]. Here we focus on the system behavior along the weakly unstable trajectory and apply the results to FeSe. We find, based on the analysis of susceptibilities along this trajectory, that the leading instability upon lowering the temperature is towards a three-component d-wave orbital nematic order. Two components are the differences between fermionic densities on dxz and dyz orbitals on hole pockets and on electron pockets, and the third one is the difference between the densities of dxy orbitals on the two electron pockets. We argue that this order is consistent with the splitting of band degeneracies, observed in recent photoemission data on FeSe by Fedorov et al. [Sci. Rep. 6, 36834 (2016)10.1038/srep36834].