Machine-Learned Phase Diagrams of Generalized Kitaev Honeycomb Magnets

We use a recently developed interpretable and unsupervised machine-learning method, the tensorial kernel support vector machine (TK-SVM), to investigate the low-temperature classical phase diagram of a generalized Heisenberg-Kitaev-Gamma (J-K-Gamma) model on a honeycomb lattice. Aside from reproducing phases reported by previous quantum and classical studies, our machine finds a hitherto missed nested zigzag-stripy order and establishes the robustness of a recently identified modulated S3 × Z3 phase, which emerges through the competition between the Kitaev and Gamma spin liquids, against Heisenberg interactions. The results imply that, in the restricted parameter space spanned by the three primary exchange interactions -- J, K, and Gamma, the representative Kitaev material α-rm RuCl3 lies close to the boundaries of several phases, including a simple ferromagnet, the unconventional S3 × Z3 and nested zigzag-stripy magnets. A zigzag order is stabilized by a finite Gammaprime and/or J3 term, whereas the four magnetic orders may compete in particular if Gammaprime is anti-ferromagnetic.