# Emergence of hierarchical modes from deep learning

arXiv: https://arxiv.org/abs/2208.09859 · HTML: https://mlsanity.com/paper/2208.09859/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 15 references that are in the map.
- First posted on arXiv 21 Aug 2022.

## Builds on (most cited first, 2 of 2)

- [The Principles of Deep Learning Theory](https://mlsanity.com/paper/2106.10165/) (2021)
- [Machine learning in spectral domain](https://mlsanity.com/paper/2005.14436/) (2020)

## Built on by (most cited first, 5 of 5)

- [Introduction to dynamical mean-field theory of randomly connected neural networks with bidirectionally correlated couplings](https://mlsanity.com/paper/2305.08459/) (2023)
- [Eight challenges in developing theory of intelligence](https://mlsanity.com/paper/2306.11232/) (2023)
- [Fermi-Bose Machine achieves both generalization and adversarial robustness](https://mlsanity.com/paper/2404.13631/) (2024)
- [Spiking mode-based neural networks](https://mlsanity.com/paper/2310.14621/) (2023)
- [Geometric origin of adversarial vulnerability in deep learning](https://mlsanity.com/paper/2509.01235/) (2025)

## Abstract (arXiv)

Large-scale deep neural networks consume expensive training costs, but the training results in less-interpretable weight matrices constructing the networks. Here, we propose a mode decomposition learning that can interpret the weight matrices as a hierarchy of latent modes. These modes are akin to patterns in physics studies of memory networks, but the least number of modes increases only logarithmically with the network width, and becomes even a constant when the width further grows. The mode decomposition learning not only saves a significant large amount of training costs, but also explains the network performance with the leading modes, displaying a striking piecewise power-law behavior. The modes specify a progressively compact latent space across the network hierarchy, making a more disentangled subspaces compared to standard training. Our mode decomposition learning is also studied in an analytic on-line learning setting, which reveals multi-stage of learning dynamics with a continuous specialization of hidden nodes. Therefore, the proposed mode decomposition learning points to a cheap and interpretable route towards the magical deep learning.