# Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function

arXiv: https://arxiv.org/abs/1809.08848 · HTML: https://mlsanity.com/paper/1809.08848/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2018.

## Built on by (most cited first, 22 of 22)

- [Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit](https://mlsanity.com/paper/2309.16620/) (2023)
- [The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit](https://mlsanity.com/paper/2306.17759/) (2023)
- [Simplifying Transformer Blocks](https://mlsanity.com/paper/2311.01906/) (2023)
- [NoT: Federated Unlearning via Weight Negation](https://mlsanity.com/paper/2503.05657/) (2025)
- [The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks](https://mlsanity.com/paper/2311.18718/) (2023)
- [Neural Born Iteration Method For Solving Inverse Scattering Problems: 2D Cases](https://mlsanity.com/paper/2112.09831/) (2021)
- [Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models](https://mlsanity.com/paper/2511.03317/) (2025)
- [IDInit: A Universal and Stable Initialization Method for Neural Network Training](https://mlsanity.com/paper/2503.04626/) (2025)
- [JPmHC Dynamical Isometry via Orthogonal Hyper-Connections](https://mlsanity.com/paper/2602.18308/) (2026)
- [Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Case](https://mlsanity.com/paper/2201.04543/) (2022)
- [Entangled Residual Mappings](https://mlsanity.com/paper/2206.01261/) (2022)
- [Advancing Deep Residual Learning by Solving the Crux of Degradation in Spiking Neural Networks](https://mlsanity.com/paper/2201.07209/) (2021)
- [Sparser, Better, Deeper, Stronger: Improving Sparse Training with Exact Orthogonal Initialization](https://mlsanity.com/paper/2406.01755/) (2024)
- [Revisiting Glorot Initialization for Long-Range Linear Recurrences](https://mlsanity.com/paper/2505.19827/) (2025)
- [Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth](https://mlsanity.com/paper/2607.14018/) (2026)
- [Flag Varieties: A Geometric Framework for Deep Network Alignment](https://mlsanity.com/paper/2605.09861/) (2026)
- [Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning](https://mlsanity.com/paper/2607.17673/) (2026)
- [A Johnson--Lindenstrauss Framework for Randomly Initialized CNNs](https://mlsanity.com/paper/2111.02155/) (2021)
- [Dynamical Isometry for Residual Networks](https://mlsanity.com/paper/2210.02411/) (2022)
- [When Does Learning Renormalize? Sufficient Conditions for Power Law Spectral Dynamics](https://mlsanity.com/paper/2512.18209/) (2025)
- [How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences](https://mlsanity.com/paper/2605.05113/) (2026)
- [Gradland: On Phenomenal Experience, Differentiated Across Many Dimensions](https://mlsanity.com/paper/2609.09306/) (2026)

## Abstract (arXiv)

We demonstrate that in residual neural networks (ResNets) dynamical isometry is achievable irrespectively of the activation function used. We do that by deriving, with the help of Free Probability and Random Matrix Theories, a universal formula for the spectral density of the input-output Jacobian at initialization, in the large network width and depth limit. The resulting singular value spectrum depends on a single parameter, which we calculate for a variety of popular activation functions, by analyzing the signal propagation in the artificial neural network. We corroborate our results with numerical simulations of both random matrices and ResNets applied to the CIFAR-10 classification problem. Moreover, we study the consequence of this universal behavior for the initial and late phases of the learning processes. We conclude by drawing attention to the simple fact, that initialization acts as a confounding factor between the choice of activation function and the rate of learning. We propose that in ResNets this can be resolved based on our results, by ensuring the same level of dynamical isometry at initialization.