# Revisiting Deep Information Propagation: Fractal Frontier and Finite-size Effects

arXiv: https://arxiv.org/abs/2508.03222 · HTML: https://mlsanity.com/paper/2508.03222/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exponential expressivity in deep neural networks through transient chaos](https://mlsanity.com/paper/1606.05340/) (2016)
- [Deep Information Propagation](https://mlsanity.com/paper/1611.01232/) (2016)
- [Orthogonal Random Features](https://mlsanity.com/paper/1610.09072/) (2016)
- [Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations](https://mlsanity.com/paper/1903.05895/) (2019)
- [Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks](https://mlsanity.com/paper/2304.03408/) (2023)
- [ACDC: A Structured Efficient Linear Layer](https://mlsanity.com/paper/1511.05946/) (2015)
- [Reservoir Computing meets Recurrent Kernels and Structured Transforms](https://mlsanity.com/paper/2006.07310/) (2020)
- [The boundary of neural network trainability is fractal](https://mlsanity.com/paper/2402.06184/) (2024)
- [Asymptotic Stability in Reservoir Computing](https://mlsanity.com/paper/2206.03854/) (2022)
- [Comparison of Reservoir Computing topologies using the Recurrent Kernel approach](https://mlsanity.com/paper/2401.14557/) (2024)
- [Mapping the Edge of Chaos: Fractal-Like Boundaries in The Trainability of Decoder-Only Transformer Models](https://mlsanity.com/paper/2501.04286/) (2025)

## Built on by (most cited first, 2 of 2)

- [Bulk-boundary decomposition of neural networks](https://mlsanity.com/paper/2511.02003/) (2025)
- [Affine Geometry of Gaussian ReLU Networks via Conditional Kac-Rice Formulas](https://mlsanity.com/paper/2609.32695/) (2026)

## Abstract (arXiv)

Information propagation characterizes how input correlations evolve across layers in deep neural networks. This framework has been well studied using mean-field theory, which assumes infinitely wide networks. However, these assumptions break down for practical, finite-size networks. In this work, we study information propagation in randomly initialized neural networks with finite width and reveal that the boundary between ordered and chaotic regimes exhibits a fractal structure. This shows the fundamental complexity of neural network dynamics, in a setting that is independent of input data and optimization. To extend this analysis beyond multilayer perceptrons, we leverage recently introduced Fourier-based structured transforms, and show that information propagation in convolutional neural networks also follow the same behavior. In practice, our investigation highlights the importance of finite network depth with respect to the tradeoff between separation and robustness.