# Towards Critical Branching Mechanism in Recurrent Neural Networks

arXiv: https://arxiv.org/abs/2606.10384 · HTML: https://mlsanity.com/paper/2606.10384/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 36 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [Optimal Machine Intelligence at the Edge of Chaos](https://mlsanity.com/paper/1909.05176/) (2019)
- [Self-Organized Criticality in the Brain](https://mlsanity.com/paper/2102.09124/) (2021)
- [Inferring collective dynamical states from widely unobserved systems](https://mlsanity.com/paper/1608.07035/) (2016)
- [Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions in LSTM Networks](https://mlsanity.com/paper/2505.20030/) (2025)
- [Toward a Physics of Deep Learning and Brains](https://mlsanity.com/paper/2509.22649/) (2025)

## Abstract (arXiv)

Criticality has been proposed as a key organizing principle in biological neural systems, yet its origin and relevance in artificial neural networks remain unclear. We analyze hidden-state dynamics in trained long short-term memory (LSTM) networks and show that small networks near their optimal training epochs (steps) exhibit scale-free avalanche statistics and branching parameters close to unity, indicative of near-critical dynamics, while larger models remain subcritical. To explain the coexistence of subcritical branching with robust 1/fβ noise, we introduce a mixture branching process framework that links heterogeneous branching dynamics to long-range temporal correlations. These results identify critical-like behavior in LSTMs as an emergent, capacity-dependent dynamical regime.