# Some Theoretical Results on Layerwise Effective Dimension Oscillations in Finite Width ReLU Networks

arXiv: https://arxiv.org/abs/2507.07675 · HTML: https://mlsanity.com/paper/2507.07675/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 3 references that are in the map.
- First posted on arXiv 10 Jul 2025.

## Builds on (most cited first, 3 of 3)

- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [Exponential expressivity in deep neural networks through transient chaos](https://mlsanity.com/paper/1606.05340/) (2016)
- [Rank Diminishing in Deep Neural Networks](https://mlsanity.com/paper/2206.06072/) (2022)

## Abstract (arXiv)

We analyze the layerwise effective dimension (rank of the feature matrix) in fully-connected ReLU networks of finite width. Specifically, for a fixed batch of m inputs and random Gaussian weights, we derive closed-form expressions for the expected rank of the m× n hidden activation matrices. Our main result shows that mathbbE[EDim(ℓ)]=m[1-(1-2/π)ℓ]+O(e-c m) so that the rank deficit decays geometrically with ratio 1-2 / π ≈ 0.3634. We also prove a sub-Gaussian concentration bound, and identify the "revival" depths at which the expected rank attains local maxima. In particular, these peaks occur at depths ℓk^*≈(k+1/2)π/log(1/ρ) with height ≈ (1-e-π/2) m ≈ 0.79m. We further show that this oscillatory rank behavior is a finite-width phenomenon: under orthogonal weight initialization or strong negative-slope leaky-ReLU, the rank remains (nearly) full. These results provide a precise characterization of how random ReLU layers alternately collapse and partially revive the subspace of input variations, adding nuance to prior work on expressivity of deep networks.