# Symmetry without a manifold: intrinsic dimension on orbits

arXiv: https://arxiv.org/abs/2609.17926 · HTML: https://mlsanity.com/paper/2609.17926/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 30 references that are in the map.
- First posted on arXiv 15 Sep 2026.

## Builds on (most cited first, 26 of 26)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [Estimating the intrinsic dimension of datasets by a minimal neighborhood information](https://mlsanity.com/paper/1803.06992/) (2018)
- [Omnigrok: Grokking Beyond Algorithmic Data](https://mlsanity.com/paper/2210.01117/) (2022)
- [A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations](https://mlsanity.com/paper/2302.03025/) (2023)
- [The Quantization Model of Neural Scaling](https://mlsanity.com/paper/2303.13506/) (2023)
- [Explaining grokking through circuit efficiency](https://mlsanity.com/paper/2309.02390/) (2023)
- [Grokking modular arithmetic](https://mlsanity.com/paper/2301.02679/) (2023)
- [Feature emergence via margin maximization: case studies in algebraic tasks](https://mlsanity.com/paper/2311.07568/) (2023)
- [Superposition Yields Robust Neural Scaling](https://mlsanity.com/paper/2505.10465/) (2025)
- [Grokking Group Multiplication with Cosets](https://mlsanity.com/paper/2312.06581/) (2023)
- [Grokking Modular Polynomials](https://mlsanity.com/paper/2406.03495/) (2024)
- [Neural Scaling Laws Rooted in the Data Distribution](https://mlsanity.com/paper/2412.07942/) (2024)
- [Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking](https://mlsanity.com/paper/2509.21519/) (2025)
- [On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking](https://mlsanity.com/paper/2602.16849/) (2026)
- [A Neural Scaling Law from Lottery Ticket Ensembling](https://mlsanity.com/paper/2310.02258/) (2023)
- [Neural Networks Provably Learn Spectral Representations for Group Composition](https://mlsanity.com/paper/2606.02993/) (2026)
- [Quiet Feature Learning in Algorithmic Tasks](https://mlsanity.com/paper/2505.03997/) (2025)
- [Beyond Neural Collapse: Task-Intrinsic Geometry Governs Neural Representations in Modular Arithmetic](https://mlsanity.com/paper/2606.08985/) (2026)
- [Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations](https://mlsanity.com/paper/2607.13749/) (2026)

## Abstract (arXiv)

The standard geometric derivation of neural scaling exponents takes the intrinsic dimension of a data manifold as its input. On modular addition in mathbbZp that derivation has no input. The exact algebraic solution is an orbit of mathbbZp acting by isometries. Transitivity alone makes the ratio statistic underlying the standard dimension estimator a point mass, so the estimator is undefined, and here the two nearest neighbour distances coincide exactly. Breaking the symmetry at scale ε returns a number, but one that tracks 1/ε with no scale free plateau. We show that the failure is general, since on any finite orbit of a group acting by isometries the estimator reports the resolution at which the set is probed rather than a dimension. What replaces the power law is exponential in hidden width, L(h)=L_∞+Aexp(-c,hα), with R2 between 0.982 and 0.995 against 0.857 to 0.906 for a power law admitting the same floor and fitted under the same protocol. Where the data supply is sufficient the rate belongs to the regulariser rather than to the group, since weight decay moves c by a factor of 47 while group order moves it by 1.10, a residual below seed to seed resolution, for every fixed α between 0.75 and 2. The critical width falls with group order rather than rising, against capacity counting that assigns a fixed number of neurons to each irreducible representation.