# On the geometry and topology of representations: the manifolds of modular addition

arXiv: https://arxiv.org/abs/2512.25060 · HTML: https://mlsanity.com/paper/2512.25060/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 31 Dec 2025.

## Builds on (most cited first, 26 of 26)

- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [Convergent Learning: Do different neural networks learn the same representations?](https://mlsanity.com/paper/1511.07543/) (2015)
- [The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks](https://mlsanity.com/paper/2306.17844/) (2023)
- [A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations](https://mlsanity.com/paper/2302.03025/) (2023)
- [Ripser: efficient computation of Vietoris-Rips persistence barcodes](https://mlsanity.com/paper/1908.02518/) (2019)
- [Grokking modular arithmetic](https://mlsanity.com/paper/2301.02679/) (2023)
- [Asymptotic theory of in-context learning by linear attention](https://mlsanity.com/paper/2405.11751/) (2024)
- [Domain Adaptive Neural Networks for Object Recognition](https://mlsanity.com/paper/1409.6041/) (2014)
- [Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks](https://mlsanity.com/paper/2406.02550/) (2024)
- [Feature emergence via margin maximization: case studies in algebraic tasks](https://mlsanity.com/paper/2311.07568/) (2023)
- [To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets](https://mlsanity.com/paper/2310.13061/) (2023)
- [Grokking Group Multiplication with Cosets](https://mlsanity.com/paper/2312.06581/) (2023)
- [Fourier Circuits in Neural Networks and Transformers: A Case Study of Modular Arithmetic with Multiple Inputs](https://mlsanity.com/paper/2402.09469/) (2024)
- [MMD-B-Fair: Learning Fair Representations with Statistical Testing](https://mlsanity.com/paper/2211.07907/) (2022)
- [Towards a unified and verified understanding of group-operation networks](https://mlsanity.com/paper/2410.07476/) (2024)
- [Uncovering a Universal Abstract Algorithm for Modular Addition in Neural Networks](https://mlsanity.com/paper/2505.18266/) (2025)
- [Fair Representation Learning for Continuous Sensitive Attributes using Expectation of Integral Probability Metrics](https://mlsanity.com/paper/2505.06435/) (2025)
- [(How) Can Transformers Predict Pseudo-Random Numbers?](https://mlsanity.com/paper/2502.10390/) (2025)
- [Can Deep Reinforcement Learning Solve Erdos-Selfridge-Spencer Games?](https://mlsanity.com/paper/1711.02301/) (2017)
- [Position: We Need An Algorithmic Understanding of Generative AI](https://mlsanity.com/paper/2507.07544/) (2025)
- [Modular addition without black-boxes: Compressing explanations of MLPs that compute numerical integration](https://mlsanity.com/paper/2412.03773/) (2024)
- [Topological Data Analysis of Neural Network Layer Representations](https://mlsanity.com/paper/2208.06438/) (2022)

## Built on by (most cited first, 3 of 3)

- [Sequential Group Composition: A Window into the Mechanics of Deep Learning](https://mlsanity.com/paper/2602.03655/) (2026)
- [Steered LLM Activations are Non-Surjective](https://mlsanity.com/paper/2604.09839/) (2026)
- [Mechanistic Interpretability for Neural Networks: Circuits, Sparse Features and Symbolic Reasoning](https://mlsanity.com/paper/2607.07316/) (2026)

## Abstract (arXiv)

The Clock and Pizza interpretations, associated with architectures differing in either uniform or learnable attention, were introduced to argue that different architectural designs can yield distinct circuits for modular addition. In this work, we show that this is not the case, and that both uniform attention and trainable attention architectures implement the same algorithm via topologically and geometrically equivalent representations. Our methodology goes beyond the interpretation of individual neurons and weights. Instead, we identify all of the neurons corresponding to each learned representation and then study the collective group of neurons as one entity. This method reveals that each learned representation is a manifold that we can study utilizing tools from topology. Based on this insight, we can statistically analyze the learned representations across hundreds of circuits to demonstrate the similarity between learned modular addition circuits that arise naturally from common deep learning paradigms.