# Sequential Group Composition: A Window into the Mechanics of Deep Learning

arXiv: https://arxiv.org/abs/2602.03655 · HTML: https://mlsanity.com/paper/2602.03655/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 49 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Builds on (most cited first, 30 of 37)

- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Transformers Learn Shortcuts to Automata](https://mlsanity.com/paper/2210.10749/) (2022)
- [Repeat After Me: Transformers are Better than State Space Models at Copying](https://mlsanity.com/paper/2402.01032/) (2024)
- [SGD on Neural Networks Learns Functions of Increasing Complexity](https://mlsanity.com/paper/1905.11604/) (2019)
- [Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit](https://mlsanity.com/paper/2207.08799/) (2022)
- [Open Problems in Mechanistic Interpretability](https://mlsanity.com/paper/2501.16496/) (2025)
- [The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks](https://mlsanity.com/paper/2306.17844/) (2023)
- [Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning](https://mlsanity.com/paper/2012.09839/) (2020)
- [A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations](https://mlsanity.com/paper/2302.03025/) (2023)
- [Representational Strengths and Limitations of Transformers](https://mlsanity.com/paper/2306.02896/) (2023)
- [Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks](https://mlsanity.com/paper/1904.13262/) (2019)
- [Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity](https://mlsanity.com/paper/2106.15933/) (2021)
- [Grokking modular arithmetic](https://mlsanity.com/paper/2301.02679/) (2023)
- [Transformers, parallel computation, and logarithmic depth](https://mlsanity.com/paper/2402.09268/) (2024)
- [Language Models Use Trigonometry to Do Addition](https://mlsanity.com/paper/2502.00873/) (2025)
- [Understanding Transformer Reasoning Capabilities via Graph Algorithms](https://mlsanity.com/paper/2405.18512/) (2024)
- [Saddle-to-Saddle Dynamics in Diagonal Linear Networks](https://mlsanity.com/paper/2304.00488/) (2023)
- [Training Dynamics of In-Context Learning in Linear Attention](https://mlsanity.com/paper/2501.16265/) (2025)
- [Pre-trained Large Language Models Use Fourier Features to Compute Addition](https://mlsanity.com/paper/2406.03445/) (2024)
- [Learning Compositional Functions with Transformers from Easy-to-Hard Data](https://mlsanity.com/paper/2505.23683/) (2025)
- [Feature emergence via margin maximization: case studies in algebraic tasks](https://mlsanity.com/paper/2311.07568/) (2023)
- [Separations in the Representational Capabilities of Transformers and Recurrent Architectures](https://mlsanity.com/paper/2406.09347/) (2024)
- [Grokking Group Multiplication with Cosets](https://mlsanity.com/paper/2312.06581/) (2023)
- [Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks](https://mlsanity.com/paper/2312.08550/) (2023)
- [Fourier Circuits in Neural Networks and Transformers: A Case Study of Modular Arithmetic with Multiple Inputs](https://mlsanity.com/paper/2402.09469/) (2024)

## Built on by (most cited first, 10 of 10)

- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts](https://mlsanity.com/paper/2605.01148/) (2026)
- [Neural Networks Provably Learn Spectral Representations for Group Composition](https://mlsanity.com/paper/2606.02993/) (2026)
- [The Rank the Task Demands: A Causal Rank Law for Matrix Memories Trained on Group Composition](https://mlsanity.com/paper/2609.12259/) (2026)
- [Functorial Neural Architectures from Higher Inductive Types](https://mlsanity.com/paper/2603.16123/) (2026)
- [Singular Learning and Occam's Razor in Deep Monomial Networks](https://mlsanity.com/paper/2606.28464/) (2026)
- [A Spectral Theory of Grokking: Weight Decay induces Feature Learning](https://mlsanity.com/paper/2609.26679/) (2026)
- [RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theory](https://mlsanity.com/paper/2609.28625/) (2026)
- [Tracking States or Tracking Cosets? An Algebraic Account of Learned State Tracking](https://mlsanity.com/paper/2609.29951/) (2026)
- [Group-Invariant Statistics Determine Embedding Geometry: Harmonic Analysis of Representations from Bach to the Night Sky](https://mlsanity.com/paper/2610.00647/) (2026)

## Abstract (arXiv)

How do neural networks trained over sequences acquire the ability to perform structured operations, such as arithmetic, geometric, and algorithmic computation? To gain insight into this question, we introduce the sequential group composition task. In this task, networks receive a sequence of elements from a finite group encoded in a real vector space and must predict their cumulative product. This task can be order-sensitive and cannot be solved by a linear model. Our analysis isolates the roles of the group structure, encoding statistics, and sequence length in shaping learning. We prove that two-layer networks from vanishing initialization learn this task one irreducible representation of the group at a time in an order determined by the Fourier statistics of the encoding. To perfectly learn the task, these networks require a hidden width exponential in the sequence length k. In contrast, we construct deeper architectures that exploit associativity to dramatically improve this scaling: recurrent neural networks can compose elements sequentially in k steps, while multilayer networks can compose adjacent pairs in parallel in log k layers. Overall, the sequential group composition task offers a tractable window into the mechanics of deep learning.