# Learning the symmetric group: large from small

arXiv: https://arxiv.org/abs/2502.12717 · HTML: https://mlsanity.com/paper/2502.12717/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 23 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Builds on (most cited first, 12 of 12)

- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks](https://mlsanity.com/paper/2306.17844/) (2023)
- [A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations](https://mlsanity.com/paper/2302.03025/) (2023)
- [Investigating the Limitations of Transformers with Simple Arithmetic Tasks](https://mlsanity.com/paper/2102.13019/) (2021)
- [Linear algebra with transformers](https://mlsanity.com/paper/2112.01898/) (2021)
- [What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding](https://mlsanity.com/paper/2010.04903/) (2020)
- [Understanding Addition in Transformers](https://mlsanity.com/paper/2310.13121/) (2023)
- [PatternBoost: Constructions in Mathematics with a Little Help from AI](https://mlsanity.com/paper/2411.00566/) (2024)
- [What is my math transformer doing? -- Three results on interpretability and generalization](https://mlsanity.com/paper/2211.00170/) (2022)
- [Compact Proofs of Model Performance via Mechanistic Interpretability](https://mlsanity.com/paper/2406.11779/) (2024)
- [Can Transformers Learn to Solve Problems Recursively?](https://mlsanity.com/paper/2305.14699/) (2023)

## Built on by (most cited first, 3 of 3)

- [PermuFormer: Multi-Task Pretraining for Permutation Representation in Algebraic Combinatorics](https://mlsanity.com/paper/2609.25438/) (2026)
- [Tracing the Path to Grokking: Embeddings, Dropout, and Network Activation](https://mlsanity.com/paper/2507.11645/) (2025)
- [Learning the Word Problem: Geodesic Lengths and Cryptographic Applications](https://mlsanity.com/paper/2607.26241/) (2026)

## Abstract (arXiv)

Machine learning explorations can make significant inroads into solving difficult problems in pure mathematics. One advantage of this approach is that mathematical datasets do not suffer from noise, but a challenge is the amount of data required to train these models and that this data can be computationally expensive to generate. Key challenges further comprise difficulty in a posteriori interpretation of statistical models and the implementation of deep and abstract mathematical problems. We propose a method for scalable tasks, by which models trained on simpler versions of a task can then generalize to the full task. Specifically, we demonstrate that a transformer neural-network trained on predicting permutations from words formed by general transpositions in the symmetric group S10 can generalize to the symmetric group S25 with near 100% accuracy. We also show that S10 generalizes to S16 with similar performance if we only use adjacent transpositions. We employ identity augmentation as a key tool to manage variable word lengths, and partitioned windows for training on adjacent transpositions. Finally we compare variations of the method used and discuss potential challenges with extending the method to other tasks.