# Prime Fourier Embeddings: A Principled Basis for Modular Arithmetic

arXiv: https://arxiv.org/abs/2606.23044 · HTML: https://mlsanity.com/paper/2606.23044/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Builds on (most cited first, 12 of 12)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [Clifford Neural Layers for PDE Modeling](https://mlsanity.com/paper/2209.04934/) (2022)
- [Grokking modular arithmetic](https://mlsanity.com/paper/2301.02679/) (2023)
- [Transformers Can Do Arithmetic with the Right Embeddings](https://mlsanity.com/paper/2405.17399/) (2024)
- [FoNE: Precise Single-Token Number Embeddings via Fourier Features](https://mlsanity.com/paper/2502.09741/) (2025)
- [Language Models Encode the Value of Numbers Linearly](https://mlsanity.com/paper/2401.03735/) (2024)
- [Numbers Already Carry Their Own Embeddings](https://mlsanity.com/paper/2606.14108/) (2026)

## Abstract (arXiv)

Numbers have algebraic structure that standard neural embeddings often fail to expose. We introduce Prime Fourier Embeddings (PFE), which encode integers as prime-indexed (cos, sin) pairs derived from the harmonic analysis of Q, providing a pre-structured representation in which modular arithmetic reduces to selecting the relevant prime channel rather than discovering algebraic structure from scratch. We prove that any linear map equivariant with respect to the product group action on PFE must be block-diagonal with one independent block per prime -- a consequence of Schur's lemma applied to the resulting character decomposition. For square-free composite moduli, the Chinese Remainder Theorem predicts which prime channels are task-relevant. Both predictions are confirmed empirically: ablation studies show specialization ratios exceeding 500x between task-relevant and task-irrelevant channels, with perfect in-distribution test accuracy across all square-free composite moduli tested.