mlsanityData to 14 Oct 2026

Convergent Evolution: How Different Language Models Learn Similar Number Representations

arXiv 2604.20817 ↗First posted 22 Apr 2026cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Language models trained on natural text learn to represent numbers using periodic features with dominant periods at T=2, 5, 10. In this paper, we identify a two-tiered hierarchy of these features: while Transformers, Linear RNNs, LSTMs, and classical word embeddings trained in different ways all learn features that have period-T spikes in the Fourier domain, only some learn geometrically separable features that can be used to linearly classify a number mod-T. To explain this incongruity, we prove that Fourier domain sparsity is necessary but not sufficient for mod-T geometric separability. Empirically, we investigate when model training yields geometrically separable features, finding that the data, architecture, optimizer, and tokenizer all play key roles. In particular, we identify two different routes through which models can acquire geometrically separable features: they can learn them from complementary co-occurrence signals in general language data, including text-number co-occurrence and cross-number interaction, or from multi-token (but not single-token) addition problems. Overall, our results highlight the phenomenon of convergent evolution in feature learning: A diverse range of models learn similar features from different training signals.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 22 Apr 2026.

Builds on

24 of 35 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  3. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  4. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  5. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  6. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  7. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  8. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 629
  9. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  10. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 259
  11. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  12. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
12 more
  1. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 156
  2. 2023Grokking modular arithmeticcited by 83
  3. 2025Language Models Use Trigonometry to Do Additioncited by 70
  4. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Modelcited by 46
  5. 2024Pre-trained Large Language Models Use Fourier Features to Compute Additioncited by 39
  6. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 39
  7. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 33
  8. 2024Language Models Encode Numbers Using Digit Representations in Base 10cited by 32
  9. 2026Symmetry in language statistics shapes the geometry of model representationscited by 24
  10. 2025FoNE: Precise Single-Token Number Embeddings via Fourier Featurescited by 22
  11. 2024Linear Recursive Feature Machines provably recover low-rank matricescited by 16
  12. 2023A polar prediction model for learning to represent visual transformationscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Conceptscited by 10
  2. 2026TabFM: A Zero-Shot Foundation Model for Tabular Datacited by 6
  3. 2026Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifoldscited by 5
  4. 2026RepBench: Compiling Benchmarks into Capability Representations for Large Language Modelscited by 0
  5. 2026Global Divergence, Local Convergence: Representation Geometry in SSMs and Transformerscited by 0
  6. 2026TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Modelscited by 0