# (How) Can Transformers Predict Pseudo-Random Numbers?

arXiv: https://arxiv.org/abs/2502.10390 · HTML: https://mlsanity.com/paper/2502.10390/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2025.

## Built on by (most cited first, 6 of 6)

- [On the geometry and topology of representations: the manifolds of modular addition](https://mlsanity.com/paper/2512.25060/) (2025)
- [Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate](https://mlsanity.com/paper/2605.21486/) (2026)
- [Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch](https://mlsanity.com/paper/2607.03660/) (2026)
- [Transformers in Pseudo-Random Number Generation: A Dual Perspective on Theory and Practice](https://mlsanity.com/paper/2508.01134/) (2025)
- [Learning Pseudorandom Numbers with Transformers: Permuted Congruential Generators, Curricula, and Interpretability](https://mlsanity.com/paper/2510.26792/) (2025)
- [Noise in Diffusion Models Is a Learnable Input](https://mlsanity.com/paper/2608.02575/) (2026)

## Abstract (arXiv)

Transformers excel at discovering patterns in sequential data, yet their fundamental limitations and learning mechanisms remain crucial topics of investigation. In this paper, we study the ability of Transformers to learn pseudo-random number sequences from linear congruential generators (LCGs), defined by the recurrence relation xt+1 = a xt + c ;mod; m. We find that with sufficient architectural capacity and training data variety, Transformers can perform in-context prediction of LCG sequences with unseen moduli (m) and parameters (a,c). By analyzing the embedding layers and attention patterns, we uncover how Transformers develop algorithmic structures to learn these sequences in two scenarios of increasing complexity. First, we investigate how Transformers learn LCG sequences with unseen (a, c) but fixed modulus; and demonstrate successful learning up to m = 232. We find that models learn to factorize m and utilize digit-wise number representations to make sequential predictions. In the second, more challenging scenario of unseen moduli, we show that Transformers can generalize to unseen moduli up to mtest = 216. In this case, the model employs a two-step strategy: first estimating the unknown modulus from the context, then utilizing prime factorizations to generate predictions. For this task, we observe a sharp transition in the accuracy at a critical depth d= 3. We also find that the number of in-context sequence elements needed to reach high accuracy scales sublinearly with the modulus.