mlsanityData to 7 Oct 2026

The Randomness Floor: Measuring Intrinsic Non-Randomness in Language Model Token Distributions

arXiv 2604.22771 ↗First posted 29 Mar 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Language models cannot be random. This paper introduces Entropic Deviation (ED), the normalised KL divergence between a model's token distribution and the uniform distribution, and measures it systematically across 31,200 generations spanning seven models, two architectures (transformer and state space), nine prompt categories, three temperatures, and five languages. Under semantically neutral prompts (empty strings, random characters, nonsense syllables) transformers still exhibit ED of approximately 0.30, meaning that 88-93% of the non-randomness observed under semantic prompts is intrinsic to the learned weights rather than induced by context. Three transformer families (Gemma, Llama, Qwen) converge on nearly identical ED values despite different training data and vocabularies. A state space model (Mamba2) reveals a qualitatively different regime: twice the ED, three times lower within-sequence variance, and massive sensitivity to temperature (r = -0.78) where transformers are nearly immune (r < 0.05). Cross-lingual experiments with Qwen-32B show a stable gradient across five languages (English, Japanese, Chinese, Polish, Arabic) that does not correlate with token fertility and persists when two languages sharing an identical tokeniser subset are compared. These findings establish a structural lower bound on randomness in pretrained language models, characterise how this bound differs across architectures, and demonstrate that language itself modulates the bound independently of tokenisation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 9 of its 11 references that are in the map. First posted on arXiv 29 Mar 2026.

Builds on

9 of 11 references.

References · 9

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  7. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  8. 2025Deterministic or probabilistic? The psychology of LLMs as random number generatorscited by 6
  9. 2026Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributionscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Probabilistic Calibration Is a Trainable Capability in Language Models
  2. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale
  3. 2026Is my model perplexed for the right reason? Contrasting LLMs' Benchmark Behavior with Token-Level Perplexity
  4. 2023Diversity Measures: Domain-Independent Proxies for Failure in Language Model Queries
  5. 2023A Latent Space Theory for Emergent Abilities in Large Language Models
  6. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Models
  7. 2023Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets
  8. 2023Self-Consistency of Large Language Models under Ambiguity