# The Illusion of Stochasticity in LLMs

arXiv: https://arxiv.org/abs/2604.06543 · HTML: https://mlsanity.com/paper/2604.06543/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 8 Apr 2026.

## Builds on (most cited first, 30 of 32)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://mlsanity.com/paper/2412.21187/) (2024)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games](https://mlsanity.com/paper/2411.13543/) (2024)
- [Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs](https://mlsanity.com/paper/2402.14903/) (2024)
- [Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity](https://mlsanity.com/paper/2510.01171/) (2025)
- [Amortizing intractable inference in large language models](https://mlsanity.com/paper/2310.04363/) (2023)
- [Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey](https://mlsanity.com/paper/2503.15850/) (2025)
- [Transformers need glasses! Information over-squashing in language tasks](https://mlsanity.com/paper/2406.04267/) (2024)
- [Inverse Scaling in Test-Time Compute](https://mlsanity.com/paper/2507.14417/) (2025)
- [Forcing Diffuse Distributions out of Language Models](https://mlsanity.com/paper/2404.10859/) (2024)
- [Number Cookbook: Number Understanding of Language Models and How to Improve It](https://mlsanity.com/paper/2411.03766/) (2024)
- [LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities](https://mlsanity.com/paper/2504.16078/) (2025)
- [EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration](https://mlsanity.com/paper/2410.06238/) (2024)
- [LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations](https://mlsanity.com/paper/2412.01441/) (2024)
- [The CLRS-Text Algorithmic Reasoning Language Benchmark](https://mlsanity.com/paper/2406.04229/) (2024)
- [Heads, Tails, and AI Fails: LLMs, Randomness, and Human Judgments](https://mlsanity.com/paper/2406.00092/) (2024)
- [Enough Coin Flips Can Make LLMs Act Bayesian](https://mlsanity.com/paper/2503.04722/) (2025)
- [Do LLMs Play Dice? Exploring Probability Distribution Sampling in Large Language Models for Behavioral Simulation](https://mlsanity.com/paper/2404.09043/) (2024)
- [Fourier Head: Helping Large Language Models Learn Complex Probability Distributions](https://mlsanity.com/paper/2410.22269/) (2024)
- [A Comparison of Large Language Model and Human Performance on Random Number Generation Tasks](https://mlsanity.com/paper/2408.09656/) (2024)
- [Playing games with Large language models: Randomness and strategy](https://mlsanity.com/paper/2503.02582/) (2025)
- [Flipping Against All Odds: Reducing LLM Coin Flip Bias via Verbalized Rejection Sampling](https://mlsanity.com/paper/2506.09998/) (2025)
- [Efficient numeracy in language models through single-token number embeddings](https://mlsanity.com/paper/2510.06824/) (2025)

## Built on by (most cited first, 9 of 9)

- [Probabilistic Calibration Is a Trainable Capability in Language Models](https://mlsanity.com/paper/2605.11845/) (2026)
- [Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models](https://mlsanity.com/paper/2607.15277/) (2026)
- [Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization](https://mlsanity.com/paper/2605.19782/) (2026)
- [UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs](https://mlsanity.com/paper/2606.06622/) (2026)
- [Structured Inference with Large Language Gibbs](https://mlsanity.com/paper/2606.19264/) (2026)
- [Mitigating Scoring Bias in LLM-as-a-Judge via Random Number Generation](https://mlsanity.com/paper/2608.05726/) (2026)
- [Sample What You Say: Aligning Language Models to Sample the Distributions They State](https://mlsanity.com/paper/2609.34929/) (2026)
- [Shared and structured inputs undermine collective random choice by reasoning AI agents](https://mlsanity.com/paper/2610.09667/) (2026)
- [Big-Big-SycoBench: A Benchmark to Detect Sycophancy Through Model Responses to Valid vs. Invalid Arguments](https://mlsanity.com/paper/2610.15634/) (2026)

## Abstract (arXiv)

In this work, we demonstrate that reliable stochastic sampling is a fundamental yet unfulfilled requirement for Large Language Models (LLMs) operating as agents. Agentic systems are frequently required to sample from distributions, often inferred from observed data, a process which needs to be emulated by the LLM. This leads to a distinct failure point: while standard RL agents rely on external sampling mechanisms, LLMs fail to map their internal probability estimates to their stochastic outputs. Through rigorous empirical analysis across multiple model families, model sizes, prompting styles, and distributions, we demonstrate the extent of this failure. Crucially, we show that while powerful frontier models can convert provided random seeds to target distributions, their ability to sample directly from specific distributions is fundamentally flawed.