# Heads, Tails, and AI Fails: LLMs, Randomness, and Human Judgments

arXiv: https://arxiv.org/abs/2406.00092 · HTML: https://mlsanity.com/paper/2406.00092/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 37 references that are in the map.
- First posted on arXiv 31 May 2024.

## Builds on (most cited first, 4 of 4)

- [SmartPlay: A Benchmark for LLMs as Intelligent Agents](https://mlsanity.com/paper/2310.01557/) (2023)
- [Can large language models explore in-context?](https://mlsanity.com/paper/2403.15371/) (2024)
- [The Theory is Predictive, but is it Complete? An Application to Human Perception of Randomness](https://mlsanity.com/paper/1706.06974/) (2017)
- [Frozen Binomials on the Web: Word Ordering and Language Conventions in Online Text](https://mlsanity.com/paper/2003.03612/) (2020)

## Built on by (most cited first, 26 of 26)

- [Vision Language Models are Biased](https://mlsanity.com/paper/2505.23941/) (2025)
- [Benchmarking Distributional Alignment of Large Language Models](https://mlsanity.com/paper/2411.05403/) (2024)
- [Base Models Beat Aligned Models at Randomness and Creativity](https://mlsanity.com/paper/2505.00047/) (2025)
- [Enough Coin Flips Can Make LLMs Act Bayesian](https://mlsanity.com/paper/2503.04722/) (2025)
- [A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments](https://mlsanity.com/paper/2509.25609/) (2025)
- [String Seed of Thought: Prompting LLMs for Distribution-Faithful and Diverse Generation](https://mlsanity.com/paper/2510.21150/) (2025)
- [The Illusion of Stochasticity in LLMs](https://mlsanity.com/paper/2604.06543/) (2026)
- [Deterministic or probabilistic? The psychology of LLMs as random number generators](https://mlsanity.com/paper/2502.19965/) (2025)
- [CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting](https://mlsanity.com/paper/2505.18102/) (2025)
- [Flipping Against All Odds: Reducing LLM Coin Flip Bias via Verbalized Rejection Sampling](https://mlsanity.com/paper/2506.09998/) (2025)
- [Strategic Algorithmic Monoculture: Experimental Evidence from Coordination Games](https://mlsanity.com/paper/2604.09502/) (2026)
- [B-score: Detecting biases in large language models using response history](https://mlsanity.com/paper/2505.18545/) (2025)
- [One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions](https://mlsanity.com/paper/2607.10252/) (2026)
- [Evaluating Binary Decision Biases in Large Language Models: Implications for Fair Agent-Based Financial Simulations](https://mlsanity.com/paper/2501.16356/) (2025)
- [LLM Agents Are Hypersensitive to Nudges](https://mlsanity.com/paper/2505.11584/) (2025)
- [LLMs Prompted for Graphs: Hallucinations and Generative Capabilities](https://mlsanity.com/paper/2409.00159/) (2024)
- [Can Hallucinations Help? Boosting LLMs for Drug Discovery](https://mlsanity.com/paper/2501.13824/) (2025)
- [Evaluating the Quality of Randomness and Entropy in Tasks Supported by Large Language Models](https://mlsanity.com/paper/2510.12080/) (2025)
- [FLIPS: Instance-Fingerprinting for LLMs via Pseudo-random Sequences](https://mlsanity.com/paper/2606.03330/) (2026)
- [Position: Behavioral Systems Require Behavioral Tests](https://mlsanity.com/paper/2608.18081/) (2026)
- [In Search of the Ingredients of Open-Endedness: Replicating Picbreeder with Large Vision-Language Models](https://mlsanity.com/paper/2605.23908/) (2026)
- [UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs](https://mlsanity.com/paper/2606.06622/) (2026)
- [Mutual Debiasing via Dual-Seed Comparison for Probabilistic Sampling in Large Language Models](https://mlsanity.com/paper/2608.26161/) (2026)
- [Sample What You Say: Aligning Language Models to Sample the Distributions They State](https://mlsanity.com/paper/2609.34929/) (2026)
- [Ask a Language Model for Lottery Numbers: Concentration in Repeated Six-of-49 Outputs](https://mlsanity.com/paper/2610.00052/) (2026)
- [Shared and structured inputs undermine collective random choice by reasoning AI agents](https://mlsanity.com/paper/2610.09667/) (2026)

## Abstract (arXiv)

Randomness is central to human cognition and to many applications in which large language models are deployed, yet probabilistic token generation does not imply that LLMs can produce unbiased random sequences. We study how contemporary LLMs generate binary random sequences using the classic behavioral-science paradigm of simulated coin flips. Across single flips, 20-flip sequences, n-gram statistics, run lengths, alternation rates, and next-flip predictability, we compare model outputs to both true Bernoulli baselines and human data from prior work. We find that LLMs reproduce several canonical human randomness biases, including over-alternation, aversion to long runs, and first-flip biases, but often amplify them or introduce model-specific distortions. Increasing temperature reduces some rigid patterns but does not eliminate systematic structure. We further investigate over-alternation through prompt-framing experiments, continuation tests, corpus searches, and residual-stream probes, finding evidence that this bias is not explained by direct memorization or tokenization alone. These results show that LLMs are unreliable generators of randomness and imperfect simulators of human randomness judgments, with implications for applications requiring stochasticity, fair sampling, or behavioral simulation.