# Base Models Beat Aligned Models at Randomness and Creativity

arXiv: https://arxiv.org/abs/2505.00047 · HTML: https://mlsanity.com/paper/2505.00047/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 30 Apr 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 26 of 26)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://mlsanity.com/paper/2312.01552/) (2023)
- [Playing repeated games with Large Language Models](https://mlsanity.com/paper/2305.16867/) (2023)
- [Does Writing with Language Models Reduce Content Diversity?](https://mlsanity.com/paper/2309.05196/) (2023)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [Homogenization Effects of Large Language Models on Human Creative Ideation](https://mlsanity.com/paper/2402.01536/) (2024)
- [The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism](https://mlsanity.com/paper/2407.10457/) (2024)
- [A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing](https://mlsanity.com/paper/2310.08433/) (2023)
- [Capturing Failures of Large Language Models via Human Cognitive Biases](https://mlsanity.com/paper/2202.12299/) (2022)
- [Benchmarking Distributional Alignment of Large Language Models](https://mlsanity.com/paper/2411.05403/) (2024)
- [One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity](https://mlsanity.com/paper/2411.04427/) (2024)
- [Large Language Models Assume People are More Rational than We Really are](https://mlsanity.com/paper/2406.17055/) (2024)
- [Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias](https://mlsanity.com/paper/2308.00225/) (2023)
- [Evaluating the Diversity and Quality of LLM Generated Content](https://mlsanity.com/paper/2504.12522/) (2025)
- [AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text](https://mlsanity.com/paper/2410.04265/) (2024)
- [Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writers](https://mlsanity.com/paper/2309.12570/) (2023)
- [Exploring Precision and Recall to assess the quality and diversity of LLMs](https://mlsanity.com/paper/2402.10693/) (2024)
- [Heads, Tails, and AI Fails: LLMs, Randomness, and Human Judgments](https://mlsanity.com/paper/2406.00092/) (2024)
- [What Are the Odds? Language Models Are Capable of Probabilistic Reasoning](https://mlsanity.com/paper/2406.12830/) (2024)
- [Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modeling](https://mlsanity.com/paper/2407.02446/) (2024)
- [LLM Strategic Reasoning: Agentic Study through Behavioral Game Theory](https://mlsanity.com/paper/2502.20432/) (2025)
- [Instruction Following without Instruction Tuning](https://mlsanity.com/paper/2409.14254/) (2024)
- [Large Language Models Playing Mixed Strategy Nash Equilibrium Games](https://mlsanity.com/paper/2406.10574/) (2024)

## Built on by (most cited first, 30 of 51)

- [KL-Regularized Reinforcement Learning is Designed to Mode Collapse](https://mlsanity.com/paper/2510.20817/) (2025)
- [LLM Probability Concentration: How Alignment Shrinks the Generative Horizon](https://mlsanity.com/paper/2506.17871/) (2025)
- [Creative Preference Optimization](https://mlsanity.com/paper/2505.14442/) (2025)
- [PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments](https://mlsanity.com/paper/2603.23231/) (2026)
- [Stop Automating Peer Review Without Rigorous Evaluation](https://mlsanity.com/paper/2605.03202/) (2026)
- [Where does output diversity collapse in post-training?](https://mlsanity.com/paper/2604.16027/) (2026)
- [When Two LLMs Debate, Both Think They'll Win](https://mlsanity.com/paper/2505.19184/) (2025)
- [Don't Throw Away Your Pretrained Model](https://mlsanity.com/paper/2510.09913/) (2025)
- [Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Models](https://mlsanity.com/paper/2505.16245/) (2025)
- [Optimizing Diversity and Quality through Base-Aligned Model Collaboration](https://mlsanity.com/paper/2511.05650/) (2025)
- [CREATE: Testing LLMs for Associative Creativity](https://mlsanity.com/paper/2603.09970/) (2026)
- [Reaching Beyond the Mode: RL for Distributional Reasoning in Language Models](https://mlsanity.com/paper/2603.24844/) (2026)
- [Unlocking LLM Creativity in Science through Analogical Reasoning](https://mlsanity.com/paper/2605.11258/) (2026)
- [Annotations Mitigate Post-Training Mode Collapse](https://mlsanity.com/paper/2605.09995/) (2026)
- [LLMs Exhibit Significantly Lower Uncertainty in Creative Writing Than Professional Writers](https://mlsanity.com/paper/2602.16162/) (2026)
- [Cognitive models can reveal interpretable value trade-offs in language models](https://mlsanity.com/paper/2506.20666/) (2025)
- [Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks](https://mlsanity.com/paper/2601.21339/) (2026)
- [Generation Space Size: Understanding and Calibrating Open-Endedness of LLM Generations](https://mlsanity.com/paper/2510.12699/) (2025)
- [The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safety](https://mlsanity.com/paper/2601.06116/) (2026)
- [PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matching](https://mlsanity.com/paper/2603.18363/) (2026)
- [No Single Best Model for Diversity: Learning a Router for Sample Diversity](https://mlsanity.com/paper/2604.02319/) (2026)
- [Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMs](https://mlsanity.com/paper/2605.11128/) (2026)
- [Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learning](https://mlsanity.com/paper/2606.03962/) (2026)
- [Are LLMs becoming similarly creative? Evidence from three years of models](https://mlsanity.com/paper/2608.19437/) (2026)
- [Effective Reinforcement Learning for Reasoning in Language Models](https://mlsanity.com/paper/2505.17218/) (2025)
- [IDRBench: Understanding the Capability of Large Language Models on Interdisciplinary Research](https://mlsanity.com/paper/2507.15736/) (2025)
- [Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning](https://mlsanity.com/paper/2511.01490/) (2025)
- [D-Models and E-Models: Diversity-Stability Trade-offs in the Sampling Behavior of Large Language Models](https://mlsanity.com/paper/2601.17865/) (2026)
- [Diversity Without Fidelity: A Solver-Sampler Mismatch in Multi-Agent LLM Negotiation Simulation](https://mlsanity.com/paper/2604.11840/) (2026)
- [Can You Keep a Secret? Involuntary Information Leakage in Language Model Writing](https://mlsanity.com/paper/2605.10794/) (2026)

## Abstract (arXiv)

Alignment has quickly become a default ingredient in LLM development, with techniques such as reinforcement learning from human feedback making models act safely, follow instructions, and perform ever-better on complex tasks. While these techniques are certainly useful, we propose that they should not be universally applied and demonstrate a range of tasks on which base language models consistently outperform their popular aligned forms. Particularly, we study tasks that require unpredictable outputs, such as random number generation, mixed strategy games (rock-paper-scissors and hide-and-seek), and creative writing. In each case, aligned models tend towards narrow behaviors that result in distinct disadvantages, for instance, preferring to generate "7" over other uniformly random numbers, becoming almost fully predictable in some game states, or prioritizing pleasant writing over creative originality. Across models tested, better performance on common benchmarks tends to correlate with worse performance on our tasks, suggesting an effective trade-off in the required capabilities.