mlsanityData to 14 Oct 2026

Base Models Beat Aligned Models at Randomness and Creativity

arXiv 2505.00047 ↗First posted 30 Apr 2025cs.CLcited by 51

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Alignment has quickly become a default ingredient in LLM development, with techniques such as reinforcement learning from human feedback making models act safely, follow instructions, and perform ever-better on complex tasks. While these techniques are certainly useful, we propose that they should not be universally applied and demonstrate a range of tasks on which base language models consistently outperform their popular aligned forms. Particularly, we study tasks that require unpredictable outputs, such as random number generation, mixed strategy games (rock-paper-scissors and hide-and-seek), and creative writing. In each case, aligned models tend towards narrow behaviors that result in distinct disadvantages, for instance, preferring to generate "7" over other uniformly random numbers, becoming almost fully predictable in some game states, or prioritizing pleasant writing over creative originality. Across models tested, better performance on common benchmarks tends to correlate with worse performance on our tasks, suggesting an effective trade-off in the required capabilities.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 46 references that are in the map. First posted on arXiv 30 Apr 2025.

Builds on

26 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning4 Dec
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 24

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2023Whose Opinions Do Language Models Reflect?cited by 773
  3. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  4. 2023Playing repeated games with Large Language Modelscited by 217
  5. 2023Does Writing with Language Models Reduce Content Diversity?cited by 192
  6. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 178
  7. 2024Homogenization Effects of Large Language Models on Human Creative Ideationcited by 127
  8. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 116
  9. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 99
  10. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 98
  11. 2024Benchmarking Distributional Alignment of Large Language Modelscited by 61
  12. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 58
12 more
  1. 2024Large Language Models Assume People are More Rational than We Really arecited by 54
  2. 2023Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Biascited by 53
  3. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  4. 2024AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Textcited by 42
  5. 2023Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writerscited by 32
  6. 2024Exploring Precision and Recall to assess the quality and diversity of LLMscited by 29
  7. 2024Heads, Tails, and AI Fails: LLMs, Randomness, and Human Judgmentscited by 26
  8. 2024What Are the Odds? Language Models Are Capable of Probabilistic Reasoningcited by 26
  9. 2024Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modelingcited by 20
  10. 2025LLM Strategic Reasoning: Agentic Study through Behavioral Game Theorycited by 17
  11. 2024Instruction Following without Instruction Tuningcited by 12
  12. 2024Large Language Models Playing Mixed Strategy Nash Equilibrium Gamescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 38.

51 papers.

Sort:

By area:

  1. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 31
  2. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  3. 2025Creative Preference Optimizationcited by 25
  4. 2026PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environmentscited by 14
  5. 2026Stop Automating Peer Review Without Rigorous Evaluationcited by 14
  6. 2026Where does output diversity collapse in post-training?cited by 11
  7. 2025When Two LLMs Debate, Both Think They'll Wincited by 10
  8. 2025Don't Throw Away Your Pretrained Modelcited by 10
  9. 2025Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Modelscited by 9
  10. 2025Optimizing Diversity and Quality through Base-Aligned Model Collaborationcited by 8
  11. 2026CREATE: Testing LLMs for Associative Creativitycited by 7
  12. 2026Reaching Beyond the Mode: RL for Distributional Reasoning in Language Modelscited by 6
38 more
  1. 2026Unlocking LLM Creativity in Science through Analogical Reasoningcited by 6
  2. 2026Annotations Mitigate Post-Training Mode Collapsecited by 5
  3. 2026LLMs Exhibit Significantly Lower Uncertainty in Creative Writing Than Professional Writerscited by 4
  4. 2025Cognitive models can reveal interpretable value trade-offs in language modelscited by 3
  5. 2026Within-Model vs Between-Prompt Variability in Large Language Models for Creative Taskscited by 3
  6. 2025Generation Space Size: Understanding and Calibrating Open-Endedness of LLM Generationscited by 2
  7. 2026The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safetycited by 2
  8. 2026PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matchingcited by 2
  9. 2026No Single Best Model for Diversity: Learning a Router for Sample Diversitycited by 2
  10. 2026Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMscited by 2
  11. 2026Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learningcited by 2
  12. 2026Are LLMs becoming similarly creative? Evidence from three years of modelscited by 2
  13. 2025Effective Reinforcement Learning for Reasoning in Language Modelscited by 1
  14. 2025IDRBench: Understanding the Capability of Large Language Models on Interdisciplinary Researchcited by 1
  15. 2025Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuningcited by 1
  16. 2026D-Models and E-Models: Diversity-Stability Trade-offs in the Sampling Behavior of Large Language Modelscited by 1
  17. 2026Diversity Without Fidelity: A Solver-Sampler Mismatch in Multi-Agent LLM Negotiation Simulationcited by 1
  18. 2026Can You Keep a Secret? Involuntary Information Leakage in Language Model Writingcited by 1
  19. 2026Narrative Flattening: How Post-Training Compresses Thematic, Affective, and Stylistic Variation in LLM Fictioncited by 1
  20. 2026AI Coding Agents in Social Science: Methodologically Diverse, Empirically Consistent, Interpretively Vulnerablecited by 1
  21. 2026Towards Physical Intuitions for Alignment Dynamics: A Case Study With Randomness Crystallizationcited by 1
  22. 2026Is Convergence Inevitable? Tracing Output Homogeneity Back to Base Modelscited by 1
  23. 2026Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learningcited by 1
  24. 2025Sample, Align, Synthesize: Graph-Based Response Synthesis with ConGrscited by 0
  25. 2026Triviality Corrected Endogenous Rewardcited by 0
  26. 2026Activation Steering for Synthetic Data Generation: The Role of Diversity in Downstream Safety Detectioncited by 0
  27. 2026Fine-Tuning Improves Information Conveyance in Language Modelscited by 0
  28. 2026IDEAFix: Evaluation Framework for Creative Defixation Prompting in LLMscited by 0
  29. 2026GENIE: A Fine-Grained Measure for Noveltycited by 0
  30. 2026Decomposer: Learning to Decompile Symbolic Music to Programscited by 0
  31. 2026Mitigating Scoring Bias in LLM-as-a-Judge via Random Number Generationcited by 0
  32. 2026CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversitycited by 0
  33. 2026Pretrained Persona Mixture Models and Tandem Models for Human Simulationcited by 0
  34. 2026Breaking Homogeneity: Diversifying Persona Sets for Creative LLM Outputscited by 0
  35. 2026Sample What You Say: Aligning Language Models to Sample the Distributions They Statecited by 0
  36. 2026Gacha Decoding: Eliciting Diverse Generations Through Instruction Followingcited by 0
  37. 2026Divergence controls entropy in distillationcited by 0
  38. 2026Design Creativity Bench: Measuring creativity in LLM-Generated UIcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Creativity Has Left the Chat: The Price of Debiasing Language Models
  2. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Design
  3. 2026Alignment Makes Language Models Normative, Not Descriptive
  4. 2024Your Weak LLM is Secretly a Strong Teacher for Alignment
  5. 2024A Probability--Quality Trade-off in Aligned Language Models and its Relation to Sampling Adaptors
  6. 2024Toward Optimal LLM Alignments Using Two-Player Games
  7. 2024Revisiting the Superficial Alignment Hypothesis
  8. 2026Playing with Words, Improving with Rewards: Training Language Models for Creative Association