# Creativity Has Left the Chat: The Price of Debiasing Language Models

arXiv: https://arxiv.org/abs/2406.05587 · HTML: https://mlsanity.com/paper/2406.05587/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 8 Jun 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [RRHF: Rank Responses to Align Language Models with Human Feedback without tears](https://mlsanity.com/paper/2304.05302/) (2023)
- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [Mitigating the Alignment Tax of RLHF](https://mlsanity.com/paper/2309.06256/) (2023)
- [GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models](https://mlsanity.com/paper/2303.10130/) (2023)
- [Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.05199/) (2023)
- [On the Creativity of Large Language Models](https://mlsanity.com/paper/2304.00008/) (2023)
- [The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2311.00168/) (2023)
- [RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models](https://mlsanity.com/paper/2311.09641/) (2023)
- [Explaining Large Language Models Decisions Using Shapley Values](https://mlsanity.com/paper/2404.01332/) (2024)
- [Fine-tuning Language Models with Generative Adversarial Reward Modelling](https://mlsanity.com/paper/2305.06176/) (2023)

## Built on by (most cited first, 29 of 29)

- [Echoes in AI: Quantifying lack of plot diversity in LLM outputs](https://mlsanity.com/paper/2501.00273/) (2024)
- [Cascade Reward Sampling for Efficient Decoding-Time Alignment](https://mlsanity.com/paper/2406.16306/) (2024)
- [Preference Optimization with Multi-Sample Comparisons](https://mlsanity.com/paper/2410.12138/) (2024)
- [CBEval: A framework for evaluating and interpreting cognitive biases in LLMs](https://mlsanity.com/paper/2412.03605/) (2024)
- [Diversity-Rewarded CFG Distillation](https://mlsanity.com/paper/2410.06084/) (2024)
- [Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLM](https://mlsanity.com/paper/2508.05775/) (2025)
- [Towards Building Specialized Generalist AI with System 1 and System 2 Fusion](https://mlsanity.com/paper/2407.08642/) (2024)
- [Creativity in AI: Progresses and Challenges](https://mlsanity.com/paper/2410.17218/) (2024)
- [The Drama Machine: Simulating Character Development with LLM Agents](https://mlsanity.com/paper/2408.01725/) (2024)
- [LLM Review: Enhancing Creative Writing via Blind Peer Review Feedback](https://mlsanity.com/paper/2601.08003/) (2026)
- [Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner](https://mlsanity.com/paper/2508.15044/) (2025)
- [LLMs Exhibit Significantly Lower Uncertainty in Creative Writing Than Professional Writers](https://mlsanity.com/paper/2602.16162/) (2026)
- [Towards a Science Exocortex](https://mlsanity.com/paper/2406.17809/) (2024)
- [AttentionDefense: Leveraging System Prompt Attention for Explainable Defense Against Novel Jailbreaks](https://mlsanity.com/paper/2504.12321/) (2025)
- [Localizing Persona Representations in LLMs](https://mlsanity.com/paper/2505.24539/) (2025)
- [Exploring the Effects of Alignment on Numerical Bias in Large Language Models](https://mlsanity.com/paper/2601.16444/) (2026)
- [Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks](https://mlsanity.com/paper/2601.21339/) (2026)
- [The Reasoning-Creativity Trade-off: Toward Creativity-Driven Problem Solving](https://mlsanity.com/paper/2601.00747/) (2026)
- [Detecting RLVR Training Data via Structural Convergence of Reasoning](https://mlsanity.com/paper/2602.11792/) (2026)
- [Probabilistic Calibration Is a Trainable Capability in Language Models](https://mlsanity.com/paper/2605.11845/) (2026)
- [Automated Creativity Evaluation of Language Models Across Open-Ended Tasks](https://mlsanity.com/paper/2606.11762/) (2026)
- [IDRBench: Understanding the Capability of Large Language Models on Interdisciplinary Research](https://mlsanity.com/paper/2507.15736/) (2025)
- [Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety](https://mlsanity.com/paper/2509.12936/) (2025)
- [Designing AI-Resilient Assessments Using Interconnected Problems: A Theoretically Grounded and Empirically Validated Framework](https://mlsanity.com/paper/2512.10758/) (2025)
- [Galton's Law of Mediocrity: Why Large Language Models Regress to the Mean and Fail at Creativity in Advertising](https://mlsanity.com/paper/2509.25767/) (2025)
- [A Representation-Level Assessment of Bias Mitigation in Foundation Models](https://mlsanity.com/paper/2604.08561/) (2026)
- [Selective Safety Steering via Value-Filtered Decoding](https://mlsanity.com/paper/2605.14746/) (2026)
- [The Assistant as a Privileged Persona: A canonical reference in cross-persona self-recognition](https://mlsanity.com/paper/2606.00545/) (2026)
- [The Judge Is Not Its Twin: Post-training makes a model's writing more predictable but barely moves its taste, as a judge, toward predictable writing](https://mlsanity.com/paper/2609.32196/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have revolutionized natural language processing but can exhibit biases and may generate toxic content. While alignment techniques like Reinforcement Learning from Human Feedback (RLHF) reduce these issues, their impact on creativity, defined as syntactic and semantic diversity, remains unexplored. We investigate the unintended consequences of RLHF on the creativity of LLMs through three experiments focusing on the Llama-2 series. Our findings reveal that aligned models exhibit lower entropy in token predictions, form distinct clusters in the embedding space, and gravitate towards "attractor states", indicating limited output diversity. Our findings have significant implications for marketers who rely on LLMs for creative tasks such as copywriting, ad creation, and customer persona generation. The trade-off between consistency and creativity in aligned models should be carefully considered when selecting the appropriate model for a given application. We also discuss the importance of prompt engineering in harnessing the creative potential of base models.