mlsanityData to 7 Oct 2026

Creativity Has Left the Chat: The Price of Debiasing Language Models

arXiv 2406.05587 ↗First posted 8 Jun 2024cs.CL cs.AIcited by 29

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Large Language Models (LLMs) have revolutionized natural language processing but can exhibit biases and may generate toxic content. While alignment techniques like Reinforcement Learning from Human Feedback (RLHF) reduce these issues, their impact on creativity, defined as syntactic and semantic diversity, remains unexplored. We investigate the unintended consequences of RLHF on the creativity of LLMs through three experiments focusing on the Llama-2 series. Our findings reveal that aligned models exhibit lower entropy in token predictions, form distinct clusters in the embedding space, and gravitate towards "attractor states", indicating limited output diversity. Our findings have significant implications for marketers who rely on LLMs for creative tasks such as copywriting, ad creation, and customer persona generation. The trade-off between consistency and creativity in aligned models should be carefully considered when selecting the appropriate model for a given application. We also discuss the importance of prompt engineering in harnessing the creative potential of base models.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 29 references that are in the map. First posted on arXiv 8 Jun 2024.

Builds on

17 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Learning to summarize from human feedback2 Sep

Other references · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  7. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  8. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
  9. 2023Mitigating the Alignment Tax of RLHFcited by 181
  10. 2023GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Modelscited by 174
  11. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  12. 2023On the Creativity of Large Language Modelscited by 57
4 more
  1. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  2. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Modelscited by 32
  3. 2024Explaining Large Language Models Decisions Using Shapley Valuescited by 12
  4. 2023Fine-tuning Language Models with Generative Adversarial Reward Modellingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 8; 2026: 12.

29 papers.

Sort:

By area:

  1. 2024Echoes in AI: Quantifying lack of plot diversity in LLM outputscited by 47
  2. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  3. 2024Preference Optimization with Multi-Sample Comparisonscited by 16
  4. 2024CBEval: A framework for evaluating and interpreting cognitive biases in LLMscited by 12
  5. 2024Diversity-Rewarded CFG Distillationcited by 9
  6. 2024Towards Building Specialized Generalist AI with System 1 and System 2 Fusioncited by 8
  7. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 8
  8. 2024Creativity in AI: Progresses and Challengescited by 7
  9. 2024The Drama Machine: Simulating Character Development with LLM Agentscited by 5
  10. 2025Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Alignercited by 4
  11. 2026LLM Review: Enhancing Creative Writing via Blind Peer Review Feedbackcited by 4
  12. 2026LLMs Exhibit Significantly Lower Uncertainty in Creative Writing Than Professional Writerscited by 4
17 more
  1. 2024Towards a Science Exocortexcited by 3
  2. 2025AttentionDefense: Leveraging System Prompt Attention for Explainable Defense Against Novel Jailbreakscited by 3
  3. 2025Localizing Persona Representations in LLMscited by 3
  4. 2026Exploring the Effects of Alignment on Numerical Bias in Large Language Modelscited by 3
  5. 2026Within-Model vs Between-Prompt Variability in Large Language Models for Creative Taskscited by 3
  6. 2026Probabilistic Calibration Is a Trainable Capability in Language Modelscited by 2
  7. 2026Automated Creativity Evaluation of Language Models Across Open-Ended Taskscited by 2
  8. 2025IDRBench: Understanding the Capability of Large Language Models on Interdisciplinary Researchcited by 1
  9. 2025Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safetycited by 1
  10. 2025Designing AI-Resilient Assessments Using Interconnected Problems: A Theoretically Grounded and Empirically Validated Frameworkcited by 1
  11. 2026The Reasoning-Creativity Trade-off: Toward Creativity-Driven Problem Solvingcited by 1
  12. 2025Galton's Law of Mediocrity: Why Large Language Models Regress to the Mean and Fail at Creativity in Advertisingcited by 0
  13. 2026Detecting RLVR Training Data via Structural Convergence of Reasoningcited by 0
  14. 2026A Representation-Level Assessment of Bias Mitigation in Foundation Modelscited by 0
  15. 2026Selective Safety Steering via Value-Filtered Decodingcited by 0
  16. 2026The Assistant as a Privileged Persona: A canonical reference in cross-persona self-recognitioncited by 0
  17. 2026The Judge Is Not Its Twin: Post-training makes a model's writing more predictable but barely moves its taste, as a judge, toward predictable writingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Base Models Beat Aligned Models at Randomness and Creativity
  2. 2025Creative Preference Optimization
  3. 2025Creativity Benchmark: A benchmark for marketing creativity for large language models
  4. 2025Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedback
  5. 2026CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity
  6. 2026Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
  7. 2023Aligning Large Language Models through Synthetic Feedback
  8. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thought