mlsanityData to 14 Oct 2026

Creative Preference Optimization

arXiv 2505.14442 ↗First posted 20 May 2025cs.CL cs.AIcited by 25

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

While Large Language Models (LLMs) have demonstrated impressive performance across natural language generation tasks, their ability to generate truly creative content-characterized by novelty, diversity, surprise, and quality-remains limited. Existing methods for enhancing LLM creativity often focus narrowly on diversity or specific tasks, failing to address creativity's multifaceted nature in a generalizable way. In this work, we propose Creative Preference Optimization (CrPO), a novel alignment method that injects signals from multiple creativity dimensions into the preference optimization objective in a modular fashion. We train and evaluate creativity-augmented versions of several models using CrPO and MuCE, a new large-scale human preference dataset spanning over 200,000 human-generated responses and ratings from more than 30 psychological creativity assessments. Our models outperform strong baselines, including GPT-4o, on both automated and human evaluations, producing more novel, diverse, and surprising generations while maintaining high output quality. Additional evaluations on NoveltyBench further confirm the generalizability of our approach. Together, our results demonstrate that directly optimizing for creativity within preference frameworks is a promising direction for advancing the creative capabilities of LLMs without compromising output quality.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 113 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

34 of 113 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Evaluating the Diversity and Quality of LLM Generated Content16 Apr
  2. 2025Modifying Large Language Model Post-Training for Diverse Creative Writing21 Mar
  3. 2025Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMs27 Feb

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  7. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  8. 2018Hierarchical Neural Story Generationcited by 1,193
  9. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  10. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  11. 2023Does Writing with Language Models Reduce Content Diversity?cited by 192
  12. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 172
19 more
  1. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 150
  2. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 141
  3. 2024Homogenization Effects of Large Language Models on Human Creative Ideationcited by 127
  4. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 125
  5. 2024Are Large Language Models Capable of Generating Human-Level Narratives?cited by 79
  6. 2022Putting GPT-3's Creativity to the (Alternative Uses) Testcited by 62
  7. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 51
  8. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
  9. 2024Echoes in AI: Quantifying lack of plot diversity in LLM outputscited by 48
  10. 2025We're Different, We're the Same: Creative Homogeneity Across LLMscited by 42
  11. 2023LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzlescited by 41
  12. 2023Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMscited by 36
  13. 2024Creative Beam Search: LLM-as-a-Judge For Improving Response Generationcited by 14
  14. 2024Enhancing Creativity in Large Language Models through Associative Thinking Strategiescited by 13
  15. 2024Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzlescited by 8
  16. 2024Creative Problem Solving in Large Language and Vision Models -- What Would it Take?cited by 7
  17. 2020The societal and ethical relevance of computational creativitycited by 4
  18. 2025S-DAT: A Multilingual, GenAI-Driven Framework for Automated Divergent Thinking Assessmentcited by 3
  19. 2020A blindspot of AI ethics: anti-fragility in statistical predictioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 18.

25 papers.

Sort:

By area:

  1. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 31
  2. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  3. 2025DiverseGRPO: Mitigating Mode Collapse in Image Generation via Diversity-Aware GRPOcited by 17
  4. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoningcited by 11
  5. 2025Multi-Actor Generative Artificial Intelligence as a Game Enginecited by 8
  6. 2025Geometry of Knowledge Allows Extending Diversity Boundaries of Large Language Modelscited by 3
  7. 2026Time-Annealed Perturbation Sampling: Diverse Generation for Diffusion Language Modelscited by 3
  8. 2025Generation Space Size: Understanding and Calibrating Open-Endedness of LLM Generationscited by 2
  9. 2025Enhancing Diversity of LLM-Generated Educational Taskscited by 2
  10. 2026An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shiftcited by 2
  11. 2026Exploring Diverse Generation Paths via Inference-time Stiefel Activation Steeringcited by 2
  12. 2026Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMscited by 2
13 more
  1. 2026Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learningcited by 2
  2. 2026Scaling Participation in Modular AI Systemscited by 2
  3. 2026Shared Nature, Unique Nurture: PRISM for Pluralistic Reasoning via In-context Structure Modelingcited by 1
  4. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  5. 2026The Limits of Automatic Evaluation of Creativity in Large Language Modelscited by 1
  6. 2026Creative Generation via Multi-Agent Debate: Does Debate Suppress Diversity?cited by 1
  7. 2026CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledgecited by 0
  8. 2026Large Language Models Align with the Human Brain during Creative Thinkingcited by 0
  9. 2026Discovering Novel LLM Experts via Task-Capability Coevolutioncited by 0
  10. 2026POLARIS: Guiding Small Models to Write Long Storiescited by 0
  11. 2026GENIE: A Fine-Grained Measure for Noveltycited by 0
  12. 2026AGC-Bench: Measuring Artificial General Creativitycited by 0
  13. 2026CLIN: an Objective Framework for Evaluating Creativity in Short Persian Literary Textcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Creativity Has Left the Chat: The Price of Debiasing Language Models
  2. 2025Creativity Benchmark: A benchmark for marketing creativity for large language models
  3. 2025CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity
  4. 2025Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Models
  5. 2025CreBench: Human-Aligned Creativity Evaluation from Idea to Process to Product
  6. 2026Playing with Words, Improving with Rewards: Training Language Models for Creative Association
  7. 2025Curiosity-Driven LLM-as-a-judge for Personalized Creative Judgment
  8. 2025Beyond Correctness: Evaluating Subjective Writing Preferences Across Cultures