mlsanityData to 14 Oct 2026

Modifying Large Language Model Post-Training for Diverse Creative Writing

arXiv 2503.17126 ↗First posted 21 Mar 2025cs.CL cs.LGcited by 40

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

As creative writing tasks do not have singular correct answers, large language models (LLMs) trained to perform these tasks should be able to generate diverse valid outputs. However, LLM post-training often focuses on improving generation quality but neglects to facilitate output diversity. Hence, in creative writing generation, we investigate post-training approaches to promote both output diversity and quality. Our core idea is to include deviation -- the degree of difference between a training sample and all other samples with the same prompt -- in the training objective to facilitate learning from rare high-quality instances. By adopting our approach to direct preference optimization (DPO) and odds ratio preference optimization (ORPO), we demonstrate that we can promote the output diversity of trained models while minimally decreasing quality. Our best model with 8B parameters could achieve on-par diversity as a human-created dataset while having output quality similar to the best instruction-tuned models we examined, GPT-4o and DeepSeek-R1. We further validate our approaches with a human evaluation, an ablation, and a comparison to an existing diversification approach, DivPO.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 43 references that are in the map. First posted on arXiv 21 Mar 2025.

Builds on

33 of 43 references.

References · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2018Hierarchical Neural Story Generationcited by 1,193
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  8. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  9. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  10. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 301
  11. 2023Does Writing with Language Models Reduce Content Diversity?cited by 192
  12. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 150
21 more
  1. 2023Co-Writing with Opinionated Language Models Affects Users' Viewscited by 148
  2. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 129
  3. 2024Homogenization Effects of Large Language Models on Human Creative Ideationcited by 127
  4. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 125
  5. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 120
  6. 2020Trading Off Diversity and Quality in Natural Language Generationcited by 96
  7. 2022Same Author or Just Same Topic? Towards Content-Independent Style Representationscited by 86
  8. 2019Comparison of Diverse Decoding Methods from Conditional Language Modelscited by 78
  9. 2022POTATO: The Portable Text Annotation Toolcited by 64
  10. 2023Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creationcited by 64
  11. 2016A Simple, Fast Diverse Decoding Algorithm for Neural Generationcited by 58
  12. 2024Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scorescited by 50
  13. 2024Echoes in AI: Quantifying lack of plot diversity in LLM outputscited by 48
  14. 2023Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Votingcited by 46
  15. 2024EDT: Improving Large Language Models' Generation by Entropy-based Dynamic Temperature Samplingcited by 45
  16. 2023Quality-Diversity through AI Feedbackcited by 40
  17. 2021Directed Diversity: Leveraging Language Embedding Distances for Collective Creativity in Crowd Ideationcited by 23
  18. 2023How Far Can We Extract Diverse Perspectives from Large Language Models?cited by 22
  19. 2023Unmasking Nationality Bias: A Study of Human Perception of Nationalities in AI-Generated Articlescited by 16
  20. 2024Search Engines in an AI Era: The False Promise of Factual and Verifiable Source-Cited Responsescited by 15
  21. 2024Guiding and Diversifying LLM-Based Story Generation via Answer Set Programmingcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 26.

40 papers.

Sort:

By area:

  1. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  2. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 45
  3. 2025LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writingcited by 42
  4. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  5. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 31
  6. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  7. 2025Creative Preference Optimizationcited by 25
  8. 2025Post-training Large Language Models for Diverse High-Quality Responsescited by 19
  9. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoningcited by 11
  10. 2025Bootstrapping Task Spaces for Self-Improvementcited by 10
  11. 2025Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Modelscited by 9
  12. 2025Optimizing Diversity and Quality through Base-Aligned Model Collaborationcited by 8
28 more
  1. 2025Measuring LLM Novelty As The Frontier Of Original And High-Quality Outputcited by 6
  2. 2026DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writingcited by 6
  3. 2026DiscoverLLM: From Executing Intents to Discovering Themcited by 6
  4. 2026LLM Review: Enhancing Creative Writing via Blind Peer Review Feedbackcited by 5
  5. 2025ConlangCrafter: Constructing Languages with a Multi-Hop LLM Pipelinecited by 4
  6. 2026Time-Annealed Perturbation Sampling: Diverse Generation for Diffusion Language Modelscited by 3
  7. 2026Learning Dynamics in RL Post-Training for Language Modelscited by 2
  8. 2026Exploring Diverse Generation Paths via Inference-time Stiefel Activation Steeringcited by 2
  9. 2026Inducing Sustained Creativity and Diversity in Large Language Modelscited by 2
  10. 2026No Single Best Model for Diversity: Learning a Router for Sample Diversitycited by 2
  11. 2026Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMscited by 2
  12. 2026Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learningcited by 2
  13. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  14. 2026Narrative Flattening: How Post-Training Compresses Thematic, Affective, and Stylistic Variation in LLM Fictioncited by 1
  15. 2026Retrieval, Reward, and Training Protocols: What Matters in Training Search Agents?cited by 1
  16. 2026Recovering Diversity Without Losing Alignment: A DPO Recipe for Post-Trained LLMscited by 1
  17. 2026Creative Generation via Multi-Agent Debate: Does Debate Suppress Diversity?cited by 1
  18. 2025Pluralistic Behavior Suite: Stress-Testing Multi-Turn Adherence to Custom Behavioral Policiescited by 0
  19. 2026Discovering Novel LLM Experts via Task-Capability Coevolutioncited by 0
  20. 2026LLM Jaggedness Unlocks Scientific Creativitycited by 0
  21. 2026Elias in the Lighthouse, Again? Diagnosing Low Diversity in LLM Storiescited by 0
  22. 2026On the Diversity of Analogy Making in Large Language Modelscited by 0
  23. 2026CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversitycited by 0
  24. 2026PTCG: Persona-guided Tree-based Counterargument Generationcited by 0
  25. 2026Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMscited by 0
  26. 2026Adversarial Closed-Loop Curriculum for Evolving Role-Playing Agentscited by 0
  27. 2026Improving the Diversity of LLM Outputs without a Trade-offcited by 0
  28. 2026Explicit Trajectory Diversity for RL-Based Post-Training of LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving Diversity in LLM Short Story Generation
  2. 2025Mind the Gap: Conformative Decoding to Improve Output Diversity of Instruction-Tuned Large Language Models
  3. 2025Evaluating the Diversity and Quality of LLM Generated Content
  4. 2025Diverse Preference Optimization
  5. 2025What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuning
  6. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Models
  7. 2026Diversity in Large Language Models under Supervised Fine-Tuning
  8. 2026"I've Seen How This Goes": Characterizing Diversity via Progressive Conditional Surprise