mlsanityData to 14 Oct 2026

NoveltyBench: Evaluating Language Models for Humanlike Diversity

arXiv 2504.05228 ↗First posted 7 Apr 2025cs.CLcited by 66

Not on the map. Why?

Abstract (arXiv)

Language models have demonstrated remarkable capabilities on standard benchmarks, yet they struggle increasingly from mode collapse, the inability to generate diverse and novel outputs. Our work introduces NoveltyBench, a benchmark specifically designed to evaluate the ability of language models to produce multiple distinct and high-quality outputs. NoveltyBench utilizes prompts curated to elicit diverse answers and filtered real-world user queries. Evaluating 20 leading language models, we find that current state-of-the-art systems generate significantly less diversity than human writers. Notably, larger models within a family often exhibit less diversity than their smaller counterparts, challenging the notion that capability on standard benchmarks translates directly to generative utility. While prompting strategies like in-context regeneration can elicit diversity, our findings highlight a fundamental lack of distributional diversity in current models, reducing their utility for users seeking varied responses and suggesting the need for new training and evaluation paradigms that prioritize diversity alongside quality.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

29 of 46 references.

References · 29

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  4. 2018Hierarchical Neural Story Generationcited by 1,193
  5. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  6. 2023Whose Opinions Do Language Models Reflect?cited by 773
  7. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  8. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  9. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  10. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  11. 20242 OLMo 2 Furiouscited by 340
  12. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
17 more
  1. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 209
  2. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 130
  3. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 129
  4. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 116
  5. 2024Is Temperature the Creativity Parameter of Large Language Models?cited by 93
  6. 2019Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challengecited by 86
  7. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 70
  8. 2024AI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuancescited by 60
  9. 2025Diverse Preference Optimizationcited by 52
  10. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
  11. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  12. 2024Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attributioncited by 40
  13. 2024Detecting Mode Collapse in Language Models via Narrationcited by 24
  14. 2024Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personascited by 23
  15. 2023Large Language Model (LLM) Bias Index -- LLMBIcited by 14
  16. 2024Growing a Tail: Increasing Output Diversity in Large Language Modelscited by 9
  17. 2024Every Answer Matters: Evaluating Commonsense with Probabilistic Measurescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 24; 2026: 42.

66 papers.

Sort:

By area:

  1. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  2. 2025Self-Improving Language Models for Evolutionary Program Synthesis: A Case Study on ARC-AGIcited by 36
  3. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 31
  4. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 29
  5. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 20
  6. 2025What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Modelscited by 14
  7. 2025G2: Guided Generation for Enhanced Output Diversity in LLMscited by 14
  8. 2025String Seed of Thought: Prompting LLMs for Distribution-Faithful and Diverse Generationcited by 11
  9. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewardscited by 10
  10. 2025MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigmcited by 9
  11. 2025Frankentext: Stitching random text fragments into long-form narrativescited by 8
  12. 2025Optimizing Diversity and Quality through Base-Aligned Model Collaborationcited by 8
38 more
  1. 2025Measuring Combinatorial Creativity in Large Language Models and Humanscited by 7
  2. 2025Polychromic Objectives for Reinforcement Learningcited by 7
  3. 2026Diversity Collapse in Multi-Agent LLM Systems: Structural Coupling and Collective Failure in Open-Ended Idea Generationcited by 7
  4. 2025Avoidance Decoding for Diverse Multi-Branch Story Generationcited by 6
  5. 2026DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writingcited by 6
  6. 2026One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RLcited by 6
  7. 2025Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativitycited by 5
  8. 2026Annotations Mitigate Post-Training Mode Collapsecited by 5
  9. 2026Language of Thought Shapes Output Diversity in Large Language Modelscited by 4
  10. 2026Diversity in Large Language Models under Supervised Fine-Tuningcited by 4
  11. 2025Geometry of Knowledge Allows Extending Diversity Boundaries of Large Language Modelscited by 3
  12. 2025The Ramon Llull's Thinking Machine for Automated Ideationcited by 3
  13. 2026Time-Annealed Perturbation Sampling: Diverse Generation for Diffusion Language Modelscited by 3
  14. 2026Beyond Semantic Manipulation: Token-Space Attacks on Reward Modelscited by 3
  15. 2026The One-Word Census: Answer-Choice Conformity Across 105 Language Modelscited by 3
  16. 2025What Shapes a Creative Machine Mind? Comprehensively Benchmarking Creativity in Foundation Modelscited by 2
  17. 2025Beware of Reasoning Overconfidence: Pitfalls in the Reasoning Process for Multi-solution Taskscited by 2
  18. 2025Enhancing Diversity of LLM-Generated Educational Taskscited by 2
  19. 2026The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safetycited by 2
  20. 2026Inducing Sustained Creativity and Diversity in Large Language Modelscited by 2
  21. 2026No Single Best Model for Diversity: Learning a Router for Sample Diversitycited by 2
  22. 2026Probabilistic Calibration Is a Trainable Capability in Language Modelscited by 2
  23. 2026AI as a Tool for Simulation-Based Experiments in Literary Studiescited by 2
  24. 2026Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learningcited by 2
  25. 2026Representational Similarity and Model Behavior in Multi-Agent Interactioncited by 2
  26. 2025LLM-NAS: LLM-driven Hardware-Aware Neural Architecture Searchcited by 1
  27. 2026Shared Nature, Unique Nurture: PRISM for Pluralistic Reasoning via In-context Structure Modelingcited by 1
  28. 2026Ex Ante Evaluation of AI-Induced Idea Diversity Collapsecited by 1
  29. 2026Anchorless Diversification for Parallel LLM Ideationcited by 1
  30. 2026AI Coding Agents in Social Science: Methodologically Diverse, Empirically Consistent, Interpretively Vulnerablecited by 1
  31. 2026Can LLM Agents Discover? Evaluating Creativity on ML Engineering Taskscited by 1
  32. 2026Creative Generation via Multi-Agent Debate: Does Debate Suppress Diversity?cited by 1
  33. 2025Cultural Alien Sampler: Open-ended art generation balancing originality and coherencecited by 0
  34. 2025Magellan: Guided MCTS for Latent Space Exploration and Novelty Generationcited by 0
  35. 2026Flow of Spans: Generalizing Language Models to Dynamic Span-Vocabulary via GFlowNetscited by 0
  36. 2026CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledgecited by 0
  37. 2026Triviality Corrected Endogenous Rewardcited by 0
  38. 2026Assessing the Creativity of Large Language Models: Testing, Limits, and New Frontierscited by 0

50 most cited of 66 All 66 (JSON)