mlsanityData to 7 Oct 2026

Benchmarking Language Model Creativity: A Case Study on Code Generation

arXiv 2407.09007 ↗First posted 12 Jul 2024cs.CLcited by 38

Not on the map. Why?

Abstract (arXiv)

As LLMs become increasingly prevalent, it is interesting to consider how ``creative'' these models can be. From cognitive science, creativity consists of at least two key characteristics: convergent thinking (purposefulness to achieve a given goal) and divergent thinking (adaptability to explore new environments or constraints) citeprunco2003critical. In this work, we introduce a framework for quantifying LLM creativity that incorporates the two design ingredients: (1) We introduce DENIAL PROMPTING which pushes LLMs to develop more creative solutions to a given problem by incrementally imposing new constraints on the previous solution, compelling LLMs to adopt new strategies. (2) We define NEOGAUGE, a metric that quantifies both convergent and divergent thinking in the generated creative responses by LLMs. We test the proposed framework on Codeforces problems, which serve as both a natural dataset for coding tasks and a collection of prior human solutions. We quantify NEOGAUGE for various proprietary and open-source models and find that even the most creative model, GPT-4, still falls short of demonstrating human-like creativity. We also experiment with advanced reasoning strategies (MCTS, self-correction, etc.) and observe no significant improvement in creativity. As a by-product of our analysis, we release NEOCODER dataset for reproducing our results on future models.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 35 of its 66 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

35 of 66 references.

References · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
23 more
  1. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  2. 2023Large Language Modelscited by 774
  3. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  4. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  5. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  6. 2023Planning with Large Language Models for Code Generationcited by 179
  7. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 168
  8. 2024CodeGemma: Open Code Models Based on Gemmacited by 145
  9. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 127
  10. 2023Self-planning Code Generation with Large Language Modelscited by 121
  11. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  12. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 98
  13. 2021CREAK: A Dataset for Commonsense Reasoning over Entity Knowledgecited by 88
  14. 2024LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Playcited by 65
  15. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
  16. 2024Consistency-diversity-realism Pareto fronts of conditional image generative modelscited by 35
  17. 2023Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writerscited by 32
  18. 2024Exploring Precision and Recall to assess the quality and diversity of LLMscited by 27
  19. 2023Competition-Level Problems are Effective LLM Evaluatorscited by 25
  20. 2024CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generationcited by 19
  21. 2024Improving Diversity of Commonsense Generation by Large Language Models via In-Context Learningcited by 19
  22. 2023GEAR: Augmenting Language Models with Generalizable and Efficient Tool Resolutioncited by 17
  23. 2024AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogiescited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 20; 2026: 14.

38 papers.

Sort:

By area:

  1. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
  2. 2024Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Contextcited by 25
  3. 2024CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraintscited by 22
  4. 2025How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Codecited by 9
  5. 2025Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Modelscited by 9
  6. 2026CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challengescited by 9
  7. 2024Creativity in AI: Progresses and Challengescited by 7
  8. 2025Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variabilitycited by 6
  9. 2025Hell or High Water: Evaluating Agentic Recovery from External Failurescited by 6
  10. 2025Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Modelscited by 6
  11. 2026CREATE: Testing LLMs for Associative Creativitycited by 6
  12. 2025Measuring LLM Novelty As The Frontier Of Original And High-Quality Outputcited by 5
26 more
  1. 2025SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learningcited by 5
  2. 2026Free Lunch for Pass@k? Low Cost Diverse Sampling for Diffusion Language Modelscited by 5
  3. 2025The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation from Recognition to Reasoningcited by 4
  4. 2025Cooking Up Creativity: Enhancing LLM Creativity through Structured Recombinationcited by 3
  5. 2025Exploring Consciousness in LLMs: A Systematic Survey of Theories, Implementations, and Frontier Riskscited by 3
  6. 2026Beyond Divergent Creativity: A Human-Based Evaluation of Creativity in Large Language Modelscited by 3
  7. 2025What Shapes a Creative Machine Mind? Comprehensively Benchmarking Creativity in Foundation Modelscited by 2
  8. 2026No Single Best Model for Diversity: Learning a Router for Sample Diversitycited by 2
  9. 2025GuessBench: Sensemaking Multimodal Creativity in the Wildcited by 1
  10. 2025Reasoning Beyond the Obvious: Evaluating Divergent and Convergent Thinking in LLMs for Financial Scenarioscited by 1
  11. 2025Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safetycited by 1
  12. 2026TeamLLM: A Human-Like Team-Oriented Collaboration Framework for Multi-Step Contextualized Taskscited by 1
  13. 2026Can LLM Agents Discover? Evaluating Creativity on ML Engineering Taskscited by 1
  14. 2024Initial Development and Evaluation of the Creative Artificial Intelligence through Recurring Developments and Determinations (CAIRDD) Systemcited by 0
  15. 2025Leveraging Large Models to Evaluate Novel Content: A Case Study on Advertisement Creativitycited by 0
  16. 2025Learning Extrapolative Sequence Transformations from Markov Chainscited by 0
  17. 2025Galton's Law of Mediocrity: Why Large Language Models Regress to the Mean and Fail at Creativity in Advertisingcited by 0
  18. 2025Does Less Hallucination Mean Less Creativity? An Empirical Investigation in LLMscited by 0
  19. 2025VOYAGER: A Training Free Approach for Generating Diverse Datasets using LLMscited by 0
  20. 2026PROMPT2BOX:Improving LLM Weakness Discovery and Specificity Estimation by Uncovering Entailment Structure among Promptscited by 0
  21. 2026CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledgecited by 0
  22. 2026Large Language Models Align with the Human Brain during Creative Thinkingcited by 0
  23. 2026How Creative Are Large Language Models in Generating Molecules?cited by 0
  24. 2026Can Language Models Imagine Without Seeing? Ekphrasis: Measuring Visual Creative Ideation in Text-Only LLMscited by 0
  25. 2026UNSPECIFIC: General Constraint Synthesis for Breaking Copy-and-Paste Shortcut in LLM Instruction Followingcited by 0
  26. 2026Is Imagination Derived from Hallucination? A Cross-Taxonomy Evaluation of Imagination and Hallucination in Large Language Modelscited by 0