mlsanityData to 14 Oct 2026

Assessing the Creativity of Large Language Models: Testing, Limits, and New Frontiers

arXiv 2605.13450 ↗First posted 13 May 2026cs.AI cs.CL cs.HCcited by 0

Not on the map. Why?

Abstract (arXiv)

Measuring the creativity of large language models (LLMs) is essential for designing methods that can improve creativity and for enhancing our scientific understanding of this ability. To accomplish this, it has become common in recent years to administer tests of human creativity to LLMs. Although these tests provide a convenient and fully automated way to score "creativity," their validity as measures of machine creativity has not been established, and these tests already have limited validity as predictors of human creativity. To address this problem, we conduct the first large-scale, systematic study assessing the effectiveness of human creativity tests for predicting the creative achievement of LLMs across three target constructs: creative writing, divergent thinking, and scientific ideation. We find that the Divergent Association Task (DAT) and the Conditional DAT are the best predictors of creative writing and divergent thinking, respectively, but that test effectiveness varies significantly by construct, and no single test predicts all constructs well. Moreover, contrary to popular belief, no existing test reliably predicts scientific ideation ability. Motivated by this problem, we introduce the Divergent Remote Association Test (DRAT), a vocabulary-space test that assesses both convergent and divergent thinking in a single instrument. The DRAT is the first and only creativity test for LLMs that is a significant predictor of scientific ideation ability, demonstrating robustness across major design choices. Furthermore, the performance gain of the DRAT is not recoverable from any linear combination of the Divergent Association Task and the Remote Associates Test, indicating that assessing divergent and convergent thinking in the same test is essential to reliably predicting scientific ideation ability.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 59 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

24 of 59 references.

References · 24

  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  3. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  5. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  6. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  7. 2017Advances in Pre-Training Distributed Word Representationscited by 207
  8. 2023SciMON: Scientific Inspiration Machines Optimized for Noveltycited by 179
  9. 2025Open Problems in Mechanistic Interpretabilitycited by 179
  10. 2023EQ-Bench: An Emotional Intelligence Benchmark for Large Language Modelscited by 80
  11. 2025NoveltyBench: Evaluating Language Models for Humanlike Diversitycited by 66
  12. 2022Putting GPT-3's Creativity to the (Alternative Uses) Testcited by 62
12 more
  1. 2025The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideascited by 62
  2. 2025OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalizationcited by 50
  3. 2023Probing the Creativity of Large Language Models: Can models produce divergent semantic association?cited by 33
  4. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 29
  5. 2024Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Contextcited by 28
  6. 2023Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Datasetcited by 17
  7. 2025Measuring Combinatorial Creativity in Large Language Models and Humanscited by 7
  8. 2025Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Modelscited by 7
  9. 2026CREATE: Testing LLMs for Associative Creativitycited by 7
  10. 2025Transformational Creativity in Science: A Graphical Theorycited by 5
  11. 2025SYNTHIA: Novel Concept Design with Affordance Compositioncited by 4
  12. 2026Beyond Divergent Creativity: A Human-Based Evaluation of Creativity in Large Language Modelscited by 3

Built on by

Not cited yet.