mlsanityData to 14 Oct 2026

Automated Creativity Evaluation for Large Language Models: A Reference-Based Approach

arXiv 2504.15784 ↗First posted 22 Apr 2025cs.CL cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Creative writing is a key capability of Large Language Models (LLMs), with potential applications in literature, storytelling, and various creative domains. However, evaluating the creativity of machine-generated texts remains a significant challenge, as existing methods either rely on costly manual annotations or fail to align closely with human assessments. In this paper, we propose an effective automated evaluation method based on the Torrance Test of Creative Writing (TTCW), which evaluates creativity as product. Our method employs a reference-based Likert-style approach, scoring generated creative texts relative to high-quality reference texts across various tests. Experimental results demonstrate that our method significantly improves the alignment between LLM evaluations and human assessments, achieving a pairwise accuracy of 0.75 (+15%).

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 9 of its 23 references that are in the map. First posted on arXiv 22 Apr 2025.

Builds on

9 of 23 references.

References · 9

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  2. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  4. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  5. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  6. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 172
  7. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 99
  8. 2022Putting GPT-3's Creativity to the (Alternative Uses) Testcited by 62
  9. 2023The Next Chapter: A Study of Large Language Models in Storytellingcited by 58

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 13.

20 papers.

Sort:

By area:

  1. 2025LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writingcited by 42
  2. 2026DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reportscited by 34
  3. 2025Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewardscited by 11
  4. 2025WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Qualitycited by 11
  5. 2025Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluatorcited by 10
  6. 2025ProactiveEval: A Unified Evaluation Framework for Proactive Dialogue Agentscited by 9
  7. 2026LLM Review: Enhancing Creative Writing via Blind Peer Review Feedbackcited by 5
  8. 2025ConlangCrafter: Constructing Languages with a Multi-Hop LLM Pipelinecited by 4
  9. 2026Beyond Divergent Creativity: A Human-Based Evaluation of Creativity in Large Language Modelscited by 3
  10. 2026Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generationcited by 2
  11. 2026Are We Truly Innovating? A Qualitative and Quantitative Study of Originality in AI Research Paperscited by 1
  12. 2026DEFINED: A Data-Efficient Computational Framework for Fine-Grained Creativity Assessment in Debate Scenarioscited by 1
8 more
  1. 2026Why Large Language Models and Humans Converge and Diverge in Evaluating Creativitycited by 1
  2. 2025Universe of Thoughts: Enabling Creative Reasoning with Large Language Modelscited by 0
  3. 2026Evaluation Framework for AI Creativity: A Case Study Based on Story Generationcited by 0
  4. 2026Creativity in AI as Emergence from Domain-Limited Generative Modelscited by 0
  5. 2026UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learningcited by 0
  6. 2026IDEAFix: Evaluation Framework for Creative Defixation Prompting in LLMscited by 0
  7. 2026CLIN: an Objective Framework for Evaluating Creativity in Short Persian Literary Textcited by 0
  8. 2026Decoupled Analysis-Judging: An Automated Creativity Evaluator Using LLMs in Complex Multi-step Creativity Taskscited by 0