# Rethinking Creativity Evaluation: A Critical Analysis of Existing Creativity Evaluations

arXiv: https://arxiv.org/abs/2508.05470 · HTML: https://mlsanity.com/paper/2508.05470/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2025.

## Built on by (most cited first, 14 of 14)

- [BILLY: Steering Large Language Models via Merging Persona Vectors for Creative Generation](https://mlsanity.com/paper/2510.10157/) (2025)
- [Where does output diversity collapse in post-training?](https://mlsanity.com/paper/2604.16027/) (2026)
- [LLMs Exhibit Significantly Lower Uncertainty in Creative Writing Than Professional Writers](https://mlsanity.com/paper/2602.16162/) (2026)
- [On the Creativity of AI Agents](https://mlsanity.com/paper/2604.13242/) (2026)
- [Do Large Language Models Always Tell The Same Stories?](https://mlsanity.com/paper/2606.17350/) (2026)
- [The Limits of Automatic Evaluation of Creativity in Large Language Models](https://mlsanity.com/paper/2608.23705/) (2026)
- [Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks](https://mlsanity.com/paper/2608.30047/) (2026)
- [AGC-Bench: Measuring Artificial General Creativity](https://mlsanity.com/paper/2607.01152/) (2026)
- [Creative Transformation in Literary Texts: Modelling Change Across Representational Levels](https://mlsanity.com/paper/2607.28513/) (2026)
- [CLIN: an Objective Framework for Evaluating Creativity in Short Persian Literary Text](https://mlsanity.com/paper/2608.30754/) (2026)
- [Is Imagination Derived from Hallucination? A Cross-Taxonomy Evaluation of Imagination and Hallucination in Large Language Models](https://mlsanity.com/paper/2609.22152/) (2026)
- [The Effects of Incremental Instruction Delivery on Language-Model Creative Writing](https://mlsanity.com/paper/2609.33738/) (2026)
- [Evaluating LLM Creativity through Long-Tail Performance](https://mlsanity.com/paper/2610.12998/) (2026)
- [DUALVIEW: Leveraging Broad and Output-Specific Evidence for LLM-based Multi-Output Text Regression](https://mlsanity.com/paper/2610.19206/) (2026)

## Abstract (arXiv)

We examine, analyze, and compare four representative creativity measures--perplexity, LLM-as-a-Judge, the Creativity Index (CI; measuring n-gram overlap with web corpora), and syntactic templates (detecting repetition of common part-of-speech patterns)--across the diverse creative domains, such as creative writing, unconventional problem-solving, and research ideation. For each domain, we compile datasets with human-aligned creative and uncreative examples and evaluate each metric's ability to discriminate between the two sets. Our analyses reveal limited consistency both across domains and metrics, as metrics that distinguish creativity in one domain fail in others (e.g., CI correctly distinguishes in creative writing but fails in problem-solving), and different metrics often disagree on the same data points (e.g., CI suggests one set to be more creative, while perplexity indicates the other set to be more creative.) We highlight key limitations, such as perplexity reflecting fluency rather than novelty; LLM-as-a-Judge producing inconsistent judgments under minor prompt variations and exhibiting bias towards particular labels; CI primarily measuring lexical diversity, with high sensitivity to implementation choices; and syntactic templates being ineffective in settings dominated by formulaic language. Our findings underscore the need for more robust, generalizable evaluation frameworks that better align with human judgments of creativity.