mlsanityData to 14 Oct 2026

Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluator

arXiv 2505.19236 ↗First posted 25 May 2025cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Creativity evaluation remains a challenging frontier for large language models (LLMs). Current evaluations heavily rely on inefficient and costly human judgments, hindering progress in enhancing machine creativity. While automated methods exist, ranging from psychological testing to heuristic- or prompting-based approaches, they often lack generalizability or alignment with human judgment. To address these issues, we propose a novel pairwise-comparison framework for assessing textual creativity that leverages shared contextual instructions to improve evaluation consistency. We introduce CreataSet, a large-scale dataset with 100K+ human-level and 1M+ synthetic creative instruction-response pairs spanning diverse open-domain tasks. Through training on CreataSet, we develop an LLM-based evaluator named CrEval. CrEval demonstrates remarkable superiority over existing methods in alignment with human judgments. Experimental results underscore the indispensable significance of integrating both human and synthetic data to train highly robust evaluators, and showcase the practical utility of CrEval in boosting the creativity of LLMs.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 52 of its 90 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

52 of 90 references.

References · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  11. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  12. 2025Gemma 3 Technical Reportcited by 1,824
38 more
  1. 2024A Survey on LLM-as-a-Judgecited by 1,423
  2. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  4. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  5. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
  6. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  7. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  8. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  9. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  10. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  11. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 172
  12. 2023Generative Judge for Evaluating Alignmentcited by 169
  13. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  14. 2025WritingBench: A Comprehensive Benchmark for Generative Writingcited by 106
  15. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 101
  16. 2023EQ-Bench: An Emotional Intelligence Benchmark for Large Language Modelscited by 80
  17. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  18. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 70
  19. 2022Putting GPT-3's Creativity to the (Alternative Uses) Testcited by 62
  20. 2023Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generationcited by 56
  21. 2023A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluatorscited by 45
  22. 2024COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuningcited by 45
  23. 2024CodeJudge: Evaluating Code Generation with Large Language Modelscited by 44
  24. 2024AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Textcited by 42
  25. 2024Assessing and Understanding Creativity in Large Language Modelscited by 35
  26. 2023MacGyver: Are Large Language Models Creative Problem Solvers?cited by 34
  27. 2023X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspectscited by 28
  28. 2024Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing?cited by 27
  29. 2025AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computationcited by 25
  30. 2025Automated Creativity Evaluation for Large Language Models: A Reference-Based Approachcited by 20
  31. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  32. 2023Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Datasetcited by 17
  33. 2024Can We Use Large Language Models to Fill Relevance Judgment Holes?cited by 17
  34. 2020Introducing Aspects of Creativity in Automatic Poetry Generationcited by 14
  35. 2024Can Large Language Models Serve as Evaluators for Code Summarization?cited by 12
  36. 2024Large Language Models show both individual and collective creativity comparable to humanscited by 11
  37. 2023HAUSER: Towards Holistic and Automatic Evaluation of Simile Generationcited by 5
  38. 2024Steering Large Language Models to Evaluate and Amplify Creativitycited by 4

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writingcited by 6
  2. 2026Restoring Exploration after Post-Training: Latent Exploration Decoding for Large Reasoning Modelscited by 4
  3. 2026Beyond Divergent Creativity: A Human-Based Evaluation of Creativity in Large Language Modelscited by 3
  4. 2026Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMscited by 2
  5. 2026Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generationcited by 2
  6. 2026DEFINED: A Data-Efficient Computational Framework for Fine-Grained Creativity Assessment in Debate Scenarioscited by 1
  7. 2026Do Large Language Models Always Tell The Same Stories?cited by 1
  8. 2026GENIE: A Fine-Grained Measure for Noveltycited by 0
  9. 2026StorySpark: Module-wise Evolutionary Search for Story Premise Generationcited by 0
  10. 2026Evaluating LLM Creativity through Long-Tail Performancecited by 0