# CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges

arXiv: https://arxiv.org/abs/2603.11863 · HTML: https://mlsanity.com/paper/2603.11863/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 54 references that are in the map.
- First posted on arXiv 12 Mar 2026.

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://mlsanity.com/paper/2507.19457/) (2025)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [A Tutorial on Principal Component Analysis](https://mlsanity.com/paper/1404.1100/) (2014)
- [Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)](https://mlsanity.com/paper/2510.22954/) (2025)
- [Open-Endedness is Essential for Artificial Superhuman Intelligence](https://mlsanity.com/paper/2406.04268/) (2024)
- [Confabulation: The Surprising Value of Large Language Model Hallucinations](https://mlsanity.com/paper/2406.04175/) (2024)
- [Benchmarking Language Model Creativity: A Case Study on Code Generation](https://mlsanity.com/paper/2407.09007/) (2024)
- [Reducing Tool Hallucination via Reliability Alignment](https://mlsanity.com/paper/2412.04141/) (2024)
- [A Survey on Large Language Model Hallucination via a Creativity Perspective](https://mlsanity.com/paper/2402.06647/) (2024)
- [Assessing and Understanding Creativity in Large Language Models](https://mlsanity.com/paper/2401.12491/) (2024)
- [EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations](https://mlsanity.com/paper/2410.22821/) (2024)
- [CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval](https://mlsanity.com/paper/2411.12644/) (2024)
- [Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows](https://mlsanity.com/paper/2512.16969/) (2025)
- [AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators](https://mlsanity.com/paper/2508.09101/) (2025)
- [CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraints](https://mlsanity.com/paper/2410.04197/) (2024)
- [CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation](https://mlsanity.com/paper/2404.08806/) (2024)
- [CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios](https://mlsanity.com/paper/2506.13977/) (2025)
- [MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation](https://mlsanity.com/paper/2505.17123/) (2025)
- [Shifting Perspectives: Steering Vectors for Robust Bias Mitigation in LLMs](https://mlsanity.com/paper/2503.05371/) (2025)
- [Toward Artificial Open-Ended Evolution within Lenia using Quality-Diversity](https://mlsanity.com/paper/2406.04235/) (2024)
- [DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writing](https://mlsanity.com/paper/2601.09609/) (2026)

## Built on by (most cited first, 9 of 9)

- [TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrieval](https://mlsanity.com/paper/2604.21806/) (2026)
- [Learning from Contrasts: Synthesizing Reasoning Paths from Diverse Search Trajectories](https://mlsanity.com/paper/2604.11365/) (2026)
- [Calibrating Model-Based Evaluation Metrics for Summarization](https://mlsanity.com/paper/2604.17200/) (2026)
- [DiffoR: A Unified Continuous Generative Framework for Universal Ordinal Regression](https://mlsanity.com/paper/2606.07599/) (2026)
- [StatefulDiscovery: Evidence-Calibrated Claim Formation in Open-Ended Scientific Discovery](https://mlsanity.com/paper/2606.11851/) (2026)
- [Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks](https://mlsanity.com/paper/2608.30047/) (2026)
- [Advancing Creative Physical Intelligence in Large Multimodal Models](https://mlsanity.com/paper/2605.26396/) (2026)
- [Customer-Agent: Overcoming Context Limitations in Ultra-Long Shopping Trajectories via Tool-Augmented Agents and RLVR](https://mlsanity.com/paper/2606.07995/) (2026)
- [Notes2Skills: From Lab Notebooks to Certainty-Aware Scientific Agent Skills](https://mlsanity.com/paper/2606.11897/) (2026)

## Abstract (arXiv)

The saturation of high-quality pre-training data has shifted research focus toward evolutionary systems capable of continuously generating novel artifacts, leading to the success of AlphaEvolve. However, the progress of such systems is hindered by the lack of rigorous, quantitative evaluation. To tackle this challenge, we introduce CreativeBench, a benchmark for evaluating machine creativity in code generation, grounded in a classical cognitive framework. Comprising two subsets -- CreativeBench-Combo and CreativeBench-Explore -- the benchmark targets combinatorial and exploratory creativity through an automated pipeline utilizing reverse engineering and self-play. By leveraging executable code, CreativeBench objectively distinguishes creativity from hallucination via a unified metric defined as the product of quality and novelty. Our analysis of state-of-the-art models reveals distinct behaviors: (1) scaling significantly improves combinatorial creativity but yields diminishing returns for exploration; (2) larger models exhibit ``convergence-by-scaling,'' becoming more correct but less divergent; and (3) reasoning capabilities primarily benefit constrained exploration rather than combination. Finally, we propose EvoRePE, a plug-and-play inference-time steering strategy that internalizes evolutionary search patterns to consistently enhance machine creativity.