# Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Models

arXiv: https://arxiv.org/abs/2510.12110 · HTML: https://mlsanity.com/paper/2510.12110/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 51 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Builds on (most cited first, 18 of 18)

- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023)
- [Advances in Pre-Training Distributed Word Representations](https://mlsanity.com/paper/1712.09405/) (2017)
- [Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges](https://mlsanity.com/paper/2406.12624/) (2024)
- [Generative artificial intelligence enhances creativity but reduces the diversity of novel content](https://mlsanity.com/paper/2312.00506/) (2023)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [Are Large Language Models Capable of Generating Human-Level Narratives?](https://mlsanity.com/paper/2407.13248/) (2024)
- [Putting GPT-3's Creativity to the (Alternative Uses) Test](https://mlsanity.com/paper/2206.08932/) (2022)
- [We're Different, We're the Same: Creative Homogeneity Across LLMs](https://mlsanity.com/paper/2501.19361/) (2025)
- [Benchmarking Language Model Creativity: A Case Study on Code Generation](https://mlsanity.com/paper/2407.09007/) (2024)
- [CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraints](https://mlsanity.com/paper/2410.04197/) (2024)
- [How AI Ideas Affect the Creativity, Diversity, and Evolution of Human Ideas: Evidence From a Large, Dynamic Experiment](https://mlsanity.com/paper/2401.13481/) (2024)
- [Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Dataset](https://mlsanity.com/paper/2306.11167/) (2023)
- [Automating psychological hypothesis generation with AI: when large language models meet causal graph](https://mlsanity.com/paper/2402.14424/) (2024)
- [Does ChatGPT Have a Poetic Style?](https://mlsanity.com/paper/2410.15299/) (2024)
- [Probing the contents of semantic representations from text, behavior, and brain data using the psychNorms metabase](https://mlsanity.com/paper/2412.04936/) (2024)

## Built on by (most cited first, 7 of 7)

- [Creativity Bias: How Machine Evaluation Struggles with Creativity in Literary Translations](https://mlsanity.com/paper/2605.13596/) (2026)
- [Are LLMs becoming similarly creative? Evidence from three years of models](https://mlsanity.com/paper/2608.19437/) (2026)
- [Beyond Reproduction: A Paired-Task Framework for Assessing LLM Comprehension and Creativity in Literary Translation](https://mlsanity.com/paper/2604.18169/) (2026)
- [Ex Ante Evaluation of AI-Induced Idea Diversity Collapse](https://mlsanity.com/paper/2605.06540/) (2026)
- [Assessing the Creativity of Large Language Models: Testing, Limits, and New Frontiers](https://mlsanity.com/paper/2605.13450/) (2026)
- [CLIN: an Objective Framework for Evaluating Creativity in Short Persian Literary Text](https://mlsanity.com/paper/2608.30754/) (2026)
- [Evaluating LLM Creativity through Long-Tail Performance](https://mlsanity.com/paper/2610.12998/) (2026)

## Abstract (arXiv)

The evaluation of LLMs' creativity represents a crucial research domain, though challenges such as data contamination and costly human assessments often impede progress. Drawing inspiration from human creativity assessment, we propose PACE, asking LLMs to generate Parallel Association Chains to Evaluate their creativity. PACE minimizes the risk of data contamination and offers a straightforward, highly efficient evaluation, as evidenced by its strong correlation with Chatbot Arena Creative Writing rankings (Spearman's ρ = 0.739, p < 0.001) across various proprietary and open-source models. A comparative analysis of associative creativity between LLMs and humans reveals that while high-performing LLMs achieve scores comparable to average human performance, professional humans consistently outperform LLMs. Furthermore, linguistic analysis reveals that both humans and LLMs exhibit a trend of decreasing concreteness in their associations, and humans demonstrating a greater diversity of associative patterns.