# Do LLMs Agree on the Creativity Evaluation of Alternative Uses?

arXiv: https://arxiv.org/abs/2411.15560 · HTML: https://mlsanity.com/paper/2411.15560/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 23 Nov 2024.

## Builds on (most cited first, 18 of 18)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges](https://mlsanity.com/paper/2406.12624/) (2024)
- [A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing](https://mlsanity.com/paper/2310.08433/) (2023)
- [Putting GPT-3's Creativity to the (Alternative Uses) Test](https://mlsanity.com/paper/2206.08932/) (2022)
- [Assessing and Understanding Creativity in Large Language Models](https://mlsanity.com/paper/2401.12491/) (2024)
- [Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructions](https://mlsanity.com/paper/2408.08781/) (2024)
- [Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis](https://mlsanity.com/paper/2407.10899/) (2024)
- [Less is More for Long Document Summary Evaluation by LLMs](https://mlsanity.com/paper/2309.07382/) (2023)
- [Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks](https://mlsanity.com/paper/2310.19740/) (2023)
- [Creativity and Machine Learning: A Survey](https://mlsanity.com/paper/2104.02726/) (2021)
- [A Closer Look into Automatic Evaluation Using Large Language Models](https://mlsanity.com/paper/2310.05657/) (2023)
- [Creative Beam Search: LLM-as-a-Judge For Improving Response Generation](https://mlsanity.com/paper/2405.00099/) (2024)
- [Automatic Assessment of Divergent Thinking in Chinese Language with TransDis: A Transformer-Based Language Model Approach](https://mlsanity.com/paper/2306.14790/) (2023)

## Built on by (most cited first, 6 of 6)

- [Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM](https://mlsanity.com/paper/2503.14478/) (2025)
- [Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation](https://mlsanity.com/paper/2608.01666/) (2026)
- [IDEAFix: Evaluation Framework for Creative Defixation Prompting in LLMs](https://mlsanity.com/paper/2606.00875/) (2026)
- [Recipes for Creativity: Iterative Generation and Evaluation in Large Language Models](https://mlsanity.com/paper/2608.07243/) (2026)
- [Using Poly-Encoders for Computationally Efficient Automated Creativity Assessment](https://mlsanity.com/paper/2608.26165/) (2026)
- [Evaluating LLM Creativity through Long-Tail Performance](https://mlsanity.com/paper/2610.12998/) (2026)

## Abstract (arXiv)

This paper investigates whether large language models (LLMs) show agreement in assessing creativity in responses to the Alternative Uses Test (AUT). While LLMs are increasingly used to evaluate creative content, previous studies have primarily focused on a single model assessing responses generated by the same model or humans. This paper explores whether LLMs can impartially and accurately evaluate creativity in outputs generated by both themselves and other models. Using an oracle benchmark set of AUT responses, categorized by creativity level (common, creative, and highly creative), we experiment with four state-of-the-art LLMs evaluating these outputs. We test both scoring and ranking methods and employ two evaluation settings (comprehensive and segmented) to examine if LLMs agree on the creativity evaluation of alternative uses. Results reveal high inter-model agreement, with Spearman correlations averaging above 0.7 across models and reaching over 0.77 with respect to the oracle, indicating a high level of agreement and validating the reliability of LLMs in creativity assessment of alternative uses. Notably, models do not favour their own responses, instead they provide similar creativity assessment scores or rankings for alternative uses generated by other models. These findings suggest that LLMs exhibit impartiality and high alignment in creativity evaluation, offering promising implications for their use in automated creativity assessment.