# The Generative Energy Arena (GEA): Incorporating Energy Awareness in Large Language Model (LLM) Human Evaluations

arXiv: https://arxiv.org/abs/2507.13302 · HTML: https://mlsanity.com/paper/2507.13302/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 17 Jul 2025.

## Builds on (most cited first, 22 of 22)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference](https://mlsanity.com/paper/2310.03003/) (2023)
- [Benchmark Data Contamination of Large Language Models: A Survey](https://mlsanity.com/paper/2406.04244/) (2024)
- [GenAI Arena: An Open Evaluation Platform for Generative Models](https://mlsanity.com/paper/2406.04485/) (2024)
- [How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inference](https://mlsanity.com/paper/2505.09598/) (2025)
- [Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussions](https://mlsanity.com/paper/2405.20267/) (2024)
- [Cost-of-Pass: An Economic Framework for Evaluating Language Models](https://mlsanity.com/paper/2504.13359/) (2025)
- [The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization](https://mlsanity.com/paper/2505.06371/) (2025)
- [Holistically Evaluating the Environmental Impact of Creating Language Models](https://mlsanity.com/paper/2503.05804/) (2025)
- [Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling](https://mlsanity.com/paper/2501.08219/) (2025)
- [Improving Your Model Ranking on Chatbot Arena by Vote Rigging](https://mlsanity.com/paper/2501.17858/) (2025)
- [Unveiling Environmental Impacts of Large Language Model Serving: A Functional Unit View](https://mlsanity.com/paper/2502.11256/) (2025)
- [Inference economics of language models](https://mlsanity.com/paper/2506.04645/) (2025)
- [am-ELO: A Stable Framework for Arena-based LLM Evaluation](https://mlsanity.com/paper/2505.03475/) (2025)
- [Speed and Conversational Large Language Models: Not All Is About Tokens per Second](https://mlsanity.com/paper/2502.16721/) (2025)

## Abstract (arXiv)

The evaluation of large language models is a complex task, in which several approaches have been proposed. The most common is the use of automated benchmarks in which LLMs have to answer multiple-choice questions of different topics. However, this method has certain limitations, being the most concerning, the poor correlation with the humans. An alternative approach, is to have humans evaluate the LLMs. This poses scalability issues as there is a large and growing number of models to evaluate making it impractical (and costly) to run traditional studies based on recruiting a number of evaluators and having them rank the responses of the models. An alternative approach is the use of public arenas, such as the popular LM arena, on which any user can freely evaluate models on any question and rank the responses of two models. The results are then elaborated into a model ranking. An increasingly important aspect of LLMs is their energy consumption and, therefore, evaluating how energy awareness influences the decisions of humans in selecting a model is of interest. In this paper, we present GEA, the Generative Energy Arena, an arena that incorporates information on the energy consumption of the model in the evaluation process. Preliminary results obtained with GEA are also presented, showing that for most questions, when users are aware of the energy consumption, they favor smaller and more energy efficient models. This suggests that for most user interactions, the extra cost and energy incurred by the more complex and top-performing models do not provide an increase in the perceived quality of the responses that justifies their use.