# ElecBench: a Power Dispatch Evaluation Benchmark for Large Language Models

arXiv: https://arxiv.org/abs/2407.05365 · HTML: https://mlsanity.com/paper/2407.05365/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 53 references that are in the map.
- First posted on arXiv 7 Jul 2024.

## Builds on (most cited first, 27 of 27)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection](https://mlsanity.com/paper/1907.09693/) (2019)
- [On Evaluating Adversarial Robustness of Large Vision-Language Models](https://mlsanity.com/paper/2305.16934/) (2023)
- [Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models](https://mlsanity.com/paper/2111.02840/) (2021)
- [Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models](https://mlsanity.com/paper/2205.10770/) (2022)
- [Does Writing with Language Models Reduce Content Diversity?](https://mlsanity.com/paper/2309.05196/) (2023)
- [Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2401.17263/) (2024)
- [Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models](https://mlsanity.com/paper/2304.03738/) (2023)
- [Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods](https://mlsanity.com/paper/2404.00282/) (2024)
- [Towards the Science of Security and Privacy in Machine Learning](https://mlsanity.com/paper/1611.03814/) (2016)
- [Eight Things to Know about Large Language Models](https://mlsanity.com/paper/2304.00612/) (2023)
- [Post-processing for Individual Fairness](https://mlsanity.com/paper/2110.13796/) (2021)
- [Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation](https://mlsanity.com/paper/2306.05783/) (2023)
- [RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning](https://mlsanity.com/paper/2401.08326/) (2024)
- [Assessing and Understanding Creativity in Large Language Models](https://mlsanity.com/paper/2401.12491/) (2024)
- [Prevent the Language Model from being Overconfident in Neural Machine Translation](https://mlsanity.com/paper/2105.11098/) (2021)
- [Terminology-Aware Translation with Constrained Decoding and Large Language Model Prompting](https://mlsanity.com/paper/2310.05824/) (2023)

## Built on by (most cited first, 13 of 13)

- [Enhancing LLMs for Power System Simulations: A Feedback-driven Multi-agent Framework](https://mlsanity.com/paper/2411.16707/) (2024)
- [EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving](https://mlsanity.com/paper/2509.17677/) (2025)
- [Large Language Model Powered Automated Modeling and Optimization of Active Distribution Network Dispatch Problems](https://mlsanity.com/paper/2507.21162/) (2025)
- [EF-LLM: Energy Forecasting LLM with AI-assisted Automation, Enhanced Sparse Prediction, Hallucination Detection](https://mlsanity.com/paper/2411.00852/) (2024)
- [EngTrace: A Symbolic Benchmark for Verifiable Process Supervision of Engineering Reasoning](https://mlsanity.com/paper/2511.01650/) (2025)
- [Context-aware LLM-based AI Agents for Human-centered Energy Management Systems in Smart Buildings](https://mlsanity.com/paper/2512.25055/) (2025)
- [From Natural Language to Solver-Ready Power System Optimization: An LLM-Assisted, Validation-in-the-Loop Framework](https://mlsanity.com/paper/2508.08147/) (2025)
- [Knowledge boundary probing and demand-guided intervention for LLM-based power system code generation](https://mlsanity.com/paper/2605.31478/) (2026)
- [Power Systems Agent Benchmark: Executable Evaluation of AI Agents in Electric Power Engineering](https://mlsanity.com/paper/2606.20950/) (2026)
- [How Do Tool-Augmented LLM Agents Perform on Real-World Energy Analytics Tasks?](https://mlsanity.com/paper/2606.26346/) (2026)
- [A Multimodal Dataset for Large Language Model Applications in the Energy Domain](https://mlsanity.com/paper/2607.11459/) (2026)
- [Seed-Anchored Budget-Bounded Graph Rendering for Question Answering on Industry-Standard Power-Grid Information and Exchange Models](https://mlsanity.com/paper/2609.02011/) (2026)
- [Training Numerical Intelligence via Auto-Diagnosis and Skill Discovery](https://mlsanity.com/paper/2610.03872/) (2026)

## Abstract (arXiv)

In response to the urgent demand for grid stability and the complex challenges posed by renewable energy integration and electricity market dynamics, the power sector increasingly seeks innovative technological solutions. In this context, large language models (LLMs) have become a key technology to improve efficiency and promote intelligent progress in the power sector with their excellent natural language processing, logical reasoning, and generalization capabilities. Despite their potential, the absence of a performance evaluation benchmark for LLM in the power sector has limited the effective application of these technologies. Addressing this gap, our study introduces "ElecBench", an evaluation benchmark of LLMs within the power sector. ElecBench aims to overcome the shortcomings of existing evaluation benchmarks by providing comprehensive coverage of sector-specific scenarios, deepening the testing of professional knowledge, and enhancing decision-making precision. The framework categorizes scenarios into general knowledge and professional business, further divided into six core performance metrics: factuality, logicality, stability, security, fairness, and expressiveness, and is subdivided into 24 sub-metrics, offering profound insights into the capabilities and limitations of LLM applications in the power sector. To ensure transparency, we have made the complete test set public, evaluating the performance of eight LLMs across various scenarios and metrics. ElecBench aspires to serve as the standard benchmark for LLM applications in the power sector, supporting continuous updates of scenarios, metrics, and models to drive technological progress and application.