# TokenPowerBench: Benchmarking the Power Consumption of LLM Inference

arXiv: https://arxiv.org/abs/2512.03024 · HTML: https://mlsanity.com/paper/2512.03024/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 40 references that are in the map.
- First posted on arXiv 2 Dec 2025.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference](https://mlsanity.com/paper/2310.03003/) (2023)
- [MLPerf Tiny Benchmark](https://mlsanity.com/paper/2106.07597/) (2021)
- [MLPerf Training Benchmark](https://mlsanity.com/paper/1910.01500/) (2019)
- [How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inference](https://mlsanity.com/paper/2505.09598/) (2025)
- [MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI](https://mlsanity.com/paper/2410.12032/) (2024)
- [Revolution or Hype? Seeking the Limits of Large Models in Hardware Design](https://mlsanity.com/paper/2509.04905/) (2025)
- [ReChisel: Effective Automatic Chisel Code Generation by LLM with Reflection](https://mlsanity.com/paper/2505.19734/) (2025)
- [ChatModel: Automating Reference Model Design and Verification with LLMs](https://mlsanity.com/paper/2506.15066/) (2025)

## Built on by (most cited first, 22 of 22)

- [Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures](https://mlsanity.com/paper/2604.09048/) (2026)
- [Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms](https://mlsanity.com/paper/2608.28044/) (2026)
- [The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project](https://mlsanity.com/paper/2603.21354/) (2026)
- [KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving](https://mlsanity.com/paper/2604.16682/) (2026)
- [WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs](https://mlsanity.com/paper/2607.02391/) (2026)
- [Request-Level Energy Attribution for Batched LLM Serving](https://mlsanity.com/paper/2608.00026/) (2026)
- [Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levels](https://mlsanity.com/paper/2608.25053/) (2026)
- [Selecting Language Models for Social Science: Start Small, Start Open, and Validate](https://mlsanity.com/paper/2601.10926/) (2026)
- [The Compression Paradox in LLM Inference: Provider-Dependent Energy Effects of Prompt Compression](https://mlsanity.com/paper/2603.23528/) (2026)
- [Cross-Family Speculative Decoding for Polish Language Models on Apple~Silicon: An Empirical Evaluation of Bielik~11B with UAG-Extended MLX-LM](https://mlsanity.com/paper/2604.16368/) (2026)
- [CiteRadar: A Citation Intelligence Platform for Researcher Profiling and Geographic Visualization](https://mlsanity.com/paper/2604.25057/) (2026)
- [Long Live the Librarian! A Persistent Search Sub-Agent for Energy-Efficient Multi-Agent Software Engineering Systems](https://mlsanity.com/paper/2605.27787/) (2026)
- [A Picture is Worth a Thousand Tokens: How Vision Language Models Cut AI Energy Costs While Improving Accuracy](https://mlsanity.com/paper/2608.07427/) (2026)
- [TokenPowerSandbox: Evidence-Gated CPU-First Screening for Energy-Aware LLM Serving](https://mlsanity.com/paper/2608.18149/) (2026)
- [More Accurate or More Efficient? Evaluating Locally Deployed Compact Open-Weight Language Models for Mathematical Reasoning](https://mlsanity.com/paper/2608.22048/) (2026)
- [Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures](https://mlsanity.com/paper/2608.25096/) (2026)
- [RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2609.05760/) (2026)
- [Where Should Agents Live? Energy-Memory Characterization of Agentic AI for the Edge-Cloud Continuum](https://mlsanity.com/paper/2609.18283/) (2026)
- [Analytical Power-Aware Provisioning for Prefill-Decode Disaggregated AI Inference](https://mlsanity.com/paper/2609.24639/) (2026)
- [Greenpixie's AI Token Methodology: Assessing the Energy, Water and CO2-eq Impact of AI Tokens for Open and Closed Weight Models](https://mlsanity.com/paper/2609.33965/) (2026)
- [Revisiting On-policy Adversarial Black-Box Distillation: Calibrating Groupwise Reward Geometry for Effective Advantage Construction](https://mlsanity.com/paper/2609.39757/) (2026)
- [Language-Conditioned Token and Reasoning Efficiency in Large Language Models: A Paired Cross-Lingual Study Protocol](https://mlsanity.com/paper/2610.04295/) (2026)

## Abstract (arXiv)

Large language model (LLM) services now answer billions of queries per day, and industry reports show that inference, not training, accounts for more than 90% of total power consumption. However, existing benchmarks focus on either training/fine-tuning or performance of inference and provide little support for power consumption measurement and analysis of inference. We introduce TokenPowerBench, the first lightweight and extensible benchmark designed for LLM-inference power consumption studies. The benchmark combines (i) a declarative configuration interface covering model choice, prompt set, and inference engine, (ii) a measurement layer that captures GPU-, node-, and system-level power without specialized power meters, and (iii) a phase-aligned metrics pipeline that attributes energy to the prefill and decode stages of every request. These elements make it straight-forward to explore the power consumed by an LLM inference run; furthermore, by varying batch size, context length, parallelism strategy and quantization, users can quickly assess how each setting affects joules per token and other energy-efficiency metrics. We evaluate TokenPowerBench on four of the most widely used model series (Llama, Falcon, Qwen, and Mistral). Our experiments cover from 1 billion parameters up to the frontier-scale Llama3-405B model. Furthermore, we release TokenPowerBench as open source to help users to measure power consumption, forecast operating expenses, and meet sustainability targets when deploying LLM services.