# Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms

arXiv: https://arxiv.org/abs/2608.28044 · HTML: https://mlsanity.com/paper/2608.28044/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 28 Aug 2026.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 25 of 25)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference](https://mlsanity.com/paper/2310.03003/) (2023)
- [DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency](https://mlsanity.com/paper/2408.00741/) (2024)
- [Energy Considerations of Large Language Model Inference and Efficiency Optimizations](https://mlsanity.com/paper/2504.17674/) (2025)
- [Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems](https://mlsanity.com/paper/2407.04014/) (2024)
- [TokenPowerBench: Benchmarking the Power Consumption of LLM Inference](https://mlsanity.com/paper/2512.03024/) (2025)
- [The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Compute](https://mlsanity.com/paper/2505.14733/) (2025)
- [Intelligence per Watt: Measuring Intelligence Efficiency of Local AI](https://mlsanity.com/paper/2511.07885/) (2025)
- [Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scaling](https://mlsanity.com/paper/2509.20241/) (2025)
- [Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models](https://mlsanity.com/paper/2502.05610/) (2025)
- [Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures](https://mlsanity.com/paper/2504.11750/) (2025)
- [SweetSpot: An Analytical Model for Predicting Energy Efficiency of LLM Inference](https://mlsanity.com/paper/2602.05695/) (2026)
- [Beyond Test-Time Compute Strategies: Advocating Energy-per-Token in LLM Inference](https://mlsanity.com/paper/2603.20224/) (2026)
- [Wattchmen: Watching the Wattchers -- High Fidelity, Flexible GPU Energy Modeling](https://mlsanity.com/paper/2603.26435/) (2026)
- [The Energy Cost of Execution-Idle in GPU Clusters](https://mlsanity.com/paper/2604.04745/) (2026)
- [TaxBreak: Unmasking the Hidden Costs of LLM Inference Through Overhead Decomposition](https://mlsanity.com/paper/2603.12465/) (2026)
- [Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflows](https://mlsanity.com/paper/2604.09611/) (2026)
- [Position: LLM Inference Should Be Evaluated as Energy-to-Token Production](https://mlsanity.com/paper/2605.11733/) (2026)
- [Fine-Grained Power and Energy Attribution on AMD GPU/APU-Based Exascale Nodes](https://mlsanity.com/paper/2604.06056/) (2026)

## Built on by (most cited first, 2 of 2)

- [Who Pays for the KV Cache? Attributing Shared AI Inference Spend Across Kubernetes and LLM Provider Bills](https://mlsanity.com/paper/2609.24991/) (2026)
- [Greenpixie's AI Token Methodology: Assessing the Energy, Water and CO2-eq Impact of AI Tokens for Open and Closed Weight Models](https://mlsanity.com/paper/2609.33965/) (2026)

## Abstract (arXiv)

Large language model (LLM) inference serving is priced by tokens, but GPU energy is consumed over inference windows. This accounting mismatch makes token-normalized metrics incomplete, since average output-token energy can decrease even when total request energy increases. We characterize this behavior with a decomposed energy model: a fixed one-time prefill with a fixed generation setup cost, while each output-token generation step adds marginal step energy. We evaluate this LLM inference energy model on NVIDIA H100 and H200 GPUs across dense and mixture-of-experts (MoE) models, reporting both request energy and token energy as functions of model type (M), phase (P), batch size (B), context length (C), and output length (N). For Llama-3.2-1B on H200 at batch-16 and context-4K, increasing output length from 10 to 512 tokens reduces token energy from 7.46 to 0.72 J/token while total batched inference-window energy increases from 1.19 to 5.93 kJ. Batching also reduces token energy, but the gain is context-bounded: at 10 output tokens, the batch-16 to batch-1 gain falls from 6.31x at context-512 to 1.17x at context-4K. MoE models amplify this effect: sparse routing and fragmented expert execution increase fixed energy at low concurrency, while batching spreads that energy across more generated tokens and substantially narrows the dense-vs.-MoE token-energy gap. These results show that energy-aware serving should jointly optimize both request energy and token energy, rather than only reducing per-token energy cost.