mlsanityData to 14 Oct 2026

Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms

arXiv 2608.28044 ↗First posted 28 Aug 2026cs.PF cs.DC cs.LGcited by 2

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Large language model (LLM) inference serving is priced by tokens, but GPU energy is consumed over inference windows. This accounting mismatch makes token-normalized metrics incomplete, since average output-token energy can decrease even when total request energy increases. We characterize this behavior with a decomposed energy model: a fixed one-time prefill with a fixed generation setup cost, while each output-token generation step adds marginal step energy. We evaluate this LLM inference energy model on NVIDIA H100 and H200 GPUs across dense and mixture-of-experts (MoE) models, reporting both request energy and token energy as functions of model type (M), phase (P), batch size (B), context length (C), and output length (N). For Llama-3.2-1B on H200 at batch-16 and context-4K, increasing output length from 10 to 512 tokens reduces token energy from 7.46 to 0.72 J/token while total batched inference-window energy increases from 1.19 to 5.93 kJ. Batching also reduces token energy, but the gain is context-bounded: at 10 output tokens, the batch-16 to batch-1 gain falls from 6.31x at context-512 to 1.17x at context-4K. MoE models amplify this effect: sparse routing and fragmented expert execution increase fixed energy at low concurrency, while batching spreads that energy across more generated tokens and substantially narrows the dense-vs.-MoE token-energy gap. These results show that energy-aware serving should jointly optimize both request energy and token energy, rather than only reducing per-token energy cost.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 25 of its 36 references that are in the map. First posted on arXiv 28 Aug 2026.

Builds on

25 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026TaxBreak: Unmasking the Hidden Costs of LLM Inference Through Overhead Decomposition12 Mar

Other references · 24

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  6. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  7. 2019MLPerf Inference Benchmarkcited by 199
  8. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 170
  9. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  10. 2025Energy Considerations of Large Language Model Inference and Efficiency Optimizationscited by 55
  11. 2024Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systemscited by 28
  12. 2025TokenPowerBench: Benchmarking the Power Consumption of LLM Inferencecited by 22
12 more
  1. 2025The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Computecited by 15
  2. 2025Intelligence per Watt: Measuring Intelligence Efficiency of Local AIcited by 14
  3. 2025Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scalingcited by 13
  4. 2025Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Modelscited by 11
  5. 2025Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architecturescited by 8
  6. 2026SweetSpot: An Analytical Model for Predicting Energy Efficiency of LLM Inferencecited by 6
  7. 2026Beyond Test-Time Compute Strategies: Advocating Energy-per-Token in LLM Inferencecited by 5
  8. 2026Wattchmen: Watching the Wattchers -- High Fidelity, Flexible GPU Energy Modelingcited by 3
  9. 2026The Energy Cost of Execution-Idle in GPU Clusterscited by 3
  10. 2026Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflowscited by 2
  11. 2026Position: LLM Inference Should Be Evaluated as Energy-to-Token Productioncited by 2
  12. 2026Fine-Grained Power and Energy Attribution on AMD GPU/APU-Based Exascale Nodescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Who Pays for the KV Cache? Attributing Shared AI Inference Spend Across Kubernetes and LLM Provider Billscited by 0
  2. 2026Greenpixie's AI Token Methodology: Assessing the Energy, Water and CO2-eq Impact of AI Tokens for Open and Closed Weight Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use
  2. 2026Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures
  3. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling
  4. 2026Beneath the Tokens: A Performance Engineering Study of Multi-Token Prediction in GPU-Accelerated LLM Inference
  5. 2026The 1/W Law: An Analytical Study of Context-Length Routing Topology and GPU Generation Gains for LLM Inference Energy Efficiency
  6. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  7. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference
  8. 2026Silicon Showdown: Performance, Efficiency, and Ecosystem Barriers in Consumer-Grade LLM Inference