mlsanityData to 14 Oct 2026

Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling

arXiv 2501.08219 ↗First posted 14 Jan 2025cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

LLM inference exhibits substantial variability across queries and execution phases, yet inference configurations are often applied uniformly. We present a measurement-driven characterization of workload heterogeneity and energy-performance behavior of LLM inference under GPU dynamic voltage and frequency scaling (DVFS). We evaluate five decoder-only LLMs (1B-32B parameters) across four NLP benchmarks using a controlled offline setup. We show that lightweight semantic features predict inference difficulty better than input length, with 44.5% of queries achieving comparable quality across model sizes. At the hardware level, the decode phase dominates inference time (77-91%) and is largely insensitive to GPU frequency. Consequently, reducing GPU frequency from 2842 MHz to 180 MHz achieves an average of 42% energy savings with only a 1-6% latency increase. We further provide a use case with an upper-bound analysis of the potential benefits of combining workload-aware model selection with phase-aware DVFS, motivating future energy-efficient LLM inference systems.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 34 of its 41 references that are in the map. First posted on arXiv 14 Jan 2025.

Builds on

34 of 41 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2023Mistral 7Bcited by 3,164
22 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  2. 2023A Survey of Large Language Modelscited by 2,576
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  9. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  10. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  11. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  12. 2019Green AIcited by 627
  13. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  14. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  15. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  16. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  17. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  18. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  19. 2019MLPerf Inference Benchmarkcited by 199
  20. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  21. 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 79
  22. 2023Reducing Energy Bloat in Large Model Trainingcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 10.

19 papers.

Sort:

By area:

  1. 2025Cost-of-Pass: An Economic Framework for Evaluating Language Modelscited by 35
  2. 2025AutoJudge: Judge Decoding Without Manual Annotationcited by 13
  3. 2025Streaming, Fast and Slow: Cognitive Load-Aware Streaming for Efficient LLM Servingcited by 3
  4. 2025AGFT: An Adaptive GPU Frequency Tuner for Real-Time LLM Inference Optimizationcited by 2
  5. 2025GPT-OSS-20B: A Comprehensive Deployment-Centric Analysis of OpenAI's Open-Weight Mixture of Experts Modelcited by 2
  6. 2026Towards Green AI: Decoding the Energy of LLM Inference in Software Developmentcited by 2
  7. 2025Performance Trade-offs of Optimizing Small Language Models for E-Commercecited by 1
  8. 2026KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Servingcited by 1
  9. 2026The Illusion of Power Capping in LLM Decode: A Phase-Aware Energy Characterisation Across Attention Architecturescited by 1
  10. 2026Energy-Efficient GPU DVFS for Fine-Tuning of SLMs on Resource-constrained Embedded Devicescited by 1
  11. 2025Climate Finance Benchcited by 0
  12. 2025The Generative Energy Arena (GEA): Incorporating Energy Awareness in Large Language Model (LLM) Human Evaluationscited by 0
7 more
  1. 2025Smaller Models, Smarter Rewards: A Two-Sided Approach to Process and Outcome Rewardscited by 0
  2. 2026Balancing Sustainability And Performance: The Role Of Small-Scale LLMs In Agentic Artificial Intelligence Systemscited by 0
  3. 2026Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Servingcited by 0
  4. 2026EnergyLens: Interpretable Closed-Form Energy Models for Multimodal LLM Inference Servingcited by 0
  5. 2026Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardwarecited by 0
  6. 2026Efficient Resource Optimization for Split Federated Learningcited by 0
  7. 2026Where Should Agents Live? Energy-Memory Characterization of Agentic AI for the Edge-Cloud Continuumcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems
  2. 2026Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use
  3. 2026Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures
  4. 2025Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modeling
  5. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency
  6. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving
  7. 2023Efficient LLM inference solution on Intel GPU
  8. 2026Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms