mlsanityData to 7 Oct 2026

SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving

arXiv 2408.05235 ↗First posted 5 Aug 2024cs.DC cs.AI cs.ARcited by 18

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

As Large Language Models (LLMs) gain traction, their reliance on power-hungry GPUs places ever-increasing energy demands, raising environmental and monetary concerns. Inference dominates LLM workloads, presenting a critical challenge for providers: minimizing energy costs under Service-Level Objectives (SLOs) that ensure optimal user experience. In this paper, we present textitthrottLL'eM, a framework that reduces energy consumption while meeting SLOs through the use of instance and GPU frequency scaling. textitthrottLL'eM features mechanisms that project future KV cache usage and batch size. Leveraging a Machine-Learning (ML) model that receives these projections as inputs, textitthrottLL'eM manages performance at the iteration level to satisfy SLOs with reduced frequencies and instance sizes. We show that the proposed ML model achieves R2 scores greater than 0.97 and miss-predicts performance by less than 1 iteration per second on average. Experimental results on LLM inference traces show that textitthrottLL'eM achieves up to 43.8% lower energy consumption and an energy efficiency improvement of at least 1.71× under SLOs, when compared to NVIDIA's Triton server.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 45 of its 77 references that are in the map. First posted on arXiv 5 Aug 2024.

Builds on

45 of 77 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems4 Jul
  2. 2024Queue management for slo-oriented large language model serving5 Jun
  3. 2024A Survey on Efficient Inference for Large Language Models22 Apr
  4. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  6. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  8. 2023Fairness in Serving Large Language Models31 Dec
  9. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
  10. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May
  11. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2024Mixtral of Expertscited by 1,785
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
22 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  5. 2022Efficiently Scaling Transformer Inferencecited by 529
  6. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  7. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  8. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  9. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  10. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  11. 2019MLPerf Inference Benchmarkcited by 195
  12. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  13. 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 183
  14. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169
  15. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 168
  16. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  17. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
  18. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  19. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  20. 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 59
  21. 2022ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputscited by 32
  22. 2020Prediction-Based Power Oversubscription in Cloud Platformscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 14.

18 papers.

Sort:

By area:

  1. 2025CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIMcited by 5
  2. 2026Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architecturescited by 5
  3. 2026DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFScited by 4
  4. 2025AGFT: An Adaptive GPU Frequency Tuner for Real-Time LLM Inference Optimizationcited by 2
  5. 2025VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routingcited by 2
  6. 2025Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implicationscited by 2
  7. 2026PowerSlider: Exploiting Phase Asymmetry for LLM Serving under Demand Responsecited by 2
  8. 2026PASCAL: A Phase-Aware Scheduling Algorithm for Serving Reasoning-based Large Language Modelscited by 1
  9. 2026ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Servingcited by 1
  10. 2026WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMscited by 1
  11. 2026Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levelscited by 1
  12. 2026PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scalingcited by 1
6 more
  1. 2026Networking-Aware Energy Efficiency in Agentic AI Inference: A Surveycited by 0
  2. 2026Enabling Spatially Fine-Grained DVFS in Neural Processing Units for Energy-Efficient LLM Servingcited by 0
  3. 2026Energy-Efficient LLM Serving via Disaggregated Attention--FFN and Flexible Frequency Scalingcited by 0
  4. 2026Efficient Resource Optimization for Split Federated Learningcited by 0
  5. 2026Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Servingcited by 0
  6. 2026A principled approach for energy-efficient training via phase-aware GPU frequency tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling
  2. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  3. 2023Efficient LLM inference solution on Intel GPU
  4. 2026PALS: Power-Aware LLM Serving for Mixture-of-Experts Models
  5. 2026EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimization
  6. 2024Inference Performance Optimization for Large Language Models on CPUs
  7. 2026A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints
  8. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency