mlsanityData to 7 Oct 2026

DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency

arXiv 2408.00741 ↗First posted 1 Aug 2024cs.AI cs.AR cs.DCcited by 106

Not on the map. Why?

Cited by 51 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The rapid evolution and widespread adoption of generative large language models (LLMs) have made them a pivotal workload in various applications. Today, LLM inference clusters receive a large number of queries with strict Service Level Objectives (SLOs). To achieve the desired performance, these models execute on power-hungry GPUs causing the inference clusters to consume large amount of energy and, consequently, result in excessive carbon emissions. Fortunately, we find that there is a great opportunity to exploit the heterogeneity in inference compute properties and fluctuations in inference workloads, to significantly improve energy-efficiency. However, such a diverse and dynamic environment creates a large search-space where different system configurations (e.g., number of instances, model parallelism, and GPU frequency) translate into different energy-performance trade-offs. To address these challenges, we propose DynamoLLM, the first energy-management framework for LLM inference environments. DynamoLLM automatically and dynamically reconfigures the inference cluster to optimize for energy and cost of LLM serving under the service's performance SLOs. We show that at a service-level, DynamoLLM conserves 53% energy and 38% operational carbon emissions, and reduces 61% cost to the customer, while meeting the latency SLOs.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Builds on 25 of its 87 references that are in the map. First posted on arXiv 1 Aug 2024.

Builds on

25 of 87 references.

References · 25

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  7. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  8. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  9. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  10. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  11. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 168
  12. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 160
13 more
  1. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  2. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 83
  3. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  4. 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 78
  5. 2023LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Modelscited by 71
  6. 2020EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inferencecited by 69
  7. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 64
  8. 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 59
  9. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  10. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 43
  11. 2022Dynamic GPU Energy Optimization for Machine Learning Training Workloadscited by 12
  12. 2019One Size Does Not Fit All: Quantifying and Exposing the Accuracy-Latency Trade-off in Machine Learning Cloud Service APIs via Tolerance Tierscited by 7
  13. 2020Prediction-Based Power Oversubscription in Cloud Platformscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 44; 2026: 50.

106 papers.

Sort:

By area:

  1. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  2. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  3. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
  4. 2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 34
  5. 2025The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimizationcited by 31
  6. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  7. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 28
  8. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  9. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 21
  10. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  11. 2025Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platformscited by 18
  12. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selectioncited by 16
38 more
  1. 2024Optimal Query Allocation in Extractive QA with LLMs: A Learning-to-Defer Framework with Theoretical Guaranteescited by 15
  2. 2024APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Servingcited by 15
  3. 2026LLM-42: Enabling Determinism in LLM Inference with Verified Speculationcited by 14
  4. 2025Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scalingcited by 12
  5. 2026Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacentercited by 11
  6. 2024The Unseen AI Disruptions for Power Grids: LLM-Induced Transientscited by 10
  7. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  8. 2026Where Do the Joules Go? Diagnosing Inference Energy Consumptioncited by 10
  9. 2024Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environmentscited by 9
  10. 2025LithOS: An Operating System for Efficient Machine Learning on GPUscited by 9
  11. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocitycited by 9
  12. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inferencecited by 8
  13. 2025Niyama : Breaking the Silos of LLM Inference Servingcited by 6
  14. 2025Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspectivecited by 5
  15. 2026FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Servingcited by 5
  16. 2026Beyond Test-Time Compute Strategies: Advocating Energy-per-Token in LLM Inferencecited by 5
  17. 2026Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architecturescited by 5
  18. 2026SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusterscited by 5
  19. 2025A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMscited by 4
  20. 2025gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttlingcited by 4
  21. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizingcited by 4
  22. 2025Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systemscited by 4
  23. 2025Trusted AI Agents in the Cloudcited by 4
  24. 2024Improving training time and GPU utilization in geo-distributed language model trainingcited by 3
  25. 2025MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraintscited by 3
  26. 2025RAP: Runtime Adaptive Pruning for LLM Inferencecited by 3
  27. 2025ElasticMoE: An Efficient Auto Scaling Method for Mixture-of-Experts Modelscited by 3
  28. 2025From Tokens to Layers: Redefining Stall-Free Scheduling for MoE Serving with Layered Prefillcited by 3
  29. 2026Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Trainingcited by 3
  30. 2025Efficient LLM Serving on Hybrid Real-time and Best-effort Requestscited by 2
  31. 2025Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Schedulingcited by 2
  32. 2025Equinox: Holistic Fair Scheduling in Serving Large Language Modelscited by 2
  33. 2025Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rankcited by 2
  34. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Servingcited by 2
  35. 2026Reducing Compute Waste in LLMs through Kernel-Level DVFScited by 2
  36. 2025Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implicationscited by 2
  37. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestrationcited by 2
  38. 2026Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflowscited by 2

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems
  2. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling
  3. 2025Taming the Titans: A Survey of Efficient LLM Inference Serving
  4. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  5. 2024WebLLM: A High-Performance In-Browser LLM Inference Engine
  6. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models
  7. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  8. 2023Efficient LLM Inference on CPUs