mlsanityData to 7 Oct 2026

Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloads

arXiv 2407.00010 ↗First posted 25 Apr 2024cs.DC cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Both the training and use of Large Language Models (LLMs) require large amounts of energy. Their increasing popularity, therefore, raises critical concerns regarding the energy efficiency and sustainability of data centers that host them. This paper addresses the challenge of reducing energy consumption in data centers running LLMs. We propose a hybrid data center model that uses a cost-based scheduling framework to dynamically allocate LLM tasks across hardware accelerators that differ in their energy efficiencies and computational capabilities. Specifically, our workload-aware strategy determines whether tasks are processed on energy-efficient processors or high-performance GPUs based on the number of input and output tokens in a query. Our analysis of a representative LLM dataset, finds that this hybrid strategy can reduce CPU+GPU energy consumption by 7.5% compared to a workload-unaware baseline.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 15 of its 38 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

15 of 38 references.

References · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Mistral 7Bcited by 3,065
  4. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  5. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
  6. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 246
  7. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  8. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 168
  9. 2021Compute and Energy Consumption Trends in Deep Learning Inferencecited by 61
  10. 2021Carbon-Aware Computing for Datacenterscited by 40
  11. 2021Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenterscited by 31
  12. 2024The Landscape and Challenges of HPC Research and LLMscited by 22
3 more
  1. 2024LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantizationcited by 14
  2. 2022Treehouse: A Case For Carbon-Aware Datacenter Softwarecited by 12
  3. 2023Energy-Efficient GPU Clusters Scheduling for Deep Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 6.

17 papers.

Sort:

By area:

  1. 2025A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrievalcited by 31
  2. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 28
  3. 2024Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systemscited by 27
  4. 2024The Price of Prompting: Profiling Energy Use in Large Language Models Inferencecited by 22
  5. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
  6. 2025Intelligence per Watt: Measuring Intelligence Efficiency of Local AIcited by 13
  7. 2025FairBatching: Fairness-Aware Batch Formation for LLM Inferencecited by 6
  8. 2024Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Cachingcited by 3
  9. 2025Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implicationscited by 2
  10. 2025Perturbative Gradient Training: A novel training paradigm for bridging the gap between deep neural networks and physical reservoir computingcited by 1
  11. 2026ConsRoute:Consistency-Aware Adaptive Query Routing for Cloud-Edge-Device Large Language Modelscited by 1
  12. 2026KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Servingcited by 1
5 more
  1. 2025An Evaluation of LLMs Inference on Popular Single-board Computerscited by 0
  2. 20263D-Learning: Diffusion-Augmented Distributionally Robust Decision-Focused Learningcited by 0
  3. 2026Networking-Aware Energy Efficiency in Agentic AI Inference: A Surveycited by 0
  4. 2026A Physics-Aware Framework for Short-Term GPU Power Forecasting of AI Data Centerscited by 0
  5. 2026Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Servingcited by 0