# Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloads

arXiv: https://arxiv.org/abs/2407.00010 · HTML: https://mlsanity.com/paper/2407.00010/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 38 references that are in the map.
- First posted on arXiv 25 Apr 2024.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning](https://mlsanity.com/paper/2002.05651/) (2020)
- [Sustainable AI: Environmental Implications, Challenges and Opportunities](https://mlsanity.com/paper/2111.00364/) (2021)
- [From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference](https://mlsanity.com/paper/2310.03003/) (2023)
- [Compute and Energy Consumption Trends in Deep Learning Inference](https://mlsanity.com/paper/2109.05472/) (2021)
- [Carbon-Aware Computing for Datacenters](https://mlsanity.com/paper/2106.11750/) (2021)
- [Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters](https://mlsanity.com/paper/2109.01313/) (2021)
- [The Landscape and Challenges of HPC Research and LLMs](https://mlsanity.com/paper/2402.02018/) (2024)
- [LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization](https://mlsanity.com/paper/2403.01136/) (2024)
- [Treehouse: A Case For Carbon-Aware Datacenter Software](https://mlsanity.com/paper/2201.02120/) (2022)
- [Energy-Efficient GPU Clusters Scheduling for Deep Learning](https://mlsanity.com/paper/2304.06381/) (2023)

## Built on by (most cited first, 17 of 17)

- [A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrieval](https://mlsanity.com/paper/2503.05659/) (2025)
- [TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms](https://mlsanity.com/paper/2501.02600/) (2025)
- [Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems](https://mlsanity.com/paper/2407.04014/) (2024)
- [The Price of Prompting: Profiling Energy Use in Large Language Models Inference](https://mlsanity.com/paper/2407.16893/) (2024)
- [Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression](https://mlsanity.com/paper/2505.19433/) (2025)
- [Intelligence per Watt: Measuring Intelligence Efficiency of Local AI](https://mlsanity.com/paper/2511.07885/) (2025)
- [FairBatching: Fairness-Aware Batch Formation for LLM Inference](https://mlsanity.com/paper/2510.14392/) (2025)
- [Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching](https://mlsanity.com/paper/2410.14740/) (2024)
- [Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications](https://mlsanity.com/paper/2601.08833/) (2025)
- [Perturbative Gradient Training: A novel training paradigm for bridging the gap between deep neural networks and physical reservoir computing](https://mlsanity.com/paper/2506.04523/) (2025)
- [ConsRoute:Consistency-Aware Adaptive Query Routing for Cloud-Edge-Device Large Language Models](https://mlsanity.com/paper/2603.21237/) (2026)
- [KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving](https://mlsanity.com/paper/2604.16682/) (2026)
- [An Evaluation of LLMs Inference on Popular Single-board Computers](https://mlsanity.com/paper/2511.07425/) (2025)
- [3D-Learning: Diffusion-Augmented Distributionally Robust Decision-Focused Learning](https://mlsanity.com/paper/2602.02943/) (2026)
- [Networking-Aware Energy Efficiency in Agentic AI Inference: A Survey](https://mlsanity.com/paper/2604.07857/) (2026)
- [A Physics-Aware Framework for Short-Term GPU Power Forecasting of AI Data Centers](https://mlsanity.com/paper/2605.04074/) (2026)
- [Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Serving](https://mlsanity.com/paper/2609.23085/) (2026)

## Abstract (arXiv)

Both the training and use of Large Language Models (LLMs) require large amounts of energy. Their increasing popularity, therefore, raises critical concerns regarding the energy efficiency and sustainability of data centers that host them. This paper addresses the challenge of reducing energy consumption in data centers running LLMs. We propose a hybrid data center model that uses a cost-based scheduling framework to dynamically allocate LLM tasks across hardware accelerators that differ in their energy efficiencies and computational capabilities. Specifically, our workload-aware strategy determines whether tasks are processed on energy-efficient processors or high-performance GPUs based on the number of input and output tokens in a query. Our analysis of a representative LLM dataset, finds that this hybrid strategy can reduce CPU+GPU energy consumption by 7.5% compared to a workload-unaware baseline.