# Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems

arXiv: https://arxiv.org/abs/2407.04014 · HTML: https://mlsanity.com/paper/2407.04014/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 23 of its 51 references that are in the map.
- First posted on arXiv 4 Jul 2024.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Efficiently Scaling Transformer Inference](https://mlsanity.com/paper/2211.05102/) (2022)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning](https://mlsanity.com/paper/2002.05651/) (2020)
- [Sustainable AI: Environmental Implications, Challenges and Opportunities](https://mlsanity.com/paper/2111.00364/) (2021)
- [From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference](https://mlsanity.com/paper/2310.03003/) (2023)
- [Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline](https://mlsanity.com/paper/2305.13144/) (2023)
- [Compute and Energy Consumption Trends in Deep Learning Inference](https://mlsanity.com/paper/2109.05472/) (2021)
- [Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference](https://mlsanity.com/paper/2403.20306/) (2024)
- [Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters](https://mlsanity.com/paper/2109.01313/) (2021)
- [Clover: Toward Sustainable AI with Carbon-Aware Machine Learning Inference Service](https://mlsanity.com/paper/2304.09781/) (2023)
- [Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloads](https://mlsanity.com/paper/2407.00010/) (2024)
- [Toward Sustainable GenAI using Generation Directives for Carbon-Friendly Large Language Model Inference](https://mlsanity.com/paper/2403.12900/) (2024)
- [Treehouse: A Case For Carbon-Aware Datacenter Software](https://mlsanity.com/paper/2201.02120/) (2022)
- [Towards Environmentally Equitable AI via Geographical Load Balancing](https://mlsanity.com/paper/2307.05494/) (2023)
- [Exploding AI Power Use: an Opportunity to Rethink Grid Planning and Management](https://mlsanity.com/paper/2311.11645/) (2023)
- [Model-driven Cluster Resource Management for AI Workloads in Edge Clouds](https://mlsanity.com/paper/2201.07312/) (2022)
- [Adapting Datacenter Capacity for Greener Datacenters and Grid](https://mlsanity.com/paper/2301.03148/) (2023)
- [Energy-Efficient GPU Clusters Scheduling for Deep Learning](https://mlsanity.com/paper/2304.06381/) (2023)

## Built on by (most cited first, 28 of 28)

- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)
- [Energy Considerations of Large Language Model Inference and Efficiency Optimizations](https://mlsanity.com/paper/2504.17674/) (2025)
- [SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving](https://mlsanity.com/paper/2408.05235/) (2024)
- [From Prompts to Power: Measuring the Energy Footprint of LLM Inference](https://mlsanity.com/paper/2511.05597/) (2025)
- [Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2503.16585/) (2025)
- [SweetSpot: An Analytical Model for Predicting Energy Efficiency of LLM Inference](https://mlsanity.com/paper/2602.05695/) (2026)
- [SWEnergy: An Empirical Study on Energy Efficiency in Agentic Issue Resolution Frameworks with SLMs](https://mlsanity.com/paper/2512.09543/) (2025)
- [Beyond Test-Time Compute Strategies: Advocating Energy-per-Token in LLM Inference](https://mlsanity.com/paper/2603.20224/) (2026)
- [Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use](https://mlsanity.com/paper/2601.22362/) (2026)
- [GPT-OSS-20B: A Comprehensive Deployment-Centric Analysis of OpenAI's Open-Weight Mixture of Experts Model](https://mlsanity.com/paper/2508.16700/) (2025)
- [Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications](https://mlsanity.com/paper/2601.08833/) (2025)
- [Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflows](https://mlsanity.com/paper/2604.09611/) (2026)
- [Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms](https://mlsanity.com/paper/2608.28044/) (2026)
- [Addressing the sustainable AI trilemma: a case study on LLM agents and RAG](https://mlsanity.com/paper/2501.08262/) (2025)
- [Small Talk, Big Impact: The Energy Cost of Thanking AI](https://mlsanity.com/paper/2601.22357/) (2026)
- [Pimp My LLM: Leveraging Variability Modeling to Tune Inference Hyperparameters](https://mlsanity.com/paper/2602.17697/) (2026)
- [KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving](https://mlsanity.com/paper/2604.16682/) (2026)
- [WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs](https://mlsanity.com/paper/2607.02391/) (2026)
- [Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations](https://mlsanity.com/paper/2607.09172/) (2026)
- [Opt4GPTQ: Co-Optimizing Memory and Computation for 4-bit GPTQ Quantized LLM Inference on Heterogeneous Platforms](https://mlsanity.com/paper/2511.19438/) (2025)
- [Token Management in Multi-Tenant AI Inference Platforms](https://mlsanity.com/paper/2603.00356/) (2026)
- [SEALing the Gap: A Reference Framework for LLM Inference Carbon Estimation via Multi-Benchmark Driven Embodiment](https://mlsanity.com/paper/2603.02949/) (2026)
- [From Perception to Action: Can UI Interventions Foster Sustainable LLM Chatbot](https://mlsanity.com/paper/2606.10861/) (2026)
- [From Caveman to Expert Analyst: Energy Consumption of Variable LLM Tasks](https://mlsanity.com/paper/2608.12350/) (2026)
- [TokenPowerSandbox: Evidence-Gated CPU-First Screening for Energy-Aware LLM Serving](https://mlsanity.com/paper/2608.18149/) (2026)
- [Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling](https://mlsanity.com/paper/2609.19499/) (2026)
- [Analytical Power-Aware Provisioning for Prefill-Decode Disaggregated AI Inference](https://mlsanity.com/paper/2609.24639/) (2026)
- [Greenpixie's AI Token Methodology: Assessing the Energy, Water and CO2-eq Impact of AI Tokens for Open and Closed Weight Models](https://mlsanity.com/paper/2609.33965/) (2026)

## Abstract (arXiv)

The rapid adoption of large language models (LLMs) has led to significant advances in natural language processing and text generation. However, the energy consumed through LLM model inference remains a major challenge for sustainable AI deployment. To address this problem, we model the workload-dependent energy consumption and runtime of LLM inference tasks on heterogeneous GPU-CPU systems. By conducting an extensive characterization study of several state-of-the-art LLMs and analyzing their energy and runtime behavior across different magnitudes of input prompts and output text, we develop accurate (R^2>0.96) energy and runtime models for each LLM. We employ these models to explore an offline, energy-optimal LLM workload scheduling framework. Through a case study, we demonstrate the advantages of energy and accuracy aware scheduling compared to existing best practices.