# EcoServe: Designing Carbon-Aware AI Inference Systems

arXiv: https://arxiv.org/abs/2502.05043 · HTML: https://mlsanity.com/paper/2502.05043/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2025.

## Built on by (most cited first, 19 of 19)

- [Online Scheduling for LLM Inference with KV Cache Constraints](https://mlsanity.com/paper/2502.07115/) (2025)
- [Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scaling](https://mlsanity.com/paper/2509.20241/) (2025)
- [Agentic Services Computing](https://mlsanity.com/paper/2509.24380/) (2025)
- [VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routing](https://mlsanity.com/paper/2509.04827/) (2025)
- [PowerSlider: Exploiting Phase Asymmetry for LLM Serving under Demand Response](https://mlsanity.com/paper/2608.21719/) (2026)
- [Hardwired-Neurons Language Processing Units as General-Purpose Cognitive Substrates](https://mlsanity.com/paper/2508.16151/) (2025)
- [CarbonEdge: Carbon-Aware Deep Learning Inference Framework for Sustainable Edge Computing](https://mlsanity.com/paper/2603.27420/) (2026)
- [KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving](https://mlsanity.com/paper/2604.16682/) (2026)
- [GAR: Carbon-Aware Routing for LLM Inference via Constrained Optimization](https://mlsanity.com/paper/2605.11603/) (2026)
- [Energy-Aware Routing to Large Reasoning Models](https://mlsanity.com/paper/2601.00823/) (2025)
- [Environmental Footprint of GenAI Research: Insights from the Moshi Foundation Model](https://mlsanity.com/paper/2604.11154/) (2026)
- [AI Inference as Relocatable Electricity Demand: A Latency-Constrained Energy-Geography Framework](https://mlsanity.com/paper/2604.27855/) (2026)
- [Greening AI Inference with Accuracy and Latency-aware User Incentives](https://mlsanity.com/paper/2605.27309/) (2026)
- [Evaluation of ML Resource Utilization Requires Model Life Cycle Assessment](https://mlsanity.com/paper/2606.07632/) (2026)
- [Enabling Spatially Fine-Grained DVFS in Neural Processing Units for Energy-Efficient LLM Serving](https://mlsanity.com/paper/2607.16473/) (2026)
- [AgentDecarbonizer: Carbon-Aware Execution for AI Agents](https://mlsanity.com/paper/2608.20566/) (2026)
- [Toward Sustainable Distributed LLM Inference: A Systems Synthesis and Research Agenda for an Energy-, Carbon-, and Cache-Aware llm-d Control Plane](https://mlsanity.com/paper/2609.05565/) (2026)
- [A principled approach for energy-efficient training via phase-aware GPU frequency tuning](https://mlsanity.com/paper/2609.24205/) (2026)
- [Dissecting the Carbon-SLO Tension in Geo-Distributed LLM Serving](https://mlsanity.com/paper/2610.14571/) (2026)

## Abstract (arXiv)

The rapid increase in LLM ubiquity and scale levies unprecedented demands on computing infrastructure. These demands not only incur large compute and memory resources but also significant energy, yielding large operational and embodied carbon emissions. In this work, we present three main observations based on modeling and traces from the production deployment of two Generative AI services in a major cloud service provider. First, while GPUs dominate operational carbon, host processing systems (e.g., CPUs, memory, storage) dominate embodied carbon. Second, offline, batch inference accounts for a significant portion (up to 55%) of serving capacity. Third, there are different levels of heterogeneity across hardware and workloads for LLM inference. Based on these observations, we design EcoServe, a carbon-aware resource provision and scheduling framework for LLM serving systems. It is based on four principles - Reduce, Reuse, Rightsize, and Recycle (4R). With a cross-stack ILP formulation and design, we demonstrate that EcoServe can lower carbon emissions by up to 47%, compared to performance, energy, and cost-optimized design points, while maintaining performance targets and SLOs.