# TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms

arXiv: https://arxiv.org/abs/2501.02600 · HTML: https://mlsanity.com/paper/2501.02600/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 79 references that are in the map.
- First posted on arXiv 5 Jan 2025.

## Builds on (most cited first, 25 of 25)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [LLM in a flash: Efficient Large Language Model Inference with Limited Memory](https://mlsanity.com/paper/2312.11514/) (2023)
- [From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference](https://mlsanity.com/paper/2310.03003/) (2023)
- [A Study of Generative Large Language Model for Medical Research and Healthcare](https://mlsanity.com/paper/2305.13523/) (2023)
- [Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention](https://mlsanity.com/paper/2403.19708/) (2024)
- [Fast Distributed Inference Serving for Large Language Models](https://mlsanity.com/paper/2305.05920/) (2023)
- [LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism](https://mlsanity.com/paper/2404.09526/) (2024)
- [DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency](https://mlsanity.com/paper/2408.00741/) (2024)
- [SpotServe: Serving Generative Large Language Models on Preemptible Instances](https://mlsanity.com/paper/2311.15566/) (2023)
- [Evaluating Quantized Large Language Models](https://mlsanity.com/paper/2402.18158/) (2024)
- [Preble: Efficient Distributed Prompt Scheduling for LLM Serving](https://mlsanity.com/paper/2407.00023/) (2024)
- [Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction](https://mlsanity.com/paper/2404.08509/) (2024)
- [Stateful Large Language Model Serving with Pensieve](https://mlsanity.com/paper/2312.05516/) (2023)
- [Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference](https://mlsanity.com/paper/2403.20306/) (2024)
- [ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching](https://mlsanity.com/paper/2403.17312/) (2024)
- [LLM Inference Serving: Survey of Recent Advances and Opportunities](https://mlsanity.com/paper/2407.12391/) (2024)
- [Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloads](https://mlsanity.com/paper/2407.00010/) (2024)
- [Towards Environmentally Equitable AI via Geographical Load Balancing](https://mlsanity.com/paper/2307.05494/) (2023)
- [The Unseen AI Disruptions for Power Grids: LLM-Induced Transients](https://mlsanity.com/paper/2409.11416/) (2024)

## Built on by (most cited first, 29 of 29)

- [ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Serving](https://mlsanity.com/paper/2502.00937/) (2025)
- [Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspective](https://mlsanity.com/paper/2509.10371/) (2025)
- [MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints](https://mlsanity.com/paper/2504.09345/) (2025)
- [Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Training](https://mlsanity.com/paper/2601.17654/) (2026)
- [VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routing](https://mlsanity.com/paper/2509.04827/) (2025)
- [Rearchitecting Datacenter Lifecycle for AI: A TCO-Driven Framework](https://mlsanity.com/paper/2509.26534/) (2025)
- [PowerSlider: Exploiting Phase Asymmetry for LLM Serving under Demand Response](https://mlsanity.com/paper/2608.21719/) (2026)
- [PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling](https://mlsanity.com/paper/2609.09662/) (2026)
- [Hardwired-Neurons Language Processing Units as General-Purpose Cognitive Substrates](https://mlsanity.com/paper/2508.16151/) (2025)
- [Improving AI Efficiency in Data Centres by Power Dynamic Response](https://mlsanity.com/paper/2510.11119/) (2025)
- [Vulcan: Instance-specialized, Verifiable Systems Heuristics Through LLM-driven Search](https://mlsanity.com/paper/2512.25065/) (2025)
- [Quantization-Aware Collaborative Inference for Large Embodied AI Models](https://mlsanity.com/paper/2602.13052/) (2026)
- [KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving](https://mlsanity.com/paper/2604.16682/) (2026)
- [Guard: Scalable Straggler Detection and Node Health Management for Large-Scale Training](https://mlsanity.com/paper/2605.17879/) (2026)
- [ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving](https://mlsanity.com/paper/2606.00735/) (2026)
- [Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference](https://mlsanity.com/paper/2606.02430/) (2026)
- [SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling](https://mlsanity.com/paper/2502.14617/) (2025)
- [StreamWise: Serving Multi-Modal Generation in Real-Time at Scale](https://mlsanity.com/paper/2603.05800/) (2026)
- [Networking-Aware Energy Efficiency in Agentic AI Inference: A Survey](https://mlsanity.com/paper/2604.07857/) (2026)
- [Designing Datacenter Power Delivery Hierarchies for the AI Era](https://mlsanity.com/paper/2605.16255/) (2026)
- [SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Serving](https://mlsanity.com/paper/2606.11244/) (2026)
- [Enabling Spatially Fine-Grained DVFS in Neural Processing Units for Energy-Efficient LLM Serving](https://mlsanity.com/paper/2607.16473/) (2026)
- [Beyond Prefill-Decode Disaggregation: Dissecting LLM Inference for Heterogeneous Platforms via Dynamic Operator Scheduling](https://mlsanity.com/paper/2607.25498/) (2026)
- [AtumAI: A Principled Framework for Agentic Generation of Datacenter Control-Plane Policies](https://mlsanity.com/paper/2608.02569/) (2026)
- [Toward Sustainable Distributed LLM Inference: A Systems Synthesis and Research Agenda for an Energy-, Carbon-, and Cache-Aware llm-d Control Plane](https://mlsanity.com/paper/2609.05565/) (2026)
- [Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving](https://mlsanity.com/paper/2609.11133/) (2026)
- [A principled approach for energy-efficient training via phase-aware GPU frequency tuning](https://mlsanity.com/paper/2609.24205/) (2026)
- [HybridInfer: Thermal-Aware Reinforcement-Learning Tier Routing for On-Device, Edge, and Cloud LLM Inference](https://mlsanity.com/paper/2609.30270/) (2026)
- [AID: A Framework for AI Infrastructure Dynamics](https://mlsanity.com/paper/2610.04801/) (2026)

## Abstract (arXiv)

The rising demand for generative large language models (LLMs) poses challenges for thermal and power management in cloud datacenters. Traditional techniques often are inadequate for LLM inference due to the fine-grained, millisecond-scale execution phases, each with distinct performance, thermal, and power profiles. Additionally, LLM inference workloads are sensitive to various configuration parameters (e.g., model parallelism, size, and quantization) that involve trade-offs between performance, temperature, power, and output quality. Moreover, clouds often co-locate SaaS and IaaS workloads, each with different levels of visibility and flexibility. We propose TAPAS, a thermal- and power-aware framework designed for LLM inference clusters in the cloud. TAPAS enhances cooling and power oversubscription capabilities, reducing the total cost of ownership (TCO) while effectively handling emergencies (e.g., cooling and power failures). The system leverages historical temperature and power data, along with the adaptability of SaaS workloads, to: (1) efficiently place new GPU workload VMs within cooling and power constraints, (2) route LLM inference requests across SaaS VMs, and (3) reconfigure SaaS VMs to manage load spikes and emergency situations. Our evaluation on a large GPU cluster demonstrates significant reductions in thermal and power throttling events, boosting system efficiency.