Rearchitecting Datacenter Lifecycle for AI: A TCO-Driven Framework
Not on the map. Why?
Abstract (arXiv)
The rapid rise of large language models (LLMs) has been driving an enormous demand for AI inference infrastructure, mainly powered by high-end GPUs. While these accelerators offer immense computational power, they incur high capital and operational costs due to frequent upgrades, dense power consumption, and cooling demands, making total cost of ownership (TCO) for AI datacenters a critical concern for cloud providers. Unfortunately, traditional datacenter lifecycle management (designed for general-purpose workloads) struggles to keep pace with AI's fast-evolving models, rising resource needs, and diverse hardware profiles. In this paper, we rethink the AI datacenter lifecycle scheme across three stages: building, hardware refresh, and operation. We show how design choices in power, cooling, and networking provisioning impact long-term TCO. We also explore refresh strategies aligned with hardware trends. Finally, we use operation software optimizations to reduce cost. While these optimizations at each stage yield benefits, unlocking the full potential requires rethinking the entire lifecycle. Thus, we present a holistic lifecycle management framework that coordinates and co-optimizes decisions across all three stages, accounting for workload dynamics, hardware evolution, and system aging. Our system reduces the TCO by up to 40% over traditional approaches. Using our framework we provide guidelines on how to manage AI datacenter lifecycle for the future.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 34 of its 103 references that are in the map. First posted on arXiv 30 Sep 2025.
Builds on
34 of 103 references.
References · 34
- 2017Attention Is All You Needcited by 60,361
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
- 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
- 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
22 more
- 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
- 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
- 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
- 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 170
- 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 163
- 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
- 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
- 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
- 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 87
- 2024Fast On-device LLM Inference with NPUscited by 62
- 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 61
- 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 55
- 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decodingcited by 55
- 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
- 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 32
- 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
- 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 29
- 2025TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inferencecited by 14
- 2025Power Stabilization for AI Training Datacenterscited by 9
- 2024Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancingcited by 7
- 2023DxPU: Large Scale Disaggregated GPU Pools in the Datacentercited by 2
- 2025System-performance and cost modeling of Large Language Model training and inferencecited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.