mlsanityData to 7 Oct 2026

Toward Sustainable Distributed LLM Inference: A Systems Synthesis and Research Agenda for an Energy-, Carbon-, and Cache-Aware llm-d Control Plane

arXiv 2609.05565 ↗First posted 3 Sep 2026cs.DC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language model (LLM) sustainability is increasingly a serving-systems problem, not only a training problem. In production, energy and carbon impact depend on more than model size: workload shape, batching, key-value (KV) cache reuse, prefill/decode placement, model and accelerator choice, power state, geographic carbon intensity, and service-level objectives (SLOs) all matter. Recent systems papers study many of these factors separately. This paper connects those results and asks a practical engineering question: what do they imply when the decision point is a distributed inference control plane such as llm-d? The contribution here is synthesis, not a new set of benchmark results. Reported performance, energy, carbon, and cost improvements remain the results of the cited papers and systems. I group the literature into recurring design patterns and use those patterns to sketch a Sustainable Inference Control Plane (SICP) for llm-d. The proposed control plane would consider latency, energy, carbon, cache reuse, serving cost, and quality when routing and scaling, while keeping TTFT/TPOT SLOs as hard constraints. I also outline an evaluation framework based on SLO-satisfied goodput per joule and per gram CO2e, together with a reproducible experimental plan. The main observation from connecting the literature is that sustainable LLM inference is unlikely to come from one "green" model or one accelerator; it is more naturally treated as a control problem across model, phase, cache, hardware, replica, region, and time.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 27 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

19 of 27 references.

References · 19

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  3. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  6. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  7. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  8. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  9. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  10. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  11. 2025Energy Considerations of Large Language Model Inference and Efficiency Optimizationscited by 54
  12. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43
7 more
  1. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 28
  2. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 21
  3. 2025EcoServe: Designing Carbon-Aware AI Inference Systemscited by 18
  4. 2026M*: A Modular, Extensible, Serving System for Multimodal Modelscited by 3
  5. 2026PowerSlider: Exploiting Phase Asymmetry for LLM Serving under Demand Responsecited by 2
  6. 2026XWind: A Cross-site Router for Large Language Model Inference Serving at Renewable Energy Farmscited by 1
  7. 2026Routing LLM Inference to the Cleanest Grid in Real Timecited by 1

Built on by

Not cited yet.