mlsanityData to 14 Oct 2026

Tropical: Enhancing SLO Attainment in Disaggregated LLM Serving via SLO-Aware Multiplexing

arXiv 2606.16264 ↗First posted 15 Jun 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

To guarantee service quality in transformer based large language model (LLM) serving, it is essential to meet the latency constraints of both the prefill phase (measured by Time-to-First-Token, TTFT) and the decode phase (measured by Time-per-Output-Token, TPOT). Non-disaggregated serving places prefill and decode on the same worker, while disaggregated serving places the prefill and decode on isolated workers. However, no single architecture excels in both TTFT and TPOT metrics. After conducting a root cause analysis, we concluded that indisaggregated LLM serving, prefill execution has minimal interference with decode execution but result in high queuing times. In contrast,non-disaggregated LLM serving effectively reduces queuing times but introduces significant interference between prefills and decodes. In order to leverage the best aspects of both non-disaggregated anddisaggregated LLM serving, we have designed and implemented Tropical.Tropical introduces an sevice-level objectives (SLO)-aware multiplexing strategy that balances the queuing time and the interference, enabling the LLM serving to achieve high TTFT and TPOT SLOs simultaneously. Our evaluation of real-world datasets reveals that Tropical outperforms both state-of-the-art non-disaggregated and disaggregated LLM serving systems, achieving up to 2.09 more requests within a 90% SLO attainment. Specially, compared to the disaggregated LLM serving system, Tropicalimproves P90 TTFT performance by 9 with only an 15% reduction in P90 TPOT. Against the non-disaggregated LLM serving systems, Tropicaldelivers a 2.8 performance improvement in P90 TPOT while maintaining the same P90 TTFT.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 35 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

27 of 35 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scale15 Aug
  2. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool25 Jun
  3. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  4. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  5. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services25 Apr
  6. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  7. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving2 Apr
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  10. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
  11. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs2 Nov
  12. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  10. 2024InternLM2 Technical Reportcited by 397
  11. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  12. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
3 more
  1. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 66
  2. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  3. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference
  2. 2026SARA: SLO-Aware Resource Allocation for Disaggregated Agentic LLM Services
  3. 2025VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routing
  4. 2025AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding
  5. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMs
  6. 2026Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference
  7. 2026Stage-Aware Communication Scheduling for Disaggregated LLM Serving
  8. 2024Queue management for slo-oriented large language model serving